Frazier, 67, died Monday night after a brief battle with liver cancer.
The two-time heavyweight champion and longtime adversary of Muhammad Ali opened his gym here in 1968. Financial difficulties forced its closure in 2008.
It was widely reported that Frazier was cash-strapped in recent years. After his death, current WBC welterweight champion Floyd Mayweather tweeted that he planned to cover Frazier's funeral costs.
A day after his death, the only reminders of the gym remaining at In and Out were a pair of tattered boxing gloves said to have belonged to Frazier, and a photograph of the former champion delivering his unforgiving left hook.
On the front window, a painted pair of boxing gloves let those in passing cars know that "Knockout Prices" awaited inside.
Outside the store Tuesday, Charles Smith of West Philadelphia recalled a brief encounter with the former champion in the gym. Smith said he could feel the "power" in Frazier's fist when they embraced.
"He said, 'I'm not trying to hurt you, I'm greeting," Smith recalled. "I said, 'I know you're greeting.' A lot of memories here."
After his career ended in 1981, Frazier and his son, Marvis, began to train young fighters at the gym. One of them is undefeated welterweight Mike Jones, who was brought to the gym at age 15 by his father.
Jones trained with the Fraziers for three years before he began his professional career.
"It was rough," Jones said, "rough neighborhood. But that didn't turn me away from it. That brought me more closer to it."
To get the attention of the gym's namesake, Jones said, you had to earn it.
First Jones trained by himself, then he was trained by Val Colbert before the Fraziers took notice.
"As they saw me grow, they saw how much power I had, and the future I could have in this sport," Jones said. "They started working with me more and more."
As Frazier had been in his career, Jones quickly became a knockout artist. The 28-year-old has recorded 18 of his 24 wins by knockout.
He said the Fraziers taught him to stay planted when delivering heavy punches to build a strong foundation.
"Anything I could take from them, I was all ears," said Jones, who is scheduled to fight Argentina's Sebastian Lujan on Dec. 3 at New York's Madison Square Garden.
Frazier used to survey the scene from behind a window while he was perched high above the first floor. Now a cash register sits there, ready to record the next purchase.
Frank Jones of North Philadelphia (no relation to Mike) said the gym was once an important part of the neighborhood. It provided an escape and a safe haven for young people to learn the sport, he said.
"It's a landmark," Jones said. "Of course, they miss it."
Contact staff writer Matt Breen at firstname.lastname@example.org or @matt_breen on Twitter.