Now, four decades later, Frazier and Ali - the former deceased, the latter trapped by Parkinson's disease - have been reunited, returning to the headlines in the city where they seldom displayed any brotherly love toward each other.
Even as Philadelphia officials last week revealed their intentions to erect a statue to honor Frazier, who died in November at 67, the late boxer's family was pushing to have his longtime North Philadelphia gym preserved as a historic site.
"It's moving forward. The family is very excited that this great Philadelphia champion will be remembered in bronze," Pete Lyde, Frazier's son-in-law, said when reached by phone Saturday.
Meanwhile, the Cherry Hill home where Ali, who earlier lived on City Avenue, resided in the early 1970s went on the market last month. And on Thursday, at a National Constitution Center ceremony, the 70-year-old boxer, once the world's most recognizable and controversial figure but now barely able to speak or walk, will receive that organization's 2012 Liberty Medal.
The Ali Philadelphia will see at Thursday's ceremony is, nearly 30 years after his Parkinson's diagnosis, increasingly frail. Wearing a white suit, he turned up on his wife Lonnie's arm at the London Olympics last month and while there made an appearance to raise funds for the Muhammad Ali Center in his native Louisville. But as a recent magazine ad he did for Louis Vuitton displayed, Ali still has that twinkle in his eyes.
Ali apparently also made an unpublicized return to Philadelphia last spring. Khalia, one of his seven daughters (he also has two sons) is married to a local man, Spencer Wertheimer, and, according to the Jerusalem Post, Ali was here in April for their son's bar mitzvah.
"Our family is very excited about him getting this award and hopefully we will be there," Lyde said. "People here remember that Ali was a great boxer, but he was also great humanitarian who did good all around the world."
Ali and Frazier were Philadelphians in 1971 when they met at Madison Square Garden in "The Fight of the Century," a matchup of unbeaten heavyweights that J. Russell Peltz, the longtime Philadelphia boxing promoter and historian, called "the greatest sporting event in the history of the world."
A little more than two years after Frazier won a decision in that ballyhooed battle, just before Ali gained redemption in their second and third meetings, the latter sold his South Jersey residence and moved elsewhere.
But in that brief interlude, the two men whose divergent images came to personify both sides in that Vietnam-torn era's cultural warfare created both history and bitter enmity right here.
Frazier, of course, had been a Philadelphian since 1958, when, fleeing racism in his native South Carolina, he arrived as a 14-year-old. A simple, rough-hewn man with a determination as powerful as his fists, he would become a symbol for all those blue-collar Americans who felt alienated by the '60s and the prominent antiwar movement Ali in many ways spawned and represented.
Ali, meanwhile, was the much-beloved, much-despised antihero, a champion of the counterculture and the outspoken king of a sport where outspokenness was rare.
The world's most famous man, Ali settled here in 1970, just before the end of his three-year exile from the sport he'd dominated since upsetting champion Sonny Liston in 1964. In 1967, the undefeated heavyweight king was stripped of his title when, citing religious beliefs, he refused induction into the Army.
Surviving through TV appearances, mostly in New York, and college lectures, Ali found Philadelphia to be a convenient location.
"I wanted to get out of Chicago because I was in New York twice a week and I found myself living in airplanes, which I hate," Ali told Philadelphia Magazine at the time. "New York was too busy. And Newark and Trenton, I looked in those places but there was nothing I liked, so I stayed with Philly."
Ali, who always attracted an eclectic mix of devotees, hangers-on, and shady characters, had befriended Major Coxson, an alleged local drug dealer with a lengthy criminal record. Through him, Ali found the house on City Avenue. Later, in part to escape the city wage tax on the $2.5 million he would earn in Ali-Frazier I, he sold it and moved to the secluded Cherry Hill estate at 1121 Winding Dr.
For Frazier, Ali's relocation to his adopted hometown was another slap in the face. After winning the heavyweight title in Ali's absence, Frazier had loaned money to the struggling deposed champion. But when the two men agreed to the 1971 fight, Ali appeared to forget about that largesse.
In public appearances, and in interviews with Philadelphia's newspapers and TV stations, the charismatic Ali belittled Frazier as stupid and ugly. He labeled him an Uncle Tom, someone who had sold out to white interests. (Later, before 1975's "Thrilla in Manila," he even called him a gorilla and mocked him with a toy ape.)
Ali subsequently offered various reasons for his behavior. He would contend that the name-calling was just a way to promote their fight.
Or, he sometimes said, Frazier had continued to call him "Clay" even though he had changed his name from Cassius Clay years before when he converted to Islam.
"A lot of it was cruel," Peltz said. "Joe felt like he had reached out to Ali when he needed help and he didn't think that was the right way to pay him back. He held a grudge for the rest of his life."
Many years later, after Ali's voice was silenced by the illness, a still bitter Frazier seemed to delight in his rival's condition.
"God's shut him up," Frazier said. "He can't talk no more because he was saying the wrong things. He was always making fun of me, telling me I'm a dummy. Tell me now, which one talks worse? He's finished and I'm still here."
What made it all worse was their proximity here. Once, according to the Philadelphia Magazine story, Ali and Frazier accidentally bumped into each other while training in Fairmount Park.
Posturing for their entourages, the two exchanged shouted threats before agreeing to settle the matter privately at a Philadelphia Police Athletic League gym.
Ali showed up, as did the media and several thousand fans who had heard about the event. But Frazier never appeared, triggering another round of Ali insults.
Because interest in the first Ali-Frazier fight was so intense, the national media flocked here to seek out the combatants. The matchup - a sporting and cultural phenomenon - was on everyone's lips, and tickets were treasured prizes.
Frazier's manager, Yank Durham, gave Peltz his complimentary tickets and asked the young promoter to sell them for him. One was a much-prized ringside seat.
Peltz, who would attend the famous bout, thought about keeping it for himself but finally sold it to a South Philadelphia bar owner for its $150 face value.
"The night of the fight I'm in the balcony looking down at the ring through binoculars," Peltz recalled. "I see the guy I sold it to and he's surrounded by Frank Sinatra, Burt Lancaster, and all these celebrities. And I said to myself, 'What a schmuck you are!' "
Frazier won the fight, which concluded with both men bruised and battered though considerably richer.
Ali, his family growing, had outgrown the City Avenue house between Haverford Avenue and 72d Street. Coxson had told him about a large, secluded home for sale near his own in Cherry Hill.
"I didn't like it because I figured it was too far from Philly," Ali told the magazine. "I like to live around people and everything. But I got to hanging around with the Major a lot [and eventually he purchased the Tuscan-style villa for $115,000]."
In June 1973, Coxson was executed gangland-style in his own house. No one has ever been charged with the crime , although police believed it may have been the result of a heroin deal that went bad.
Later that year, Ali sold his home for $175,000 and moved elsewhere, eventually settling in Michigan, though he would continue to train in remote Deer Lake, Pa., near Pottsville.
Frazier would live in the Philadelphia area the rest of his life, and he and Ali would meet twice again - Ali winning a Garden rematch in 1974 and the famed "Thrilla in Manila" a year later.
In retirement, Ali's fame and reputation expanded and softened. He remains an international figure, one more respected than ever, as his Liberty Medal would imply.
Frazier, meanwhile, spent his final years hounded by financial difficulties and puzzled, some said, by Ali's status as a secular saint.
Several years ago, Ali apologized for the attacks on his competitor, and Frazier, publicly at least, accepted.
"But deep down," Peltz said, "I don't think Joe ever reconciled."
Said Lyde, Frazier's son-in-law: "That was between him and Joe. That was then. There's no animosities between the families now. He's part of the boxing family. In fact, my wife and Ali's daughter Khalia, who lives in this area, are the best of friends."
Contact staff writer Frank Fitzpatrick at 215-854-5068, ffitzpatrick@phillynews.com, or on Twitter @philafitz. Read his blog, "Giving 'Em Fitz," at www.philly.com/fitz
