Frazier died Nov. 7 from cancer at age 67.
The South Carolina service had none of the celebrity trappings of Frazier's funeral Monday in Philadelphia, where Jesse Jackson delivered the eulogy to a congregation including Muhammad Ali, former heavyweight champ Larry Holmes, and promoter Don King.
Also absent from the service at Bethesda Christian Fellowship on St. Helena Island was Frazier's body. In place of his casket, two large portraits of Frazier stood at the church altar - one of him wearing the Olympic gold medal he won in 1964; the other taken with his massive heavyweight championship belt slung over his shoulder.
After the church service, dozens more gathered for an outdoor memorial at a waterfront park overlooking the Beaufort River. Frazier's daughter, Jacqui Frazier Lyde of Philadelphia, told the crowd she regretted that she was unable to bring his casket back to South Carolina.
"I feel like, as the breeze is blowing, my father's spirit is here," Lyde said. "Because this is his home."
Lyde said her father was being buried in a blue suit and patent leather shoes. "He looked like one of God's men."
Frazier spent his first 15 years in Beaufort on a farm where his parents grew corn, watermelon, and okra. Frazier's father was proud that he was not a sharecropper. He owned his land.
By age 6, Frazier was in the fields helping his brothers and sisters pick tomatoes and other crops. He began driving his father's pickup truck when he was 7. His mother would later recall Frazier started to fight around the age of 9.
Frazier's father encouraged the brawling, saying he could grow up to be the next Joe Louis, and Frazier started training with whatever materials he had at hand. The fighter later said he gave daily beatings to his homemade punching bag for several years.
Frazier was expelled from school in the ninth grade when he fought a white student for calling his mother names. He got a job working construction that helped him build his body and earn enough money to leave the South. In 1959, at age 15, Frazier bought a ticket and boarded a bus to New York to begin training as a boxer in earnest.
"I left the South as soon as I found out about the North," Frazier later told a biographer.
At the outdoor service, which ended with the traditional boxers' send-off of a ringing bell, Beaufort Mayor Billy Keyserling lamented that Frazier had to leave the segregated South to achieve success.
Frazier's home state made some amends a year ago when he returned to Beaufort to be awarded the Order of the Palmetto, South Carolina's highest civilian honor.