The woman said she'd come to let the man know she had taken care of a gas bill, that he and Frazier wouldn't have to use a kerosene heater any longer.
Authorities did not identify the elderly man, and his family sped off to find out where he was being treated.
Morgan Harley, 40, lives across the street and said he called 9-1-1 just before 9 a.m. and tried to rush into the home to rescue Frazier but was pushed back by the smoke.
"It was just too much," Harley said.
Frazier lived on Stanley Street, near James G. Blaine Elementary School, for more than 50 years, and neighbors referred to her as "Mom" but said she didn't come out all that often. Frazier had at least one son who had died, a niece said, and neighbors believed that Frazier also had a grandson.
"I don't think she had too much family left," said George Glover, 88.
Fire Chief Richard Davison said firefighters got the blaze under control by 9:10 a.m. and investigators had not determined the fire's cause. Numerous people who gathered on the street said the home had no heat and that Frazier and her housemate were using a kerosene heater to keep warm.
"I know she was having a real hard time with the cold," Harley said.
Frazier's death marks the 24th fire fatality so far this year, Davison said. The city had 25 fire fatalities last year.
On Twitter: @JasonNark