Now? Now he is talking from Livingston, Mont., headed for another long, relaxing day of fishing on the Yellowstone River. That is his life now, cramming in all the things he wanted to do more of over those years than his job, family and summer camps allowed, trying to broaden his horizons beyond the hardwood.
Taking it easy hasn't been easy. Bob Brown has one of those gaudy records that look like a misprint (476-154 as a high school coach), has been placed in five hall of fames around New England, has written for local newspapers and had awards named after him. Even in retirement, his footprint keeps expanding, from former players now coaching neighboring high schools and colleges right on up to the pro game.
In June, Steve Clifford, whose first coaching gig was as Brown's assistant at Saint Anselm College, was named the Charlotte Bobcats coach at age 51. By then, his son's name was floated around in several municipalities beyond this one, and it seemed only a matter of time - and contract length - before he would be hired by one of the 13 teams that went into the offseason looking for a new head coach.
Four years as Boston University's head coach was the closest the father ever came to where his son is now, and it convinced him that he was out of his league even then.
"I think my niche, my best coaching, was high school and Division III," Bob Brown said. "That really fit me, the person. It was teaching kids life lessons through the medium of basketball. And I liked that because I thought I could help kids. But at the pro level, it's a business. And in Division I, it's a business. That to me was totally different. Even now I still help out some kids and some coaches, but it's a teaching thing. I still enjoy that."
Already 52, Brett Brown will never have his father's opulent resume. Except for short stints as a head coach in Australia that included its 2012 Olympic team, his career has been spent behind the NBA front lines, as an assistant, a player-personnel director, a hands-on guru of footwork and positioning.
He can judge talent. He can also farm it to its full potential. Tony Parker blossomed with his help. Manu Ginobili eased into NBA nuances with his guidance.
"Where he's like me is that I always tried to do a job with the best effort I could," said the old coach. "Regardless of how long it took. And I think he thinks a similar way.
"But I also think he's been extremely fortunate to have been around Rick Pitino . . . And Lindsay Gaze, who is the Australian basketball guru. And Gregg Popovich and the San Antonio Spurs. He's been able to grow and learn from those people. That's how we're different.
"Brett started off like all of us who start out coaching with the big picture of X's-and-O's," the old coach said. "I think by being around all those top coaches, he realized that you're only as good as your players. You have to develop your players to the point where they can compete at this level.
"And that's where we're different. He has taken his knowledge of basketball way beyond anything I have learned or understood. The pro game is so different from the high school or college game. I sit down and talk to him now and I'm blown away by how much he has learned over the years."
Enough to stick his toes back in perhaps? Brett once asked for the old man's help when he was hired as a head coach Down Under. Isn't there a natural role for the Downeast's most prominent basketball scholar down in Philly?
"No, no, no," Bob Brown said, laughing hard. "I'm done coaching. I'm 75 now. I'm out here fishing. I'm just going to be one of those old guys who sits back, has all the answers and is smart enough to shut my mouth."
On Twitter: @samdonnellon