But the former San Antonio Spurs assistant remains excited about the challenge of rebuilding one of the league's storied franchises.
Brown realizes that there will be few improvements - if any - this season with the Sixers in tank mode, aiming for a high draft pick.
But the 52-year-old coach says his goal is to run a disciplined program. He's putting a huge emphasis on player development, conditioning, and changing the culture above everything else.
"You know at some point in the not-so-distant future, [winning] will resurrect itself as the judgment day," said Brown, who signed a four-year guaranteed contract in August. "It will be our report card: Did you win or did you lose? That's not the report card this year. It's how we go about our business.
"The process is going to be far more important than the result."
But that won't change Brown's coaching approach.
He expects the Sixers to give their maximum effort in games, practices, and film study. Brown also intends to keep coaching them up.
"We screw up, they understand," he said. "When they do something good, they are praised. It's that black and white. It's that simple.
"I think that if we focus on that process, then the results will slowly become [reality.]"
This season, though, the results could be lopsided based on the Sixers' preseason performances.
They finished 2-5, with one of those victories coming against Bilbao Basket, a Spanish team. The lost their last four preseason contests to Brooklyn, Charlotte, Cleveland, and Minnesota by an average of 22.5 points.
Those outcome could have been worse.
Deron Williams, Jason Terry, Paul Pierce, and Andrei Kirilenko were among seven Nets who did not play in a 127-97 decision on Oct. 14. On Wednesday, the Timberwolves benched starters Kevin Love, Nikola Pekovic, and Ricky Rubio in the second half of a 125-102 rout.
"Over the past month since we started playing games, you really started to understand the enormity of the job at hand," Brown said. "But I also say that with excitement.
"There were times during the whole process where you would see tremendous signs of improvement with some of our young guys."
Regardless of what the record is, Brown will get a pass this season. The excitement begins next summer when the Sixers will possibly have two lottery picks and substantial salary-cap space. Then the expectations will grow.
Brown will be judged by wins and losses by Year 3 regardless of who general manager Sam Hinkie puts on the roster.
With that will come the pressure to produce. And if he doesn't start winning, he will be considered a failure.
So even though he has a guaranteed deal, Brown could destroy any future head coaching opportunities by struggling in the long term in Philadelphia.
Some critics have said that he would have benefited from staying with San Antonio for another season, where he may have been the lead assistant coach. That move would have made him an attractive head coaching candidate next summer for teams in better shape than the Sixers.
"I feel like the risk-reward in this city is extremely special and very attractive to me," Brown said of taking the job. "I've always appreciated and enjoyed a challenge. I like competition."