It's the kind of atmosphere O'Brien has fostered with his quarterbacks, and one that stands in contrast to his public persona as a fiery, intense competitor.
"He's that Boston guy. He grew up in Andover [Mass.]," Hoyer said Thursday. "He's just tough and a hard worker, very passionate about what he does. But also there's a great humorous side to him. When you're meeting 10 hours a day, you got to lighten the mood. That's one of the things I'll miss about him most."
O'Brien, of course, is five days from taking on the gargantuan task of replacing Paterno as Penn State's next head coach. But first there is a football game - something called Super Bowl XLVI - that has the 42-year-old preoccupied.
Since being named Paterno's successor last month, O'Brien has had to juggle both jobs. This week, he has dealt with preparation for Sunday's game, a diluted recruiting class, and unending questions about his overall predicament. But he handled the barrage with poise and the occasional joke.
O'Brien said he gets his Irish temper from his mother, but his sense of humor from growing up in the Boston area and from his two older brothers.
"They have great senses of humor," O'Brien said. "It's like that New England sense of humor - a little bit of a wise guy. Hopefully, people take it the right way."
O'Brien isn't likely to show this side as much with the Nittany Lions. At the NFL level, coaches and players are sometimes more like peers.
"It's like brothers teasing each other," said Hoyer, Brady's backup. "Obviously, the relationship is a little bit different when you go to college."
O'Brien coached at the collegiate level for 14 seasons, the last time at Duke in 2005-06. So he has experience with coaching teenagers. But he has never been a head coach at that level. He and others say he will have no problems being a disciplinarian.
"I've been able to coach every type of player - the 34-year-old guy that's the best player at his position, the 17-year-old guy that was a true freshman and coming in just out of high school," said O'Brien, who is married and has two sons. "So I have a pretty good idea about how to coach individual players. And I do think it's about each individual player. Each guy needs a different way to be motivated and taught."
O'Brien was New England's quarterbacks coach when Hoyer arrived in 2009. With Brady being Brady, the coach was able to spend additional time with the rookie. Hoyer said O'Brien's gift was being able to take a complex offense and simplify it for him.
"He'll be great with young guys coming in because he was willing to spend the extra time with me," Hoyer said. "All he has to do was give Tom the game plan."
But after a certain point he expects results. And that's where the intense O'Brien kicks in, the one who was caught on camera barking at Brady on the sideline during the season.
"He's always got energy. He's always ready to go," third-string quarterback Ryan Mallett said. "So he's going to push you to be your best every day."
Despite his Ivy League education - O'Brien went to Brown, like Paterno - he's blue-collar, those close to him say. That is, if he wore collared shirts.
"Is he the most polished head coach-looking guy? No, no, he's not," Patriots tight-end coach Brian Ferentz said. "But you know what, that doesn't make you a good coach."
When Ferentz watched O'Brien's introductory news conference at Penn State, he said to the other Patriots coaches who watched that it was only the second time he had ever seen O'Brien wear a tie.
"The third time was on our flight out here," Ferentz said. "So that's going to be a little different for him."
Paterno often wore ties, and always did on Saturdays. The Nittany Lions blue blazer is a trademark in State College. O'Brien has already been criticized by some Penn State followers because he doesn't look the part. He said he understands but hopes that will change.
"There will always be naysayers," O'Brien said. "If I was out there, I would probably be a naysayer and ask, 'Who is this guy?' "
But O'Brien has said all the right things and has been adamant that he will uphold Penn State's traditions, such as its iconic plain blue and white uniforms.
"Without naming names, there's probably been some coaches that came in and out of [Big Ten] programs that perhaps didn't really understand where they were, didn't fit into the culture, the background, the tradition, the history," said Ferentz, who played at Iowa under his father, Kirk.
Rich Rodriguez infamously went into Michigan and tried to turn the Wolverines program upside down. Mallett left for Arkansas almost as soon as he arrived.
"I'm glad I left before he changed all that stuff," he said.
Kirk Ferentz, once considered by some to be the heir apparent to Paterno, was born in Western Pennsylvania. Brian Ferentz said that much of his family is from that area and that an uncle and a few cousins played for Paterno.
"There's nowhere in the Big Ten that traditions are more prevalent than at Penn State," Ferentz said. "You need to embrace it."
Perhaps one day Nittany Lions fans will wear traditional masks of O'Brien - as they often did of Paterno - with embellished cleft chins.
"His," Hoyer said of O'Brien's chin, "is going to be hard to beat."
Contact staff writer Jeff McLane at 215-854-4745, firstname.lastname@example.org or @Jeff_McLane on Twitter.