A clip reel of those, er, boisterous home videos sold the show, and though the TV version of his family's a little different - in real life, he has two older brothers, not a brother and a sister - the stories are ones "that I grew up with and wanted to re-explore," according to Goldberg.
So much so that when a reporter at a Television Critics Association event this summer took issue with a storyline involving his character's grandfather (played by George Segal), Goldberg pointed out that the incident in question had happened to his actual Pops.
Not that there isn't a downside to mining one's loved ones for material.
The Penn Charter grad, whose writing credits include the big-screen "Fanboys" and the twice-canceled Fox comedy "Breaking In," admitted that he's feeling more defensive than usual about this project.
"I've discovered, surprisingly, that the hardest thing about this is that when critics attack the show and say certain things, I take it personally. Whereas on my last show, I'd go, 'OK, that's a good point. I don't disagree.'
"But on this one, because it is my family and I'm putting myself out there, you take it more personally," said Goldberg in an interview in Beverly Hills this summer. "So I'm just trying not to read anything, because I don't want to get my feelings hurt."
His physician father, played on "The Goldbergs" by Jeff Garlin ("Curb Your Enthusiasm"), died about five years ago, but "my mom [played by Wendi McLendon-Covey, of "Reno 911!"] is "beyond thrilled. This is the greatest thing for her ever," he said.
"In a lot of ways, it's a way to resurrect my dad. And also therapy, digging up the past," Goldberg said. "I think if I'd done it right after [he died], it would've been hard. Now I can just pull out all the funny things. Especially casting Jeff, who is like my dad. He channels my dad. It's amazing. There are certain things that he says in this pilot that my dad said and he delivers it the same exact way. It's amazing."
Goldberg, whose wife, a therapist, has known him since he was 16 - they met at a summer theater program for high-school students at Northwestern University - said "she's really proud" of what he's done with "The Goldbergs."
"In her mind, I took something that wasn't the most positive experience all the time and made it a comedy, and made something really great out of something that maybe wasn't the most healthy thing all of the time," he said.
In the meantime, he may be depriving his own children of future comedy material by raising them in a home with less shouting and where, presumably, he doesn't remove his pants at the front door on arriving home, as Garlin does in the pilot.
As a parent, "I don't know what I'm doing," Goldberg said. "I look at how my dad raised me and I know that's not what to do. So I know that. And a lot of the time, I'll sneak over to [my wife] and say, 'What do I do?' And she'll tell me.
"There's a lot of tantrums, and the way we deal with it is so different. My dad would have called us a moron and screamed at us. And we have timeouts and a lot of talking and discussions. It's really weird."
On Twitter: @elgray