"Mentally, I've never gotten over it," Bunning said last week. "It was as close as I ever got."
Even more hurtful, perhaps, were the absences from family that baseball demanded.
"[For] all the times I missed something that you thought was important," an emotional Bunning told his children during his 1996 Hall induction speech, "I want you to know that I loved you then. And I hope you know I love you now."
But Bunning is fortunate. His career included one day - ironically, a Father's Day - that produced memories so rich they can still mask, though never erase, his disappointments.
Fifty years ago, on June 21, 1964, the Phillies ace made history, throwing the National League's first perfect game in 84 years, the first in any regular season since 1922.
"It's nice," he said, "to have something like that to remember about 1964."
That Father's Day performance at Shea Stadium in New York, he now understands, was a gift to his five daughters and four sons, a well-timed payback for all the dances, games, and graduations he missed.
Since leaving the Senate in 2010, Bunning, 82 now with hair as white as a new Phillies uniform, has had plenty of time for his children, who include two sets of twins and some notable careers.
Barbara is a lawyer. David is a federal judge. Bill is an Air Force veteran who flew in the first Gulf War. Jim Jr. is a former football player at Indiana. Amy worked as political aide to her father.
Some of them, along with many of his grandchildren and great-grandchildren - 24 in all - will be with Bunning on Sunday as, 50 Father's Days later, the Phillies commemorate his perfect game.
Bunning will throw the ceremonial first pitch at Citizens Bank Park, and former teammates Dick Allen, Cookie Rojas, and John Herrnstein will be there. The current team will wear 1964 throwback uniforms during their game against the Chicago Cubs.
There were only seven Bunning children on Father's Day 1964. Twins Amy and David were born in 1966. Only one, eldest daughter Barbara, was there to see her father make baseball history. The others watched its conclusion on television.
Countless times since then, the entire clan has viewed that game's ninth inning - the only bit of film that survives.
"And all of them," Bunning said, "understand very clearly what a special Father's Day that was."
With her husband due to pitch the opener of a Phils-Mets doubleheader, Mary Bunning drove to New York that morning with daughter Barbara and Gail Cater, the wife of Phillies first baseman Danny Cater.
They briefly visited the 1964 World's Fair adjacent to Shea Stadium and then, with temperatures already in the 90s, retreated to the brand-new ballpark for the 1 p.m. game.
At his Manhattan hotel, Bunning rose early, breakfasted, and went to Mass. He was delighted to hear a TV weatherman forecast temperatures in the mid-90s.
"I loved the heat," he said. "I sweated a lot, and I'd go through three jerseys every game. But I welcomed it because I knew I was in better shape than the hitters."
In the visitors' clubhouse, after donning a gray road jersey over a long-sleeved T-shirt, he smiled when he saw the lineup card. Manager Casey Stengel had benched George Altman, the one hitter on the anemic Mets who concerned Bunning.
His mound opponent would be Tracy Stallard. In 1961, the righthander had surrendered Roger Maris' 61st home run. Soon, Stallard would be a footnote to history again.
In his first Philadelphia season after nine in Detroit, Bunning, 32, was at his peak. He was 6-2 with an ERA under 2.50. Tellingly, on May 29, he'd been perfect for 62/3 innings against Houston before a blooper fell in front of Wes Covington.
Though he'd been hit hard the previous Wednesday in Chicago, Bunning earned a rare save a day later when manager Gene Mauch used him to get the final two outs in a 6-3 win.
Now, following his bullpen session, Bunning told catcher Gus Triandos he felt sharp.
Triandos had his doubts in the first inning, when the pitcher hung two offerings, fat curveballs that Jim Hickman fouled off.
"I told Gus if they weren't hitting those pitches, this could be one heck of a day," Bunning said.
The Mets went down easily until, with one out in the fifth, Jesse Gonder smoked a ball into the hole. Second baseman Tony Taylor dove, stopped it, and threw out the leaden-legged catcher.
The Phils led, 6-0, when, after 18 straight Mets had been retired, Bunning broke baseball tradition. In the dugout, he shouted to teammates: "C'mon, dive. Do something out there. Let's get this perfect game."
"The pressure builds not just on the pitcher, but on the fielders," he explained. "I was trying to relieve it by talking."
It was a lesson he learned July 20, 1958. On that sweltering Sunday at Fenway Park, Bunning, then a Tiger, no-hit the Red Sox, getting Ted Williams for the final out. (The Red Sox haven't been no-hit at home since.)
"The only game I ever got Williams out four times," Bunning said. "Four pop-ups."
He'd been so emotionally drained by pressure that day that he collapsed afterward. Six years later, he was, if not cool, at least collected.
"He was silly on the mound whenever I went out to talk to him," Triandos, who died last year, recalled in 1989. "He was jabbering like a magpie."
Meanwhile, back at the Cherry Hill apartment Bunning rented in-season, baby-sitter Mary Fran Hoffman summoned Bunning's other children to watch their father on Channel 6's telecast. Though Phillies announcer By Saam never mentioned the perfect-game possibility, Hoffman knew enough baseball to recognize what was happening.
A hundred miles to the north, so did Mauch. After the sixth, he shored up Bunning's support, removing Covington, inserting Bobby Wine at short, and switching Rojas to left field.
There were only a few subsequent scares. In the seventh, third baseman Allen cleanly fielded Ron Hunt's smash. And with two outs in the eighth, Bunning, whose 90 pitches included just 21 balls, ran a full count on Mets outfielder Bob Taylor.
A borderline 3-2 slider froze Taylor. Umpire Ed Sudol called it a strike.
"It was there," Bunning said.
By the bottom of the ninth, the 32,026 fans were on their feet cheering for the Phillies pitcher. On radio, equally excited Mets broadcaster Bob Murphy constantly reminded his listeners of the moment's significance.
"Not since 1880," he told them, "has there been a perfect game in the National League."
Shortstop Charley Smith led off. On a 2-1 pitch, he looped a foul that Wine caught behind third. With that, Altman marched out of the dugout to bat for Amado Samuel.
Bunning bore down. The burly outfielder slapped his first pitch hard to right. Johnny Callison raced toward it, but the ball curved foul. Altman fouled back the next pitch, then swung and missed at the next. Two outs.
With pitcher Tom Sturdivant due up, Stengel summoned another lefthanded pinch-hitter, .074-hitting backup catcher John Stephenson.
"When I first faced him earlier that season, Mauch told me, 'This guy can't spell curveball,' " Bunning recalled. "That's all he was getting."
Stephenson flailed at the first pitch, pulling off badly.
"I must have faced him 18 or 20 times . . . and I never did get a hit," said Stephenson, who became more successful as a scout and college coach. "He always pitched me on the outside."
That's where Bunning threw the second pitch, a called strike.
"You can never possibly come any closer and not get it," Murphy told his listeners. "Jim Bunning is one strike away from a perfect game!"
Stephenson, thinking Bunning had set him up with two curves, geared for a fastball.
Fooled, he stabbed halfheartedly at a third straight breaking ball and missed.
It was 3:19 p.m.
Bunning slammed his fist triumphantly into his glove and strode toward Triandos. The other Phillies hustled to join them.
Back in Cherry Hill, Hoffman had to deal with excited children as well as with calls from neighbors, relatives, and reporters.
In New York, sportswriters and photographers swarmed Bunning's locker. He answered and posed long into Game 2, an 8-2 Phillies win that was 18-year-old Rick Wise's first victory.
Daily News beat man Stan Hochman joked to Bunning that an asterisk ought to accompany his masterpiece since it came against the woeful Mets.
A CBS-TV page, dispatched to the ballpark, waited until the clubhouse crush subsided. Then he asked Bunning if he would make a cameo appearance on The Ed Sullivan Show that night at 8.
Bunning, a militant proponent of players' rights, responded instantly.
He agreed to do it for $1,000.
Late that afternoon, Bunning and his traveling companions left the Queens ballpark and headed for Manhattan.
"Everybody said we should go to Toots Shor's, so we did," he recalled. "But it was closed for Father's Day."
With no backup plan, they proceeded to the theater at 52d and Broadway, where Sullivan's weekly TV show was filmed.
There, Bunning met golfer Ken Venturi, who that same day had won the last U.S. Open concluded with a 36-hole final. Venturi had nearly died in the grueling heat at suburban Washington's Congressional Country Club.
"He said, 'I finally do something to get my picture on the front page and you have to go and knock me off,' " Bunning said.
Both athletes were introduced early by Sullivan. They stood, acknowledged the audience's applause, and sat back down.
When the show ended at 9 p.m., the weary and hungry travelers headed home.
"We still hadn't eaten," Bunning remembered. "So we stopped at the first restaurant we hit."
It was at a New Jersey Turnpike rest stop, in a booth at a Howard Johnson's, where Bunning's perfect Father's Day ended.