He studied the Daily Racing Form as if it were the Talmud and was convinced he could decipher it better than most. Despite this devotion to handicapping, he rarely cashed a ticket. The more he lost, the more he felt compelled to explain why. It was cruel fate's fault, not Joe's, as Joe was happy to explain to anyone within earshot.
Sometimes he lost because he'd been shut out at the window. On other occasions it was a last-second change of heart, a bum tip, or a ticket-seller's mistake. Once he blamed a bird that flew past his "mortal lock" at the top of the stretch. Most common, he failed because the race had been fixed.
To encounter Joe at the track was to guarantee yourself a share of his ill fortune. That's why his fellow racetrack regulars learned to avoid him. My friends and I, unfortunately, never mastered their evasive techniques. Inevitably, as if from out of nowhere, he'd emerge alongside us.
"You guys ain't gonna believe what just happened," he'd moan.
At that moment, we knew it was time to tear up our tickets and go.
The colorful world Joe Bummer and his Runyonesque peers so miserably inhabited, one where large numbers of degenerate gamblers, well-groomed blue bloods, and Average Joes mixed democratically, no longer exists.
Horse racing has long since fallen from the lofty pinnacle it used to share with baseball and another sport now on life support, boxing. And all the Sport of Kings' men can't put it back together again.
Racetrack attendance is abysmal. The Preakness, the Kentucky Derby, and Breeders' Cup weekend are among the few notable exceptions, days when racing's glory is briefly reawakened.
Don't be fooled by the 164,906 fans who showed up at Churchill Downs two weeks ago. The Derby is as much about mint juleps, hats, and celebrities as it is horses. And 160,000 of those fans in Louisville aren't likely to venture inside a track again until next May.
The rest of the year, racetracks tend to be empty and depressing places whose clientele is almost exclusively sour old pensioners.
At Parx in Bensalem, for example, the vast racing complex typically is a ghost town, an afterthought to the adjacent casino, where you can lose money quicker, cleaner, and easier.
"We're just not going to get people to come back on a regular Wednesday or Thursday. It's not going to happen," Maryland Jockey Club president Tom Chuckas said before Saturday's Preakness. "We have to get past the idea that convincing people to come to the track is the key."
There's no mystery about racing's woes.
Competition from casinos, Internet gaming, and a much more vibrant sports scene has drawn away the casual horse fan. Small fields, short odds, and the sport's ongoing drug issues have alienated many of the more serious.
Once-grand race facilities such as Garden State, Hollywood Park, and Bowie have closed, as have scores of others across the United States. Despite a plethora of off-site and online options, pari-mutuel wagering has dipped in six of the last seven years. The number of races also continues to dwindle. There were 45,000 in the United States in 2012, according to the Jockey Club. That's barely half as many as in 1989.
It's only in old photographs and in YouTube videos where horse racing's former allure can be grasped.
There, in the decades between the world wars, large, lively crowds, mostly male and hatted, press toward the rail, craning their necks for a glimpse of the super horse du jour. The excitement and spectacle are palpable.
Along with movie theaters and baseball stadiums, race tracks then were Americans' favorite entertainment venues. Compared to the era's utilitarian ballparks and stadiums, they were palaces, complete with such amenities as full-service restaurants, valet parking, and sparkling clubhouses.
Horse racing had a cachet that appealed to a wide range of fans, from blue blood to bum. Dressing up for a day at the races was a social event the masses shared.
The sport was illegal in Pennsylvania then, but in 1942 Eugene Mori opened Garden State Park in Delaware Township - later renamed Cherry Hill. Thirty-two thousand fans showed up on opening day, and for years afterward weekday crowds of 15,000 to 20,000 were commonplace.
When Triple Crown winner Whirlaway ran there in 1942, tens of thousands streamed to the track from Philadelphia, many arriving in horse-drawn carriages to circumvent tight wartime gas restrictions.
Sure, these tracks were the only places beyond Nevada's borders where betting was legal. But gambling was far from their only allure.
The Sport of Kings still held enormous appeal for a nation with equine roots. People knew and understood horses. Notable thoroughbreds such as Seabiscuit, Man O' War, and War Admiral - the latter two owned by Delaware County's Samuel Riddle - occupied the headlines as frequently as baseball's stars.
But don't weep for horse racing. It had its day, and it was a long and bright one. Darkness arrives for everything and everyone.
In 2014 it survives as a niche sport whose appeal seems to broaden only when a "people's horse" - Funny Cide; Smarty Jones; or, now, California Chrome - emerges as a Triple Crown threat.
When those compelling stories fade, so does racetrack attendance.
At that point, all that remains is the graying generation of railbirds, those grizzled devotees who still smoke, still clutch rolled-up newspapers in their hands, still prefer a live ticket-seller to a computer.
They'll be dead and gone soon. When they are, you won't find many people willing to bet on the future of America's racetracks.
Except maybe Joe Bummer.