Years later, when White was president of the National League, buttoned-up became his enduring image. He was wary of the media, disdainful, hiding his hostility behind a flurry of press releases, content to do his thing so far behind the scenes he was in a different area code.
And now White has written a fascinating book about his playing days, about his broadcasting career, about his turbulent time as National League president from 1989-94, about his problems with commissioner Fay Vincent.
The book is called "Uppity." That's shorthand for the degrading phrase, "uppity black guy" that bigots used to describe African-Americans who dared to speak out against discrimination and other issues.
White handled the name-calling. Shrugged off the threat of sticks and stones, too. Spoke out about segregated housing in spring training in the early '60s, and got the Cardinals to do something about it
"I wasn't alone," White said. "Bob Gibson was in on it. Curt Flood, George Crowe, Dr. Ralph Wimbish, who was president of the St. Petersburg NAACP.
"Things changed because the owner realized he wasn't going to sell a lot of Budweiser beer until things changed. Plus, Arizona was trying to lure teams out there for spring training."
White, a Phillie from 1966 to '68, is a tough, defensive interview, still wary after all these years. If they gave out Gold Gloves for smothering controversial hits, White would have an attic full of them. He offers clues to explain his attitude.
"I was invited to speak to a convention of black football coaches," he recalled. "It was in Atlanta and I flew down there. Ron Dickerson, the Temple coach, asked me a question about racism in baseball.