The voice goes on to say that, "You can sleep with [black people]. You can bring them in; you can do whatever you want. The little I ask you is not promote it on [Instagram] . . . and not bring them to my games."
Stiviano had posted photos of herself with Magic Johnson and the Dodgers' Matt Kemp on Instagram. They both are African-Americans.
In his own defense, Sterling has pointed out that Stiviano is a defendant in a lawsuit brought against her family by his family. He has said the tape does not reflect his views, beliefs or feelings.
He has not said, "It wasn't me."
So if it is proved to be that it is Sterling's voice on the recording, we'll see if Silver has the stones to deliver a heavy punishment to one of his 30 bosses.
"There are broad powers in place under the NBA's constitution and bylaws that include a range of sanctions," Silver said. "All of those will be considered depending on the findings of our investigation."
Silver might as well get ready to institute those broad powers because, given Sterling's history - which includes a 2006 federal housing lawsuit settled for a record $2.7 million for discriminating against African-Americans and Hispanics, and several claims by Clippers employees of racial bias - it is well-documented that he has a disdain for African-Americans.
In a sports league in which more than 75 percent of the players are African-American, that's a heavy issue for a rookie commissioner to address.
What Silver decides to do will be the first building block in his legacy as leader of the NBA. For the many good things that former commissioner David Stern did for the NBA, addressing issues with Sterling was not one of them.
"No room for Donald Sterling in our league," said Miami Heat star LeBron James, the most popular basketball player in the world. "As commissioner of our league, you have to make a stand and you have to be very aggressive with it.
"I don't know what it's going to be, but you just can't have that in our league."
A lot of people associated with the NBA, ones who represent a lot of the positive images the league wants to project, are speaking out against Sterling.
For Silver and the owners to ignore that would be economic suicide for a league that depends on the support of African-Americans more than any other.
And while I agree that the league is going to have to do something dramatic and unprecedented to one of its owners, I also know that - considering Sterling is a billionaire - no record fine or suspension meted out by Silver will be more than a splash of water off Sterling's back.
The only thing the NBA could do that would affect Sterling in a meaningful way is the one thing it has no power to do: kick him out of an extremely exclusive club by forcing him to sell the Clippers.
There is no real legal recourse for stupidity, and any attempt to force Sterling for any reason other than a financial inability to successfully operate the Clippers would be tossed out in flames by a court.
Ironically, Sterling has been one of the most progressive NBA owners when it comes to hiring African-Americans for leadership positions.
But it's not the dichotomy in Sterling's actions that is under review.
Some have suggested that the Clippers players should boycott the rest of their playoff series with the Golden State Warriors.
Yesterday in Oakland, the Clippers players wore their warmup shirts but then huddled at center court and tossed them to the floor, revealing their red Clippers jerseys being worn inside out to hide the team logo. The players also wore black wristbands or armbands and they all wore black socks.
I don't believe it's the players' responsibility to punish Sterling - especially when it will ultimately penalize them more than it will their owner.
The players are under contract to play. If they refuse to play out of protest, they violate the terms of their contract and effectively void it.
Sacrificing millions of dollars in salary and the pursuit of a lifelong dream to win an NBA championship is not something I would do simply because my boss doesn't want his girlfriend to have me over for a public barbecue.
Ultimately, it will not be the league or the players who will decide whether Sterling is adequately punished for his alleged actions.
That will be up to the people who always claim they have no power: the fans.
If fans are that worked up over Sterling, then boycott his team, stop going to games, stop watching it on television, stop buying its merchandise.
Only the fans have the clout to make it financially uncomfortable enough for Sterling to give up his ownership of the Clippers.
If fans stop showing up at the Staples Center so that Sterling cannot recoup the hundreds of millions of dollars it takes to operate an NBA franchise; if the fans stop watching on television so he cannot ask for large advertising fees; if the fans reject his product to the point where it becomes toxic, only then might Sterling be moved to step away from the team.
But that would take sacrifice from fans, a sacrifice they have historically proved they are not willing to make.
The Clippers should be a title contender for the next several seasons.
For a fan base of a loser franchise that has long suffered under the enormous cloud of the success of the Lakers, this is an extraordinarily addictive golden era of fantastic basketball.
I wonder how many Clippers fans are truly willing to walk the walk about punishing Sterling when the only true solution would be their willingness to step away from supporting a quality team.
No matter what Silver does if Sterling is found to be the voice on the tape, it will ultimately be just a cosmetic slap on the wrist.
But if only 3,000 people show up at Staples Center for Game 5 of the Clippers-Warriors series tomorrow night, Sterling will begin to sweat.
And if Clippers fans then continued to show commitment to their resentment of him no matter how good the product is on the floor, Sterling will eventually walk away from the team.
Soon, from both the NBA and NBA fans, we'll find out who is truly upset about Sterling's alleged comments and who just wants to talk about being upset by them.