Addressing the crowd in the gymnasium on Arcadia's campus in Glenside, Sotomayor said she decided to write the memoir because she felt people had formed an "idealized" image of her that wasn't entirely accurate.
"A reporter, very early on in my term, asked me how my 'happy childhood' had affected my success," she said. "That really gave me pause, because I had never thought of my childhood in traditionally happy terms."
Before the speech, many students said they found Sotomayor's life inspiring.
"I thought it was motivating - riveting, really," said Netera Bickle, 19, who is studying to be an actress. "As a minority myself, it's inspiring to see that she overcame all these struggles and did so much with her life."
Elsie Contreras, 18, shares Sotomayor's background as a Latina from New York City. She said she was put off by the memoir's "overdramatic" portrayal of Sotomayor's young life.
"It doesn't sound that sad. It sounds like an average Hispanic life," said Contreras, who aspires to be an inner-city teacher. "It's hard to relate to it because she made it sound so bad."
Sotomayor elaborated on some of those experiences Tuesday, recalling when she arrived at Princeton only to realize "this was a world not just different, but totally alien to me. This world of prep schools and people who traveled on vacations at spring break and Christmas."
Emphasizing the importance of a liberal arts education, the justice said that things like school prestige, "summa cum laude, Phi Beta Kappa - those things matter in this world, and they put you in a position that's both a blessing and a responsibility. That's what class is about."
She said the justice system, and the high court specifically, would benefit from more diversity of experiences. None of the Supreme Court justices has a background in criminal defense, she said, adding: "You don't have to be a bleeding-heart liberal to understand that that has value in an institution upholding the Constitution with promises of rights to the accused."
And on a more local scale, she said, "we are failing in delivering equality in justice." Funding for legal aid and criminal defense "should be a priority. We're forgetting the message of our justice system when we [cut that]."
Sotomayor surprised the audience - and unnerved her security guards - by strolling through the crowd, shaking hands and pausing to be photographed with students during the question-and-answer session.
She also generated quite a few laughs.
Asked if she feels alienated serving with people who don't share her humble background, Sotomayor replied: "Some of my best friends are rich and white." She was quick to label that a joke, adding: "You can't take a brush and write off anyone because of how they grew up."