"I made up my mind that they were all going to go to jail," Brotman said.
He summoned HUD officials into court to chastise them for not being vigilant enough to notice the fraud, which cost taxpayers millions of dollars.
"These were officials who should have seen what was going on and stopped it," Brotman said. "I said to them, 'I'm only sorry that I don't have you in front of me as indicted or convicted people. You are the ones that should be going to jail.'
"I gave them hell, calmly," he said.
In the Virgin Islands, where he was assigned as a visiting judge in 1980, he is known as the federal judge who returned in the midst of the devastation from Hurricane Hugo to supervise restoration of the courts and the "rule of law," as he said.
He is also known for supervising the construction of a new courthouse on St. Croix. It opened in 1991.
But he is probably best known for the case involving the prison on St. Thomas, a sweeping prison-rights case that has gone on since 1992. That was the case that became his grand finale on Thursday.
"It was an unconstitutional jail," Brotman said. Overcrowding, insufficient medical care, violence - "they weren't getting enough to eat. There were problems with the guards."
"It was a constant battle to make the government do the things it should to bring it back to a constitutional jail," he said. Brotman held four Virgin Islands governors in contempt of court.
"He's incredibly hardworking and very thoughtful," said American Civil Liberties lawyer Eric Balaban, who has represented the prisoners in the case for most of his career.
"And he's done things in this case that are unusual. He goes to the facility on a regular basis," Balaban said. "He lets the prisoners see him and lets the officers see him."
On Thursday, Balaban and the lawyer for the governor of the Virgin Islands asked Brotman to approve a settlement in the case - the last case Brotman has been carrying.
"The jail is coming along pretty well now," Brotman said in an interview in his Camden chambers last month.
As of Thursday, there were 780 docket entries in the case - the last being Brotman's order approving the settlement.
After he approved the settlement, courthouse staff in St. Thomas and St. Croix crowded into the courtroom there, taking turns talking into a video camera to bid a fond farewell to the judge.
"I feel very humble," Brotman said Thursday afternoon. "They came into the courtroom and proceeded to say nice things about me. It was very, very nice and heart filling."
In 1997, then-U.S. Chief Justice William H. Rehnquist, knowing Brotman's background in military intelligence, appointed him to the Foreign Intelligence Surveillance Court for a seven-year term.
That's the supersecret court that has been in the news since June, when government contractor Edward Snowden leaked the court's order requiring a Verizon subsidiary to provide a daily feed of domestic call records to the National Security Agency.
The court reviews requests for surveillance warrants for suspected foreign agents in the United States.
No comment, Brotman said when asked about the Snowden case, but he said his work on the court "was one of the most interesting and important things I ever did."
Brotman was serving on the court in 2001. After the 9/11 attacks, the court grew from seven members to 11.
"I knew what I was doing was really, really important because it affected the security of my country.
"I had to understand what was going on and what was being asked of me, but [had to strike a balance] because the rights of the person involved have to be considered," he said.
Brotman grew up in Brotmanville, a small village near Vineland established by his grandfather, a Russian immigrant who moved his coat factory from Brooklyn to South Jersey, providing jobs for immigrants and helping them settle in a new land.
In 1942, a year after Japan attacked Pearl Harbor, Brotman, then 18, left Yale University to enlist in the military. He studied Burmese and the culture, politics and economy of Southeast Asia and China, and was assigned to the Office of Strategic Services, a World War II military intelligence agency that evolved into the Central Intelligence Agency.
During the Korean War, he was called to active duty, joining the Armed Forces Security Agency, an intelligence agency.
In 1952, he set up a law practice in Vineland. President Gerald R. Ford named him a federal judge in 1975.
In 1990, Brotman went on senior status. He was eligible to retire on a full pension, but he, like seven other judges on the New Jersey federal bench, continued to work, essentially free, as he would earn the same by staying home.
Until 2005, he carried a full caseload, but he had cut back since. Now, with the Virgin Islands prison case out of his hands, he'll have time to pack up his chambers, sort through his photos, travel with his wife, Suzanne, and spend time with his two children, Richard and Alison Braem.
"We've been married 62 years, and we enjoy each other," said Brotman, who now lives in Voorhees.
He said he regretted none of his decisions.
"I was never afraid to make a decision, and it sort of came naturally to me. I was able to put everything together, weighing the pros and cons."
That includes ordering at restaurants. "My wife agonizes," he said. "It takes her so long to order. I never had that problem."
Contact Jane Von Bergen at email@example.com, @JaneVonBergen on Twitter, or at 215-854-2769. Read her workplace blog at www.philly.com/jobbing