A PHA spokeswoman declined to explain why the two left, saying simply, "PHA and these employees have parted ways."
James and Caldwell could not be reached for comment.
The changes were handled by Kelvin Jeremiah, the agency's newly installed executive director. He started work at PHA only 10 months ago as head of audits and compliance, and replaced Michael P. Kelly as director June 15.
Kelly resigned to take a new job overseeing public housing in Washington. Days later, it was disclosed that the move had been precipitated by an investigation into his admitted affair with a PHA employee, who also resigned.
Though Kelly was cleared of wrongdoing, the incident was embarrassing for federal officials who had hired him to reform PHA after Greene was fired in September 2010 for secretly settling three sexual-harassment cases for more than $600,000.
PHA, which was viewed as being in a state of administrative disarray when Greene left, is operating under the direct control of the U.S. Department of Housing and Urban Development.
Jereon Brown, a HUD spokesman, said the departures of James and Caldwell were not related to Kelly's problems.
PHA, he said, needs to bring in "the right talent in the right place" to ensure it operates smoothly when PHA is eventually returned to local control.
"There have been changes, there may be more changes," Brown said. "Any personnel changes at PHA are focused toward the future and enhancing the operations there."
Brown said several of the top managers at PHA had long-term contracts, but those have been phased out.
"HUD has been going toward 'at-will' employment there . . . so that you can bring in new people," Brown said.
Kelly, Brown said, "brought in some people. Now we're going to go after some other high-quality people."
Contact Jennifer Lin at 215-854-5659 or jlin@phillynews.com, or follow on Twitter @j_linq.
