State investigators found suspicious patterns of changed answers on tests at Philadelphia's Roosevelt Middle School

Theodore Roosevelt Middle School in East Germantown. (Michael Bryant/Staff)
Theodore Roosevelt Middle School in East Germantown. (Michael Bryant/Staff)
Posted: July 25, 2011

When one seventh-grade student at Philadelphia's Roosevelt Middle School took state reading and math exams in 2009, answers were erased 35 times - changed from wrong to right every time.

The odds of that happening naturally are more remote than 1 in 100 trillion. Winning the Powerball lottery, whose odds are still an astronomical 1 in 195 million, is much easier.

Eighty-two students at Roosevelt - one-quarter of the school's seventh and eighth graders tested - had statistically suspicious patterns of changing answers on state tests in 2009, state-appointed investigators have found.

The East Germantown school - hailed by Superintendent Arlene C. Ackerman last year as a district pacesetter - came under scrutiny in May after several Roosevelt teachers told The Inquirer that they had witnessed deliberate test security breaches there in the last few years.

The teachers said they had seen answers written on a blackboard, students being given improper assistance, and other violations.

A subsequent Philadelphia School District investigation of 2011 testing in the spring found no proof of improprieties at Roosevelt. The inquiry did not involve an analysis of erasures or other statistical measures.

The 2009 investigation, based on an in-depth analysis of multiple indicators by Data Recognition Corp. of Minnesota, surfaced a few days later after sitting on a shelf in the state Education Department for two years.

After learning of its contents, Ronald Tomalis, the new state education secretary, asked districts where school tests had been flagged to look into the matter and report back to him. And he ordered similar statistical studies of 2010 and 2011 state tests.

An analysis by The Inquirer of the 2009 report's student-level data found that some Roosevelt students' ability to fix their wrong entries was uncanny.

Nineteen students had nearly perfect erasure patterns - at least nine out of 10 times they adjusted an answer from wrong to right. Six of the students were perfect on such fixes.

This is the sort of performance that in just two years lifted the school's test scores a stunning 52 points in math on a 100-point scale and 51 in reading on the state tests. The improvement was the best - by a considerable margin - of any comparable school in the district.

After questions were raised about this performance, the district defended its test security system as "robust" and said it included training in test protocol and random visits of 75 percent of all district and charter schools during PSSAs.

Stefanie Ressler, the school's principal during the testing, who has since been reassigned to a larger and better-performing school, has not responded to numerous inquiries to discuss Roosevelt's performance. Last year, she described the gains as "a miracle" at a School District event.

But a woman whose job placed her inside Roosevelt classrooms for three years said the test violations were "no secret." Aisha Mickeals, who was a behavioral health worker at Roosevelt, said she had witnessed cheating frequently, including during the 2009 testing.

School staff would "walk right up behind them to see what they had and shake their head no and point," Mickeals said. "And the kid would erase and fill in the correct answer."

Her descriptions dovetailed with the accounts of teachers, who asked that their names not be used for fear of reprisals.

Mickeals described the school during her three years there as a chaotic place, with few consequences for students who acted out. Though she lives in the same neighborhood as the school on East Washington Lane, Mickeals said she refused to send her children there.

"The kids were just running through the halls," Mickeals said. "No structure. Fighting. Play fighting. Braiding hair. Eating hoagies. Jumping rope. It would look like something from a movie."

She does not fault the teachers, Mickeals said.

"I think the teachers that were there were good teachers and capable of doing the job," she said. "But there was no structure and no control in the school. The kids just ran wild."

During testing, Mickeals was assigned to a resource room where some students - usually ones with behavior problems - took tests together.

Every afternoon, Mickeals and other behavioral health workers gathered to discuss the accomplishments and challenges of the day. At times, she mentioned the testing problems.

"Those were things that were brought to the table - about them getting help on the tests when they weren't supposed to or, you know, passing classes that we clearly knew they failed," she said.

But nothing ever came of the discussions, and she never went to the principal or district about what she had seen, Mickeals said.

"I needed my job," she said.

Mickeals was laid off in June 2010 when the district cut its contract with Northern Home for Children, the nonprofit organization that provided behavioral health services at Roosevelt. She now works outside the city, for another company.

She said she was not surprised at the results contained in the state report, which places special emphasis on erasures.

"Of particular interest are the schools that were flagged by the erasure analysis indicator," the report said. ". . . These results may strongly suggest that a testing irregularity occurred at the school."

The Inquirer found that among the Roosevelt students with unexpected erasure patterns, about 1,500 answers had been changed. Eight out of 10 times, the correction was from a wrong answer to the right one, a rate of improvement far higher than the state average.

Briefed this year about the Roosevelt situation, Gregory J. Cizek, a professor of educational measurement and evaluation at the University of North Carolina and an expert on cheating, said: "The conclusion is their scores aren't credible."

It is unusual that large numbers of students at a single school have such dramatic improvement, Cizek said, and "these kids should be retested under state-monitored conditions to confirm the validity of their scores."

The report also stressed that statistical evidence does not prove there was cheating but can form part of a pattern that suggests cheating.

"Alone," the report said, "each index may suggest an irregular testing behavior that should be investigated. Together, they provide more support to the notion that test results were suspiciously earned."

In the state's report, Roosevelt stood out as one of a number of district schools with numerous abnormal indicators.

The school's seventh-grade test results were identified as aberrant on seven measures of study. Eighth-grade tests were flagged six times.

The report documented abnormal scores in reading in both grades at the school, as well as unexpected increases in the number of students passing the test from the year before. It also found that the highly improbable erasures had occurred in both grades in reading and math.

The report assigned a "threat level" to each measure - a simplified 0-100 scale to make complex numbers more easily digestible.

For each school, threat scores above 10 were flagged as improbable. The higher the number, the less expected the result. A score of 10 places the likelihood of the unusual result at 1 in 10,000. At 20, it leaps to 1 in 100 million.

Roosevelt's unusual erasure patterns received a threat score of 39.1 in eighth-grade reading and 37.2 in seventh-grade reading.

In math, the number of wrong-to-right erasures by the students was deemed to defy odds considerably greater than 1 in 100 million.

According to the report, "Any school-level testing irregularities may render the [adequate yearly progress] status of a school invalid."

Across the district, 88 schools - a third of all schools - were flagged in the report for erasures, with nearly half marked for both subjects. About 40 of those schools showed aberrant erasures in more than one grade.

Four of the district's schools were assigned the highest threat score of 99.9 for erasures - meaning the odds were astronomical against such an erasure pattern's occurring by chance. Those schools - Olney Elementary, Tilden Middle, Barratt Middle, and Strawberry Mansion High - had a combined 278 students whose tests were considered unusual.

Roosevelt is one of 20 district schools identified as having abnormal erasure patterns involving at least 10 flagged tests in one subject and one grade. But that may be a conservative tally.

Unlike threat scores given to schools, the data analysis was far more lenient on student erasures. Only students with threat scores of 50 or greater were counted. "Using a flagging criterion of 10 resulted in too many students being flagged," the report said. "We wanted to keep the number of flagged students to a manageable number."

Once school districts reexamine suspicious test results and report back to Tomalis, a larger state investigation can follow, he said.

"Some of the indications are that there are things we certainly need to follow up on," Tomalis said in an Inquirer interview this month.


Contact staff writer Kristen Graham at 215-854-5146, kgraham@phillynews.com, or @newskag on Twitter. Read her blog, "Philly School Files," at www.philly.com/schoolfiles

comments powered by Disqus
|
|
|
|
|