"There's a different community here, and part of it I'm not in touch with. It's a black community, and I'm white," he said. "When we moved here, everyone was white, everyone was Catholic, and everyone had kids."
Ayrer moved to Willingboro with the first wave of homeowners in South Jersey's Levittown. Levitt, who built a similar development in Bucks County, initially refused to sell to African Americans.
The New Jersey Supreme Court opened access in the 1960s after hearing a complaint filed by an African American family who wanted to take advantage of the Cape Cods and ranchers selling for $13,000.
When Ayrer took office in 1980, all five members of the all-Democratic Township Council were white, but in the '90s, the racial makeup shifted. Over the last decade, Ayrer said, he has been the only Caucasian on council.
"It was just a bunch of folks working together . . .. No one really deals on the basis of race in this town," said Ayrer, who, with his wife, Clara, raised four children in the four-bedroom Cape Cod they bought in 1960.
The 2010 census found 73 percent of Willingboro's population was African American, 17 percent was white, and 9 percent was Hispanic.
Ayrer, who also served three stints as mayor, is among the longest-serving officials in New Jersey. He has been among the dozen or so officials honored each year by the New Jersey League of Municipalities for serving longer than two decades. The service record, however, goes to Surf City Mayor Len Connors, who the league says has been in that post for nearly 50 years.
Ayrer said he stayed on because "it became a way of life."
Before he retired from his job in 1996, Ayrer, who has a Ph.D. in education, prepared tests and did research for the Philadelphia School District. He is active in Willingboro's Corpus Christi Roman Catholic Church, where he is a deacon.
He will be missed when he leaves council, Mayor Eddie Campbell Jr. said.
"He was a very good person to work with. He was our eye-on-the-budget type of person, always dependable, always there, never missing meetings," Campbell said. "He's 80 and deserves to retire. I'm 81, and I know what it's like."
Campbell, who has been in office 16 years, will become the town's longest-serving official when Ayrer retires March 18.
Lavonne Bebler Johnson, a former mayor and the chairperson of the town's Democratic Committee, says she, too, is sorry to hear Ayrer is retiring.
"Such integrity, such devotion," she said.
The committee will review applications for Ayrer's replacement and provide names for the council to consider. The appointment would run until the November election, when voters will decide who will hold Ayrer's one-year unexpired term until the end of 2015.
Ayrer got his start in politics when he was appointed to council after Bill Kearns, another longtime councilman, resigned.
The year before that, Ayrer made an unsuccessful run for a seat on the Burlington County Board of Chosen Freeholders.
"I got walloped," he recalled with a chuckle. "I was interested in environmental issues, and someone called me up and asked me to run for freeholder."
In Willingboro, Ayrer found winning elections easier. The races are uncontested, he said, because Republicans generally do not field a candidate in the Democratic stronghold. Occasionally, he said, he faced a primary challenge.
Perhaps the biggest challenge Ayrer faced as a councilman was the deterioration of the Willingboro Plaza, a once-busy shopping center off Route 130. Attractive in its heyday, it fell victim to competition from malls and ended with empty stores. Ayrer said the council had to get the buildings razed, entice new businesses to open, and bring in community college campuses and a state-of-the-art library.
"A dozen years ago, it was just such an eyesore," he said. Now, he said, he is pleased with the bustling Town Center in the plaza space.
However, the largest business in that center, Express Scripts, recently announced it would move. "It's a killer," Ayrer said, adding that the college campuses and other businesses were doing well.
The largest controversy in his tenure, Ayrer said, was whether the dry town should allow liquor sales.
"People who didn't want it made a fuss and said they didn't want drunks coming to the town," he said.
The problem was resolved by limiting alcohol sales to restaurants. "We don't have bars," he said.
After retiring from politics, Ayrer plans to remain in Willingboro, reading more books, watching more classic movies, and enjoying the deck he recently added to his Cape Cod.