Republican U.S. Rep. Jon Runyan announced in November that he would not seek a third term.
The district has been dominated by Republicans for decades but is one of the most closely divided in the country. President Obama has won it twice.
In another closely watched South Jersey race, State Sen. Jeff Van Drew (D., Cape May) announced Monday that he would not challenge Republican U.S. Rep. Frank LoBiondo in the Second District.
Runyan's abrupt announcement that he wasn't running "means we've lost fund-raising for the past three months" while local Republicans search for a candidate, Brown said. "If you're not someone who can write yourself quarter-million-dollar checks - and I can't - the challenges aren't philosophical, they're practical."
Brown, who operates a title insurance firm in Marlton, donates to charity his $8,000 annual salary as Evesham's part-time mayor. He also is the kicking coach for the NFL Baltimore Ravens.
The Democratic leadership in the Third District has endorsed Burlington County Freeholder Aimee Belgard for the open seat.
Brown said he would support any Republican nominee except Steve Lonegan, a former mayor of Bogota, Bergen County, who was the Republican nominee in October's special election to fill the vacancy left in the U.S. Senate by the death of Democrat Frank R. Lautenberg.
Lonegan, a conservative and tea party favorite, lost to former Newark Mayor Cory Booker. In January, he scuttled plans to retire from politics and announced his intention to seek the Third District nomination.
Brown referred Tuesday to Lonegan as a "carpetbagger" who "doesn't know the district," and called his nomination bid "craziness." Lonegan has said he is purchasing a house in Lavallette, Ocean County.
Brown said he was "not closing the door" on higher office, such as the state Senate, given the opportunity and time to raise money.
"I now have a pretty good idea of the process," he said.
Van Drew's decision in the Second District is likely to clear the way for Bill Hughes Jr., son of longtime U.S. Rep. William Hughes, to secure the Democratic nomination.
The chairs of the Democratic Party committees in six counties said last month that they would withhold an endorsement in the race until Van Drew decided.
"The state of New Jersey is more challenged now than at almost any time in our recent history," Van Drew said in a statement released Monday.
He added that he thought he could make more of a "tangible difference in South Jersey" by staying in the Senate, where he is a member of the budget committee and chairman of the urban and community affairs committee.
Hughes said in an interview that Democratic leaders had expressed "excitement about what happens next."
"This is going to be a real race," Hughes said. "It starts now."
His campaign raised $185,000 in the fourth quarter of 2013, while LoBiondo had $1.1 million cash on hand as of Dec. 31, according to Federal Election Commission data.
Gov. Christie endorsed LoBiondo last month and said he was "all in" on the congressman, who has held the seat since 1995. The district spans parts of Burlington and Gloucester Counties through Cape May.
Inquirer staff writer Andrew Seidman contributed to this article.