That problem, coupled with increased sales of imported fruits and vegetables, has financially hurt farmers, according to a report released in March by the Partnership for a New American Economy, and Agriculture Coalition for Immigration Reform.
Nationally, the share of produce that was imported grew by nearly 80 percent between 1998 and 2012, the report said.
"Domestic producers have to have access to a steady workforce - they need a new more flexible guest-worker program," said Ray Gilmer, a spokesman for the United Fresh Produce Association, a national trade group with 1,300 members. "If they have that, they can compete with anyone in the world."
The amount of fresh produce consumed by Americans increased more than 10 percent from 1998 to 2012, but "our farmers don't have the labor force to meet the demand," said John Feinblatt, chairman of the Partnership for a New American Economy, a bipartisan organization of more than 500 mayors and business leaders.
Tom Holtzhauser knows the problem firsthand. He usually hires eight to 15 Mexican laborers to trim his peach trees at his farm in Mullica Hill, Gloucester County.
This season, though after about 15 years of having their help, none were available. He and a cousin trimmed all 9,000 trees themselves over the winter. "I need these guys to survive," said Holtzhauser, owner of Holtzhauser Farms and director of the New Jersey Peach Promotion Council.
The current H2-A system, administered by the U.S. Department of Labor, provides temporary visas for workers from 63 countries to fill seasonal agricultural jobs.
It requires farmers to prove that they are unable to hire U.S. workers and that the employment of migrants would not adversely affect the wages and working conditions of local workers in similar jobs.
Some 99,813 H2-A visas were issued last year, 85,248 in 2012, 83,844 in 2011, and 79,011 in 2010, according to Department of Labor statistics. But many more who work on farms are undocumented.
"What's most essential isn't expanding the number of workers who come into the U.S.," said Art Read, general counsel for Friends of Farmworkers, a legal service organization for low-wage farmworkers. "It's dealing with the employment authorization documents of the undocumented population.
"It is a mistake to talk merely about expanding the foreign-worker program," said Read, who is based in Philadelphia. "You want to deal with the workers who are here, making sure they can work and farmers can hire them."
The U.S. Senate last year passed immigration reform that included a new agricultural guest-worker visa program to help farmers more easily secure a workforce and allow undocumented workers to apply for a "blue card."
The workers would hold the card up to eight years while continuing farmwork and could obtain lawful permanent residence after five years and possible citizenship after an additional five years. The reform effort has stalled in the House.
Pennsylvania has more than 60,700 farmer laborers; about 36,000 of them are seasonal workers who work less than 150 days a year, and nearly all of those come from other countries, according to federal statistics. New Jersey has 24,300 farm laborers and about 14,700 seasonal workers.
Many migrant workers move from Florida to Georgia to North Carolina and New Jersey, tending and harvesting crops as the weather warms.
"H2-A doesn't work for farmers," said Ed Wengryn, research associate for the New Jersey Farm Bureau. Much of the current number of incoming workers "is absorbed before the harvest" by farms in other states such as California and Florida.
Growers must reduce the number of foreign hires by the number of local workers but are often left with insufficient hands at harvest time because the domestic helpers quit early, said Bill Troxell, a spokesman for the Pennsylvania Farm Bureau. By that time, it's too late to hire migrants.
"We need a system that's grower-friendly," said Troxell. "We have 1.3 million [foreign] farm laborers in the U.S., but 75 to 80 percent of them are illegal."
Many carry documents that appear to be genuine, "but they aren't," he said. "You can't question someone just because they're Hispanic and leave yourself open to charges of discrimination.
"But if you hire someone who is illegal," Troxell said, "you're subject to fines and could have workers taken away during the harvest season."
The need for workers - and competition for their services - will become more intense as the economy improves since many migrants shift to higher-paying jobs in the hospitality, construction, and landscaping industries.
"I provide housing on the farm and have a beautiful two-story house with four color TVs, cable, showers, washing machines, and dryers," Holtzhauser said. "I have learned how to treat my men.
"But I've seen some farmers offering them more than the minimum wage to come to their farms," he said. "In the last couple years, I've seen someone come in to talk with my men while they were sitting outside - he was looking to steal them. There's a definite shortage."
Workers are paid the minimum wage of $8.25 in New Jersey and $7.25 in Pennsylvania, but some farmers pay more to make sure they hold on to their labor force. Sheppard pays a minimum wage of $11.06 an hour. Holtzhauser pays $8.25.
"The issue of setting wage rates is a critical aspect of immigration reform - an aspect that the employer community is least interested in," Read said.
The farmers "want a simplified process," said August Wuillermin, who has a 250-acre vegetable farm in Hammonton. "But the issue has become a political football.
"If there's reform, you'll probably see a hybrid that nobody will be happy with," he said. "Then, you'll see larger amounts of imports coming in."
There's already been "a glut of imports," Wuillermin said. "We're getting squeezed."
If farmers don't get the price they need, they "have to make a decision and could end up chopping a lot" of produce, he said.
To hold on to their share of the market, American farmers must have easy access to laborers, growers say.
"There's no machine for harvesting peppers, cucumbers, and lettuce," Sheppard said. "If they aren't harvested, they won't be hauled in trucks, they won't be stocked on supermarket shelves, or show up in restaurants.
"If you deported [all foreign laborers], the economy would collapse," he said.
The migrant workers "are the backbone of our backbone," Holtzhauser said. "If you don't have them, you'll have acres and acres of fruits and vegetables that won't be harvested."