Republican Rep. F. James Sensenbrenner Jr. of Wisconsin suggested that some political appointees at Justice should be removed immediately, and asked Holder when he was going to "clean up this mess."
He added, "If you don't get to the bottom of this," the only other option might be "impeachment."
Between the fall of 2009 and January 2011, federal Bureau of Alcohol, Tobacco, Firearms and Explosives agents purposely allowed weapons to be illegally purchased and circulated on both sides of the U.S.-Mexico border in hopes that they could be tracked to drug-cartel leaders.
Instead, the operation ended up arming the cartels.
'For years to come'
Two of the weapons turned up at the scene of a shooting in which a U.S. Border Patrol agent was killed in southern Arizona a year ago. Scores of others were reportedly used in violent crimes in Mexico.
"It is an unfortunate reality that we will continue to feel the effects of this flawed operation for years to come," Holder said. "Guns lost during this operation will continue to show up at crime scenes on both sides of the border."
He added, in language pointed at Republicans on the panel, that "as we work to avoid future losses and further mistakes, it is unfortunate that some have used inflammatory and inappropriate rhetoric about one particular tragedy that occurred near the Southwest border in an effort to score political points."
Holder has maintained that he was unaware of Fast and Furious until months after Agent Brian Terry's death last December.
He has since requested a Justice Department inspector general's investigation and ordered his employees never to open similar operations that include illegal gun "walking."
Republicans have countered that he should have known much earlier, and if he did not, it suggests that he is incompetent to continue as the nation's chief law enforcement officer. They also have hinted that Holder might even have approved or condoned the gun-walking tactics under Fast and Furious.
Under questioning by Rep. Jason Chaffetz (R., Utah), Holder said he had not spoken with President Obama, other Obama cabinet members, or the president of Mexico about Fast and Furious.
"You don't have 15 minutes to pick up the phone?" Chaffetz said.
Holder said that others in his department had discussed Fast and Furious with top U.S. and Mexican officials and that he had made some "personnel changes" at ATF headquarters.
On the Democratic side, Rep. Mike Quigley of Illinois described Fast and Furious as a "horrible screw-up."
Among the Justice officials the Republicans want ousted is Lanny A. Breuer, the assistant attorney general in charge of the department's criminal division.
On Wednesday, Iowa Sen. Charles E. Grassley, the ranking Republican on the Senate Judiciary Committee, said in a floor speech that "it is time for him to go," noting that Breuer had helped draft an erroneous letter to Congress about Fast and Furious.
Michael Chertoff, the former Homeland Security secretary who early in the George W. Bush administration had the job Breuer now holds, spoke out on behalf of Breuer, whom Chertoff has known for more than a dozen years. "He's a very able, dedicated prosecutor," Chertoff said. "He acknowledged he was mistaken in not bringing this issue up to the attorney general at an earlier point in time, and that's an admirable thing to do."
Tracy Schmaler, the chief Justice spokeswoman, said Holder "continues to have confidence" in Breuer.
Rep. Lamar Smith (R., Texas), chairman of the House Judiciary Committee, said, "Many questions remain as to how such a reckless and dangerous law enforcement program was allowed to operate under the Justice Department."
This article contains information from the Associated Press.