Joel Ward scored a hat trick as the Flyers fell to 3-9, their worst 12-game start in franchise history.
"It's embarrassing. We're not playing to our potential," captain Claude Giroux said.
After the second period, Holmgren went into the locker room and chastised the players.
"Once they scored the first goal, we kind of just stopped playing," he said. "We seem like we're afraid to play the game right now, afraid to make plays, afraid to battle for pucks, afraid to skate after pucks.
"It's tough to watch."
With 14 minutes, 29 seconds left, four fights erupted, including a matchup between the goalies as Ray Emery pounded the Caps' Braden Holtby.
The fans roared.
Emery was penalized two minutes for leaving the crease, two minutes for instigating, five minutes for fighting, and a 10-minute game misconduct. (Goalie Steve Mason, who started the game, was forced to reenter.)
The other fights: Wayne Simmonds and Tom Wilson; Vinny Lecavalier and Steve Olesky; and Brayden Schenn and Alexander Urbom. All told, 74 minutes in penalties were called for the brawl, with 54 minutes assessed to the Flyers. Lecavalier will miss Saturday's game with a facial injury, Holmgren said.
"When you're getting slapped around like that [on the scoreboard], it's a response," Holmgren said. "Do I have an issue with it? Probably not. It's a response from an embarrassed hockey team."
"They care. They're frustrated. That's basically what it boils down to," coach Craig Berube said of the brawl.
As for the "Fire Holmgren!" chants and boos, the general manager said he understood the fans' frustration level.
"If I was sitting in the stands, I'm not sure I'd be chanting, but I might be thinking the same thing," Holmgren said. "It's part of the business."
Earlier, the Flyers actually controlled most of the first period; they did not allow the Caps a shot until 4:56 remained in the opening period.
From that point, the Flyers looked disinterested until the end of the second period. Their competitive level disappeared. So did their back-checking. During one 20-minute stretch, they were outshot, 15-3.
Washington (6-7) scored five second-period goals, three against Emery, who replaced Mason with 17:16 left in the middle stanza. Mason allowed three goals on eight shots, but couldn't be faulted for any of them.
Each of the first three goals came after Flyers turnovers.
Steve Downie, in his Flyers debut, was scoreless and lost a fight to Aaron Volpatti. Downie, who 17 penalty minutes, was cut about his left eye in the bout and suffered a concussion, Holmgren said.