It was the Flyers' sixth straight loss (0-4-2) in Denver since 2002.
The Flyers reached the season's halfway point - 41 games - with a 20-17-4 record.
The Avs nearly took a 3-1 lead with 8 minutes, 59 seconds left in regulation, but Jamie McGinn's goal was waved off. It was ruled that goalie Steve Mason had the puck covered and that the whistle had been blown.
Mason (29 saves) was outstanding in the loss. "I thought the effort was there tonight, but we just didn't get the bounces," he said.
Colorado had taken a 2-0 lead when Ryan O'Reilly's close-range shot deflected off Mason's pad, fluttered over him, and one-hopped into the net with 10:43 left in the second period.
Wayne Simmonds cut it to 2-1 about 41/2 minutes later as Andrej Meszaros' point shot appeared to carom off the winger's knee and past Varlamov, the former Washington Capitals goaltender. It was Simmonds' team-high 13th goal and his seventh tally in the last five games.
Earlier, Colorado capitalized on Sean Couturier's neutral-zone turnover to start a three-on-two and take a 1-0 lead with 8:46 remaining in the first period.
McGinn, whose brother, Tye, plays for the Phantoms, knocked in a rebound of Matt Duchene's shot to give the Avalanche the lead. Mason, who made a handful of acrobatic saves early in the game, had no chance to make the stop.
The Flyers were listless for most of the period until they applied pressure on Varlamov during a power play, firing five shots, including a breakaway by Steve Downie, playing his first game against his former team. Varlamov stopped Downie's breakaway backhander with 4:50 left in the first.
With about three minutes to go in the first, Raffl was knocked hard into the boards by Colorado's Gabriel Landeskog, and he woozily left the ice. The Flyers later announced he had a concussion and was out indefinitely.
Landeskog was not penalized for the hit.
With Raffl out, several players took turns playing left wing on the No. 1 line, and the top three lines were juggled.
Coach Craig Berube was pleased with the effort from a team that was missing two key forwards, Raffl and Matt Read.
"There were no passengers tonight," he said. "They all worked hard."
Before Thursday, the Flyers had scored at least four goals in seven of their last eight games, and they were on a 7-1-1 run. They averaged 1.47 goals in their first 15 games, and 3.6 goals in their next 25 games.