Carl Hagelin returned to the Rangers' lineup after sitting out a three-game suspension for his elbow on Alfredsson
The Rangers broke through in the second with three goals after going six straight periods without scoring.
"We showed some desperation tonight," Rangers defenseman Marc Staal said.
Chris Neil gave Ottawa a 1-0 lead in the first with a power-play goal and Jason Spezza cut it to 3-2 with 39 seconds left to play in the third. Neil was planted at the edge of Henrik Lundqvist's crease along with teammate Zack Smith when Ottawa's power play put it ahead, 1-0, at 7:05. Sergei Gonchar's point shot deflected in off Neil's skate as he tried to jump out of the way.
That got the frenzied, towel-waving crowd into a lather that only increased when Neil answered Brandon Prust's challenge for a fight 8 minutes later.
The momentum seemed to be completely in Ottawa's favor when the first intermission hit, especially with goalie Craig Anderson locked into a lengthy shutout streak and looking sharp once again.
But the Senators lost their focus in the second period and referees Steve Kozari and Tim Peel gave the Rangers four power plays.
Stepan tied the game, 1-1, during the first one at 8:55, taking two whacks at a sweet pass from Richards to beat Anderson. That ended the Rangers' scoring drought at 145 minutes, 27 seconds.
A call that Ottawa fans will question helped put New York ahead for good.
New York received a lengthy, 5-on-3 power play after Nick Foligno was sent off for goaltender interference - replays showed two Rangers players guiding him into Lundqvist - and Richards took advantage of the extra space by skating off the wall and beating Anderson high at 17:08.
That was soon followed by a back-breaking goal from Chris Kreider, the first of his NHL career. He sneaked a shot through Anderson with 40.3 seconds to play in the period to make it 3-1.
"I didn't really think that we played particularly well or worked particularly hard," Ottawa coach Paul MacLean said.
In another game *
At Chicago, Mike Smith made 39 saves and the Phoenix Coyotes eliminated the Blackhawks from the Western Conference playoffs with a 4-0 victory in Game 6.
Oliver Ekman-Larsson scored in the second period and Gilbert Brule, Antoine Vermette and Kyle Chipchura had goals in the third as the Coyotes won a first-round series for the first time since moving to Phoenix for the 1996-97 season. It is the first series triumph for the franchise since 1987, when it was still the Winnipeg Jets.
The Coyotes won all three games at the United Center - Game 6 was the first not to be decided in overtime - and now move on to play Nashville in the conference semifinals.