Wrexham's Liberato Cacace and Oliver Rathbone scored in quick succession in the first half. The second-tier team looked headed for victory with a 3-1 lead in a breathless second half before Callum Hudson-Odoi struck in the 76th and 89th minutes.
Forest boss Sean Dyche had selected a virtual reserve team with eight changes from Tuesday's Premier League win over West Ham United.
But he sent on reinforcements in Neco Williams, Nicolas Dominguez, and Morgan Gibbs-White for the second half, and it paid off as Igor Jesus pulled one back in the 64th minute, nodding in a cross off the post.
Dominic Hyam restored Wrexham's two-goal advantage when he dove to head home a free kick in the 74th minute before Hudson-Odoi's heroics.
But Wrexham prevailed in the shootout after extra time ended without a goal and the victory was another highlight for a team that has climbed from non-league football to the second tier in three seasons under the ownership of Hollywood A-listers Ryan Reynolds and Rob McElhenney. Canadian Reynolds was at Friday's game.
"It was a rollercoaster game," Wrexham boss Phil Parkinson told TNT Sports. "The late goal knocked the stuffing out of us a bit... We had to rally the troops a bit. The whole ground had gone flat."
Reflecting on the shootout, he added: "I didn't watch it. I was staring at the crowd and waiting for their reaction... Arthur has produced the save and will be the hero, and rightly so."
Championship side Oxford United beat League Two's MK Dons on penalties after their match ended 1-1, while a couple of League One teams moved on, with Wigan Athletic beating Championship side Preston North End 1-0, and Port Vale defeating League Two's Fleetwood Town 1-0.