Nuno Espirito Santo's side twice blew the lead as Jarrod Bowen's double was cancelled out by an Igor Thiago brace to force extra time at the London Stadium.
But in the shootout, Brentford winger Ouattara attempted a chipped 'Panenka' penalty, and saw it saved by West Ham keeper Alphonse Areola.
It was an awful mistake by the Burkina Faso international, and West Ham made him pay. Bowen, Valentin Castellanos, Callum Wilson, Tomas Soucek and Konstantinos Mavropanos all converted their spot-kicks, ensuring West Ham will host Leeds in April for a place in the semi-finals.
The East London club's first FA Cup quarter-final for 10 years is a welcome respite in a difficult season.
They sit third bottom of the Premier League table, behind Nottingham Forest on goal difference, with nine games to save themselves from crashing into the Championship.
Seventh in the Premier League and in contention for European qualification, Brentford missed the chance to reach the quarter-finals for the first time since 1989.
Haunted by the threat of relegation, Nuno made seven changes to the side that won at Fulham in the league last week as he prioritised their survival bid.
West Ham took the lead in the 19th minute when Mateus Fernandes' cross to the far post was headed down by Tomas Soucek, and Bowen reacted quickest to steer past Brentford keeper Caoimhin Kelleher.
Brentford drew level nine minutes later as Thiago glanced Nathan Collins' header into the net with his chest, the goal surviving a VAR check for a potential handball and offside.
The Hammers moved back ahead in the 34th minute when Adama Traore was tripped inside the penalty area by Michael Kayode. Andy Madley didn't give the spot-kick, but VAR official Constantine Hatzidakis told the referee to consult the pitch-side monitor, and he changed his mind after watching the incident again.
Unfazed by the lengthy delay, Bowen sent Kelleher the wrong way from the spot.
Collins' header was cleared off the line by Ollie Scarles before West Ham's Axel Disasi missed a golden opportunity, scuffing his chance from close-range after Kelleher denied Soucek.
Kelleher made another fine save from Soucek on the stroke of half-time.
Bowen was unable to complete a hat-trick as Kelleher produced yet another good stop from the England forward in the second half.
West Ham's misses came back to haunt them in the 81st minute when Brentford equalised to force extra time. Crysencio Summerville conceded the penalty with a push on Kayode, and Thiago stepped up to fire home from the spot.
The Brazilian forward has 21 goals in all competitions in his breakthrough campaign as he pushes for a place in his country's World Cup squad.
Having used all their substitutes, West Ham finished extra-time with 10 men after Summerville hobbled off in the closing moments.
However, they still managed to prevail in the shootout.