Home captain Mitchell Marsh made a defiant 88 but Australia, chasing 297 for victory, could never recover from the Maharaj-inflicted blows and were bundled out for 198 inside 41 overs.
Put into bat, South Africa posted 296-8 after Aiden Markram, Matthew Breetzke and skipper Temba Bavuma produced half-centuries.
Travis Head was the pick of the Australia bowlers, claiming 4-57, but Maharaja proved nearly unplayable when the hosts returned to bat.
"I made my debut in 2016 in Australia. So it's a wonderful opportunity to get my first five-for here," Maharaj said after collecting the player of the match award.
"I just tried to utilise the condition. The wicket was a little bit sticky with the new ball. I just tried to utilise that and got the reward."
South Africa got off to a strong start after Markram (82) added 92 runs for the opening stand with Ryan Rickelton, who made 33.
Rickelton got an lbw decision against him reversed but was dismissed by Head, while Markram fell near the halfway stage of their innings.
Bavuma (65) and Breetzke (57) combined in a 92-run stand, though South Africa fell short of the 300-mark.
When Australia began their chase, Head hit Nandre Burger for five fours in an over as the hosts reached the 50-mark in five overs.
South Africa struck back with spin, and once Prenelan Subrayen dismissed Head, Maharaj turned the game on its head, claiming five wickets in 26 deliveries.
Marnus Labuschagne and Alex Carey were trapped lbw, Cameron Green and Aaron Hardie had their off-stumps knocked back, while Josh Inglis had his defence breached as Australia slumped to 89-6.
With wickets tumbling at the other end, Marsh, who hit 10 fours, had to temper his natural aggression.
Ben Dwarshuis contributed 33 down the order, but it was of little significance.
The teams move to Mackay for the second ODI of the three-match series on Friday.
Australia prevailed 2-1 in the preceding T20 series between the sides.