Mina Tanaka scored a first-half brace and set up Maika Hamano's goal after the break before Moeka Minami completed the rout with a 75th-minute header in the opening match of the invitational SheBelieves Cup.
Three of the goals came from defensive errors, leaving interim coach Sermanni fuming.
"Probably the most disappointing thing was... I just felt some of the goals that we gave up were very poor, and that's kind of un-Australian," the Scot said in the post-match press conference.
"We want to be a team that's hard to beat, and to be honest today we really gave up some soft goals. But overall, more than a fair result."
A poor clearance opened the door for Tanaka to score with a clever flick in the sixth minute, and she grabbed her second near the half-hour mark after defenders Clare Hunt and Alanna Kennedy clattered into each other in the box chasing a high ball.
Minami nodded home the fourth after beating goalkeeper Mackenzie Arnold to the ball as she attempted to punch clear.
Japan could easily have had another two goals while the Matildas had only one shot for the entire match, substitute Charlotte Grant hitting the bar in the closing stages.
Sermanni said the match was a "pretty huge wake-up call" for Australia, who host the Asian Cup in just over a year's time.
"I suppose the positive thing is now it gives us a chance to say, 'well, OK, come this time next year when we're just about getting into the Asian Cup, what do we need to do to beat teams like Japan?'"
The Matildas' preparations for the four-team SheBelieves Cup were overshadowed by injured captain Sam Kerr's court case in London and a pay dispute between the players and Football Australia.
Kerr was found not guilty of racially abusing a police officer in a trial that gripped her home nation.
Sermanni said the off-field issues had no impact on the team's performance.
"It would be false of me to deflect anything on the performance today from anything off the field," he said.