Israel twice went ahead before Italy appeared to seal the win when taking a two-goal lead, but a dramatic finish needed an added time goal to snatch a 5-4 victory and give them hope of qualification after missing the last two World Cups.
Gattuso has made a winning start to his reign after replacing Luciano Spalletti, following a 5-0 win over Estonia on Friday with another victory three days later but it was an altogether different performance in Hungary against Israel.
"We're crazy - we conceded some absurd goals but we will hold onto this win," Gattuso told Italian broadcaster Rai.
"Our opponents surprised us a bit, they were marking man-to-man. Today our legs weren't at their best but that's normal, it's always like this in the second match, though we need to improve."
Italy twice conceded own goals and were lucky to escape another after keeper Gianluigi Donnarumma fisted the ball into his own net but was relieved when it was disallowed for a foul on him, and at times they looked very shaky at the back.
"We're too fragile, we give away goals too easily. The boys know it but the problem is mine, not theirs," Gattuso said.
"To play as a unit, we need to improve in certain areas. My staff and I need to improve as soon as possible, I'm not criticising my players."
Italy twice came from behind thanks to Moise Kean's double, and although they almost let the win slip from their grasp, there was at least a different spirit to the team than in the 3-0 loss to Norway which cost Spalletti his job.
"Credit to the team, because we had the strength to react to every blow we took, but we can't allow easy goals like we did today," Gattuso said.
"Even though it wasn't a great day, there was a desire to fight back and that's the most important thing.
"I'll thank the team in the locker room, but if we want to achieve something important, we have to grow."
Italy, second in the group on nine points ahead of Israel on goal difference with a game in hand, are still three points off Norway who have also played four matches.
The top team in the group qualify for next year's finals with the second-placed side going into the playoffs.