The historic match celebrating 75 years of diplomatic relations took place at Japan's modest Sano International Cricket Ground outside Tokyo. It was Japan's first T20 game against a full-member nation.
India, who retained their T20 World Cup crown at home earlier this year, opted to bat first after rain and a wet outfield delayed the start and forced a five-overs-a-side shootout.
India, captained by Shreyas Iyer, struggled to score on a sluggish pitch as they made 32-4.
Japan - ranked 41st by the International Cricket Council (ICC) - pushed their more illustrious opponents all the way as they threatened a massive shock.
They were in the game until the last ball but finished on 30-3, falling agonisingly just short.
Wide controversy dominates finale
The last over was not without controversy as the India players petitioned the umpires to overturn a wide call in their favour, before winning by just two runs.
"Yeah, extremely proud, as I think I've been saying to a few other guys, that when it gets that close, it's actually a little bit tougher to take, and the emotions are running pretty high in our group," Japan skipper Kendel Kadowaki-Fleming said.
"But the fact that we were able to run the best team in the world, in my opinion, to the last ball of a game is pretty special."
Japan's 17-year-old spinner Charlie Hara-Hinze stood out with figures of 2-17 in his two overs after he opened the attack and sent back opener Abhishek Sharma for a golden duck.
Kadowaki-Fleming attempted to pull off the chase in his run-a-ball 12, but India's experienced spinners saved the heavyweights from embarrassment.
"It doesn't matter how good a team you are, because the opposition is also competitive, and the attitude and approach the Japanese team showed today was impeccable, and kudos to them," India skipper Iyer said.
"The approach was outstanding. So I'm sure that they are on the right track right now."
The build-up to what should have been a mismatch was affected by Typhoon Dujuan, with heavy rain in Sano forcing training to be cancelled on Monday.
The venue, which is also used for baseball and has a capacity of around 2,000 spectators, was dominated by India fans who had been expecting a rout.
Japan will play the opening match of men's T20 cricket at the Aichi-Nagoya Asian Games against Afghanistan on Thursday at a converted baseball field.