The pressure on the Spurs boss eased with a 2-0 win over Borussia Dortmund in the Champions League on Tuesday, but anything other than victory at relegation-threatened Burnley this weekend would renew scrutiny on him.
The Dane told reporters at his pre-match press conference on Thursday that despite the noise, support from club bosses had been "exceptional throughout the season".
"I know there's a few questions about me, but it's not about me," said the 52-year-old.
"It's about me, the staff, the players, the club, and that needs to be aligned. And that direction and alignment are very clear from ownership to the board.
"Everyone is on the same page, and there are a lot of things going in the right direction."
He added: "I have lunch with Vinai (Venkatesham, chief executive) three times a week because it's a good way to talk and get around all the small bits we need to do because we're also busy with meetings and a busy schedule."
Spurs fans chanted "You're getting sacked in the morning" after last week's 2-1 home defeat by relegation-threatened West Ham, which left them 14th in the table.
Tottenham are already out of both domestic cups, though the win against Dortmund secured a place in the Champions League play-off round.
Van de Ven is a 'Tottenham player'
The Spurs boss was asked about the future of defender Micky van de Ven, who has been linked with a move to Liverpool.
He said the Dutchman "seems happy" in London.
"He's a fantastic player and a fantastic ambassador for this club," said Frank.
"He's maybe had his best season for the club. He's growing as a leader and a big player for the future.
"I would say he is a very important player for us. Micky is a Tottenham player for us now and the future."