The 29-year-old striker joined Sunderland in 2013 but struggled to make an impact at the Stadium of Light, scoring just three goals in all competitions.
Altidore left the North East less than two years later, when he was used as part of a swap deal which allowed Jermain Defoe to join Sunderland.
But Altidore, who has won over 100 caps for the USA, has re-found his form in the MLS and admitted all was not well during his time in England.
"A lot of people don’t know, but I was going through a lot off the field in England, whether that was staying out of trouble [or with] relationships, it [spilled] into my soccer and it got to the point where the club had to intervene," Altidore told the Toronto Sun.
The hilarious reason Newcastle United goalkeeper Nick Pope went viral on Twitter following Burger King tweet
Newcastle United transfer target confirms move ahead of official announcement
Newcastle United ‘in talks’ over Chelsea duo, Toon set to walk away from ‘£60m’ discussions
Newcastle United to ‘push ahead’ with £48m double raid, Toon in ‘pole position’ for £42m Tottenham star
Tottenham Hotspur star ‘banished’ from squad amid Newcastle United speculation - reports
"And so my time at Sunderland, aside from the first two months, was really just bad in terms of off the field. Soccer was secondary. It’s something I really haven’t talked about. But it is what it is. It’s life. But looking back it now, it made me a stronger person.
"That’s the toughest part to swallow. I wasn’t able to really go and give it my all there. I was just side-tracked by other things."
Toronto head coach Greg Vanney revealed he met Altidore at a quiet pub in London in 2014 to seal the striker's move to America.
"It was in the basement, the underground portion of a pub, on a Sunday morning where nobody was going to be. And it was just the two of us, and we just chatted," said Vanney.
That conversation persuaded Altidore, who wasn't sure about the move at first, to return to the MLS following spells in Spain, Holland and England.
"I remember thinking obviously [Toronto] is a great city but I wasn’t ready to go on a team that lost every game but I liked the idea of trying to turn a franchise around that’s been historically not very good," added the striker.
"And I liked the idea of being part of the reason why it turned around. And I thought there was a big reward there, a big gain in terms of coming and really re-writing the history. That was exciting.
"Greg had his little notebook [at the pub] and he was just knocking out things. And I just thought, ‘Wow. This means a lot to him and them’.
"I left that meeting and while my mind wasn’t made up, I was definitely thinking, ‘This could be good. Why not them?’ It was just a great, honest meeting and I left it thinking, ‘Wow, playing for someone like that can for sure help me get better.
"It’s the right timing, maybe I can consider it.’ And so from that moment on, I started to consider playing in Toronto."