I think it was just cringey/embarrassing. He was a serious Shakespearean actor who wanted to do drama (as well as Loki) and all of a sudden he was in a relationship with someone like Taylor who everyone thought writes songs for teenagers and it just didn't match his serious image. They were far too public with their publicity stunt, the pictures were so staged and he even got his mother, sister and niece involved in it, he invited paps to all corners of the world to take pictures of him and Taylor being affectionate in strategically romantic locations. Then there was the I heart TS t shirt debacle and that awkward picture of Ryan & Blake and him and Taylor being weirdly affectionate, it was all just a total onslaught of 'romance' and it felt out of place. Everyone poked fun at him for it and he dropped off the radar for a year to deal with the fallout, he said he was taking a year out to focus on himself. Taylor was dating her current boyfriend Joe Alwyn while she was with (or doing a publicity stunt with, if you want to see it that way) Tom and she ended up writing a song about him.
The insane attention Tom got during that time, most of it negative, I feel like it traumatised him and resulted in him having a very negative view of the media. Now he never answers personal questions in interviews and his publicist tells journalists they can't ask him about Taylor since 5 year later that's still the topic everyone wants to ask him about. It damaged his reputation as a serious actor and I think it's just another example of a showmance gone wrong. The GQ profile piece also didn't help, it was meant to be damage limitation/reinvention or just to explain his side of the story but it made him look desperate and insecure. The journalist seemed to think he was weird and clingy/needy, he kept her until late babbling and trying to explain himself and then the next day, early in the morning, went to her hotel room (house?) to explain himself all over again and make sure she understood what he'd said. Some of his fans have speculated that he suffers from depression or low mood sometimes and the Hiddleswift situation exacerbated that. His publicist and management clearly thought that he was toxic after the debacle and wanted him to lie low for a while, I think he also wanted to do that too.
I think he's doing the kinds of jobs he wants to do. I'm looking forward to The Essex Serpent since I loved the book but it's a shame White Stork has been shelved. I thought a politician role would be perfect for him and would be a return to gritty drama after Loki. He doesn't touch his Instagram and I think he's scared off PR relationships for life after Taylor.
And I should also mention that the journalist who did the aforementioned GQ profile piece was photographed by paps giving him a goodbye hug outside his house and referred to as a 'mystery girl.' She was married and Tom had to call her up to apologise and hoped her husband wasn't offended by it.