The actor, who's been dating British musician FKA Twigs since 2014, made the cryptic revelation on the Howard Stern radio show.
He told the veteran host that he was wary of releasing details of his private life because of ‘crazy’ Twilight fans.
He described them as, "A crack group of crazies who think every single decision you're making is either creating some kind of conspiracy or...".
Speaking of his relationship, he said - "So you kind of think, like, to protect it, 'I want to create a big boundary between it', [but] then it makes it difficult for your actual relationship."
Knowing what to say and what to keep private was ‘one of the most frustrating things in the world’ he said.