Little Britain star David Walliams says rumours about X Factor mogul Simon Cowell’s sexuality will always surround him – even though he says he is straight.

“I work with Simon Cowell – a lot of people say he’s gay. He’s not but it will follow him until the day he dies. What’s the point of worrying about it?” Walliams told Metro.

“There’s nothing wrong with being gay, so if people say I am, it doesn’t bother me. I’m not going to sue about that. I’d sue if people said I was a rapist.”

Last month, during an interview to promote his new autobiography entitled, Camp David, Walliams revealed that previously he had gay experiences as a teenager and also spoke about the rumours surrounding his sexuality.

The 41-year-old told the Sun: “I’ve always been effeminate, and I think people confuse effeminacy with homosexuality, like they go hand-in-hand.

“But there are plenty of effeminate people who are straight, and plenty of butch people who are gay”.

Earlier this year Walliams, who has been a judge on Cowell’s other series, Britain’s Got Talent, joked about the press attention surrounding the 53-year-old’s sexuality during a joint appearance on Jonathan Ross’ chat show.

Walliams told Ross: “The main question I get is, ‘Is Simon gay?’ Even my mum asks. She says, ‘He’s nice. Is he gay?’… Well we know the answer to that. With the gay thing, the jury is still out.”

Cowell was seen to be visibly cringing during the jokes telling Walliams: “Most of this will be edited out.”