Once a man came to see the 8th Imam (as) and he heard people talking about his past. So he wept bitterly and said to the Imam, I’ve changed my ways and yet they still speak ill off me.
The Imam (as) responded: does it not suffice you, that I am pleased with you?
You don’t need to please the haters, haters are going to hate.
You only need to please your lord, avoid the ignorant and keep the company of the righteous.
May Allah give you support and change your image in the eyes of the people to a righteous image.