It is normal to worry and be concerned about what people think. The most important thing is that your intention remains for Allah. Whatever you do in life which is good, let it be for Allah. The hadith says the reward is all through the intentions. You know your intentions are for God if you are not sinning or prioritize what people think over God.