Yes the impact of gheeba upon those who listen to it is harmful as well
You need to continue to object to the wrong as this is your duty. Speak gently to the person but firmly that you don’t want such statements or words in your house. You should not remain silent about it. Allah knows how hard it is for you especially as he’s a guest but He will help you and reward you. Keep reminding the person that they’re eating flesh of their brother and sister. That image should be on their minds when they’re about to backbite