The Kaba is the direction that we pray, we don’t worship the Kaba. It’s different with idol worshippers who worshipped and pray to the idols. Furthermore, the idols were something that people made up. They have no holiness other than people saying this is holy. The Kaba is the first masjid of worship and it’s a place designated by God.