Why did Allah let the Ka’bah be destroyed on several occasions shouldn’t he have protected it like he did against ashabal fil?

Perhaps the reason why Ashab al-Fil were destroyed was because they had a personal problem with the Kaba, while in the other cases Kaba was destroyed was an accident and not a purposeful intent to destroy the Kaba itself.