First of all, intentionally skipping a prayer is a bigger sin than breaking one’s fast. The Kaffara for breaking one’s fast does not mean that breaking a fast is more sinful. Sometimes when the sin is very big, it doesn’t have a Kaffara (meaning there is no way to expiate for it…accept a sincere repentance then yes of course it expiates for it).
Secondly, this is not just any invitation. This is a divine invitation from the king of the universe. Breaking one’s fast is insulting to God’s position since it’s a violation of his invitation. Imagine if a king invites you to his palace, then at the palace you disrespect the king, insult him, or deliberately violate his rules. This is a big offense. Now imagine this happening with our creator.