We don’t have a hadith that says he wasn’t allowed to go. The hadith mentions that Imam Hassan and Hussain (peace be upon them) went with the angels. They didn’t mention Hazrat Abbas (a). Possibly only an infallible Imam could attend.