One way to establish that is to refer to Motawater hadiths that all Muslims have accepted. For example:
1- The hadith of Thiqlayn clearly proves the infallibility of the Ahlulbayt, since it says they shall never separate from the Qur’an till the Day of Judgment. If someone never separates from the infallible Qur’an, he must be infallible.
2- The Sahih hadiths that Imam Ali is the most knowledgeable, the most just, is always with the Qur’an, is loved by Allah and His messenger, and so on. Imam Ali’s status is not disputed by any Muslim. Yes some Sunnis claim for instance he may have upset Lady Fatima (in a fabricated incident) and irritated the Prophet, but they all agree the Prophet loved him deeply and Islam succeeded because of him. And all Muslims agree that Hassan and Hussain are the masters of the youth of paradise, and all their descendants (the Imams) were pious. There is no dispute in that. So we have certainly over the positive status of Ahlulbayt.
3- As for those enemies and hypocrites, not all Muslims agree on them. There are hundreds of reports that indicate they committed injustice—all from Sunni books. You can see these videos for that: