It’s very unfortunate that he treats you all this way, especially your mother. Islam is not just praying and fasting. A good Muslims never treats his family like that. Islam is all about good Akhlaaq. May Allah guide him.
Some recommendations:
1- You have tried to engage with him in many different ways, but I’d still recommend you brainstorm and experiment with new ways to deal with him. Be creative. It’s possible that he had a rough upbringing and was emotionally abused when he was a child. Maybe he has emotional needs. Try to show him love and compassion and see what happens. Hug him, kiss him, say nice things to him, etc… Sometimes it works.
2- If nothing works, then try to avoid him as much as possible. Avoid anything that triggers him. Just say salam to him as him if he needs anything and keep a distance from him. That would reduce the toxic environment at home.
3- Don’t underestimate the power of Du’a. Continue praying Allah changes this situation and guides him.
I recommend the follow A’mal. They are effective:
1- Do sujud for several minutes each night and with a deep breath each time say: سبحان ربي الاعلى وبحمده (subhana rabbial a3la wa be7amdeh)
2- Before sleeping, say 100 times لا اله الا الله (la ilaha illa allah) and take a deep breath each time
3- Read this Du’a:
https://duas.mobi/amp/dua/worries
4- Read this Du’a 4 times a day:
“Allahomma inni atawajjahu ilayka bi nabiyyika nabiyyer-rahma wa ahle baytihi-lladheena khtartahom ala ilmin alal alameen. Allahomma dhallel li so'obataha wa huzunataha wakfini sharraha fa innakal kafil mo'afi wal ghalibol qaher."
5- Put the Qur’an on your heart and recite verse 82 of Sura al-Israa:
وَنُنَزِّلُ مِنَ الْقُرْآنِ مَا هُوَ شِفَاءٌ وَرَحْمَةٌ لِّلْمُؤْمِنِينَ
6- Read this Dua daily:
https://www.duas.org/Amn.htm
7- Everyday pray 2 rak’as like the morning prayer, then when done gift it to Imam Mahdi. Say O Allah I offer this prayer as a gift to Imam Mahdi. Then talk to Imam Zaman and ask him to help you with this situation. This prayer is very effective
8- After the morning prayer, but your hand on your heart and say 70 times يا فتاح (ya fatta7)