The month of Muharram is indeed a month of mourning, and azadari is a slogan of Allah. To uphold this sacred slogan, we need to show that to all around us by attending azadari especially.
The right thing would to politely excuse yourself because the love of the Holy Prophet and His pure progeny carries more importance in your heart than even the nearest kith and kin. Try to remain very polite and also send your gift or visit them later on.
If you know for sure it would lead to severing family relations, and you're aware that there's no sin like mixed gathering or music, you can go for a few minutes and pray for the person.
The verse of the Holy Quran clearly states this in sura Tawba ayat 24.
Tell them, (O Prophet): "If your fathers and your sons and your brothers and your wives and your tribe and the riches you have acquired and the commerce of which you fear a slackening, and the dwellings that you love, if they are dearer to you than Allah and His Messenger and striving in His cause, then wait until Allah brings about His decree. Allah does not guide the evil-doing folk."