All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ‘alayhi wa sallam, is His slave and Messenger.
We have not found any Islamic evidence or statements from the scholars indicating that if someone does not pardon those who have wronged him, his Hajj will not be accepted. Someone who has been wronged is not obliged in Islam to pardon the person who wronged him; rather, the person who has been wronged is entitled to retaliate in kind against the oppressor. However, to pardon is much more encouraged in Islam, and it is one of the greatest means of drawing closer to Allaah The Exalted.
It is recommended for someone who has been wronged to pardon his relatives who have wronged him with even greater reason because they share family ties; such an act is rather more rewardable, Allaah willing. There is no doubt that oppression emanating from one’s relatives is harder to endure, but pardoning their oppression is much more rewardable and the fruits reaped in this case are incomparable with those reaped in case of retaliation against them. Please refer to Fataawa 84351 and 89899.
Therefore, you are advised to make an effort to pardon them, and not leave room for Satan to lure you into missing such great rewards and attaining the honorable status of those who pardon. You will reap the fruits of such a righteous deed in both the world and the Hereafter; Allaah The Exalted says (what means): {And those who strive for Us – We will surely guide them to Our ways. And indeed, Allaah is with the doers of good.} [Quran 29:69]
In fact, you did well by retaining relations with them and maintaining the ties of kinship and being polite with them. This is sufficient as far as family ties are concerned. However, you are advised to pardon them and forgive their transgressions. We implore Allaah to facilitate and accept your Hajj, to forgive your sins, and to raise your rank.
Allaah Knows best.