All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
First of all, we did not understand what you mean by saying “I swore on my Afterlife”.
In any case, what we can say in brief is that swearing by anyone or anything besides Allah is a sin that requires repentance.
Nonetheless, it is an oath that is not effective so it does not entail expiation if the person breaks it.
Al-Mawardi, from the Shaafi'i School of jurisprudence, said in al-Haawi regarding swearing by other than Allah:
“It (the oath) is not effective and it is not obligatory to fulfill it, and there is no expiration for it if one breaks his oath; this is like something agreed upon (among the scholars).” [End of quote]
Ibn Qudaamah said:
"An oath is not considered binding if the person swears by a created being or something such as the Ka‘bah, Prophets, or any other creature. An expiation is not obligatory on the oath-taker if he breaks the oath. This is the apparent meaning of the opinion of Imaam Al-Khiraqi . It is also the view of most of the Muslim jurists .” [End of quote]
Therefore, if you swore by something other than Allah not to wear some clothes and then you wore them, then you are not obliged to expiate because your oath is not binding (not effective) in principle, but you must repent to Allah for swearing by other than Him.
However, your statement 'wallah iladheem ala akhirtee’; this means that you swore by Allah. Hence, if you wore what you have sworn not to wear, while knowing it and doing it deliberately, then you are obliged to expiate once even though you wore the clothes many times, because your oath does not mean repetition.
In case you wore the clothes out of forgetfulness, then the view we adopt in our Site (Islamweb) is that whoever breaks his oath out of forgetfulness is not obliged to expiate for it [for breaking his oath out of forgetfulness]. But if you expiate in order to be on the safe side and be acquitted from sin, then it is better. The expiation is feeding ten poor people, or clothing them, or emancipating a slave. If you cannot find any of these, then you have to fast for three days.
Allah knows best.