Praise be to Allah, the Lord of the Worlds; and may His blessings and peace be upon our Prophet Muhammad and upon all his Family and Companions.
First of all you have to make sincere repentance for the sin you committed, i.e. using Riba. Allah declares war against those who engage in Riba. Allah Says (interpretation of meaning): {O you who believe! Be afraid of Allâh and give up what remains (due to you) from Ribâ (usury) (from now onward), if you are (really) believers. And if you do not do it, then take a notice of war from Allâh and His Messenger ….}[2:278-279].
Tawbah means giving up the sin totally, having deep sorrow for the violation of the ruling of Allah, and deciding firmly not to return to that sin again. Know that, the fact that the government was paying the interest does not change its rulings. Taking usury/interest-based loan itself is forbidden regardless of whether the government pays the interest or the person himself pays it; approving of the contract of usury itself is forbidden.
Now, if you do not fear committing Zina for not being married then you have to pay the loan first, even if it takes a long time, since the one who does not pay a loan though he is able to pay it, is a procrastinator and doing so is an injustice as mentioned in a sound Hadith. However, if you fear committing Zina then you have to get married first as Imam al-Dardeeri said in his book.
The Hajj has the least priority in these matters since it becomes obligatory only by capability. Allah Says (interpretation of meaning): {….And Hajj (pilgrimage to Makkah) to the House (Ka'bah) is a duty that mankind owes to Allâh, those who can afford the expenses (for one's conveyance, provision and residence);….}[3:97].
An indebted person is not a capable person unless he has permission of the creditor. However, if getting married does not become compulsory then one should perform Hajj first since performing Hajj immediately after its being obligatory is compulsory, according to the preponderant opinion of Muslim scholars.
Allah knows best.