All perfect praise be to Allaah, the Lord of the worlds. I testify that there is none worthy of worship except Allaah and that Muhammad, sallallaahu ‘alayhi wa sallam, is His slave and Messenger.
Whoever manages to catch one Rak‘ah with the Imaam in the Jumu‘ah prayer caught the prayer and should only make up for the missed Rak‘ah and he should not perform four Rak‘ahs. If a person catches only one Rak’ah of the Jumu‘ah prayer, then he caught the Jumu‘ah prayer and should not pray the Thuhr prayer instead. Not listening to the Khutbah (Friday sermon) has nothing to do with the validity of the Jumu‘ah prayer, as it is not a condition for the validity of the prayer of the Ma'moom (one being led in prayer).
Hence, as the questioner had already performed four Rak‘ahs with the intention of performing the Thuhr prayer and not two Rak‘ahs with the intention of performing the Jumu‘ah prayer, it is more prudent to perform this Thuhr prayer again (so as to be on the safe side). Scholars held different opinions regarding a worshipper who performs the Thuhr prayer on Friday before the end time of the Jumu‘ah prayer. Some believed that if a worshipper performs the Thuhr prayer in Friday before the end time of the Jumu‘ah prayer, his prayer is invalid. This is according to the Hanbali School of Fiqh, the later Shaafi‘i school of Fiqh (that was complied in Egypt) and the most correct opinion in the Maaliki School of Fiqh. You performed two Rak‘ahs from the Thuhr prayer before the end time of the Jumu‘ah prayer. Therefore, you should perform this Thuhr prayer again just to be on the safe side.
Allaah Knows best.