All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
The scholars of Islam have different opinions regarding this issue. But the most preponderant opinion is that whoever comes to the mosque while he had missed all the prayer except a bit less than a Rak'ah and he could join another Jamaa'ah (congregation) in the same mosque from the beginning of the prayer, then it is better for him to join the new Jamaa'ah.
This is the view of the Shaafi'i School of jurisprudence and Ibn Taymiyyah considered it more preponderant. He (Ibn Taymiyyah) said: “When the person catches less than a Rak'ah with the Imam, it is better for him if there is another Jamaa'ah with which he can pray a complete prayer, to pray with that Jamaa'ah.”
However, if he joins the first Jamaa'ah before the Imam's Tasleem, then he gets the reward of praying with the congregation.
Allah knows best.