Praise be to Allah, the Lord of the Worlds; and may His blessings and peace be upon our Prophet Muhammad and upon all his Family and Companions.
Allah, The Most High, ordained us to be kind and dutiful to our parents even if they are Kafirs (non-Muslims).
Allah says: "But if they (both) strive with you to make you join in worship with Me others that of which you have no knowledge, then obey them not, but behave with them in the world kindly. (Luqman 31:15)
So Allah ordered us to behave with them kindly in this world. Bukhari reported in his Sahih that Asma Bint Abu Bakr, may Allah be pleased with her, said: "My mother came to me, hoping (for my favor) during the life time of the Prophet. I asked the Prophet, "May I treat her kindly?" He replied, "Yes". So the verse of Allah and the Hadith of the Prophet (Sallallahu Alaihi wa Sallam) indicate that if one parent is Kafir, he should be treated kindly. This is of course if this does not lead to an evil at the expense of ties of kinship and kindness.
Therefore, we do not consider that you should prevent this man from seeing his children except if he is as you described in the question, that he hates Islam, disbelieves in Allah and does acts that take him out of the fold of Islam, and if you definitely think that he will influence your children, it is permissible for you to prevent him from staying alone with your children or to prevent the children from visiting him.
So if you are sure that he tried to corrupt them, then you have to absolutely forbid him from ever seeing them or contacting them by phone again.
Because in doing so, there is an Islamic benefit for your children, to preserve their belief and morals from corruption.
We ask Allah to help you educate your children in the way and manner that Allah and His Prophet (Sallallahu Alaihi wa Sallam) like.
Allah knows best.