All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger. We ask Allaah to exalt his mention as well as that of his family and all his companions.
The supplication of a fasting person when he breaks his fast is accepted by Allaah, so we advise you to supplicate for your husband in the hope that Allaah will turn him into a righteous man and make him regain his senses, if what you mentioned about him is confirmed, like abandoning the prayers and committing other prohibitions. For more benefit, please refer to Fatwa 84397.
Therefore, we advise you to be patient with him and advise him in a good and soft manner, especially when he is fasting as he is more likely to accept advice. Then if his situation improves, all perfect praise be to Allaah, otherwise you have the right to ask him to divorce you if you think that divorce is better for you. For more benefit, please refer to Fatwa 90785, 90046 and 88010.
As regards accusing him of adultery, it is not permissible to do so without a clear evidence, and in principle he is innocent from this.
Allaah Knows best.