All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
Your husband has no authoritative power over your brother’s wife, but it is obligatory to order her to cover up (i.e., to wear the Hijab) and to forbid her from displaying her body parts that should be covered (and being scantily dressed).
Therefore, if your husband commanded you to forbid her from displaying her body parts that should be covered (and being scantily dressed) and ordering her to wear the Hijab, obey him in that, unless you suffer harm or you fear great harm that will result from ordering her to wear the Hijab, then there is nothing wrong with you in abandoning ordering her to wear it while denying it in your heart. In addition, if you are waived from the obligation to order this woman to wear Hijab because you despair of the benefit of ordering her to do so, or because you fear harm from ordering her, or that it will result in greater harm if you order her, then your husband does not have the right to oblige you to do so, nor to leave you because you did not obey him in that.
Allah knows best.