All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah and that Muhammad, sallallaahu ʻalayhi wa sallam, is His slave and Messenger.
It is impermissible for your brother to keep his wife in the family house if she may be subject to fitnah (temptation) or be a source of fitnah for others, as is the case in the situation described in the question. The husband is enjoined to keep his wife safe from potential fitnah, as entailed by the duties of him being in charge of her. Allaah, The Exalted, says (what means): {Men are in charge of women by (right of) what Allaah has given one over the other and what they spend (for maintenance) from their wealth...} [Quran 4:34] As-Saʻdi wrote, “This means that men are in charge of women in terms of obliging them to fulfill the rights of Allaah over them by observing His obligations and preventing them from committing evils...”
It should be noted that the wife is entitled to a separate house so as to spare her any hardship or inconvenience of sharing the house with any person, as we highlighted in fatwa 84608. If the husband provides such accommodation for his wife, even if it within the family house (in a separate portion of the house), then there is no harm in that. Your brother should be advised to strive in providing his wife with the required accommodation even if he has to pay rent.
This wife is enjoined to keenly cover her ʻawrah (body parts that should be concealed as per the sharee'ah) before the non-mahram (marriageable) family members in the family house and avoid staying in seclusion with her husband's brothers because they are not among her mahram men. If she laxly failed to observe modesty and adherence to the relevant sharee'ah provisions governing man-woman interactions with her male in-laws, then she bears a sin for that. The husband's brothers are also enjoined to avoid being alone with their brother's wife; and whoever of them is negligent in this respect bears a sin for that. It should be noted, though, that the sins committed by mistake are pardoned. Allaah, The Exalted, says about the supplications of the believers (what means): {...Our Lord, do not impose blame upon us if we have forgotten or erred...} [Quran 2:286] It has been reported that Allaah, The Exalted, says (when the believers recite these supplications), “I have done so.” [Saheeh Muslim] Praise be to Allaah for that.
For more benefit, please, refer to fatwa 127911.
Allaah knows best.