All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
You should avoid these whispers and not pay any attention to them.
Many jurists held that the impurity at the sole (bottom) of the slippers and socks is purified by rubbing the ground, which is the view chosen by Shaykh Ibn Taymiyyah .
Hence, there is no harm on you in acting according to this view.
Moreover, walking with your dry shoes on the floor of the house does not transfer the impurity if we assume that the shoes have become impure, because the impurity is not transmitted by contact from a dry body to another dry one.
To conclude, you have to avoid these whispers and not pay any attention to them.
For more benefit that impurity is not transferred from a dry body to another dry body that it comes in contact with, please refer to Fatawa 301366, 81546, 350434, 370297 and 427297.
Allah knows best.