All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
You should ignore these whispers and not pay any attention to them. You should not consider that something is impure unless you are certain that it is impure; a certainty upon which you can swear.
Moreover, there is a difference of opinion about the transfer of impurity from something to another thing that is pure.
The Hanafi school is of the view that impurity is not transferred if its effect is not seen on the pure garment, and the like, in a way that if it (the impure garment) is squeezed, no impurity would come out of it. Hence, you may act according to this view since you are affected with whispers.
Also, little impurity is something that is absolutely excused according to the view of many scholars, such as Ibn Taymiyyah . So, you may act according to this view as well, as a way of removing hardship and repelling whispers.
Finally, we warn you again against whispers and against being giving in to them as this leads to a great evil.
Allah knows best.