All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
You should ignore your whispers and not pay any attention to them because giving in to them leads you to a great evil.
Nonetheless, we will clarify to you what you are obliged to wash in ablution and what you must not wash.
The hair that is attached to the whole of the ear is from the face and it must be washed along with the face. The same thing applies to the whiteness that is between this hair and the ear: it is from the face and it must be washed.
As for the hair that is above the ear, which is close to the head and that comes down from the head, and perhaps this is what you are asking for (and which you referred to as sideburns), then this is from the head, and it not an obligation to wash it when washing the face. Rather, it should be wiped when wiping the head.
With regard to the whiteness that is above the ear [and before the hair of the head], then it is not an obligation to wash it either since it is part of the head.
Now that you know this, then if you are in doubt about washing something that you are obliged to wash, then doubting something after having finished an act of worship does not affect its validity.
In principle, all your previous ablutions are valid, so you do not repeat any prayer. However, if you are certain that you have left out something (in ablution) that you are obliged to wash and you were ignorant about the ruling, then there is a difference of opinion among the scholars about whether or not you must repeat it.
However, since you are affected with whispers, then you can act according to the view that does not oblige you to repeat, which is the view chosen by Shaykh Ibn Taymiyyah .
For more benefit, please refer to Fataawa 330295 and 325087.
Allah knows best.