All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger.
There are three kinds of the prostration for forgetfulness: obligatory, recommended and permissible.
First: The obligatory prostration for forgetfulness:
a- If a praying person unintentionally adds one Rukoo' (bowing) or Sujood (prostration) or something similar of the actions of the prayer, he has to prostrate for forgetfulness. Ibn Mas'ood said: "The Prophet led us in prayer and he performed five Rak'ahs. When he turned around after ending the prayer, the people whispered to one another. He said: "What is the matter?" They said: 'O Prophet of Allaah! Has the prayer been increased (i.e. by Allaah)?' He replied: "No". They said: 'You performed five Rak'ahs'. So, he stood up again and performed two prostrations for forgetfulness, and then, said Salaam. After that, he said: "I am just a human being like you (i.e. I forget)” [Muslim]
b- If the praying person performs less than the required number of Rak'ahs of the prayer. `Imraan Ibn al-Husayn said: "The Prophet ended his prayer with Salaam after performing three Raka'hs during the `Asr prayer. Then, he entered his room. Thereupon, a man stood up and said: 'Has the prayer been shortened?' The Prophet came out of his room and performed the fourth Rak'ah that he forgot and then made two prostrations as prostration for forgetfulness (before concluding the prayer), and finally he concluded the prayer by saying Salaam." [Muslim]
c- If the praying person does not perform any of the obligations (Waajibaat) of the prayer such as Tasbeeh in the bowing position (Saying Subhaana Rabbiya Al-'Atheem) or in the prostration position (Saying Subhaana Rabbiya Al-A'laa), or if he does not recite the first Tashahhud (in the three or four Rak'ahs prayer).
If one intentionally does not perform the prostration for forgetfulness in these and similar cases, his/her prayer becomes void.
Second: The recommended prostration for forgetfulness: The prostration for forgetfulness is recommended if a praying person unintentionally utters a legislated mention of Allaah in a wrong position such as reciting Al-Faatihah while he/she is in the bowing down position or in the prostration.
Third: The permissible prostration for forgetfulness: The prostration for forgetfulness is permissible if a praying person leaves out any of the supererogatory mention of Allaah such as not reciting an additional portion of the Quran (after Al-Faatihah) or abandoning the recitation of the introductory supplication of the prayer.
According to Imaam Ahmad's School of jurisprudence, the prostration for forgetfulness should be done before saying Salaam to conclude the prayer in general. However, the prostration for forgetfulness should be performed after finishing the prayer if the praying person – as in the narration from 'Imraan Ibn Al-Husayn – forgets one Rak'ah or more, or if one doubts the number of Rak'ahs and thus bases on that which he is certain of, then the prostration for forgetfulness should be performed after finishing the prayer. In a narration reported by Ibn Mas'ood the Prophet said: "When anyone of you becomes uncertain during prayer (i.e. of how much he has prayed), he should try to remember the exact number and then complete what he missed of his prayer and finally perform two prostrations for forgetfulness." [Al-Bukhari] Another narration reads: “One has to complete his prayer, say Salaam and then perform the prostration for forgetfulness.”
Moreover, Imaam Muslim reported: “The Prophet completed his prayer, said Salaam, and then performed the prostration for forgetfulness.”
Furthermore, it should be mentioned that it is permissible for the prostration for forgetfulness to be performed before or after saying Salaam, since the two ways were authentically recorded in sound narrations.
Nonetheless, it is necessary to know that it is not acceptable to perform the prostration for forgetfulness in case of forgetting or leaving out any of the pillars of the prayer. This is not sufficient as leaving out any pillar nullifies the prayer and thus it must be performed again. The same ruling applies to the conditions for the validity of the prayer.
Our final advice is to follow the rulings of the prostration for forgetfulness as mentioned according to Imaam Ahmad since they are supported by evidence and they suffice your need and help you avoid any misunderstanding or confusion.
Allaah Knows best.