Electroencephalography (EEG) has recently gained popularity in user authentication systems since it is unique and less impacted by fraudulent interceptions. Although EEG is known to be sensitive to emotions, understanding the stability of brain responses to EEG-based authentication systems is challenging. In this study, we compared the effect of different emotion stimuli for the application in the EEG-based biometrics system (EBS). Initially, we pre-processed audio-visual evoked EEG potentials from the ‘A Database for Emotion Analysis using Physiological Signals’ (DEAP) dataset. A total of 21 time-domain and 33 frequency-domain features were extracted from the considered EEG signals in response to Low valence Low arousal (LVLA) and High valence low arousal (HVLA) stimuli. These features were fed as input to an XGBoost classifier to evaluate the performance and identify the significant features. The model performance was validated using leave-one-out cross-validation. The pipeline achieved high performance with multiclass accuracy of 80.97% and a binary-class accuracy of 99.41% with LVLA stimuli. In addition, it also achieved recall, precision and F-measure scores of 80.97%, 81.58% and 80.95%, respectively. For both the cases of LVLA and LVHA, skewness was the stand-out feature. We conclude that boring stimuli (negative experience) that fall under the LVLA category can elicit a more unique neuronal response than its counterpart the LVHA (positive experience). Thus, the proposed pipeline involving LVLA stimuli could be a potential authentication technique in security applications.