Abstract

Facial emotion recognition is a process that involves detecting and interpreting emotions from facial expressions. This field draws from a range of disciplines, including computer science, psychology, and neuroscience. The ability to accurately recognize emotions from facial expressions has broad implications for humancomputer interaction, healthcare, security, and marketing. This paper presents a thorough overview of the current state of the art in facial emotion recognition, covering topics such as facial expression theories, types of facial emotion recognition, datasets, techniques, evaluation metrics, and applications. Additionally, the paper addresses ethical considerations related to facial emotion recognition. The aim of this survey is to provide a comprehensive understanding of the present state of facial emotion recognition technology.

References

• MAH Akhand, Shuvendu Roy, Nazmul Siddique, Md Abdus Samad Kamal, and Tetsuya Shimamura. Facial emotion recognition using transfer learning in the deep cnn. Electronics, 10(9):1036, 2021.
• Abdulrahman Alreshidi and Mohib Ullah. Facial emotion recognition using hybrid features. In Informatics, volume 7, page 6. MDPI, 2020.
• Shadi AlZu’bi, Raed Abu Zitar, Bilal Hawashin, Samia Abu Shanab, Amjed Zraiqat, Ala Mughaid, Khaled H. Almotairi, and Laith Abualigah. A novel deep learning technique for detecting emotional impact in online education. Electronics, 11(18), 2022.
• Tarun Kumar Arora, Pavan Kumar Chaubey, Manju Shree Raman, Bhupendra Kumar, Yagnam Nagesh, PK Anjani, Hamed MS Ahmed, Arshad Hashmi, S Balamuralitharan, and Baru Debtera. Optimal facial feature based emotional recognition using deep learning algorithm. Computational Intelligence and Neuroscience, 2022, 2022.
• Lisa Feldman Barrett. How emotions are made: The secret life of the brain. Pan Macmillan, 2017.
• Ross Buck. Nonverbal behavior and the theory of emotion: the facial feedback hypothesis. Journal of Personality and social Psychology, 38(5):811, 1980.
• Prateek Chhikara, Prabhjot Singh, Rajkumar Tekchandani, Neeraj Kumar, and Mohsen Guizani. Federated learning meets human emotions: A decentralized framework for human– computer interaction for iot applications. IEEE Internet of Things Journal, 8(8):6949–6962, 2021.
• M Kalpana Chowdary, Tu N Nguyen, and D Jude Hemanth. Deep learning-based facial emotion recognition for human– computer interaction applications. Neural Computing and Applications, pages 1–18, 2021.
• Charles Darwin and Phillip Prodger. The expression of the emotions in man and animals. Oxford University Press, USA, 1998.
• Shichuan Du, Yong Tao, and Aleix M Martinez. Compound facial expressions of emotion. Proceedings of the national academy of sciences, 111(15):E1454–E1462, 2014.
• Paul Ekman. An argument for basic emotions. Cognition & emotion, 6(3-4):169–200, 1992.
• Paul Ekman andWallace V Friesen. Constants across cultures in the face and emotion. Journal of personality and social psychology, 17(2):124, 1971.
• Paul Ekman and Wallace V Friesen. Facial action coding system. Environmental Psychology &Nonverbal Behavior, 1978.
• Paul Ekman,Wallace V Friesen, and Phoebe Ellsworth. Emotion in the Human Face: Guide-lines for Research and an Integration of Findings: Guidelines for Research and an Integration of Findings. Pergamon, 1972.
• Paul Ekman, E Richard Sorenson, and Wallace V Friesen. Pan-cultural elements in facial displays of emotion. Science, 164(3875):86–88, 1969.
• Zixiang Fei, Erfu Yang, David Day-Uei Li, Stephen Butler, Winifred Ijomah, Xia Li, and Huiyu Zhou. Deep convolution network based emotion analysis towards mental health care. Neurocomputing, 388:212–227, 2020.
• Mingxing Gong. A novel performance measure for machine learning classification. International Journal of Managing Information Technology (IJMIT) Vol, 13, 2021.
• Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in representation learning: A report on three machine learning contests. In International conference on neural information processing, pages 117–124. Springer, 2013.
• Jianzhu Guo, Zhen Lei, Jun Wan, Egils Avots, Noushin Hajarolasvadi, Boris Knyazev, Artem Kuharenko, Julio C. Silveira Jacques Junior, Xavier Bar´o, Hasan Demirel, Sergio Escalera, J¨uri Allik, and Gholamreza Anbarjafari. Dominant and complementary emotion recognition from still images of faces. IEEE Access, 6:26391–26403, 2018.
• Jianzhu Guo, Zhen Lei, Jun Wan, Egils Avots, Noushin Hajarolasvadi, Boris Knyazev, Artem Kuharenko, Julio C Silveira Jacques Junior, Xavier Bar´o, Hasan Demirel, et al. Dominant and complementary emotion recognition from still images of faces. IEEE Access, 6:26391–26403, 2018.
• Kanika Gupta, Megha Gupta, Jabez Christopher, and Vasan Arunachalam. Fuzzy system for facial emotion recognition. In Intelligent Systems Design and Applications: 20th International Conference on Intelligent Systems Design and Applications (ISDA 2020) held December 12-15, 2020, pages 536– 552. Springer, 2021.
• Selen Is¸ık Ulusoy, S¸ eref Abdurrahman G¨ulseren, Nermin O¨ zkan, and Cu¨neyt Bilen. Facial emotion recognition deficits in patients with bipolar disorder and their healthy parents. General Hospital Psychiatry, 65:9–14, 2020.
• Ping Jiang, Bo Wan, Quan Wang, and Jiang Wu. Fast and efficient facial expression recognition using a gabor convolutional network. IEEE Signal Processing Letters, 27:1954– 1958, 2020.
• Dorota Kami´nska, Kadir Aktas, Davit Rizhinashvili, Danila Kuklyanov, Abdallah Hussein Sham, Sergio Escalera, Kamal Nasrollahi, Thomas B Moeslund, and Gholamreza Anbarjafari. Two-stage recognition and beyond for compound facial emotion recognition. Electronics, 10(22):2847, 2021.
• Asad Khattak, Muhammad Zubair Asghar, Mushtaq Ali, and Ulfat Batool. An efficient deep learning technique for facial emotion recognition. Multimedia Tools and Applications, 81(2):1649–1683, 2022.
• Imane Lasri, Anouar Riad Solh, and Mourad El Belkacemi. Facial emotion recognition of students using convolutional neural network. In 2019 Third International Conference on Intelligent Computing in Data Sciences (ICDS), pages 1–6, 2019.
• Chong Li, Mingzhao Yang, Yongting Zhang, and Khin Wee Lai. An intelligent mental health identification method for college students: A mixed-method study. International Journal of Environmental Research and Public Health, 19(22), 2022.
• Shan Li, Weihong Deng, and JunPing Du. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2584– 2593, 2017.
• Patrick Lucey, Jeffrey F Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. The extended cohnkanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In 2010 ieee computer society conference on computer vision and pattern recognitionworkshops, pages 94–101. IEEE, 2010.
• Michael Lyons, Miyuki Kamachi, and Jiro Gyoba. The Japanese Female Facial Expression (JAFFE) Dataset. April 1998.
• J´an Magyar, Gergely Magyar, and Peter Sincak. A cloudbased voting system for emotion recognition in humancomputer interaction. In 2018 World Symposium on Digital Intelligence for Systems and Machines (DISA), pages 109– 114, 2018.
• Abigail A Marsh, Reginald B Adams Jr, and Robert E Kleck. Why do fear and anger look the way they do? form and social function in facial expressions. Personality and Social Psychology Bulletin, 31(1):73–86, 2005.
• David Matsumoto. Culture and nonverbal behavior. The SAGE handbook of nonverbal communication, pages 219– 235, 2006.
• Golam Morshed, Hamimah Ujir, and Irwandi Hipiny. Customer’s spontaneous facial expression recognition. Indonesian Journal of Electrical Engineering and Computer Science, 22(3):1436–1445, 2021.
• Madhuka Nadeeshani, Akash Jayaweera, and Pradeepa Samarasinghe. Facial emotion prediction through action units and deep learning. In 2020 2nd International Conference on Advancements in Computing (ICAC), volume 1, pages 293–298, 2020.
• Shaldon Wade Naidoo, Nalindren Naicker, Sulaiman Saleem Patel, and Prinavin Govender. Computer vision: The effectiveness of deep learning for emotion detection in marketing campaigns. International Journal of Advanced Computer Science and Applications, 13(5), 2022.
• Heenakausar Pendhari, Sushma Nagdeoti, Sandeep Rathod, Lubna Khan, and Saurabh Vishwakarma. Compound emotions: A mixed emotion detection. Available at SSRN 4120265, 2022.
• Robert Plutchik and Henry Kellerman. Emotion, theory, research, and experience, volume 3. Academic press, 1980.
• Lance M. Rappaport, Nicole Di Nardo, Melissa A. Brotman, Daniel S. Pine, Ellen Leibenluft, Roxann Roberson-Nay, and John M. Hettema. Pediatric anxiety associated with altered facial emotion recognition. Journal of Anxiety Disorders, 82:102432, 2021.
• James A Russell. Is there universal recognition of emotion from facial expression? a review of the cross-cultural studies. Psychological bulletin, 115(1):102, 1994.
• Khadija Slimani, Khadija Lekdioui, Rochdi Messoussi, and Raja Touahni. Compound facial expression recognition based on highway cnn. In Proceedings of the New Challenges in Data Sciences: Acts of the Second Conference of the Moroccan Classification Society, pages 1–7, 2019.
• Iryna Spivak, Svitlana Krepych, Vasyl Faifura, and Serhii Spivak. Methods and tools of face recognition for the marketing decision making. In 2019 IEEE International Scientific- Practical Conference Problems of Infocommunications, Science and Technology (PIC S&T), pages 212–216, 2019.
• A Swaminathan, A Vadivel, and Michael Arock. Ferce: facial expression recognition for combined emotions using ferce algorithm. IETE Journal of Research, 68(5):3235–3250, 2022.
• Selvarajah Thuseethan, Sutharshan Rajasegarar, and John Yearwood. Complex emotion profiling: An incremental active learning based approach with sparse annotations. IEEE Access, 8:147711–147727, 2020.
• G¨uray Tonguc¸ and Betul Ozaydın Ozkara. Automatic recognition of student emotions from facial expressions during a lecture. Computers & Education, 148:103797, 2020.
• NiWayan SuryaWardhani, Masithoh Yessi Rochayani, Atiek Iriany, Agus Dwi Sulistyono, and Prayudi Lestantyo. Crossvalidation metrics for evaluating classification performance on imbalanced data. In 2019 International conference on computer, control, informatics and its applications (IC3INA), pages 14–18. IEEE, 2019.
• Yuanlun Xie, Wenhong Tian, and Tingsong Ma. A transfer learning approach to compound facial expression recognition. In 2020 4th International Conference on Advances in Image Processing, pages 95–101, 2020.
• Tian Xu, Jennifer White, Sinan Kalkan, and Hatice Gunes. Investigating bias and fairness in facial expression recognition. In European Conference on Computer Vision, pages 506–523. Springer, 2020.
• Milad Mohammad Taghi Zadeh, Maryam Imani, and Babak Majidi. Fast facial emotion recognition using convolutional neural networks and gabor filters. In 2019 5th Conference on Knowledge Based Engineering and Innovation (KBEI), pages 577–581. IEEE, 2019.