Global InformationList of facial expression databases information
A facial expression database is a collection of images or video clips with facial expressions of a range of emotions. Well-annotated (emotion-tagged) media content of facial behavior is essential for training, testing, and validation of algorithms for the development of expression recognition systems. The emotion annotation can be done in discrete emotion labels or on a continuous scale. Most of the databases are usually based on the basic emotions theory (by Paul Ekman) which assumes the existence of six discrete basic emotions (anger, fear, disgust, surprise, joy, sadness). However, some databases include the emotion tagging in continuous arousal-valence scale.
In posed expression databases, the participants are asked to display different basic emotional expressions, while in spontaneous expression database, the expressions are natural. Spontaneous expressions differ from posed ones remarkably in terms of intensity, configuration, and duration. Apart from this, synthesis of some AUs are barely achievable without undergoing the associated emotional state. Therefore, in most cases, the posed expressions are exaggerated, while the spontaneous ones are subtle and differ in appearance.
Many publicly available databases are categorized here.[1][2] Here are some details of the facial expression databases.
| Database | Facial expression | Number of Subjects | Number of images/videos | Gray/Color | Resolution, Frame rate | Ground truth | Type |
|---|---|---|---|---|---|---|---|
| FERG-3D-DB (Facial Expression Research Group 3D Database) for stylized characters [3] | angry, disgust, fear, joy, neutral, sad, surprise | 4 | 39574 annotated examples | Color | Emotion labels | Frontal pose | |
| Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) [4] | Speech: Calm, happy, sad, angry, fearful, surprise, disgust, and neutral.
Song: Calm, happy, sad, angry, fearful, and neutral. Each expression at two levels of emotional intensity. |
24 | 7356 video and audio files | Color | 1280x720 (720p) | Facial expression labels
Ratings provided by 319 human raters |
Posed |
| Extended Cohn-Kanade Dataset (CK+)[5] | neutral, sadness, surprise, happiness, fear, anger, contempt and disgust | 123 | 593 image sequences (327 sequences having discrete emotion labels) | Mostly gray | 640* 490 | Facial expression labels and FACS (AU label for final frame in each image sequence) | Posed; spontaneous smiles |
| Japanese Female Facial Expressions (JAFFE)[6] | neutral, sadness, surprise, happiness, fear, anger, and disgust | 10 | 213 static images | Gray | 256* 256 | Facial expression label | Posed |
| MMI Database[7] | 43 | 1280 videos and over 250 images | Color | 720* 576 | AU label for the image frame with apex facial expression in each image sequence | Posed and Spontaneous | |
| Belfast Database[8] | Set 1 (disgust, fear, amusement, frustration, surprise) | 114 | 570 video clips | Color | 720*576 | Natural Emotion | |
| Set 2 (disgust, fear, amusement, frustration, surprise, anger, sadness) | 82 | 650 video clips | Color | ||||
| Set 3 (disgust, fear, amusement) | 60 | 180 video clips | Color | 1920*1080 | |||
| Indian Semi-Acted Facial Expression Database (iSAFE)[9] | Happy, Sad, Fear, Surprise, Angry, Neutral, Disgust | 44 | 395 clips | Color | 1920x1080
(60 fps) |
Emotion labels | Spontaneous |
| DISFA[10] | - | 27 | 4,845 video frames | Color | 1024*768; 20 fps | AU intensity for each video frame (12 AUs) | Spontaneous |
| Multimedia Understanding Group (MUG)[11] | neutral, sadness, surprise, happiness, fear, anger, and disgust | 86 | 1462 sequences | Color | 896*896, 19fps | Emotion labels | Posed |
| Indian Spontaneous Expression Database (ISED)[12] | sadness, surprise, happiness, and disgust | 50 | 428 videos | Color | 1920* 1080, 50 fps | Emotion labels | Spontaneous |
| Radboud Faces Database (RaFD)[13] | neutral, sadness, contempt, surprise, happiness, fear, anger, and disgust | 67 | Three different gaze directions and five camera angles (8*67*3*5=8040 images) | Color | 681*1024 | Emotion labels | Posed |
| Oulu-CASIA NIR-VIS database | surprise, happiness, sadness, anger, fear and disgust | 80 | three different illumination conditions: normal, weak and dark (total 2880 video sequences) | Color | 320×240 | Posed | |
| FERG (Facial Expression Research Group Database)-DB[14] for stylized characters | angry, disgust, fear, joy, neutral, sad, surprise | 6 | 55767 | Color | 768x768 | Emotion labels | Frontal pose |
| AffectNet[15] | neutral, happy, sad, surprise, fear, disgust, anger, contempt | ~450,000 manually annotated
~ 500,000 automatically annotated |
Color | Various | Emotion labels, valence, arousal | Wild setting | |
| IMPA-FACE3D[16] | neutral frontal, joy, sadness, surprise, anger, disgust, fear, opened, closed, kiss, left side, right side, neutral sagittal left, neutral sagittal right, nape and forehead (acquired sometimes) | 38 | 534 static images | Color | 640X480 | Emotion labels | Posed |
| FEI Face Database | neutral,smile | 200 | 2800 static images | Color | 640X480 | Emotion labels | Posed |
| Aff-Wild[17][18] | valence and arousal | 200 | ~1,250,000 manually annotated | Color | Various (average = 640x360) | Valence, Arousal | In-the-Wild setting |
| Aff-Wild2[19][20] | neutral, happiness, sadness, surprise, fear, disgust, anger + valence-arousal + action units 1,2,4,6,12,15,20,25 | 458 | ~2,800,000 manually annotated | Color | Various (average = 1030x630) | Valence, Arousal, 7 basic expressions, action units for each video frame | In-the-Wild setting |
| Real-world Affective Faces Database (RAF-DB)[21][22] | 6 classes of basic emotions (Surprised, Fear, Disgust, Happy, Sad, Angry) plus Neutral and 12 classes of compound emotions (Fearfully Surprised, Fearfully Disgusted, Sadly Angry, Sadly Fearful, Angrily Disgusted, Angrily Surprised, Sadly Disgusted, Disgustedly Surprised, Happily Surprised, Sadly Surprised, Fearfully Angry, Happily Disgusted) | 29672 annotated examples | Color | Various for original dataset and 100x100 for aligned dataset | Emotion labels | Posed and Spontaneous |
- ^ "collection of emotional databases". Archived from the original on 2018-03-25.
- ^ "facial expression databases".
- ^ Aneja, Deepali, et al. "Learning to generate 3D stylized character expressions from humans." 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2018.
- ^ Livingstone & Russo (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. doi:10.1371/journal.pone.0196391
- ^ P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar and I. Matthews, "The Extended Cohn-Kanade Dataset (CK+): A complete facial expression dataset for action unit and emotion-specified expression," in 3rd IEEE Workshop on CVPR for Human Communicative Behavior Analysis, 2010
- ^ Lyons, Michael; Kamachi, Miyuki; Gyoba, Jiro (1998). The Japanese Female Facial Expression (JAFFE) Database. doi:10.5281/zenodo.3451524.
- ^ M. Valstar and M. Pantic, "Induced disgust, happiness and surprise: an addition to the MMI facial expression database," in Proc. Int. Conf. Language Resources and Evaluation, 2010
- ^ I. Sneddon, M. McRorie, G. McKeown and J. Hanratty, "The Belfast induced natural emotion database," IEEE Trans. Affective Computing, vol. 3, no. 1, pp. 32-41, 2012
- ^ Singh, Shivendra; Benedict, Shajulin (2020). "Indian Semi-Acted Facial Expression (ISAFE) Dataset for Human Emotions Recognition". In Thampi, Sabu M.; Hegde, Rajesh M.; Krishnan, Sri; Mukhopadhyay, Jayanta; Chaudhary, Vipin; Marques, Oge; Piramuthu, Selwyn; Corchado, Juan M. (eds.). Advances in Signal Processing and Intelligent Recognition Systems. Communications in Computer and Information Science. Vol. 1209. Singapore: Springer. pp. 150–162. doi:10.1007/978-981-15-4828-4_13. ISBN 978-981-15-4828-4.
- ^ S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh and J. Cohn., "DISFA: A Spontaneous Facial Action Intensity Database," IEEE Trans. Affective Computing, vol. 4, no. 2, pp. 151–160, 2013
- ^ N. Aifanti, C. Papachristou and A. Delopoulos, The MUG Facial Expression Database, in Proc. 11th Int. Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), Desenzano, Italy, April 12–14, 2010.
- ^ S L Happy, P. Patnaik, A. Routray, and R. Guha, "The Indian Spontaneous Expression Database for Emotion Recognition," in IEEE Transactions on Affective Computing, 2016, doi:10.1109/TAFFC.2015.2498174.
- ^ Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D.H.J., Hawk, S.T., & van Knippenberg, A. (2010). Presentation and validation of the Radboud Faces Database. Cognition & Emotion, 24(8), 1377—1388. doi:10.1080/02699930903485076
- ^ "Facial Expression Research Group Database (FERG-DB)". grail.cs.washington.edu. Retrieved 2016-12-06.
- ^ Mollahosseini, A.; Hasani, B.; Mahoor, M. H. (2017). "AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild". IEEE Transactions on Affective Computing. PP (99): 18–31. arXiv:1708.03985. doi:10.1109/TAFFC.2017.2740923. ISSN 1949-3045. S2CID 37515850.
- ^ "IMPA-FACE3D Technical Reports". visgraf.impa.br. Retrieved 2018-03-08.
- ^ Zafeiriou, S.; Kollias, D.; Nicolaou, M.A.; Papaioannou, A.; Zhao, G.; Kotsia, I. (2017). "Aff-Wild: Valence and Arousal 'In-the-Wild' Challenge" (PDF). 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 1980–1987. doi:10.1109/CVPRW.2017.248. ISBN 978-1-5386-0733-6. S2CID 3107614.
- ^ Kollias, D.; Tzirakis, P.; Nicolaou, M.A.; Papaioannou, A.; Zhao, G.; Schuller, B.; Kotsia, I.; Zafeiriou, S. (2019). "Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond". International Journal of Computer Vision. 127 (6–7): 907–929. arXiv:1804.10938. doi:10.1007/s11263-019-01158-4. S2CID 13679040.
- ^ Kollias, D.; Zafeiriou, S. (2019). "Expression, affect, action unit recognition: Aff-wild2, multi-task learning and arcface" (PDF). British Machine Vision Conference (BMVC), 2019. arXiv:1910.04855.
- ^ Kollias, D.; Schulc, A.; Hajiyev, E.; Zafeiriou, S. (2020). "Analysing Affective Behavior in the First ABAW 2020 Competition". 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020). pp. 637–643. arXiv:2001.11409. doi:10.1109/FG47880.2020.00126. ISBN 978-1-7281-3079-8. S2CID 210966051.
- ^ Li., S. "RAF-DB". Real-world Affective Faces Database.
- ^ Li, S.; Deng, W.; Du, J. (2017). "Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild". 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2584–2593. doi:10.1109/CVPR.2017.277. ISBN 978-1-5386-0457-1. S2CID 11413183.