Final Report Summary - MMFP (Multimodal Face Processing)
Facial image processing and analysis is the task of automatically analyzing face images in order to acquire information about the depicted persons. This includes, for example, a person’s identity, emotional state, age, and gender. There are two main objectives of this project. The first one is building a common framework to derive information from face images and the second one is joint maximization of information extraction performance. In addition, the project addresses the task of benchmarking face processing under ambient conditions.
Towards these goals, significant progress has been achieved during the second period of the project. Taking into account the impact of deep learning in the computer vision field, face representation has been updated with convolutional neural networks (CNN). Age estimation, gender classification, facial expression recognition, and face recognition systems have been revised and further developed. These developed technologies have been evaluated under ambient, challenging conditions. The contributions can be summarized as follows. (i) We have developed a face recognition system that exploits deep CNN representation and score normalization. The developed system was benchmarked in the International Challenge on Biometric Recognition in the Wild 2016. In addition, a comprehensive analysis of deep learning representations under varying conditions has been conducted. (ii) We developed a facial expression recognition system that utilizes an ensemble of static and dynamic representations. The developed system was evaluated within the Emotion Recognition in the Wild Challenge 2016. (iii) We addressed the problem of apparent age estimation. In this problem, face images have multiple age labels corresponding to the ages perceived by the annotators. The developed system has been benchmarked at the ChaLearn Apparent Age Estimation Challenge. (iv) Transfer of deep representations for specific tasks, such as age and gender classification has been studied. Benchmarks have been conducted on the challenging Adience dataset. (v) We have investigated innovative applications of facial image analysis, such as visual estimation of taste appreciation and intelligent tutoring.
Extensive collaborations with researchers in Europe, Asia, and USA have been established and several joint papers have been published. The researcher has also completed three international projects –two bilateral projects and one multinational project. Moreover, he has received additional grants from national funding resources and industry. Besides, to widen his research network in Europe further, he has been management committee member of three COST Actions. The research work during the project has been published in four peer-reviewed journal papers, 24 peer-reviewed international conference and workshop papers. Three journal papers are under review. The researcher has received a national outstanding young scientist award in 2016.
The researcher is currently an associate professor and has accomplished to receive a tenured, permanent position at the host institution. He has established his own research group, SiMiT Lab. Many activities have been carried out to disseminate the results of the conducted research to the larger scientific community, as well as to the general public, through publications in prestigious journals, conferences, public website, demo videos, participation in a technology fair, open lab days, and a magazine article.
Towards these goals, significant progress has been achieved during the second period of the project. Taking into account the impact of deep learning in the computer vision field, face representation has been updated with convolutional neural networks (CNN). Age estimation, gender classification, facial expression recognition, and face recognition systems have been revised and further developed. These developed technologies have been evaluated under ambient, challenging conditions. The contributions can be summarized as follows. (i) We have developed a face recognition system that exploits deep CNN representation and score normalization. The developed system was benchmarked in the International Challenge on Biometric Recognition in the Wild 2016. In addition, a comprehensive analysis of deep learning representations under varying conditions has been conducted. (ii) We developed a facial expression recognition system that utilizes an ensemble of static and dynamic representations. The developed system was evaluated within the Emotion Recognition in the Wild Challenge 2016. (iii) We addressed the problem of apparent age estimation. In this problem, face images have multiple age labels corresponding to the ages perceived by the annotators. The developed system has been benchmarked at the ChaLearn Apparent Age Estimation Challenge. (iv) Transfer of deep representations for specific tasks, such as age and gender classification has been studied. Benchmarks have been conducted on the challenging Adience dataset. (v) We have investigated innovative applications of facial image analysis, such as visual estimation of taste appreciation and intelligent tutoring.
Extensive collaborations with researchers in Europe, Asia, and USA have been established and several joint papers have been published. The researcher has also completed three international projects –two bilateral projects and one multinational project. Moreover, he has received additional grants from national funding resources and industry. Besides, to widen his research network in Europe further, he has been management committee member of three COST Actions. The research work during the project has been published in four peer-reviewed journal papers, 24 peer-reviewed international conference and workshop papers. Three journal papers are under review. The researcher has received a national outstanding young scientist award in 2016.
The researcher is currently an associate professor and has accomplished to receive a tenured, permanent position at the host institution. He has established his own research group, SiMiT Lab. Many activities have been carried out to disseminate the results of the conducted research to the larger scientific community, as well as to the general public, through publications in prestigious journals, conferences, public website, demo videos, participation in a technology fair, open lab days, and a magazine article.