Talent Judge Decision Prediction Based on Facial Expression Using VGG16 and Fuzzy Analysis


Fakhruddin Fikriansyah
Anggunmeka Luhur Prasasti
Marisa W. Paryasto


Expression is a human communication tool to express feelings. Currently, facial expression recognition technology continues to be developed by research to improve the quality of the technology. With this, existing models will be developed in order to predict human decisions based on facial expressions. This final project uses the modified VGG16 architecture as a facial expression classification model. The datasets used for classification are FER-2013 Modification into five expressions, namely angry, disgust, happy, neutral, and surprised, with a total data of 23910. The classification model is used to read the jury's facial expressions via video in the prediction model. The result of the expression that is read will be calculated using Fuzzy Logic in determining the jury's decision with the result of 'yes' or 'no'. The results of testing the Modified VGG16 Architecture using the best combination of parameters were obtained with epoch 100, batch size 32, learning rate 0.0001, and data split 10% for validation to obtain training accuracy of 93% and validation accuracy of 86%. The model is evaluated by testing data of 10% outside of the training data, obtaining a test accuracy of 85%. The classification report from the evaluation obtained a precision of 84%, a recall of 82%, and an f1-score of 83%. The resulting model has a good performance in classifying facial expressions compared to VGG16. The results of the judge's decision prediction using fuzzy obtained a correct prediction of 20:20 from the number of test samples.


How to Cite
Fikriansyah, F., Anggunmeka Luhur Prasasti, & Marisa W. Paryasto. (2022). Talent Judge Decision Prediction Based on Facial Expression Using VGG16 and Fuzzy Analysis. MULTINETICS, 8(2), 87–96. https://doi.org/10.32722/multinetics.v8i2.4705


  1. S. Zhang, X. Pan, Y. Cui, X. Zhao, and L. Liu, “Learning Affective Video Features for Facial Expression Recognition via Hybrid Deep Learning,” IEEE Access, vol. 7, pp. 32297–32304, 2019, doi: 10.1109/ACCESS.2019.2901521.
  2. C. Qi et al., “Facial Expressions Recognition Based on Cognition and Mapped Binary Patterns,” IEEE Access, vol. 6, pp. 18795–18803, 2018, doi: 10.1109/ACCESS.2018.2816044.
  3. G. Yue and L. Lu, “Face Recognition Based on Histogram Equalization and Convolution Neural Network,” Proceedings - 2018 10th International Conference on Intelligent Human-Machine Systems and Cybernetics, IHMSC 2018, vol. 1, pp. 336–339, 2018, doi: 10.1109/IHMSC.2018.00084.
  4. Z. Zhang and M. Li, “Research on facial expression recognition based on neural network,” Proceedings - 2020 International Conference on Computer Network, Electronic and Automation, ICCNEA 2020, pp. 78–81, 2020, doi: 10.1109/ICCNEA50255.2020.00025.
  5. B. Li and D. Lima, “Facial expression recognition via ResNet-50,” International Journal of Cognitive Computing in Engineering, vol. 2, pp. 57–64, 2021, doi: 10.1016/j.ijcce.2021.02.002.
  6. J. K. Josephine Julina and T. S. Sharmila, “Facial Emotion Recognition in Videos using HOG and LBP,” 2019 4th IEEE International Conference on Recent Trends on Electronics, Information, Communication and Technology, RTEICT 2019 - Proceedings, pp. 56–60, 2019, doi: 10.1109/RTEICT46194.2019.9016766.
  7. N. Krishnadas and N. T. Bhuvan, “Facial Expression Recognition Using VGG16and LSTM,” Turkish Online Journal of Qualitative Inquiry (TOJQI), vol. 12, no. 7, pp. 9657–9674, 2021.
  8. P. N. R. Bodavarapu and P. V. V. S. Srinivas, “Facial expression recognition for low resolution images using convolutional neural networks and denoising techniques,” Indian Journal of Science and Technology, vol. 14, no. 12, pp. 971–983, 2021, doi: 10.17485/ijst/v14i12.14.
  9. N. Nour, M. Elhebir, and S. Viriri, “F ACE E XPRESSION R ECOGNITION USING C ONVOLUTION N EURAL N ETWORK ( CNN ) M ODELS,” vol. 11, no. 1, pp. 1–11, 2020, doi: 10.5121/ijgca.2020.11401.
  10. F. Altekin and H. Demir, “Emotion Detection from Facial Expression Using Different Feature Descriptor Methods with Convolutional Neural Networks,” vol. 4, no. July, pp. 14–17, 2021.
  11. B. Ko, H. G. Kim, and H. J. Choi, “Controlled dropout: A different dropout for improving training speed on deep neural network,” 2017 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2017, vol. 2017-Janua, pp. 972–977, 2017, doi: 10.1109/SMC.2017.8122736.
  12. L. Hui and S. Yu-Jie, “Research on face recognition algorithm based on improved convolution neural network,” Proceedings of the 13th IEEE Conference on Industrial Electronics and Applications, ICIEA 2018, pp. 2802–2805, 2018, doi: 10.1109/ICIEA.2018.8398186.
  13. F. A. Isman, A. L. Prasasti, and R. A. Nugrahaeni, “Expression Classification For User Experience Testing Using Convolutional Neural Network,” in 2021 International Conference on Artificial Intelligence and Mechatronics Systems (AIMS), Apr. 2021, pp. 1–6. doi: 10.1109/AIMS52415.2021.9466088.
  14. S. Tammina, “Transfer learning using VGG-16 with Deep Convolutional Neural Network for Classifying Images,” International Journal of Scientific and Research Publications (IJSRP), vol. 9, no. 10, p. p9420, 2019, doi: 10.29322/ijsrp.9.10.2019.p9420.
  15. A. Ghosh, A. Sufian, F. Sultana, A. Chakrabarti, and D. De, Fundamental concepts of convolutional neural network, vol. 172, no. January. 2019. doi: 10.1007/978-3-030-32644-9_36.
  16. A. B. Jala, T. W. Purboyo, and R. A. Nugrahaeni, “Implementation of Convolutional Neural Network (CNN) Algorithm for Classification of Human Facial Expression in Indonesia,” 2020 International Conference on Information Technology Systems and Innovation, ICITSI 2020 - Proceedings, pp. 256–262, 2020, doi: 10.1109/ICITSI50517.2020.9264940.
  17. E. D. S. Mulyani and J. P. Susanto, “Classification of maturity level of fuji apple fruit with fuzzy logic method,” 2017 5th International Conference on Cyber and IT Service Management, CITSM 2017, 2017, doi: 10.1109/CITSM.2017.8089294.
  18. F. Y. Mulato, “Klasifikasi Kematangan Buah Jambu Biji Merah ( Psidium Guajava ) dengan Menggunakan Model Fuzzy,” pp. 1–155, 2015.
  19. B. Taha and D. Hatzinakos, “Emotion Recognition from 2D Facial Expressions,” 2019 IEEE Canadian Conference of Electrical and Computer Engineering, CCECE 2019, pp. 1–4, 2019, doi: 10.1109/CCECE.2019.8861751.