Real-Time User Attention and Typing Behavior Analysis via MediaPipe Face Mesh and Spatial Threshold Calibration
DOI:
https://doi.org/10.67308/6fjx5g72Keywords:
Face Mesh, MediaPipe, Gaze Detection, Attention Tracking, Flask, Thread Safety, Typing Analytics, HCI.Abstract
Automated user attention mapping frameworks have attained an important role in measuring human-computer interaction (HCI) metrics and online educational methods. Conventional vision-based solutions usually result in substantial computational overhead from convolutional models or do not feature precise geometric thresholds, which cannot be adjusted for different seating positions. This paper suggests an efficient and non-intrusive software system able to track the user’s screen-versus-keyboard attention. More precisely, it is capable of distinguishing when a person is looking at the computer screen and when the person is gazing at a keyboard. The solution uses a normal webcam video stream to receive data through the MediaPipe Face Mesh technology that enables automatic 3D coordinate system construction for getting a vertical head-pose offset. The two-level automatic system calibration process is carried out by means of a special model allowing data processing through asynchronous Flask web server work. In addition, to guarantee that no frame loss occurs at the hardware level, thread safety is ensured via a special mutex lock.
Downloads
References
[1] X. Zhang, Y. Sugano, M. Fritz, and A. Bulling, "Appearance-based gaze estimation in the wild," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 11, pp. 2724–2737, 2019.
[2] A. Nigam, R. Pasricha, T. Singh, and P. Pradhan, "A systematic review on AI-based proctoring systems for online examinations," Wireless Personal Communications, vol. 119, no. 4, pp. 3143–3165, 2021.
[3] L. Xu, Z. Li, Y. Gan, and H. Xia, "Soft-label guided stacked dual attention network for head pose estimation and its application to classroom gaze analysis," Scientific Reports, vol. 16, no. 1, p. 405, 2026.
[4] S. Jha and C. Busso, "Estimation of driver's gaze region from head position and orientation using probabilistic confidence regions," IEEE Transactions on Intelligent Vehicles, vol. 8, pp. 59–72, 2022.
[5] K. Meyer, A. Al-Hamadi, and R. Niese, "Real-time head pose estimation using Random Forests," Journal of Real-Time Image Processing, vol. 14, no. 3, pp. 621–633, 2018.
[6] P. Majaranta and K.-J. Räihä, "Twenty years of eye typing: systems and design issues," in Proceedings of the 2002 Symposium on Eye Tracking Research & Applications, 2002, pp. 15–22.
[7] P. Majaranta and K.-J. Räihä, "Text entry by gaze: utilizing eye-tracking," in Text Entry Systems: Mobility, Accessibility, Universality, I. S. MacKenzie and K. Tanaka-Ishii, Eds. San Francisco, CA, USA: Morgan Kaufmann, 2007, pp. 175–187.
[8] D. Mardanbegi and D. W. Hansen, "Review of eye gaze tracking systems in mobile devices," Computer Vision and Image Understanding, vol. 116, no. 11, pp. 1177–1188, 2012.
[9] A. Swiffin, J. I. Arnott, J. A. Pickering, and A. Newell, "Adaptive and predictive techniques in a communication prosthesis," Augmentative and Alternative Communication, vol. 3, no. 4, pp. 181–191, 1987.
[10] M. Kytö, B. Ens, T. Piumsomboon, G. A. Lee, and M. Billinghurst, "Pinpointing: Precise head- and eye-based target selection for augmented reality," in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018, pp. 1–14.
[11] D. Strazdas, J. Hintz, and A. Al-Hamadi, "Robo-hud: Interaction concept for contactless operation of industrial cobotic systems," Applied Sciences, vol. 11, no. 12, p. 5366, 2021.
[12] M. E. Foster, A. Gaschler, and M. Giuliani, "Automatically classifying user engagement for dynamic multi-party human–robot interaction," International Journal of Social Robotics, vol. 9, no. 5, pp. 659–674, 2017.
[13] K. Amara, M. A. Guerroudji, O. Kerdjidj, N. Zenati, and N. Ramzan, "6 DoF phantom head pose estimation-based deep learning and brain tumor segmentation for AR visualization and interaction," IEEE Sensors Journal, vol. 23, no. 20, pp. 23367–23376, 2023.
[14] P. O. Kristensson and K. Vertanen, "The potential of dwell-free eye-typing for fast assistive gaze communication," in Proceedings of the 2012 ACM Conference on Ubiquitous Computing, 2012, pp. 241–244.
[15] F. R. Willett, D. T. Avansino, L. R. Hochberg, J. M. Henderson, and K. V. Shenoy, "High-performance brain-to-text communication via handwriting," Nature, vol. 593, no. 7858, pp. 249–254, 2021.
[16] B. Dane, T. S. Chaudhari, T. M. Gupta, S. Tayade, and D. R. Thombare, "Agri Sight: AI-Powered Crop Selection and Product Recommendation System," IRJIST; https://doi.org/10.67308/irjist.073
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Arnav Yogesh Wakekar, Kartik Rameshwar Wanjari, Anis Jamal Yelurkar, Padmali Dilip Wagh, Prof. Supriya Mahatme, Rakshit Satish Zalte (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Articles published in the International Research Journal of Innovation in Science and Technology (IRJIST) are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
This license permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Authors retain the copyright of their work and grant the journal the right of first publication. Proper attribution to the authors and the journal is required for any reuse of the published content.
IRJIST on WhatsApp
