Automatic Recognition of Gesture Qualities in Speech Delivery in VR-Work-in-Progress

Abstract

Gesture use during public speaking can be analyzed as an aspect of speaker performance as well as an indicator of speaker emotions. Recognizing gesture quality rather than identifying specific gestures presents an underexplored challenge compared to traditional gesture recognition in virtual reality (VR). To that end, we use a VR headset and controllers to create a database of 162 five-second-long gestures. Next, we ask ten judges to evaluate the quality of gestures in three dimensions, i.e., dynamics, range, and between-hand distance, on a low-medium-high scale with average Fleiss Kappa 0.39 inter-rater agreement. A comparison of various classifiers revealed that the Support Vector Machine (SVM) yielded the best results, achieving classification accuracy of 70-85% for each quality dimension.

BibTeX

				
					@INPROCEEDINGS{10536589,
  author={Skrzek, Tomasz and Malawski, Filip and Tadeja, Slawomir K. and Hekiert, Daniela and Hemmerling, Daria and Igras-Cybulska, Magdalena},
  booktitle={2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)}, 
  title={Automatic Recognition of Gesture Qualities in Speech Delivery in VR-Work-in-Progress}, 
  year={2024},
  volume={},
  number={},
  pages={624-627},
  keywords={Support vector machines;Headphones;Three-dimensional displays;Virtual reality;Speech recognition;Gesture recognition;User interfaces;Gesture—Speech—Training—Machine learning;Automatic feedback—Public speaking},
  doi={10.1109/VRW62533.2024.00122}}

				
			
APA Reference

T. Skrzek, F. Malawski, S. K. Tadeja, D. Hekiert, D. Hemmerling and M. Igras-Cybulska, “Automatic Recognition of Gesture Qualities in Speech Delivery in VR-Work-in-Progress,” 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Orlando, FL, USA, 2024, pp. 624-627, doi: 10.1109/VRW62533.2024.00122.

Cyber-human Lab Contributors

Dr Sławomir Tadeja

Slawomir K. Tadeja is a postdoctoral research associate in the Cyber-Human Lab belonging to the Institute of Manufacturing at the University of Cambridge. Previously,...