Real-Time Onboard Object Detection for Augmented Reality: Enhancing Head-Mounted Display with YOLOv8

Abstract

This paper introduces a software architecture for real-time object detection using machine learning (ML) in an augmented reality (AR) environment. Our approach uses the recent state-of-the-art YOLOv8 network that runs onboard on the Microsoft HoloLens 2 head-mounted display (HMD). The primary motivation behind this research is to enable the application of advanced ML models for enhanced perception and situational awareness with a wearable, hands-free AR platform. We show the image processing pipeline for the YOLOv8 model and the techniques used to make it real-time on the resource-limited edge computing platform of the headset. The experimental results demonstrate that our solution achieves real-time processing without needing offloading tasks to the cloud or any other external servers while retaining satisfactory accuracy regarding the usual mAP metric and measured qualitative performance.

BibTeX

				
					@INPROCEEDINGS{10234302,
  author={Łysakowski, Mikołaj and Żywanowski, Kamil and Banaszczyk, Adam and Nowicki, Michał R. and Skrzypczyński, Piotr and Tadeja, Sławomir K.},
  booktitle={2023 IEEE International Conference on Edge Computing and Communications (EDGE)}, 
  title={Real-Time Onboard Object Detection for Augmented Reality: Enhancing Head-Mounted Display with YOLOv8}, 
  year={2023},
  pages={364-371},
  doi={10.1109/EDGE60047.2023.00059}}

				
			
APA Reference

M. Łysakowski, K. Żywanowski, A. Banaszczyk, M. R. Nowicki, P. Skrzypczyński and S. K. Tadeja, “Real-Time Onboard Object Detection for Augmented Reality: Enhancing Head-Mounted Display with YOLOv8,” 2023 IEEE International Conference on Edge Computing and Communications (EDGE), Chicago, IL, USA, 2023, pp. 364-371, doi: 10.1109/EDGE60047.2023.00059.

Cyber-human Lab Contributors

Dr Sławomir Tadeja

Slawomir K. Tadeja is a postdoctoral research associate in the Cyber-Human Lab belonging to the Institute of Manufacturing at the University of Cambridge. Previously...