Video2MR: Automatically Generating Mixed Reality 3D Instructions by Augmenting Extracted Motion from 2D Videos

Keiichi Ihara, Kyzyl Monteiro, Mehrad Faridan, Rubaiat Habib Kazi, Ryo Suzuki

Mixed reality instructions are great for physical training, but they take a lot of time and money to create. Video2MR turns ordinary online instructional videos into 3D avatars in mixed reality using AI motion capture, then adds guidance automatically. It highlights differences between the learner’s posture and the avatar’s, draws trajectories for key body parts, lets learners control time and speed with their own body motion, and repositions the avatar so they can see it from other angles. Built on HoloLens 2 and Azure Kinect, it supports yoga, dancing, soccer, tennis and other exercises. In our study, participants found it more engaging and playful than learning from the original 2D videos.

My role

I helped build and test the system, and ran the user study, from collecting the data to analyzing it.

Videos

Cite

@inproceedings{ihara2025video2mr,
  title     = {{Video2MR: Automatically Generating Mixed Reality 3D Instructions by Augmenting Extracted Motion from 2D Videos}},
  author    = {Ihara, Keiichi and Monteiro, Kyzyl and Faridan, Mehrad and Kazi, Rubaiat Habib and Suzuki, Ryo},
  booktitle = {Proceedings of the 30th International Conference on Intelligent User Interfaces},
  series    = {IUI '25},
  year      = {2025},
  pages     = {1548--1563},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/3708359.3712159},
  url       = {https://doi.org/10.1145/3708359.3712159}
}