ChameleonControl: Teleoperating Real Human Surrogates through Mixed Reality Gestural Guidance for Remote Hands-on Classrooms
Remote video lessons work poorly for hands-on skills, because students need to see a technique performed on a real body or object in the room. ChameleonControl instead lets a remote instructor teleoperate a real person in the classroom. The instructor’s tracked hands are overlaid in the local surrogate’s mixed reality headset, and the surrogate matches them with their own hands while the instructor talks through the lesson, so the class watches a live, physical demonstration. We deployed the system in physiotherapy and massage therapy classes and explored mechanical assembly, sign language and cooking lessons. Across three studies with 53 participants, it increased engagement and the sense of co-presence compared to conventional remote instruction.
My role
I led the project. I designed and built the system, deployed it in physiotherapy and massage therapy classes, and ran all three user studies.
How I built it
The surrogate wears a custom headset I built from a Meta Quest 2, a ZED Mini stereo camera and an iPad Air on a 3D-printed mount. The iPad shows the instructor’s face to the class, but it also covers the Quest’s own passthrough cameras, so the ZED Mini became the surrogate’s eyes: a Windows PC running Unity renders its stereo feed and streams it to the Quest over Oculus Link. The same view goes to the remote instructor, whose hands are tracked with a Leap Motion controller and drawn as virtual hands in both people’s view, so the surrogate can line their own hands up with them. The finished headset weighs 1.3 kg.
Videos
Cite
@inproceedings{faridan2023chameleoncontrol,
title = {{ChameleonControl: Teleoperating Real Human Surrogates through Mixed Reality Gestural Guidance for Remote Hands-on Classrooms}},
author = {Faridan, Mehrad and Kumari, Bheesha and Suzuki, Ryo},
booktitle = {Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems},
series = {CHI '23},
year = {2023},
numpages = {13},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
doi = {10.1145/3544548.3581381},
url = {https://doi.org/10.1145/3544548.3581381}
}