Augmented Conversation with Embedded Speech-Driven On-the-Fly Referencing in AR

Shivesh Jadon, Mehrad Faridan, Edward Mah, Rajan Vaish, Wesley Willett, Ryo Suzuki

Smartphones put information a few taps away during a conversation, but looking things up pulls your attention, and your eyes, away from the person you’re talking to. Augmented Conversation explores AR that brings references into the conversation instead. Our prototype, RealityChat, runs on HoloLens 2: it transcribes speech in real time, extracts keywords with a transformer-based language model, and shows them as interactive tags around the conversation partner. Glancing at a keyword with a short gaze dwell reveals embedded visual references such as maps, weather, photos and Wikipedia summaries. We explored the design space of visual referencing for conversations and developed seven design guidelines through a user-centered design process. An initial user study found that it reduced distraction and friction compared with smartphone searches, while still providing useful and relevant information.

My role

I co-developed RealityChat, the HoloLens prototype that won Snapchat’s Snap Creative Challenge, and ran the pilot study and the user study.

Cite

@misc{jadon2024augmented,
  title         = {{Augmented Conversation with Embedded Speech-Driven On-the-Fly Referencing in AR}},
  author        = {Jadon, Shivesh and Faridan, Mehrad and Mah, Edward and Vaish, Rajan and Willett, Wesley and Suzuki, Ryo},
  year          = {2024},
  eprint        = {2405.18537},
  archivePrefix = {arXiv},
  primaryClass  = {cs.HC},
  doi           = {10.48550/arXiv.2405.18537},
  url           = {https://arxiv.org/abs/2405.18537}
}