Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI

Hiroki Kaimoto, Kyzyl Monteiro, Mehrad Faridan, Jiatong Li, Samin Farajian, Yasuaki Kakehi, Ken Nakagaki, Ryo Suzuki

In most AR sketching tools, the physical world can affect your sketches, but your sketches can’t affect the physical world. Sketched Reality makes this work in both directions by pairing an iPad AR sketching tool with small tabletop robots (Sony Toio). Lines, walls, pendulums and springs drawn and simulated on the iPad can push, launch, collide with and constrain the real robots, and moving the robots by hand updates the sketch. The paper contributes a set of interactions and a design space for bi-directional AR sketching, and demonstrates applications in tangible physics education, explorable mechanisms, games for children and programming robots by sketching.

My role

I helped build the AR sketching tool and the robot control behind it.

Videos

Cite

@inproceedings{kaimoto2022sketched,
  title     = {{Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI}},
  author    = {Kaimoto, Hiroki and Monteiro, Kyzyl and Faridan, Mehrad and Li, Jiatong and Farajian, Samin and Kakehi, Yasuaki and Nakagaki, Ken and Suzuki, Ryo},
  booktitle = {Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology},
  series    = {UIST '22},
  year      = {2022},
  numpages  = {12},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/3526113.3545626},
  url       = {https://doi.org/10.1145/3526113.3545626}
}