Surgeons want better visual guidance and feedback, says biomedical engineer Suman Mondal of Washington University in St. Louis (USA). Standard diagnostic imaging, such as x-rays and MRIs, are too bulky to use in the operating room. That's why he and surgeons like Obeng-Gyasi are working together to develop compact techniques to see tumors better during breast cancer surgeries.

To that end, they're experimenting with augmented reality (AR) technology, perhaps most well known for its role in games like Pokemon Go. Unlike virtual reality, AR doesn't block out real surroundings to create an artificial environment. It simply lays extra information on top of the real-world environment. "Augmented reality is basically anything that adds information directly to your visual field," says Mondal.

The added visuals for medical AR can be very simple, says Lu Lan, a PhD student at Boston University (USA) who collaborates with Obeng-Gyasi. Prototype AR systems overlay 3D lines and shapes onto a video stream or the surgeon's field of view. It could also potentially allow doctors in another room to annotate images for the surgeon in real time.

AR Surgical Training Tools

Outside of the operating room, AR technologies could also be used as an educational tool for medical students. Jerzy Rozenblit, a professor in the Department of Electrical and Computer Engineering, and his group are developing an AR-based kit that students can use to train their depth perception in surgery. The kits consist of deceptively simple hand-eye coordination tasks students have to accomplish using foot-long metal surgical arms. For example, in one task, a user has to thread a hoop on a thin wire without touching the wire.

Currently, Rozenblit's training kits are limited to laparoscopy, a type of surgery commonly performed in the abdomen or pelvis with the aid of a camera inside the body. Similar kits could be developed for breast cancer surgery training, says Rozenblit.

AR-based training can also introduce students to high-risk tasks that their predecessors may have encountered only in real surgeries without prior preparation. "We can create all sorts of training scenarios that are highly repeatable," says Rozenblit. Students can gain experience dealing with rare but critical situations without the risk.