Title: Vision-based Force Estimation in Robot-assisted Surgery by Humans and Machines 

THURSDAY, FEBRUARY, 24, 2022 11:00AM-11:50AM

Abstract: Tissue handling is an important skill for surgeons to perform safe and effective surgery. In robot-assisted minimally invasive surgery  (RMIS), such skill is difficult to acquire due to lack of haptic feedback. RMIS surgeons learn to estimate tissue interaction forces through visual feedback, often over many hours of in-vivo practice. My research leverages the RMIS telesurgical robotic platform as both a sensor and actuation suite to (a) develop automated data-driven vision-based force estimates that can provide objective measures of tissue handling skills, and (b) provide multimodal robot-mediated real-time feedback to the RMIS surgeon to improve their tissue handling skill. In this talk, I will present models and algorithms for visionbased force estimation in RMIS from both human and machine perspectives. From the human perspective, I evaluate the  effect of haptic training on human teleoperators’ abilities to visually estimate forces through a telesurgical robot. From the machine perspective, I design multimodal deep learning-based methods to estimate interaction forces and deliver haptic feedback during tissue manipulation. The  results demonstrate that human teleoperators and machines can learn visual force estimation from haptic training and multimodal manipulation  data respectively, setting the stage for future work in improved methods for humanmachine skills development and autonomous robot-assisted surgery.