Deep learning can improve surgical outcomes

Negin Ghamsarian is a postdoctoral researcher at the AI in Medical Imaging lab at the ARTORG Center for Biomedical Engineering Research. She is convinced that self-supervised and semi-supervised deep learning can overcome the current limitations on the applicability of AI technologies for analysing surgical videos, so that postoperative complications can be better predicted and surgical procedures performed even more precisely.

Text: Monika Kugemann

Read the original article here. (in German)

Negin, tell us about your research.
I am currently focusing on bridging the gap between deep learning-based medical image and video analysis in the laboratory and under real-world conditions. An important aspect that is taken for granted in many deep learning applications is the availability of labelled data (i.e. data with annotations). Despite the promising performance of supervised deep learning methods in many healthcare tasks, such as surgical phase recognition, object detection, localisation and semantic segmentation, their performance depends heavily on suitable annotations. On the other hand, the different image and video acquisition devices and environmental conditions in hospitals or healthcare centres can lead to a large domain shift between different datasets and hinder the generalisability of the trained networks. In other words, a neural network trained on annotated images from one hospital may not achieve acceptable performance when evaluated on a dataset from another hospital or device.
The questions I ask in my research are therefore: how can we reduce the dependence on annotations without compromising the performance of neural networks? And how can we use the abundant unannotated data to close the gap in cross-dataset analysis?

What would be an example of this?
Take cataract surgery – the most frequently performed eye operation and a procedure in very high demand worldwide. I have been working on DL-assisted analysis of cataract surgery videos since my doctorate. Through close interdisciplinary collaboration with experienced surgeons, I was able to show that deep learning can make a significant contribution to accelerating surgical training and improving surgical outcomes through phase recognition [1], relevance-based compression [2], detection of intraoperative irregularities and prediction of postoperative complications [3], and surgical scene understanding [4-5].

Each of these tasks requires a specific level of annotation, ranging from the image to the region to the pixel level. Since the variety of videos recorded with different devices and in different hospitals is quite large, the trained networks can deliver satisfactory performance mainly for videos from the same device and hospital. Providing new annotations for each new dataset is time-consuming and costly, especially in the medical field, where expertise is crucial. This is currently the bottleneck in deep learning-based medical image analysis. While providing adequate annotations is a challenge, hospitals and healthcare centres have a wealth of unlabelled images and videos, and the usefulness of such data for improving healthcare is underestimated.
Semi-supervised learning, self-supervised learning and domain adaptation with various strategies such as pseudo-supervision, consistency regularisation and contrastive learning are ways of solving this problem. In the current phase of my research, I am developing semi-supervised learning and domain adaptation techniques tailored to the inherent characteristics of surgical videos and volumetric images such as OCT and MRI, in order to reduce the annotation requirements for reliable semantic segmentation in these domains [6].

You studied in Iran and Austria. How do you like it in Bern?
The University of Bern is excellently connected with Inselspital, which creates an ideal interdisciplinary environment for researchers in the field of AI for medicine. Access to medical datasets and feedback from physicians and surgeons are crucial components of serious research in this field, and the University offers ample opportunities for both. In addition, the AIMI lab has a first-class hardware infrastructure that enables large-scale deep learning-based evaluations, which are essential for cutting-edge research. Alongside these academic advantages, the University of Bern offers a vibrant, multicultural environment for professional development.
Being a member of CAIM and working with a team of computer scientists and medical engineers is also a great opportunity for me. It gives me the chance to exchange ideas with other researchers in my field, receive feedback on my work and learn from their expertise.

What motivates you?
I have always been fascinated by the intersection of artificial intelligence and medicine, a field in which AI can be useful rather than destructive. Developing novel methods to tackle complex medical problems is extremely rewarding, as it can contribute to advances in healthcare and have a positive impact on patients’ well-being. Deep learning can revolutionise the way we approach medical diagnosis and surgical procedures, and that is the driving force that motivates me to go the extra mile.

How important will AI be in surgery in the future?
Thanks to technological progress in surgery, operating theatres are developing into intelligent environments. Context-aware systems will be central components of this development. They can improve preoperative surgical planning, provide automatic difficulty assessment, support scheduling in the operating theatre and comprehensively interpret the surgical context.
With built-in real-time warnings and decision support, they will be of great value, especially for less experienced surgeons. Their capabilities extend to the automatic analysis of surgical videos and include functions such as indexing, documentation and the generation of postoperative reports.
AI-based systems will become indispensable in surgery in the coming years. By overcoming the existing limitations of AI-assisted medical image and video analysis, computer-assisted diagnosis and surgery will be made considerably easier in the foreseeable future.

PUBLICATIONS

[1] Ghamsarian, N., Taschwer, M., Putzgruber-Adamitsch, D., Sarny, S. and Schoeffmann, K., 2021, January. Relevance detection in cataract surgery videos by spatio-temporal action localization. In 2020 25th International Conference on Pattern Recognition (ICPR) (pp. 10720-10727). IEEE.

[2] Ghamsarian, N., Amirpourazarian, H., Timmerer, C., Taschwer, M. and Schöffmann, K., 2020, October. Relevance-based compression of cataract surgery videos using convolutional neural networks. In Proceedings of the 28th ACM International Conference on Multimedia (pp. 3577-3585).

[3] Ghamsarian, N., Taschwer, M., Putzgruber-Adamitsch, D., Sarny, S., El-Shabrawi, Y. and Schoeffmann, K., 2021. LensID: a CNN-RNN-based framework towards lens irregularity detection in cataract surgery videos. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24 (pp. 76-86). Springer International Publishing.

[4] Ghamsarian, N., Taschwer, M., Putzgruber-Adamitsch, D., Sarny, S., El-Shabrawi, Y. and Schöffmann, K., 2021. ReCal-Net: Joint Region-Channel-Wise Calibrated Network for Semantic Segmentation in Cataract Surgery Videos. In Neural Information Processing: 28th International Conference, ICONIP 2021, Sanur, Bali, Indonesia, December 8–12, 2021, Proceedings, Part III 28 (pp. 391-402). Springer International Publishing.

[5] Ghamsarian, N., Taschwer, M., Sznitman, R. and Schoeffmann, K., 2022, September. DeepPyramid: Enabling Pyramid View and Deformable Pyramid Reception for Semantic Segmentation in Cataract Surgery Videos. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 276-286). Cham: Springer Nature Switzerland.

[6] Ghamsarian, N., Tejero, J.G., Neila, P.M., Wolf, S., Zinkernagel, M., Schoeffmann, K. and Sznitman, R., 2023. Domain Adaptation for Medical Image Segmentation using Transformation-Invariant Self-Training. arXiv preprint arXiv:2307.16660.