Published: Apr 12, 2023 by Yihao Liu

Posted on LinkedIn

“First in the World” - Introducing SAMM (Segment Any Medical Model), A 3D Slicer Integration to SAM (Segment Anything Model)

“Three developers, two nights, first in the world, we made segmenting any medical image possible.” I would use this line if we were a startup to get our angel investors. However, we are some graduate students, humble ones, so we will not use it; instead, I am here to say why we should be careful to make early statements on the performance of LLMs such as SAM on medical images. (As a side note, I know we used a “big name” for our tiny extension - it just sounded very nice.)

Based on the conversations I saw recently, the performance of SAM on medical images is still in the debate, a one-sided one. Under a LinkedIn post, Dr. Andras Lasso, one of the leading engineers who developed 3D Slicer (the go-to software if you are doing anything medical image processing related), commented: “SAM works better than PowerPoint’s ‘Remove background’ tool, but the demo application’s performance on medical images is nothing to be excited about.” He backed it with an example demo comparing a 20 years old method, “Watershed” against SAM. A more complete discussion is here.

This statement aligns with our tests. I work on brain images. Using SAM, the valleys and peaks are ignored. Instead, the output is more like a smooth surface around the brain. We suspect this is because of the domain gap between the SAM database and medical images.

There have already been papers in Arxiv testing on medical images. The first is a research from Vanderbilt and Nvidia. They tried SAM on skin cancer tissues from Cancer Genome Atlas (TCGA) datasets. It showed promising results on large connected objects but poor results on dense instance segmentations.

On the other side of the debate, most people are excited because, first, it’s a new model, and people tend to get hyped. Second, it’s easy and effective. You can have a nice mask over the anatomy with simple prompts.

It seems like the consensus on the medical image is that the technology is just not for it yet. Entering SAMM, we understand there is a potential gap (or benefit) between LLM and fine medical image segmentation, that’s why we call the community to try it and improve the paradigm wherever possible. The good thing about using Slicer for this is that it already has all the existing methods with its 20 years span. This made comparison and optimization easier. There are many additional modules that can be integrated to SAM, that potentially improve its performance on medical images.

We are all looking forward to a ChatGPT moment in computer vision. SAM tackles segmentation very nicely. For now, the scene of use is very important. Expert knowledge or some sort of prior is still needed for SAM to really “segment anything.”