Support SAM2 (Inference only) as base model for all the apps

[SachidanandAlle]

• SAM2 Inference Task
  • Radiology (sam_2d and sam_3d)
  • Bundle (sam_2d and sam_3d)
  • Endoscopy (sam_2d)
  • Pathology (sam_2d)
• 3D Slicer Plugin (support ROI/BOX for SAM2/Smart Edit)
• Unit Test for SAM2 Inference

Verified SAM2 for 3D Slicer (both 2D/3D), QuPath, CVAT plugins.

[SachidanandAlle]

---------- coverage: platform linux, python 3.12.3-final-0 -----------
Name                                           Stmts   Miss  Cover
------------------------------------------------------------------
monailabel/app.py                                 54     14    74%
monailabel/client/__init__.py                      1      0   100%
monailabel/client/client.py                      347    133    62%
monailabel/config.py                              69      0   100%
monailabel/datastore/__init__.py                   0      0   100%
monailabel/datastore/cvat.py                     172     44    74%
monailabel/datastore/dicom.py                    156     45    71%
monailabel/datastore/dsa.py                      172    112    35%
monailabel/datastore/local.py                    393     33    92%
monailabel/datastore/utils/__init__.py             0      0   100%
monailabel/datastore/utils/colors.py               1      0   100%
monailabel/datastore/utils/convert.py            108     30    72%
monailabel/datastore/utils/dicom.py               73     25    66%
monailabel/datastore/xnat.py                     229     87    62%
monailabel/deepedit/__init__.py                    0      0   100%
monailabel/deepedit/handlers.py                  122     19    84%
monailabel/deepedit/interaction.py                42      7    83%
monailabel/deepedit/transforms.py                236     17    93%
monailabel/endpoints/__init__.py                   0      0   100%
monailabel/endpoints/activelearning.py            36      3    92%
monailabel/endpoints/batch_infer.py               36      2    94%
monailabel/endpoints/datastore.py                169     14    92%
monailabel/endpoints/infer.py                    102     41    60%
monailabel/endpoints/info.py                      12      0   100%
monailabel/endpoints/login.py                     25     11    56%
monailabel/endpoints/logs.py                      32      3    91%
monailabel/endpoints/model.py                     67     27    60%
monailabel/endpoints/ohif.py                      22      3    86%
monailabel/endpoints/proxy.py                     52     13    75%
monailabel/endpoints/scoring.py                   48      2    96%
monailabel/endpoints/session.py                   76      8    89%
monailabel/endpoints/train.py                     49      7    86%
monailabel/endpoints/user/__init__.py              0      0   100%
monailabel/endpoints/user/auth.py                 75     34    55%
monailabel/endpoints/wsi_infer.py                 75     16    79%
monailabel/interfaces/__init__.py                  0      0   100%
monailabel/interfaces/app.py                     396     54    86%
monailabel/interfaces/config.py                   43      6    86%
monailabel/interfaces/datastore.py                85     26    69%
monailabel/interfaces/exception.py                24      2    92%
monailabel/interfaces/tasks/__init__.py            0      0   100%
monailabel/interfaces/tasks/batch_infer.py        92      6    93%
monailabel/interfaces/tasks/infer.py              10     10     0%
monailabel/interfaces/tasks/infer_v2.py           35      3    91%
monailabel/interfaces/tasks/scoring.py            10      1    90%
monailabel/interfaces/tasks/strategy.py           10      1    90%
monailabel/interfaces/tasks/train.py              14      2    86%
monailabel/interfaces/utils/__init__.py            0      0   100%
monailabel/interfaces/utils/app.py                80     10    88%
monailabel/interfaces/utils/transform.py          56     11    80%
monailabel/interfaces/utils/wsi.py                74     23    69%
monailabel/main.py                               232     51    78%
monailabel/sam2/__init__.py                        0      0   100%
monailabel/sam2/infer.py                         247     39    84%
monailabel/scribbles/__init__.py                   0      0   100%
monailabel/scribbles/infer.py                     41      5    88%
monailabel/scribbles/transforms.py               209     63    70%
monailabel/scribbles/utils.py                     92     34    63%
monailabel/tasks/__init__.py                       0      0   100%
monailabel/tasks/activelearning/__init__.py        0      0   100%
monailabel/tasks/activelearning/epistemic.py      41      2    95%
monailabel/tasks/activelearning/first.py          17      1    94%
monailabel/tasks/activelearning/random.py         25      1    96%
monailabel/tasks/infer/__init__.py                 0      0   100%
monailabel/tasks/infer/basic_infer.py            313     50    84%
monailabel/tasks/infer/bundle.py                 205     19    91%
monailabel/tasks/scoring/__init__.py               0      0   100%
monailabel/tasks/scoring/dice.py                  33      0   100%
monailabel/tasks/scoring/epistemic.py            124     13    90%
monailabel/tasks/scoring/epistemic_v2.py         125     24    81%
monailabel/tasks/scoring/sum.py                   27     15    44%
monailabel/tasks/train/__init__.py                 0      0   100%
monailabel/tasks/train/basic_train.py            371     59    84%
monailabel/tasks/train/bundle.py                 215     52    76%
monailabel/tasks/train/handler.py                 61      4    93%
monailabel/tasks/train/utils.py                   33     27    18%
monailabel/transform/__init__.py                   0      0   100%
monailabel/transform/cache.py                     73     48    34%
monailabel/transform/post.py                     235     44    81%
monailabel/transform/pre.py                       62      1    98%
monailabel/transform/writer.py                   187     23    88%
monailabel/utils/__init__.py                       0      0   100%
monailabel/utils/async_tasks/__init__.py           0      0   100%
monailabel/utils/async_tasks/task.py              50      0   100%
monailabel/utils/async_tasks/utils.py             91      6    93%
monailabel/utils/others/__init__.py                0      0   100%
monailabel/utils/others/class_utils.py           124      7    94%
monailabel/utils/others/detection.py              28      0   100%
monailabel/utils/others/generic.py               238     66    72%
monailabel/utils/others/label_colors.py           15      0   100%
monailabel/utils/others/modelzoo_list.py           1      0   100%
monailabel/utils/others/pathology.py              91      4    96%
monailabel/utils/others/planner.py                62      4    94%
monailabel/utils/sessions.py                     115      8    93%
------------------------------------------------------------------
TOTAL                                           7763   1575    80%

[SachidanandAlle]

[SachidanandAlle]

[Joeycho]

Hi @SachidanandAlle! This is great work :) Did you also have a chance to test it (SAM2) with OHIF? Or do you work for it now?

If not yet, but you have interest in, then maybe you can take a look my recent work. I enabled SAM2 at OHIF via Monai Label.

https://www.youtube.com/watch?v=KlzAZU5GKug

[SachidanandAlle]

Monailabel ohif v3 plugin is kind of in broken state. Work in progress to fix it to support all the previous functions of monailabel and keep it similar to 3D slicer.

Once that is achieved, SAM 2D/3D versions will appear similar to monailabel deepgrow models for users to interact and annotate. ROI input may not be there initially (need to investigate how ohif supports bounding box). But feel free to help or add any missing options once RC2 with enhanced ohif plugin is released. Hopefully by next week.

We need more contributions to make monailabel better and useful for everyone.

Thanks for sharing the video. Very nice. Looking for more support and contributions for monailabel.

[Joeycho]

@SachidanandAlle Thank you for the quick reply and straightforward answer :)

I see. Yes, in the beginning, I tried to use all the models via monailabel panel at ohif, but I failed to use it. Instead, I decided to add a couple of buttons in the toolbar and invoke monailabel API directly. I still don't know how to use deepgrow and deepedit at OHIF. Therefore, I'm very interested in to reproduce them.

I used several useful utilities and tools of Monailabel, but I passed many parts of MonaiLabel. It will be great if I have a chance to explain how I did it to you, therefore you can give me an advice how to use MonaiLabel and modify it properly.

I'm already in contact with OHIF core team. Regarding my recent work, I had a call with them this Monday, and they explained their interest is more in integrating SAM/SAM2 in serverless way, like using Onnx/WebGPU resource.

What I can bring it to MonaiLabel community is the updated Ohif (now 3.9 beta, but plan to upgrade it to recent one) with working SAM2 (points, bounding box prompts, 2D/3D inference) + multiple labels. But as I explained, it might look different in code level, which makes a bit challenging to ship it to your main branch.

Do you have interest in collaboration with me? I'm working as PhD student at University Hospital Bonn, under the supervison, Prof. Philipp Vollmuth. My interest is somehow contributing MONAI Label repo with my work, and at the same time, I would like to understand it better.

[SachidanandAlle]

That's really helpful. Currently there is some work in progress to update the monailabel plugin based on latest ohifv3.
Tracking it via: #1785

Once the rc2 (hopefully in next week) is released. You can verify if basic (without ROI) works great for SAM2. And you can add your ideas (supporting ROI based) on top of that. You have definitely some cool ideas related to toolbard/roi etc.. you can help to enhance rc2 plugin based on the same.

[diazandr3s]

This is great to hear, @Joeycho
Are you joining us for the Slicer Week? Check it out: https://projectweek.na-mic.org/PW42_2025_GranCanaria/
It'd be awesome to have you there! We could dive into your project and explore some cool collaboration ideas. What do you think?