Add GroundingSAM (GroundingDino + SAM) #1720

pavel-esir · 2024-02-16T10:51:01Z

ticket: CVS-131290

review-notebook-app · 2024-02-16T10:51:07Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

call GroundingDINO in OpenVINO publish draft PR GroundingDINO works from FE added resize for GroundingDino ready for review

review-notebook-app · 2024-02-26T15:44:27Z

View / edit / reply to this conversation on ReviewNB

eaidova commented on 2024-02-26T15:44:27Z
----------------------------------------------------------------

Line #1.    %pip install -q "torch==2.1.0" "torchvision==0.16.0" --extra-index-url https://download.pytorch.org/whl/cpu

this code is not compatible with torch 2.2.0 and torchvision 0.17.0?

timm, transformers also have dependencies on torch and can unexplicitly install packages, please also install them with --extra

do you really need pycocotools and supervision for running notebook (looks like some training/evaluation dependencies)?

pavel-esir commented on 2024-02-27T10:38:34Z
----------------------------------------------------------------

Torch 2.2 tvision 0.17 works as well, i will remove manual install of torch and will rely on timm with extra index url.

About pycocotools it also seemed strange to me, but without it i end up having import error, because it's needed in visualizer

https://github.com/wenyi5608/GroundingDINO/blob/main/groundingdino/util/visualizer.py#L19

which is imported in file where model torch nn.Module is defined

https://github.com/wenyi5608/GroundingDINO/blob/main/groundingdino/models/GroundingDINO/groundingdino.py#L37

Supervision is needed for simpler annotation, but it can be done manually. I will do that and will remove dependency

pavel-esir commented on 2024-03-01T10:53:03Z
----------------------------------------------------------------

i have drawn masks manually in PIL but it turned out supervision is still imported inside for post processing groundingdino.util.inference.Model

i will rewrite this part and will remove supervision dependency

pavel-esir commented on 2024-03-06T23:49:04Z
----------------------------------------------------------------

Returned back supervision. With it code looks simpler.

review-notebook-app · 2024-02-26T15:44:28Z

View / edit / reply to this conversation on ReviewNB

eaidova commented on 2024-02-26T15:44:28Z
----------------------------------------------------------------

Line #9.        text_token_mask = torch.tensor([[

can this mask be obtained using some tool or using random? I think for real user it is hard to understand how to create and use it

pavel-esir commented on 2024-03-06T23:49:34Z
----------------------------------------------------------------

done, significantly enhanced providing example input

review-notebook-app · 2024-02-26T15:44:29Z

View / edit / reply to this conversation on ReviewNB

eaidova commented on 2024-02-26T15:44:29Z
----------------------------------------------------------------

please provide some explanation for this part. Which steps required to prepare data and postprocess results. There is a big function, but it maybe hard to understand what stated behind that if somebody wants to replicate this process

pavel-esir commented on 2024-03-06T23:51:07Z
----------------------------------------------------------------

i will provide description for this function as well as description with running combined GroundingDINO + SAM/EfficientSAM

pavel-esir commented on 2024-03-07T13:32:33Z
----------------------------------------------------------------

slightly refactored this function, added comments with explanations, and type annotations

review-notebook-app · 2024-02-26T15:44:30Z

View / edit / reply to this conversation on ReviewNB

eaidova commented on 2024-02-26T15:44:29Z
----------------------------------------------------------------

Line #1.    def predict_torch(predictor, image, transformed_boxes):

name of function confusing, do you really use pytorch for prediction?

pavel-esir commented on 2024-02-26T16:20:23Z
----------------------------------------------------------------

very obsolete name, i'll rename 🤦

review-notebook-app · 2024-02-26T15:44:31Z

View / edit / reply to this conversation on ReviewNB

eaidova commented on 2024-02-26T15:44:30Z
----------------------------------------------------------------

not clear, then what is above if it is run grounding sam? :)

I think more text needed, something:

Now, you can try apply grounding sam on own images using interactive demo. The code below provides helper functions used in demonstration

pavel-esir commented on 2024-03-07T14:01:44Z
----------------------------------------------------------------

added clarification for this part

review-notebook-app · 2024-02-26T15:44:32Z

View / edit / reply to this conversation on ReviewNB

eaidova commented on 2024-02-26T15:44:31Z
----------------------------------------------------------------

please put here little instruction how to launch (which data should be provided, what is advanced options responsibility)

pavel-esir commented on 2024-03-07T13:29:39Z
----------------------------------------------------------------

agreed and added clarifications

review-notebook-app · 2024-02-26T15:44:33Z

View / edit / reply to this conversation on ReviewNB

eaidova commented on 2024-02-26T15:44:32Z
----------------------------------------------------------------

Line #4.                input_image = gr.Image(source='upload', type="pil", value=f"{repo_dir}/assets/demo1.jpg", tool="sketch")

I think better to use gr. Examples for providing default images and labels

https://www.gradio.app/docs/examples

pavel-esir commented on 2024-03-06T22:57:59Z
----------------------------------------------------------------

replaced with gr.Interface and and specified examples arg, this gives UI similar to gr.Examples

pavel-esir · 2024-02-26T16:20:25Z

very obsolete name, i'll rename 🤦

View entire conversation on ReviewNB

pavel-esir · 2024-02-27T10:38:36Z

Torch 2.2 tvision 0.17 works as well, i will remove manual install of torch and will rely on timm with extra index url.

About pycocotools it also seemed strange to me, but without it i end up having import error, because it's needed in visualizer

https://github.com/wenyi5608/GroundingDINO/blob/main/groundingdino/util/visualizer.py#L19

which is imported in file where model torch nn.Module is defined

https://github.com/wenyi5608/GroundingDINO/blob/main/groundingdino/models/GroundingDINO/groundingdino.py#L37

Supervision is needed for simpler annotation, but it can be done manually. I will do that and will remove dependency

View entire conversation on ReviewNB

review-notebook-app · 2024-02-27T23:58:27Z

View / edit / reply to this conversation on ReviewNB

aleksandr-mokrov commented on 2024-02-27T23:58:27Z
----------------------------------------------------------------

Remove outputs with no useful information and with internal paths

pavel-esir commented on 2024-03-07T13:28:32Z
----------------------------------------------------------------

removed

review-notebook-app · 2024-02-27T23:58:28Z

View / edit / reply to this conversation on ReviewNB

aleksandr-mokrov commented on 2024-02-27T23:58:27Z
----------------------------------------------------------------

Line #2.        from groundingdino.models import build_model

Why are the import inside functions? All these function always are called.

pavel-esir commented on 2024-03-06T23:01:09Z
----------------------------------------------------------------

moved them outside

review-notebook-app · 2024-02-27T23:58:29Z

View / edit / reply to this conversation on ReviewNB

aleksandr-mokrov commented on 2024-02-27T23:58:28Z
----------------------------------------------------------------

Cleanup

pavel-esir commented on 2024-03-06T23:01:56Z
----------------------------------------------------------------

done

review-notebook-app · 2024-02-28T13:08:51Z

View / edit / reply to this conversation on ReviewNB

apaniukov commented on 2024-02-28T13:08:50Z
----------------------------------------------------------------

Line #7.        token_type_ids = torch.tensor([[0, 0, 0, 0, 0, 0]])

The same is for attention_mask and token_type_ids, they are already returned by tokenizer:

>>> pt_grounding_dino_model.tokenizer([caption], return_tensors="pt", return_special_tokens_mask=True)
{'input_ids': tensor([[ 101, 1996, 2770, 3899, 1012,  102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1]]), 'special_tokens_mask': tensor([[1, 0, 0, 0, 0, 1]])}

special_token_mask can be transformed into text_token_mask like this:

_, counts = torch.unique_consecutive(text_token_mask, return_counts=True)

text_token_mask = torch.block_diag(*[torch.ones(size, size) for size in counts])

pavel-esir commented on 2024-03-06T23:03:23Z
----------------------------------------------------------------

thanks a lot. Will apply this

pavel-esir commented on 2024-03-06T23:19:59Z
----------------------------------------------------------------

done

pavel-esir · 2024-03-06T23:51:08Z

i will provide description for this function as well as description with running combined GroundingDINO + SAM/EfficientSAM

View entire conversation on ReviewNB

…ewest

pavel-esir · 2024-03-07T13:28:20Z

removed

View entire conversation on ReviewNB

pavel-esir · 2024-03-07T13:28:33Z

removed

View entire conversation on ReviewNB

pavel-esir · 2024-03-07T13:29:40Z

agreed and added clarifications

View entire conversation on ReviewNB

pavel-esir · 2024-03-07T13:32:34Z

slightly refactored this function, added comments with explanations, and type annotatetions

View entire conversation on ReviewNB

pavel-esir · 2024-03-07T14:01:46Z

added clarification for this part

View entire conversation on ReviewNB

pavel-esir · 2024-03-07T14:03:16Z

Applied comments. @eaidova @apaniukov @aleksandr-mokrov please take a look

pavel-esir · 2024-03-07T15:43:55Z

compilation on mac fails. i will disable testing on mac

review-notebook-app · 2024-03-08T05:07:09Z

View / edit / reply to this conversation on ReviewNB

eaidova commented on 2024-03-08T05:07:08Z
----------------------------------------------------------------

Line #2.    ov_compiled_grounded_dino = core.compile_model(ov_dino_model, device.upper())

no need in upper

pavel-esir commented on 2024-03-08T07:39:55Z
----------------------------------------------------------------

done

review-notebook-app · 2024-03-08T05:12:01Z

View / edit / reply to this conversation on ReviewNB

eaidova commented on 2024-03-08T05:12:00Z
----------------------------------------------------------------

I think this text does not describe cell content, maybe better to add it into code as comment with additional description for other steps like disabling gradients and model tracing as well?

pavel-esir commented on 2024-03-08T07:40:03Z
----------------------------------------------------------------

done

eaidova · 2024-03-08T05:16:51Z

@pavel-esir could you please check correctness of notebook meta for docs?

I'm not sure that visual question answering is correct task for this notebook, also not sure that it is possible to run it using binder (binder has only 2GB RAM, are you sure that it is enough for running your notebook? do you try to run it?)

Please also add image for preview

pavel-esir · 2024-03-08T07:39:56Z

done

View entire conversation on ReviewNB

pavel-esir · 2024-03-08T07:40:05Z

done

View entire conversation on ReviewNB

pavel-esir · 2024-03-08T07:40:36Z

@added link to image, removed visual question answering, applied.

not sure that it is possible to run it using binder (binder has only 2GB RAM, are you sure that it is enough for running your notebook? do you try to run it?)

checked, indeed it consumes too much ram > 5Gb, removed binder link

notebooks/288-grounded-segment-anything/README.md

update image for GroundedSAM #1720

pavel-esir force-pushed the add_grounding_sam branch 2 times, most recently from ea4fa79 to a991ec2 Compare February 23, 2024 10:56

pavel-esir marked this pull request as ready for review February 23, 2024 10:56

initial

3624e91

call GroundingDINO in OpenVINO publish draft PR GroundingDINO works from FE added resize for GroundingDino ready for review

pavel-esir force-pushed the add_grounding_sam branch from a991ec2 to 3624e91 Compare February 23, 2024 11:16

pavel-esir added 3 commits February 23, 2024 15:26

add metada, make linter happy

87d556f

add README

fcc2b14

fix pip-conflicts

e8499a8

andrei-kochin requested review from a team, apaniukov, itrushkin and aleksandr-mokrov and removed request for a team February 26, 2024 10:36

[not not review] for test only

93c142a

pavel-esir force-pushed the add_grounding_sam branch from 7269f44 to 43b4154 Compare February 28, 2024 00:15

[do not review] for test only 2

43b4154

minor refactoring

6f6f322

pavel-esir added 2 commits March 7, 2024 00:54

Merge remote-tracking branch 'upstream/main' into add_grounding_sam_n…

2ab2565

…ewest

Merge remote-tracking branch 'upstream/main' into add_grounding_sam_n…

ffa173e

…ewest

refactored, provided description

98c274c

pavel-esir added 3 commits March 7, 2024 15:08

fix paths when EfficientSAM is used

58e9b26

fix spelling typo

2e6322b

remove skip_kernel_extension

f70d104

ignore GroundedSAM on mac

76bb043

add image to metadata, minor corrections

4f877e8

eaidova approved these changes Mar 8, 2024

View reviewed changes

pavel-esir commented Mar 8, 2024

View reviewed changes

notebooks/288-grounded-segment-anything/README.md Outdated Show resolved Hide resolved

remove binder

a137542

eaidova merged commit 376bade into openvinotoolkit:main Mar 8, 2024
13 of 15 checks passed

pavel-esir mentioned this pull request Mar 8, 2024

Fix GroundedSAM preview #1812

Merged

eaidova pushed a commit that referenced this pull request Mar 8, 2024

Fix GroundedSAM preview (#1812)

169094e

update image for GroundedSAM #1720

pavel-esir deleted the add_grounding_sam branch March 11, 2024 07:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GroundingSAM (GroundingDino + SAM) #1720

Add GroundingSAM (GroundingDino + SAM) #1720

pavel-esir commented Feb 16, 2024 •

edited

Loading

review-notebook-app bot commented Feb 16, 2024

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

pavel-esir commented Feb 26, 2024

pavel-esir commented Feb 27, 2024

review-notebook-app bot commented Feb 27, 2024 •

edited

Loading

review-notebook-app bot commented Feb 27, 2024 •

edited

Loading

review-notebook-app bot commented Feb 27, 2024 •

edited

Loading

review-notebook-app bot commented Feb 28, 2024 •

edited

Loading

pavel-esir commented Mar 6, 2024

pavel-esir commented Mar 7, 2024

pavel-esir commented Mar 7, 2024

pavel-esir commented Mar 7, 2024

pavel-esir commented Mar 7, 2024

pavel-esir commented Mar 7, 2024

pavel-esir commented Mar 7, 2024

pavel-esir commented Mar 7, 2024

review-notebook-app bot commented Mar 8, 2024 •

edited

Loading

review-notebook-app bot commented Mar 8, 2024 •

edited

Loading

eaidova commented Mar 8, 2024

pavel-esir commented Mar 8, 2024

pavel-esir commented Mar 8, 2024

pavel-esir commented Mar 8, 2024

Add GroundingSAM (GroundingDino + SAM) #1720

Add GroundingSAM (GroundingDino + SAM) #1720

Conversation

pavel-esir commented Feb 16, 2024 • edited Loading

review-notebook-app bot commented Feb 16, 2024

review-notebook-app bot commented Feb 26, 2024 • edited Loading

review-notebook-app bot commented Feb 26, 2024 • edited Loading

review-notebook-app bot commented Feb 26, 2024 • edited Loading

review-notebook-app bot commented Feb 26, 2024 • edited Loading

review-notebook-app bot commented Feb 26, 2024 • edited Loading

review-notebook-app bot commented Feb 26, 2024 • edited Loading

review-notebook-app bot commented Feb 26, 2024 • edited Loading

pavel-esir commented Feb 26, 2024

pavel-esir commented Feb 27, 2024

review-notebook-app bot commented Feb 27, 2024 • edited Loading

review-notebook-app bot commented Feb 27, 2024 • edited Loading

review-notebook-app bot commented Feb 27, 2024 • edited Loading

review-notebook-app bot commented Feb 28, 2024 • edited Loading

pavel-esir commented Mar 6, 2024

pavel-esir commented Mar 7, 2024

pavel-esir commented Mar 7, 2024

pavel-esir commented Mar 7, 2024

pavel-esir commented Mar 7, 2024

pavel-esir commented Mar 7, 2024

pavel-esir commented Mar 7, 2024

pavel-esir commented Mar 7, 2024

review-notebook-app bot commented Mar 8, 2024 • edited Loading

review-notebook-app bot commented Mar 8, 2024 • edited Loading

eaidova commented Mar 8, 2024

pavel-esir commented Mar 8, 2024

pavel-esir commented Mar 8, 2024

pavel-esir commented Mar 8, 2024

pavel-esir commented Feb 16, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 26, 2024 •

edited

Loading

review-notebook-app bot commented Feb 27, 2024 •

edited

Loading

review-notebook-app bot commented Feb 27, 2024 •

edited

Loading

review-notebook-app bot commented Feb 27, 2024 •

edited

Loading

review-notebook-app bot commented Feb 28, 2024 •

edited

Loading

review-notebook-app bot commented Mar 8, 2024 •

edited

Loading

review-notebook-app bot commented Mar 8, 2024 •

edited

Loading