convert to panoptic dataset with specified colormap

[alpereninci]

I would like to convert a dataset imported as coco detection or cvat to panoptic coco format by setting specified colormap for masks.
I used the following code snippet for that. I could not find any information how to change colormap.

from datumaro.components.dataset import Dataset
# colormap = {"0": (0,0,0), "1": (73, 77, 174), "2": (220, 20, 60), "3": (70, 130, 180), "4": (64, 170, 64)} # 0 stands for background.
dataset = Dataset.import_from("annotations.json", "coco")
# dataset = Dataset.import_from("annotations.xml", "cvat")
dataset.transform('polygons_to_masks')
dataset.export('converted/', 'coco', save_images=True)

[zhiltsov-max]

Hi. Our COCO implementation supports only instance-id-encoded masks (segments described in json and single-number indexed PNGs). If you want masks in color, consider exporting into the voc_segmentation format.

Also, in the code snippet you provided, you'll probably need to add segmentation_mode='mask', if you want to load from COCO and save masks in COCO (by default, the code respects is_crowd attribute, which will be kept untouched after polygons_to_masks).

[alpereninci]

Thank you for quick reply. As far as I understood, I need to export in voc_segmentation format for panoptic labels. It is not clear for me how I can use exported labels in panoptic coco type.
Default panoptic coco format includes color in segments_info to match png labels which is defined with categories. Also, I think your coco format conversion does not support the "is_thing" for panoptic segmentation.

[alpereninci]

Also, I would like to ask how you handle overlapping segments defined with polygons while converting from coco detection or segmentation to panoptic?

[zhiltsov-max]

As far as I understood, I need to export in voc_segmentation format for panoptic labels. It is not clear for me how I can use exported labels in panoptic coco type.

I'll try to explain in more details. Apparently, there are at least 2 COCO panoptic formats out there - one with instance-id masks (no colors, described here, p.4) and another one with class masks (in color; probably, used for the stuff task, which is not fully covered in Datumaro). Our COCO panoptic implementation supports only the first variant. Also, there is voc_segmentation format, which should be close to the second variant with class masks in color, and in which you can control class colors.

format conversion does not support the "is_thing" for panoptic segmentation.

Yes, probably, you're right. It could be a good feature request.

how you handle overlapping segments defined with polygons while converting from coco detection or segmentation to panoptic?

The process is the following. Let's consider few terms:

• Spatial annotation - a polygon, bbox or mask
• They can have a z_order property set, which controls the "depth" or the "layer" of the annotation object.
• Also, annotations can be joined into groups. Joined annotations have the same group attribute value
• We call a group of annotations an "instance" (an annotated object)

1. We collect all spatial annotations for an image and split them into groups (using the group attribute)
2. We look for the largest annotation (by the area) in each group and call it a leader. This leader represents the instance (annotated object), which is being described by the group. The group annotations are merged into a single mask (if it is required by the is_crowd attribute, or conversion mode, or if there is more than 1 annotation) or into a set of polygons (remember, COCO allows to have multiple polygons for a single object). The group inherits all the properties of its leader - including z_order, attributes, and label.
3. If crop-covered is specified, we arrange obtained instances wrt their z_order (or don't change their order, if not specified), convert them into masks, paint them into a single image (which automatically crops covered parts), then convert back into source representation (polygons or a mask).

We provide several options to control behaviour:

• --crop-covered parameter allows to decide, whether we need to drop covered portions from underlying segments (false by default)
• --segmentation-mode (respect is_crowd or keep input annotation type / force polygons / force masks)