What counts as a "SWITCH"?

[alckasoc]

Adding on to #159, I'm wondering what counts as a "SWITCH".

I looked at the docstring for the code here, but it still doesn't seem right. If I'm reading this correctly, then the following scenario would be a switch:

Instance i in frame t is assigned to track_id 0. At frame t + 1, instance i is assigned to track_id 1.

Another possibility for a switch (or actually 2 switches) is if instance i and instance j at frame t are assigned to track_ids 0 and 1, respectively. And at frame t + 1, instance i is now assigned to 1, and instance j is now assigned to 0.

Figure 1. A possible switch.

For the image above, I would think it would log the event as a switch because the first update would map Oid=0 to Hid=0 and Oid=1 to Hid=1. Then, the 2nd update will map Oid=0 to Hid=1 and Oid=0 to Hid=1 (which will probably result in some error). Because Oid=0 is now mapped to Hid-1 instead of Hid=0, shouldn't this be logged as a switch?

Thanks.

[cheind]

@jvlmdr could you comment on this?

[jvlmdr]

Hi @alckasoc !

the 2nd update will map Oid=0 to Hid=1 and Oid=0 to Hid=1 (which will probably result in some error).

The metrics enforce that the correspondence is one-to-one per frame: each ground-truth track can match to at most one predicted track, and vice versa. This is done by solving a linear assignment problem in each frame (after continuing any existing feasible correspondence).

In your example above, this prevents both oid 0 and oid 1 from being matched to hid 1 even though this would be the best match for each individual oid.

See the MOTA paper for more detail ("Evaluating multiple object tracking performance: the CLEAR MOT metrics").

[alckasoc]

Thank you.

I see now. I have another example that I was a little confused about.

In the above image, the first update matches oid 0 with hid 0 and oid 1 with hid 1. Shouldn't the second update match oid 0 with hid 1 and oid 1 with hid 0? And by doing that, it would count a switch, no? I'm confused about why the 2nd update still matches oid 0 to hid 0 and oid 1 to hid 1.

Thanks.

[jvlmdr]

MOTA assumes that there is a binary overlap criterion. The motmetrics package uses a nan in the distance matrix to represent no-overlap and a finite value to represent a potential match. When there are multiple possible matches for a track, MOTA is defined to first continue existing feasible correspondences and then solve an assignment problem to put the remaining unmatched tracks into correspondence. This is described in section 2.1.3 of the original paper: https://link.springer.com/content/pdf/10.1155/2008/246309.pdf (If you haven't already, please read this before asking more questions about the metrics.)

In the second example you've shown, the correspondence from the first frame can be continued into the second frame since all of the elements of the distance matrix are not-nan.

For an example of how the distance matrix is computed using a threshold:

py-motmetrics/motmetrics/distances.py

Line 123 in 1ad168c

return np.where(dist > max_iou, np.nan, dist)

[jvlmdr]

Are you happy for me to close this issue? Thanks!

[cheind]

Thanks @jvlmdr :)