NAN map values after training with pkummd dataset

[fmthoker]

@yjxiong
I sucessfully trained a new model with PKUMMD dataset. However during testing I am getting nan values for map. I have attached both training and testing logs below.

These are some logs at test time.
/home/thoker/detection/ssn/anet_toolkit/Evaluation/eval_detection.py:231: RuntimeWarning: invalid value encountered in true_divide
rec = this_tp / npos
/home/thoker/detection/ssn/anet_toolkit/Evaluation/eval_detection.py:231: RuntimeWarning: invalid value encountered in true_divide
rec = this_tp / npos
/home/thoker/detection/ssn/anet_toolkit/Evaluation/eval_detection.py:231: RuntimeWarning: invalid value encountered in true_divide
rec = this_tp / npos
/home/thoker/detection/ssn/anet_toolkit/Evaluation/eval_detection.py:231: RuntimeWarning: invalid value encountered in true_divide
rec = this_tp / npos
Evaluation done.
+Detection Performance on pkummd--+------+------+------+------+------+------+------+---------+
| IoU thresh | 0.50 | 0.55 | 0.60 | 0.65 | 0.70 | 0.75 | 0.80 | 0.85 | 0.90 | 0.95 | Average |
+------------+------+------+------+------+------+------+------+------+------+------+---------+
| mean AP | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan |
+------------+------+------+------+------+------+------+------+------+------+------+---------+

Just to make sure my model is converging here are the training logs after 60 epochs:

Act. FG 100.00 (84.90) Act. BG 99.93 (99.93) [76/1402]
Epoch: [61][980/1177], lr: 0.00010 Time 0.967 (1.166) Data 0.535 (0.800) Loss 0.1610 (0.2812) Act. Loss 0.128 ( 0.233) Comp. Loss 0.061 ( 0.250) Reg
. Loss 0.271 (0.228)
Act. FG 87.50 (84.88) Act. BG 99.94 (99.94)
Epoch: [61][1000/1177], lr: 0.00010 Time 1.311 (1.165) Data 0.876 (0.800) Loss 0.4598 (0.2808) Act. Loss 0.407 ( 0.233) Comp. Loss 0.204 ( 0.249) Reg
. Loss 0.322 (0.229)
Act. FG 75.00 (84.92) Act. BG 99.94 (99.94)
Epoch: [61][1020/1177], lr: 0.00010 Time 1.445 (1.163) Data 1.016 (0.799) Loss 0.2110 (0.2813) Act. Loss 0.167 ( 0.234) Comp. Loss 0.246 ( 0.249) Reg
. Loss 0.190 (0.228)
Act. FG 75.00 (84.89) Act. BG 99.94 (99.94)
Epoch: [61][1040/1177], lr: 0.00010 Time 1.517 (1.169) Data 1.090 (0.806) Loss 0.1750 (0.2805) Act. Loss 0.129 ( 0.233) Comp. Loss 0.218 ( 0.249) Reg
. Loss 0.241 (0.228)
Act. FG 87.50 (84.98) Act. BG 99.94 (99.94)
Epoch: [61][1060/1177], lr: 0.00010 Time 0.701 (1.171) Data 0.297 (0.807) Loss 0.8812 (0.2801) Act. Loss 0.837 ( 0.232) Comp. Loss 0.346 ( 0.248) Reg
. Loss 0.098 (0.228)
Act. FG 75.00 (85.04) Act. BG 99.94 (99.94)
Epoch: [61][1080/1177], lr: 0.00010 Time 0.620 (1.171) Data 0.001 (0.809) Loss 0.3701 (0.2809) Act. Loss 0.291 ( 0.233) Comp. Loss 0.116 ( 0.249) Reg
. Loss 0.674 (0.228)
Act. FG 75.00 (85.01) Act. BG 99.94 (99.94)
Epoch: [61][1100/1177], lr: 0.00010 Time 0.625 (1.170) Data 0.000 (0.807) Loss 0.2096 (0.2793) Act. Loss 0.152 ( 0.232) Comp. Loss 0.277 ( 0.248) Reg
. Loss 0.302 (0.229)
Act. FG 87.50 (85.13) Act. BG 99.94 (99.94)
Epoch: [61][1120/1177], lr: 0.00010 Time 0.618 (1.168) Data 0.058 (0.805) Loss 0.6727 (0.2788) Act. Loss 0.559 ( 0.231) Comp. Loss 0.634 ( 0.249) Reg
. Loss 0.506 (0.228)
Act. FG 62.50 (85.18) Act. BG 99.94 (99.94)
Epoch: [61][1140/1177], lr: 0.00010 Time 1.028 (1.166) Data 0.595 (0.803) Loss 0.2802 (0.2800) Act. Loss 0.196 ( 0.232) Comp. Loss 0.236 ( 0.249) Reg
. Loss 0.605 (0.228)
Act. FG 87.50 (85.10) Act. BG 99.95 (99.95)
Epoch: [61][1160/1177], lr: 0.00010 Time 1.290 (1.166) Data 0.865 (0.803) Loss 0.2358 (0.2799) Act. Loss 0.209 ( 0.232) Comp. Loss 0.136 ( 0.249) Reg
. Loss 0.133 (0.228)
Act. FG 62.50 (85.13) Act. BG 99.95 (99.95)
Test: [0/17] Time 4.963 (4.963) Loss 1.9900 (1.9900) Act. Loss 1.975 (1.975) Comp. Loss 0.070 (0.070) Act. Accuracy 75.00 (75.00) FG 87.50 BG 62.50 Reg. Loss 0.079 (0.
079)
Testing Results: Loss 4.05014 Activity Loss 4.006 Completeness Loss 0.207
Act Accuracy 56.06 FG Acc. 79.55 BG Acc. 32.58 Regression Loss 0.238
Epoch: [62][0/1177], lr: 0.00010 Time 7.005 (7.005) Data 6.562 (6.562) Loss 1.0257 (1.0257) Act. Loss 0.961 ( 0.961) Comp. Loss 0.544 ( 0.544) Reg
. Loss 0.099 (0.099)

[yjxiong]

You can modify the code to see the per-class APs. Usually, this is due to an incorrect number of classes, where the missing class has no instance detected and thus a NaN ap.