Mnist evaluation

[eyalho]

Have someone successfully trained on mnist or fashion-mnist?

I didnt manage to do it yet. Help will be great.

Tnx

[eyalho]

Just saw that this PR #54
Include MNIST implementation.

Did the experiments produce good results??

[wvangansbeke]

Hi @eyalho,

I haven't found the time to check PR #54. I want to make sure that it works before merging it. I hope you can understand. At first glance, it looks ok, but augmentations and other hyperparameters might need to be modified for MNIST.

[eyalho]

Totally make sense.
I already trained it:

• "as is": it didn't yield good result due to low number of epochs.

Accuracy of top-20 nearest neighbors on train set is 76.68
Fill memory bank for mining the nearest neighbors (val) .
Fill Memory Bank [0/209]
Fill Memory Bank [100/209]
Fill Memory Bank [200/209]
Mine the nearest neighbors (Top-5)
Accuracy of top-5 nearest neighbors on val set is 77.18

Evaluate best model based on SCAN metric at the end
{'ACC': 0.7456, 'ARI': 0.5819513405204482, 'NMI': 0.6815947066918223, 'ACC Top-5': 0.9777, 'hungarian_match': [(0, 0), (1, 8), (2, 3), (3, 7), (4, 5), (5, 4), (6, 1), (7, 2), (8, 9), (9, 6)]}

Evaluate model at the end
{'ACC': 0.1135, 'ARI': 0.0, 'NMI': 6.755387526067596e-16, 'ACC Top-5': 0.5134, 'hungarian_match': [(0, 0), (1, 9), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8), (9, 1)]}

• Next, I just changed epoch=500 and In a few hours (when train finish) I would update here with the results.

If you know to direct to specific augmentations or other hyperparameters - I can play around with them and post.

[eyalho]

for epoch=500, it did improved, but probably can be better with better augmentations.

Result of kNN evaluation is **96.71**
Accuracy of top-20 nearest neighbors on train set is 76.68 ->  **93.65**
Fill memory bank for mining the nearest neighbors (val) .
Fill Memory Bank [0/209]
Fill Memory Bank [100/209]
Fill Memory Bank [200/209]
Mine the nearest neighbors (Top-5)
Accuracy of top-5 nearest neighbors on val set is 77.18 ->  **93.43**

Evaluate best model based on SCAN metric at the end
{'ACC': 0.826, 'ARI': 0.6591190466685025, 'NMI': 0.7541952129206612, 'ACC Top-5': 0.9945, 'hungarian_match': [(0, 2), (1, 9), (2, 6), (3, 7), (4, 5), (5, 3), (6, 0), (7, 8), (8, 4), (9, 1)]}

Evaluate model at the end
Unfortunately it stopped before reached the end, so I don't have the results logged

Will continue update if I continue working on it..

[anewusername77]

for epoch=500, it did improved, but probably can be better with better augmentations.

Result of kNN evaluation is **96.71**
Accuracy of top-20 nearest neighbors on train set is 76.68 ->  **93.65**
Fill memory bank for mining the nearest neighbors (val) .
Fill Memory Bank [0/209]
Fill Memory Bank [100/209]
Fill Memory Bank [200/209]
Mine the nearest neighbors (Top-5)
Accuracy of top-5 nearest neighbors on val set is 77.18 ->  **93.43**

Evaluate best model based on SCAN metric at the end
{'ACC': 0.826, 'ARI': 0.6591190466685025, 'NMI': 0.7541952129206612, 'ACC Top-5': 0.9945, 'hungarian_match': [(0, 2), (1, 9), (2, 6), (3, 7), (4, 5), (5, 3), (6, 0), (7, 8), (8, 4), (9, 1)]}

Evaluate model at the end
Unfortunately it stopped before reached the end, so I don't have the results logged

Will continue update if I continue working on it..

hey there,
can you tell me how to organize dataset? do i have to write a dataset class and dataloader myself?

[wvangansbeke]

I will point to this issue in the readme for those who are interested. Since most methods work on more challenging datasets already, I believe MNIST is not that relevant anymore anyway. So I'm closing this for now.