Investigate using NeuralMagic as post-training step [Tracker]

[erikerlandson]

cc @ChristianMeyndt

NeuralMagic is basically a tool for analyzing a neural net model, and identifying a modified sparse topology that is much smaller and faster. It operates as a second training phase. So one trains a model and then run a tool to analyze the model, and then a second training run to fine tune the new sparse architecture.

NeuralMagic is capable of making sparse versions of a model that are 10-100 times smaller and faster. Actual results are of course dependent on specifics of the problem domain.

Once we have the training pipeline fully ported, it should be relatively easy to add a neural-magic stage to generate a sparse version of the model.

in Neural magic the typical steps are:
1. train a model 
2. convert to ONNX 
3. use their sparsify tool (it analyzes the model and show the performance improvement you can get with pruning (only available in NM yet))
4. get a sparsify recipe (a YAML file for their optimizer)
5. now train the model with their optimizer (sparseML)
6. convert to ONNX 
7. deploy with their deepsparse inference engine, working with avx2 or avx512 only at the moment

references

• https://github.com/AICoE/elyra-aidevsecops-tutorial/blob/master/docs/source/neural-magic-overview.md
• https://neuralmagic.com/
• https://github.com/neuralmagic

[pacospace]

Thanks @erikerlandson!

Neural Magic has also the possibility to fine-tune models. We could start working on a new Elyra pipeline that:

• analyze pre-trained model for NLP Question Answering from NM: https://sparsezoo.neuralmagic.com/?domain=nlp&sub_domain=question_answering&page=1;
• Prepare GPU image for training teacher network and fine-tuning sparse model. Prepare GPU image for training teacher network and fine tuning sparse model [Neural Magic] #128;
• Test GPU builds in new cluster (Test NM GPU image with KFP #135);
• Prepare custom dataset for fine-tuning sparse model notebook;
• Prepare the notebook for training teacher network using custom data and storing on Ceph (Step 2.3 - Train Dense Teacher #139);
• Prepare notebook for fine-tuning sparse model using custom data, converting to ONNX and storing on Ceph (Step 2.5 Fine Tune model #140);
• Setup Kubeflow Pipeline with 4 steps (prepare dataset + train teacher network + select pre trained model + fine tune sparse model);
• Create image, manifests and deploy NM inference model created;
• Prepare a notebook to compare results between the existing model and Neural Magic model. (Show the difference in terms of performance with respect to current solution deploying the models and testing them: Demo pipeline AICoE/elyra-aidevsecops-tutorial#449 (comment));

cc @markurtz (welcome) What could be resources for fine-tuning that model?

[ChristianMeyndt]

Hi @erikerlandson and @pacospace,
making the models smaller will help a lot for sure and Neural Magic sounds very promising!
If you need any further information on the current model training solution feel free to reach out to me.
I'm curious to see the outcome of this fine tuning!
Thanks

[erikerlandson]

Thanks Christian! The main issue I'm aware of is understanding exactly what data-set you trained your model with, because Francesco would want to provide NM with that same data to do it's version of training.

…

On Wed, Dec 8, 2021 at 9:51 AM ChristianMeyndt ***@***.***> wrote: Hi @erikerlandson <https://github.com/erikerlandson> and @pacospace <https://github.com/pacospace>, making the models smaller will help a lot for sure and Neural Magic sounds very promising! If you need any further information on the current model training solution feel free to reach out to me. I'm curious to see the outcome of this fine tuning! Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#91 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB7OOUUYWXJEQ7V2K2QKQDUP6EHPANCNFSM5HL4YI6Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[ChristianMeyndt]

We need the KPI mapping file (https://github.com/os-climate/corporate_data_pipeline/tree/main/data_input/ESG/kpi_mapping) and the annotations file (https://github.com/os-climate/corporate_data_pipeline/tree/main/data_input/ESG/annotations) as input.
And then of course we also need all the PDFs that are mentioned in the annotations file.
Probably we already have these PDFs within these 40.000 reports on S3, but I'm not sure if you can easily find them by the file name.
Else we have these 300-400 reports on our side and could also upload them somewhere.

[ChristianMeyndt]

FYI @HeatherAck @JeremyGohBNP @LeaADeleris @andraNew @OferHarari @idemir-ids @DaBeIDS @mriefer
This is the issue for NeuralMagic we talked about on Monday. It sounds really promising.

[pacospace]

We need the KPI mapping file (https://github.com/os-climate/corporate_data_pipeline/tree/main/data_input/ESG/kpi_mapping) and the annotations file (https://github.com/os-climate/corporate_data_pipeline/tree/main/data_input/ESG/annotations) as input. And then of course we also need all the PDFs that are mentioned in the annotations file. Probably we already have these PDFs within these 40.000 reports on S3, but I'm not sure if you can easily find them by the file name. Else we have these 300-400 reports on our side and could also upload them somewhere.

Thanks @ChristianMeyndt!! I will check and let you know in case I have any trouble!

[pacospace]

Overview of the work: https://docs.google.com/presentation/d/1BvrbKUaqxs9CSKrwUYVgxCpvqBZ3GXDWsTYPLSTffHs/edit#slide=id.g1125f65fa8b_0_0