New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Questions about quantization #81

Open

mxjmtxrm opened this issue Jun 10, 2024 · 1 comment

mxjmtxrm commented Jun 10, 2024 •

edited

Loading

Hi, great work!
I met some problems during 4bit weight-only quantization(--lwc).

Is there any problem if the norm is nan?
what's the best lwc hyper-parameter of LLama2 with different scales? like lwc-lr and epochs?
Does more calib data bring better results?

I quantized a llama model using different lwc hyper-parameters and received different results.

nsamples=1000, batch_size=1, epoch=2, the ppl is correct.
nsamples=2000, batch_size=8, epoch=10, the ppl is super large (40000+).
What's the problem?

SherrySwift commented Oct 31, 2024

I found NaN norm during training, too. I guess it is caused by AMP training.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment