[Bug]: Run LLMs with OpenVINO GenAI Flavor on NPU

[taikai-zz]

OpenVINO Version

Name: openvino
Version: 2024.4.0
Summary: OpenVINO(TM) Runtime
Home-page: https://docs.openvino.ai/2023.0/index.html
Author: Intel(R) Corporation
Author-email: [email protected]
License: OSI Approved :: Apache Software License
Location: /root/openvino_env/lib/python3.12/site-packages
Requires: numpy, openvino-telemetry, packaging
Required-by: openvino-tokenizers

Operating System

Ubuntu 24.04 LTS　　Linux ubuntu 6.8.0-48-generic

Device used for inference

NPU

Framework

None

Model used

TinyLlama

Issue description

Refer to official documentation：
https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide/genai-guide-npu.html

This is my hardware information

import openvino_genai as ov_genai
help(ov_genai.LLMPipeline)

As shown in the above figure: The device does not support NPU. I followed the instructions in the document and changed it to NPU, but the result was empty. Changing it to CPU or GPU restored normal operation. May I ask where I made the operation error?

There is another issue: if you check the usage rate of NPU in Ubuntu environment, such as tools like nvidia-smi

[Wan-Intel]

Did you encountered issue when using the latest version of OpenVINO™ GenAI?

You may use the latest OpenVINO™ GenAI and use NPU by following the steps in Run LLMs with OpenVINO GenAI Flavor on NPU.

[helena-intel]

In addition to using the latest OpenVINO GenAI, if you haven't exported the model with --sym, please try exporting the model with that option. This should work:

optimum-cli export openvino -m TinyLlama/TinyLlama-1.1B-Chat-v1.0 --weight-format int4 --sym --ratio 1.0 --group-size 128 TinyLlama-1.1B-Chat-v1.0

Also note that for NPU, you should add do_sample=False to the pipe.generate() call. See the documentation for more limitations/recommendations.

[taikai-zz]

optimum-cli export openvino -m Qwen/Qwen2-7B --weight-format int4 --sym --ratio 1.0 --group-size 128 Qwen2-7B

[helena-intel]

Unfortunately, there is an issue with using NPU for LLMs on Ubuntu. The NPU team is working on it; the issue is not with OpenVINO, but on the kernel/driver level. I am sorry you're running into this. We will keep you informed.

[dmatveev]

This is 32GB MTL.. Ticket was opened for TinyLLaMa, but logs are mentioning group-quantized QWEN2-7B - a completely different league

[Wan-Intel]

I also encountered issue only when using NPU.

I'll escalate the case to relevant the team and we'll provide an update as soon as possible.

[helena-intel]

@taikai-zz a new NPU driver was released today with a fix for LLM on LNL: https://github.com/intel/linux-npu-driver/releases/tag/v1.10.1 Could you check if that fixes the issue for you? We also had a new openvino-genai release this week, 2024.6, with performance improvements on NPU, so please upgrade with pip install --upgrade openvino-genai.

Also note that for running larger LLMs (>4B) you should use per-channel quantization. This note will be added to the docs too, I'm mentioning it here because I see you're using a 7B model. Instead of group-size 128, you should specify group-size -1 (note the minus sign). This is an example from the docs for Llama-2-7b: optimum-cli export openvino -m meta-llama/Llama-2-7b-chat-hf --weight-format int4 --sym --group-size -1 --ratio 1.0 --awq --scale-estimation --dataset=wikitext2 Llama-2-7b-chat-hf

[taikai-zz]

Thank you for your help. It has now returned to normal, but the speed is too slow。

There is an error in the document, please fix it

[helena-intel]

I'm glad to hear the issue is fixed! For faster speed, please see the document (same one you screenshotted) about model caching. That will speed up model loading time. Since model loading time only occurs once, it's also useful to measure inference time, by adding start = time.perf_counter() and end = time.perf_counter() before and after pipe.generate()and then showing duration asprint(end-start)`.

The group_size -1 enables channel-wise quantization, your screenshot is from the group quantization tab. Also note that I recommended this for larger models. For the 1.1B model, group quantization will work fine too. As I mentioned, this will be clarified in the docs.