This might not be an issue but a misunderstanding.
I run FastFlowLM on a Ryzen AI 7 360 PRO NPU with 64GB VRAM.
The purpose is to run visual tasks on NPU and I am selecting a model for perf/RAM among the QWen3.5 family. I try 2B, 4B, and 9B models with 4k/8k context window size to concisely read short text from a 1080x720pix PNG image.
What is confusing: On model load, the consumed RAM is slightly different - OK, but when I run inference on the models they all end up with the same 27GB consumed RAM, which is very confusing. Also the context window size does not have an effect. I expected to see less RAM consumed by smaller models.
Here are RAM consumption details:
- QWen3.5-2B
- On load: 4.2GB
- On inference: 27GB
- QWen3.5-4B
- On load: 5.5GB
- On inference: 27GB
- QWen3.5-9B
- On load: 9.0GB
- On inference: 27GB
Am I supposed to expect the same RAM consumption, no matter the LLM size: 2B/4B/9B?
This might not be an issue but a misunderstanding.
I run FastFlowLM on a Ryzen AI 7 360 PRO NPU with 64GB VRAM.
The purpose is to run visual tasks on NPU and I am selecting a model for perf/RAM among the QWen3.5 family. I try 2B, 4B, and 9B models with 4k/8k context window size to concisely read short text from a 1080x720pix PNG image.
What is confusing: On model load, the consumed RAM is slightly different - OK, but when I run inference on the models they all end up with the same 27GB consumed RAM, which is very confusing. Also the context window size does not have an effect. I expected to see less RAM consumed by smaller models.
Here are RAM consumption details:
Am I supposed to expect the same RAM consumption, no matter the LLM size: 2B/4B/9B?