[Model Support]: Unofficial MiniCPM5-2B Q4NX pack for XDNA 2

[julianmb]

Summary

Unofficial FastFlowLM / Q4NX build of openbmb/MiniCPM5-2B running on the AMD XDNA 2 NPU (no CPU/GPU decode).

• Model Weights & Precompiled AIE Kernels: huggingface.co/julianmb/MiniCPM5-2B-NPU2
• Dedicated Standalone Repro & Pipeline Repo: github.com/julianmb/minicpm5-xdna2
• Conversion Scripts & Tests: github.com/julianmb/npuhalo/tree/main/ports/minicpm5-2b
• OpenBMB Community Discussion: github.com/OpenBMB/MiniCPM/discussions/372

⚠️ Status / Triage Notice (42-Layer Runlist Timeout):

• Prefill: Passes cleanly on XDNA 2 (chunk 1/1 with 38 tokens).
• Decode: Trips {"error":"runlist failed execution (ERT_CMD_STATE_TIMEOUT)"} on the first token in libqwen3_npu.so.
• Root Cause: As confirmed and isolated by @Platano78, the qwen3 engine currently batches all layer forwards into a single monolithic xrt::runlist (working up to 36 layers on qwen3:4b, while gemma4e handles 42 layers on the same hardware).
• Reproducer: A self-contained 10-second reproducer is provided in scripts/reproduce_ert_timeout.py. Chunking decode forward passes into sub-runlists (e.g. 21 + 21 layers) in libqwen3_npu.so should unlock MiniCPM5 and other 40+ layer models.

---

Hardware Compatibility

• Verified Testbed: AMD Ryzen AI Max+ 395 (Strix Halo, 50 TOPS XDNA 2, /dev/accel/accel0)
• Expected Compatibility: Any AMD XDNA 2 silicon:
  • Strix Halo: Ryzen AI Max+ 395, 390, 385, PRO series
  • Strix Point: Ryzen AI 9 HX 375, HX 370, 365, PRO 300 series
  • Krackan / Kraken Point: Ryzen AI 7, Ryzen AI 5
• Runtime: FastFlowLM (tested with v1.0.1, v1.0.2, v1.0.4)

---

Measured & Projected Performance

• Prefill Speed (TTFT): 81.5 – 128.1 tok/s (~420 ms TTFT) — Verified passing
• Projected Decode Speed: ~63.1 – 63.6 tok/s on XDNA 2 (extrapolated from isolated single-layer GEMM latencies on the 1.7B twin; live generation currently blocked by 42-layer runlist timeout)
• Power Draw: ~2–4 W (leaves 100% of the iGPU and CPU free)
• Footprint: ~1.88 GB in NPU memory (Q4_1 / Q4NX layout)

---

What is in the Pack

• model.q4nx (~1.88 GB, Q4_1 / Q4NX)
• config.json
• Precompiled AIE kernels: attn.xclbin, mm.xclbin, layer.xclbin, dequant.xclbin
• MiniCPM5 tokenizer + chat template

---

How It Was Ported (GQA Firmware Workaround)

MiniCPM5-2B has 16 Query heads and 2 Key/Value heads ($16:2 = 8:1$ GQA ratio). The FastFlowLM AIE firmware (libmha.so) does not have a native $8:1$ kernel for $d_{head}=128$:

1. $4\times$ KV Head Replication: Replicated the 2 KV heads $4\times$ along dimension 0 into 8 KV heads ($16:8 = 2:1$ GQA ratio). Under Grouped Query Attention, this maintains exact mathematical equivalence while matching the native _gen_mha_seq_d128_q2 AIE kernel.
2. Qwen3 Runtime Routing: Dynamically dispatched $d_{head}=128$ via libqwen3_npu.so when intermediate_size == 6144.
3. Identity QK-Norm Injection: Synthetic unit RMSNorm tensors ($\gamma = 1.0$) were injected across all 42 layers in model.q4nx.

---

How to Run

mkdir -p ~/.config/flm/models
git clone https://huggingface.co/julianmb/MiniCPM5-2B-NPU2 ~/.config/flm/models/MiniCPM5-2B-NPU2

# Register under "models" in ~/.config/flm/model_list.json:
# "minicpm5:2b": {
#   "path": "~/.config/flm/models/MiniCPM5-2B-NPU2",
#   "model_type": "qwen3",
#   "tokenizer": "~/.config/flm/models/MiniCPM5-2B-NPU2"
# }

flm serve minicpm5:2b --host 127.0.0.1 --port 8001

[Platano78]

Adding some data from an independent attempt at this pack, in case it helps triage — and because
it may be relevant to whether/how this lands in the catalog.

Setup: Ryzen AI Max+ 395 (Strix Halo), NPU fw 1.1.2.65, in-tree amdxdna 0.7, kernel
7.0.0-31-generic, Ubuntu 26.04. Tried on FLM v1.0.1, v1.0.2 and v1.0.4.

Result: prefill succeeds, decode fails on the first token on all three versions:

{"error":"runlist failed execution (ERT_CMD_STATE_TIMEOUT)"}

gemma4-it:e4b answers at 12.7 t/s on the same NPU seconds either side, so the device is fine.
@julianmb has since reproduced the same failure on his own testbed and added a troubleshooting
section to the model card — details in the
HF discussion.

Two findings that might be useful upstream:

1. It looks like a per-engine depth limit rather than a platform one. Same box, same firmware,
sole model on a fresh server each time:

model	layers	engine	decode
qwen3:1.7b	28	qwen3	43.3 t/s
qwen3:4b	36	qwen3	19.8 t/s
qwen3.6-moe:35b-a3b	40	qwen3.6-moe	16.8 t/s
gemma4-it:e4b	42	gemma4e	12.7 t/s
minicpm5:2b	42	qwen3	fails

The qwen3 engine chains fine at 36 layers and fails at 42, while gemma4e handles 42 and
qwen3.6-moe handles 40. So the boundary is engine-specific and sits somewhere between 37 and 42 —
not a Strix Halo hardware ceiling.

2. It is not the kernel geometry. I substituted the 36-, 40- and 42-layer layer.xclbin
binaries from the models above into this pack — all three fail identically. That points at how the
qwen3 engine constructs its decode runlist rather than at the AIE binary.

Possibly related: #713 (Qwen3-8B slow decode) and #710 (35B per-request TTFT), if those also
trace back to per-layer runlist construction.

One practical note for anyone evaluating this: the decode figures in the description above haven't
been reproducible for me or, as of the HF thread, for the author either — worth confirming
independently before they're treated as a baseline.

[julianmb]

@Platano78 Thank you for the rigorous testing and for isolating the depth boundary — this is tremendously helpful data for FastFlowLM maintainers.

Confirming the Findings

1. Local Reproduction: We reproduced the exact same behavior on our Ryzen AI Max+ 395 testbed: prefill succeeds cleanly (chunk 1/1 with 38 tokens), but the first decode token trips runlist failed execution (ERT_CMD_STATE_TIMEOUT) inside xrt::runlist::wait().
2. Decode Figure Clarification: The 63.6 tok/s in the description was an architectural projection extrapolated from isolated layer GEMM timings on the 1.7B twin, not a sustained live generation trace in FLM. I have updated the issue description to make this explicitly clear so it is not treated as a measured baseline.
3. Per-Engine Runlist Construction: Your layer comparison (gemma4e passing at 42 vs qwen3 failing at 42) confirms the Strix Halo silicon and amdxdna driver handle 42+ layers fine. The bottleneck is strictly host-side in libqwen3_npu.so, where all 42 layer forward passes are batched into a single synchronous xrt::runlist submission before waiting.

Upstream Reproducer & Harness

To assist FastFlowLM maintainers with triaging and testing a chunked runlist fix, we set up a standalone open repository:
👉 github.com/julianmb/minicpm5-xdna2

• scripts/reproduce_ert_timeout.py: A self-contained 10-second reproducer verifying the prefill-pass vs decode-timeout behavior.
• Full pipeline scripts (expand_kv_heads.py for 16:2 -> 16:8 GQA, inject_qk_norm.py for RMSNorm unit injection).

Proposed Fix for libqwen3_npu.so

In qwen3_npu::Impl::forward(), splitting the decode sequence into sub-runlists (e.g. chunking into two 21 + 21 layer submissions with an intermediate synchronization) rather than issuing all 42 layers in one monolithic xrt::runlist should resolve the timeout and unlock 40+ layer models on XDNA 2.

[Atomic-Germ]

Is there a reason the xclbins are hosted on the repo? Atomic-Germ/Embedding-Gemma-300M-NPU2 and other very very early ones had this kind of layout. If this is trying to link against qwen3 though, that's not the way. More reasonably it needs a new set of xclbins altogether.

[julianmb]

@Atomic-Germ Good call, and you're spot on on both counts.

1. Why the .xclbin files were in the repo

The bundled .xclbin blobs were strictly an experimental convenience so early testers on Strix Halo could test the pack with a single git clone without having to manually hunt down or symlink paths. We completely agree that hosting duplicate AIE binaries in the model repo is legacy/bloat — adopting the flm-add.py symlink convention (pointing back to official system xclbins) is much cleaner and is the pattern we will follow moving forward.

2. Linking against qwen3 vs new xclbins

The motivation for trying to map to qwen3 was that MiniCPM5-2B is an exact geometry twin of Qwen3-1.7B ($d_{head}=128$, hidden 2048, intermediate 6144), with two differences:

1. $16:2$ GQA vs $16:8$ GQA: Solved mathematically via $4\times$ KV head replication ($2 \to 8$), which natively matches the compiled _gen_mha_seq_d128_q2 kernel.
2. Layer count: 42 layers vs 28 layers.

Without a public compiler toolchain to generate fresh standalone XCLBINs for custom architectures, adapting weights to fit existing compiled kernels is currently the only practical avenue for community porting.

However, as @Platano78 demonstrated above (and as we reproduced locally), substituting 36-, 40-, and 42-layer layer.xclbin binaries fails identically with ERT_CMD_STATE_TIMEOUT. The failure is in libqwen3_npu.so chaining all 42 layers into a single synchronous xrt::runlist submission before waiting (whereas gemma4e handles 42 layers fine).

So you're completely right: either FastFlowLM needs an official engine/pipeline for 42-layer Qwen-style topologies with chunked runlist dispatch, or dedicated XCLBINs and engine registration for MiniCPM5.

[Atomic-Germ]

@julianmb - we've got an open build system in OpenFlow, though we're completely replacing the closed ones. You're not barking up the wrong tree though.

[julianmb]

@Atomic-Germ Thanks for the pointer! Checking out OpenFlowLM-Next right now — the REPLACEMENT_PLAN.md and the recent PRs replacing the 141 closed .so dispatchers with open open_kernels recipes (and PR #28 with the flm-add kernel set linker) are fantastic to see.

Replacing the closed dispatchers is exactly what unlocks this: the single-runlist queue depth limit in libqwen3_npu.so was the primary wall hitting 42 layers, and having open build recipes means custom layer counts, GQA ratios (like 16:2), and hybrid blocks don't need to be shoehorned into closed binaries.

We have a Strix Halo testbed (Ryzen AI Max+ 395, 8-column XDNA 2, kernel 7.0 / in-tree amdxdna 0.7) running active telemetry and benchmarks. Very happy to test OpenFlowLM builds as the pipeline lands and contribute recipes/benchmarks for MiniCPM5 and other non-standard architectures!

[D-revv]

Additional reproduction: containerized Strix Halo, all variants fail identically

Adding our data points to strengthen the evidence that the fix must be host-side runlist chunking in libqwen3_npu.so.

Environment: AMD Ryzen AI Max+ 395 (Strix Halo), NPU fw 1.1.2.65, kernel 7.0.14-pve, XRT 2.26.0 (built from amd/xdna-driver with -noert), FLM v1.0.4 (debian13 .deb), running in a Docker container (debian:13) inside a Proxmox LXC with /dev/accel/accel0 passthrough.

Result: identical failure — prefill succeeds, first decode token trips runlist failed execution (ERT_CMD_STATE_TIMEOUT), same signature:

• ELF UUID: 2791d0d5-5f89-81c6-50fa-0451747a2b19
• txn_op_idx = 0xFFFFFFFF, ctx_pc = 0x28B060AD
• Log: Creating checkpoint at context length 37 → Start generating... → timeout at xrt::runlist::wait()

What we ruled out (all fail identically):

Variable	Tested	Result
Kernel driver	in-tree amdxdna 0.7 AND out-of-tree v0.17 (xdna-driver main)	timeout
NPU firmware	npu.dev.sbin AND npu_7.sbin (release 1.1.2.65)	timeout
force_cmdlist	0 and 1	timeout
tdr_timeout_ms	60000 (watchdog fires at 60s, job never completes)	timeout
XRT -noert	built with it	no effect (build-tolerance flag only)
FLM version	v1.0.4	timeout

Positive controls on the same NPU (device healthy):

• llama3.2:1b (16 layers): ~60–70 t/s, clean
• qwen3:1.7b (28 layers): works
• lfm2.5-tk:1.2b: works (thinking mode)

Conclusion: confirms the ~36-layer boundary in the qwen3 engine is the wall — no driver/firmware/XRT/FLM-version combination changes the outcome. The 42-layer monolithic runlist in libqwen3_npu.so needs chunking (e.g. 2×21 layers with an intermediate sync) to unlock MiniCPM5 and other 40+ layer models. Happy to test any patch or intermediate build.

[julianmb]

@D-revv Thank you for the thorough matrix — the out-of-tree driver and firmware sweeps, force_cmdlist toggle, and the 60 s TDR run are genuinely useful data. I also owe the thread two corrections to claims I made earlier.

Withdrawing overstated conclusions

1. Queue overflow is a hypothesis, not a finding. My earlier replies (and the model-card Q3) presented monolithic-runlist overflow as the confirmed root cause, with chunking as the fix. That went beyond the evidence: disassembly shows decode dispatches through xrt::runlist (execute() → wait()), but nothing we have establishes queue capacity, the failing operation, or that chunking resolves it. Likewise the "37–42 layer boundary" is an uncontrolled comparison (Qwen3-4B and this port differ in weights, vocabulary, conversion, and runtime config), and my reading of txn_op_idx/ctx_pc assumed field semantics I don't have. I've qualified the model card accordingly.

2. The port has an independent correctness problem I missed. Auditing the conversion, the served config re-labels a Llama-family model (LlamaForCausalLM upstream) as Qwen3ForCausalLM to route through libqwen3_npu.so, and the injected all-ones q_norm/k_norm tensors were described as an "identity." They are not: RMSNorm with γ=1 still normalizes Q and K, and the engine's own symbols (_send_rope_rms_weights) show RMS handling on the execution path. So even past the dispatch failure, output equivalence is unvalidated. I've corrected the loader-shim wording in both the npuhalo/ports/minicpm5-2b docs and minicpm5-xdna2.

3. Old benchmark figures can't be attributed. Per #716 (silent fallback with echoed tag), and since our harnesses never checked the response model field or /v1/models, the historical "63 tok/s" numbers should be treated as unverified for MiniCPM5. I've added served-model assertions to the harnesses and flagged the model card.

What would actually unblock this

• Which dispatch in the decode sequence fails, and how the runlist is assembled per layer (an instrumented/debug build, or guidance on enabling dispatch logging, would settle it).
• The intended semantics of q_norm/k_norm in the Qwen3 engine (applied RMSNorm vs. presence check), so the port can match the original architecture instead of shimming it.
• Whether a 42-layer Qwen-style topology is expected to work with existing XCLBINs at all, or whether dedicated kernels/registration are required (@Atomic-Germ's OpenFlowLM-Next looks like the right long-term path here).

What is solid: the failure signature reproduces across three independent setups with identical symptoms, and your sweeps show no driver/firmware/XRT-version combination changes the outcome. Testbeds on our side (bare metal Strix Halo, 8-column, kernel 7.0, in-tree amdxdna 0.7) remain available for any diagnostic build.