Qwen3.6-35B-A3B on Linux (FLM v1.0.4): ~2.7 s fixed per-request TTFT = ~4,900 serialized small NPU execs from per-expert MoE dispatch — with a validated coarsening fix (concatenated-expert GEMM)
Summary
flm serve qwen3.6-moe:35b-a3b on Linux has a fixed ~2.7 s per-request TTFT cost for small prompts (10-token prompt: ~2.7–3.0 s to first token; ~300-token: 5.85 s → marginal prefill ~105–110 tok/s is at spec). strace/perf/disassembly attribute it to the MoE prefill path dispatching ~4,900 tiny serial NPU execs per request (submit-then-block, no overlap). Decode (16.5–16.8 tok/s) is at the published spec; only the per-request fixed cost is pathological, and it dominates exactly the small-prompt/agent-loop regime.
Environment
- APU: AMD Ryzen AI MAX+ 395 (Strix Halo, XDNA2 NPU5), 128 GB LPDDR5-8000, Ubuntu 26.04, amdxdna
context_limit=64
- Runtime: FLM v1.0.4 (
flm serve qwen3.6-moe:35b-a3b --port 8098), Q4_K weights (flm_q4k_high_precision)
- Server-reported split for a 19-token prompt:
prefill_duration_ttft: 2.93 s, decoding_speed_tps: 16.48
Evidence
- Driver-level (strace, one 10-token request, 3.75 s window, single thread issues everything):
- ~4,890
DRM_IOCTL_AMDXDNA_EXEC_CMD and ~4,830 DRM_IOCTL_SYNCOBJ_TIMELINE_WAIT — 1:1, submit-then-block, zero pipelining
- ~5,400 each of
CREATE_BO / GEM_CLOSE / GET_BO_INFO — per-op buffer lifecycle
- Time-weighted: 95.2% of ioctl time is the syncobj waits (1.958 s): 96% of waits cluster at 0.10–0.40 ms (median 0.217 ms, p90 0.243 ms — flat across thousands of uniform ops → real serial small-kernel time, not launch overhead). Only 188 waits > 2 ms.
- perf: at stock env 56% of cycles in libgomp = worker spin (fixable via env; applied
OMP_WAIT_POLICY=passive GOMP_SPINCOUNT=0 → CPU 2.32 → 0.36 cores/request, TTFT unchanged — spin is background waste, not the latency cause).
- Binary analysis (libqwen3_6_moe_npu.so, not stripped): per-op dispatch is per-expert:
qwen3_6_moe_npu::Impl::prefill 0x77280 → _prefill_with_mm 0x76320 → qwen3_6_moe_expert_prefill_context::forward 0x897f0, which loops experts individually with full buffer+run lifecycle per iteration (15× buffer dtor, 7× delete, 6× run dtor in-function) → ~122 ops/layer-block × 40 blocks ≈ 4,900/request.- 22 OpenMP sites: 21
GOMP_parallel (17 fixed num_threads=4, 4 env) + 1 GOMP_parallel_sections (num_threads=3) in expert/linear prefill contexts + cpu_func — fixed counts ignore OMP_NUM_THREADS.
- Overlap attempts fail (measured):
- Exec-burst probe on a validated GEMM (M=128 K=2048 N=8192, kernel "MLIR_AIE"): 32 back-to-back launches = 1.07× speedup;
submit_all = 155 ms → each launch blocks until the previous exec completes (per-kernel single command slot; explicit run.start() rejected "bad command state"). amdxdna/XRT is strictly serial per kernel/hwctx.
- Two full-model FLM instances (separate hwctx): concurrent requests took 43 s TTFT each vs ~3 s solo (full-array execs multiplex on the single AIE array with severe contention). Not a parallelism path.
The fix (concatenated-expert dispatch) — validated kernel already exists
The fine granularity is FLM dispatching the top-8+shared experts individually. A concatenated-expert GEMM (one run per operator per layer-block) would cut ~122 → ~10–20 ops/block, est. floor ~2.7 s → ~0.3–0.7 s. We validated this exact kernel on-box: final_i8_MOE_GU_qwen3.6-moe_35b.xclbin (concatenated expert GU, N = TOP_K×2×IM_EXP) matches CPU math (rel RMSE < 0.05). CPU-side per-expert GLU (GOMP regions in expert_prefill_context::forward) should likewise become one parallel region over all experts.
What we tried that did NOT move the floor (so others don't repeat)
--pmode turbo; keep-alive reuse; hwctx context_limit=64; OMP_WAIT_POLICY=passive/GOMP_SPINCOUNT=0/OMP_NUM_THREADS=8 (env applied, no TTFT effect); BO pooling (ceiling ~0.07 s — BO ioctls are 3.5% of syscall time); async submission (impossible: per-hwctx serial); multi-instance fan-out (43 s contention).
Request
Please look at (a) making the MoE prefill dispatch concatenated/per-expert-group runs instead of per-expert runs, and (b) batching buffer-object reuse across the per-expert loop. Happy to test patches; the full corpus (inventories, disasm, classmaps, op-gen map, burst probe source) is available on request.
Full structured report also maintained locally (extraction corpus with 22-site GOMP table, driver trace artifacts, and fix-design docs). Reference: #636 (Windows, v0.9.45 — same "no persistent state / dispatch granularity" theme; this is the Linux v1.0.4 root-cause + validated fix direction).
Qwen3.6-35B-A3B on Linux (FLM v1.0.4): ~2.7 s fixed per-request TTFT = ~4,900 serialized small NPU execs from per-expert MoE dispatch — with a validated coarsening fix (concatenated-expert GEMM)
Summary
flm serve qwen3.6-moe:35b-a3bon Linux has a fixed ~2.7 s per-request TTFT cost for small prompts (10-token prompt: ~2.7–3.0 s to first token; ~300-token: 5.85 s → marginal prefill ~105–110 tok/s is at spec). strace/perf/disassembly attribute it to the MoE prefill path dispatching ~4,900 tiny serial NPU execs per request (submit-then-block, no overlap). Decode (16.5–16.8 tok/s) is at the published spec; only the per-request fixed cost is pathological, and it dominates exactly the small-prompt/agent-loop regime.Environment
context_limit=64flm serve qwen3.6-moe:35b-a3b --port 8098), Q4_K weights (flm_q4k_high_precision)prefill_duration_ttft: 2.93 s,decoding_speed_tps: 16.48Evidence
DRM_IOCTL_AMDXDNA_EXEC_CMDand ~4,830DRM_IOCTL_SYNCOBJ_TIMELINE_WAIT— 1:1, submit-then-block, zero pipeliningCREATE_BO/GEM_CLOSE/GET_BO_INFO— per-op buffer lifecycleOMP_WAIT_POLICY=passive GOMP_SPINCOUNT=0→ CPU 2.32 → 0.36 cores/request, TTFT unchanged — spin is background waste, not the latency cause).qwen3_6_moe_npu::Impl::prefill0x77280 →_prefill_with_mm0x76320 →qwen3_6_moe_expert_prefill_context::forward0x897f0, which loops experts individually with full buffer+run lifecycle per iteration (15× buffer dtor, 7× delete, 6× run dtor in-function) → ~122 ops/layer-block × 40 blocks ≈ 4,900/request.GOMP_parallel(17 fixed num_threads=4, 4 env) + 1GOMP_parallel_sections(num_threads=3) in expert/linear prefill contexts + cpu_func — fixed counts ignoreOMP_NUM_THREADS.submit_all= 155 ms → each launch blocks until the previous exec completes (per-kernel single command slot; explicitrun.start()rejected "bad command state"). amdxdna/XRT is strictly serial per kernel/hwctx.The fix (concatenated-expert dispatch) — validated kernel already exists
The fine granularity is FLM dispatching the top-8+shared experts individually. A concatenated-expert GEMM (one run per operator per layer-block) would cut ~122 → ~10–20 ops/block, est. floor ~2.7 s → ~0.3–0.7 s. We validated this exact kernel on-box:
final_i8_MOE_GU_qwen3.6-moe_35b.xclbin(concatenated expert GU, N = TOP_K×2×IM_EXP) matches CPU math (rel RMSE < 0.05). CPU-side per-expert GLU (GOMP regions inexpert_prefill_context::forward) should likewise become one parallel region over all experts.What we tried that did NOT move the floor (so others don't repeat)
--pmode turbo; keep-alive reuse; hwctxcontext_limit=64;OMP_WAIT_POLICY=passive/GOMP_SPINCOUNT=0/OMP_NUM_THREADS=8(env applied, no TTFT effect); BO pooling (ceiling ~0.07 s — BO ioctls are 3.5% of syscall time); async submission (impossible: per-hwctx serial); multi-instance fan-out (43 s contention).Request
Please look at (a) making the MoE prefill dispatch concatenated/per-expert-group runs instead of per-expert runs, and (b) batching buffer-object reuse across the per-expert loop. Happy to test patches; the full corpus (inventories, disasm, classmaps, op-gen map, burst probe source) is available on request.
Full structured report also maintained locally (extraction corpus with 22-site GOMP table, driver trace artifacts, and fix-design docs). Reference: #636 (Windows, v0.9.45 — same "no persistent state / dispatch granularity" theme; this is the Linux v1.0.4 root-cause + validated fix direction).