Skip to content

mha: size the K/V arg specs by num_KV_heads, not num_heads - #169

Open
atassis wants to merge 1 commit into
amd:develfrom
atassis:fix/mha-gqa-kv-arg-spec
Open

mha: size the K/V arg specs by num_KV_heads, not num_heads#169
atassis wants to merge 1 commit into
amd:develfrom
atassis:fix/mha-gqa-kv-arg-spec

Conversation

@atassis

@atassis atassis commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

design.py declares Q and O as (heads, S_q_pad, d) but K and V as
(num_KV_heads, S_kv_pad * d). get_arg_spec() sized all four from num_heads, so
under GQA the K/V specs are over by num_heads / num_KV_heads. On the GQA param
already in test.py (seq_len=16384, d=64, heads=8, pipelines=8, kv_heads=2) that is
8388608 against 2097152, a factor of 4.

Two things kept it hidden. num_kv_heads=0 normalizes to heads in design.py, which
makes the old formula accidentally correct for standard MHA, and run_test reads only
spec.direction, taking input buffers from the supplied tensor rather than from
spec.shape. So nothing consults the wrong value today. It surfaces inside an
OperatorSequence, where compilation/sequence.py asserts the MLIR arg count equals
the one computed here.

Added

  • iron/operators/mha/test.py: test_arg_spec_matches_design_shapes, covering GQA and
    standard MHA. It asserts against the shapes design.py declares rather than against
    literals, so it fails if either side drifts. Device-free, and deliberately not reusing
    get_params() -- the GQA case there is marked extensive, which is half of why this
    went unnoticed.

Changed

  • iron/operators/mha/op.py: get_arg_spec() sizes Q/O from num_heads and K/V from
    num_KV_heads, applying the same 0 -> num_heads normalization design.py does.

Removed

Known limitation

_calculate_seq_padding hardcodes the block size to 64, while design.py derives
S_q_pad and S_kv_pad from B_q and B_kv separately. That is correct today only
because both default to 64 and op.py never overrides them, and __post_init__ sets
self.B_q/self.B_kv which the helper ignores. Left alone here to keep this to one
defect.

design.py declares Q/O as (heads, S_q_pad, d) but K/V as
(num_KV_heads, S_kv_pad * d). get_arg_spec used num_heads for all four,
so under GQA K and V are over by num_heads/num_KV_heads: 8388608 against
2097152 on the GQA param already in test.py, a factor of 4.

Two things hid it. num_kv_heads=0 normalizes to heads, which makes the
old formula accidentally right for plain MHA, and run_test reads only
spec.direction, never spec.shape. It surfaces in an OperatorSequence,
where compilation asserts the MLIR arg count equals the computed one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant