Align tiny Mistral v0.2 config with mistralai/Mistral-7B-Instruct-v0.2

[albertvillanova]

This PR aligns the tiny-MistralForCausalLM-0.2 generator config with its reference model,
mistralai/Mistral-7B-Instruct-v0.2.

Part of #7137. Sibling of #7203, which does the same for v0.1.

Motivation

The script builds MistralConfig from architecture arguments only, so every field not passed falls back
to the class default. Four of them diverge from the reference.

The sharpest is sliding_window. MistralConfig defaults to 4096, and v0.2 is the release that
removed sliding-window attention, so the reference sets it to None. The tiny model therefore runs
sliding-window attention where the model it stands in for does not, which is not a scaled-down v0.2 but a
different attention configuration.

Solution

Mirror the reference values:

• sliding_window=None (reference disables it; class default is 4096)
• rope_theta=1000000.0 (reference override; class default is 10000.0)
• max_position_embeddings=32768 (reference value; class default is 131072)
• rms_norm_eps=1e-05 (reference value; class default is 1e-06)
• vocab_size=32000, matching len(tokenizer.vocab) today and pinned so the config no longer depends on
  the tokenizer files at generation time

Before

[config_diff] mistralai/Mistral-7B-Instruct-v0.2 vs tiny (9 differences)
  hidden_size                                      4096                               → 8
  intermediate_size                                14336                              → 32
  max_position_embeddings                          32768                              → 131072
  num_attention_heads                              32                                 → 4
  num_hidden_layers                                32                                 → 2
  num_key_value_heads                              8                                  → 2
  rms_norm_eps                                     1e-05                              → 1e-06
  rope_theta                                       1000000.0                          → 10000.0
  sliding_window                                   None                               → 4096

After

[config_diff] mistralai/Mistral-7B-Instruct-v0.2 vs tiny (5 differences)
  hidden_size                                      4096                               → 8
  intermediate_size                                14336                              → 32
  num_attention_heads                              32                                 → 4
  num_hidden_layers                                32                                 → 2
  num_key_value_heads                              8                                  → 2

Every remaining row is the deliberate size reduction. Produced with print_config_diff at
transformers==4.56.2, the version check_transformers_version() pins. The Hub repo needs regenerating
for this to take effect.

Changes

• Pin vocab_size to the reference's 32000
• Set sliding_window, rope_theta, max_position_embeddings and rms_norm_eps from the reference config

---

Note

Low Risk
Script-only change to tiny test model generation; no runtime library or production model behavior unless the Hub artifact is regenerated.

Overview
Updates the tiny Mistral v0.2 generation script so MistralConfig matches mistralai/Mistral-7B-Instruct-v0.2 on non-size fields, instead of relying on MistralConfig defaults that diverge from the reference.

vocab_size is pinned to 32000 (replacing len(tokenizer.vocab)). sliding_window=None, rope_theta=1000000.0, max_position_embeddings=32768, and rms_norm_eps=1e-05 are set explicitly—most notably disabling sliding-window attention, which v0.2 removed but the class still defaults to 4096.

After regeneration, print_config_diff should only show the intentional tiny architecture shrink (hidden size, layers, heads, etc.).

^{Reviewed by Cursor Bugbot for commit 1c912c9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[albertvillanova]

The CI model has been regenerated and the corresponding Hub PR is open. It needs to be merged once this PR is approved:

• https://huggingface.co/trl-internal-testing/tiny-MistralForCausalLM-0.2/discussions/1

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

The vocabulary size were tiny back then