fix: restore layerwise MiniCPM checkpoint loading on Transformers 5 - #1600
Open
feng1201 wants to merge 1 commit into
Open
fix: restore layerwise MiniCPM checkpoint loading on Transformers 5#1600feng1201 wants to merge 1 commit into
feng1201 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layerwise MiniCPM loading fails under the supported Transformers 5.x range. Loading the official
BAAI/bge-reranker-v2-minicpm-layerwiseconfiguration with Transformers 5.7.0 changesrope_scaling=Noneinto a defaultrope_typedictionary;_init_rope()then raisesKeyError: 'type'. This reproduces the exception reported in #1581 (the report did not specify its Transformers version).Fix the complete checkpoint-loading path in the layerwise finetuning model:
nn.initoperations so Transformers 5 does not overwrite loaded parameters, and rebuild the non-persistent rotary buffers after loading from the meta device.The latter steps are necessary to load successfully and preserve model outputs after resolving the initial configuration error. Model weights, biases, padding rows, normalization weights and sharing relationships are checked across checkpoint round trips.
Validation on CPU, Python 3.11.15 / Torch 2.14.0:
get_modelloader, one optimizer step, and all head types with shared/unshared embeddings.Trainer.processing_classattribute and reproduce identically on unchanged source.AutoConfig.from_pretrainedat revision47b5332b296c4d8cb6ee2c60502cc62a0d708881, resized to a tiny model: construction and shared-weight check pass.git diff --checkand dependency consistency checks pass.Full-size GPU finetuning and Flash Attention were not run. No dependency changes. Developed with AI assistance.