feat: ModernBERT - #54
Merged
Merged
Conversation
- Fix hardcoded pad_token_id=1 in attention masks to use tokenizer.pad_token_id (supports both RoBERTa and ModernBERT) - Add mean pooling for ModernBERT in clone detection models (ModernBERT has no pooler head; mean pooling is recommended) - Parameterize model_type and tokenizer_name in all shell scripts with backward-compatible defaults - Affected tasks: Clone-detection-POJ104, Clone-detection-CodeNet, Code-classification-POJ104, Code-classification-CodeNet
Generate 88 bash scripts in experiments_downstream/, one per downstream-task + model combination. Covers 4 baselines (CodeBERT, GraphCodeBERT, ContraBERT_C, ContraBERT_G) and 4 InvPT-trained models across all 7 downstream tasks (with CodeNet subsets). Includes gen_all.py generator script for reproducibility.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ModernBERT Support
Add ModernBERT as a supported model architecture for both pre-training and downstream evaluation.
Pre-training
model_typeconfig parameter (roberta|modernbert) to select encoder architecturepoolingconfig parameter (cls|mean) for contrastive embedding extractionModelTypeandPoolingStrategytype aliases inmodeling/_types.pymodel_typeandpoolingthroughPretrainConfig,ContrastiveTrainer, and CLIexperiments/modernbert_base.yamlexample configdoc/modernbert.mdwith setup and usage documentationDownstream Evaluation
pad_token_id=1in attention mask construction to usetokenizer.pad_token_id(supports both RoBERTa pad=1 and ModernBERT pad=0)Clone-detection-POJ104,Clone-detection-CodeNet) for ModernBERT compatibilitymodel_typeandtokenizer_namein all downstreamrun.shandrun_aug_test.shscripts with backward-compatible defaultsExperiment Scripts
experiments_downstream/with 88 generated bash scripts (viagen_all.py) covering all 8 models x 7 tasks (with CodeNet subsets)downstream/structure