Skip to content

feat: self-contrast for rows without augmentation - #53

Merged
EYH0602 merged 6 commits into
mainfrom
fix/self-contrast
Feb 11, 2026
Merged

feat: self-contrast for rows without augmentation#53
EYH0602 merged 6 commits into
mainfrom
fix/self-contrast

Conversation

@EYH0602

@EYH0602 EYH0602 commented Feb 11, 2026

Copy link
Copy Markdown
Member

Summary

  • Fix: Instead of dropping rows that have no code transformation (transformed is None), use self-contrast — copy the original code into the transformed field so the model sees two views of the same code with different MLM masks. This creates "easy" contrastive pairs that act as implicit curriculum learning alongside the harder invariant-contrast pairs.
  • Config flag: Add self_contrast (default true) to PretrainConfig and both CLI subcommands (run --no-self-contrast, pretrain --no-self-contrast) to allow disabling the behavior.
  • Experiment configs: All supcon/ and grouped/ YAML configs updated to use the full dataset (data/csn.jsonl) with self_contrast: true.
  • Documentation: Added doc/self-contrast.md with analysis of why self-contrast is feasible and beneficial (MLM masking as augmentation, implicit curriculum, data efficiency, SupCon compatibility).

Motivation

Transformation operators can fail on certain code samples (e.g., no for-loops to convert, no local variables to rename). Previously these rows were silently dropped, wasting a large fraction of the dataset. Self-contrast retains every row by leveraging independent MLM masking as a natural data augmentation — analogous to SimCSE's dropout-as-augmentation strategy.

Changes

File Change
modeling/pretrain.py Replace dataset.filter with dataset.map that fills None → original code (when self_contrast=True)
modeling/config.py Add self_contrast: bool = True to PretrainConfig
modeling/cli.py Add --self-contrast / --no-self-contrast to both run and pretrain subcommands
experiments/**/*.yaml Switch dataset to data/csn.jsonl, add self_contrast: true
doc/self-contrast.md Design doc explaining the approach

Test plan

  • Run pre-training with self_contrast: true (default) and verify no rows are dropped
  • Run with --no-self-contrast and verify rows without augmentation are filtered out
  • Compare training loss curves between self-contrast and no-self-contrast
  • Evaluate downstream task performance (clone detection, defect detection) for both settings

@EYH0602 EYH0602 changed the title Fix/self contrast feat: self-contrast for rows without augmentation Feb 11, 2026
@EYH0602
EYH0602 merged commit d876e4b into main Feb 11, 2026
3 checks passed
@EYH0602
EYH0602 deleted the fix/self-contrast branch February 28, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant