Skip to content

grpo_multi_student_fresh_process_test races its 100 ms callback timeout #129

Description

@deepfates

test/grpo_multi_student_fresh_process_test.exs ("fresh BEAM resumes only the active student and preserves distinct portable results") failed once in the stranger.check lane on PR #127 (commit 45edc41), which changes no code the test reads:

assert first =~ "grpo_step_outcome_unknown"
left:  "{:error, {:reinforcement_callback_timeout, :final_model_artifact, 100}}\n"

The test runs a child BEAM with {:hang_after_step, "fresh-base-b"} and a 100 ms reinforcement callback timeout, then expects the child's stdout to contain both grpo_step_outcome_unknown and reinforcement_callback_timeout. On a loaded runner the timeout fired before the step outcome was recorded, so only the timeout error printed. The same lane passes on main and on PR #126, so this is a timing race in the test, not a regression.

To fix: read run_fresh/2, Imp.Test.FileMultiStudentGRPOTrainer, and the Imp.Clients.TrainingJob path that emits both messages, and decide whether the expected order is guaranteed by code or only by timing. Make it deterministic (have the fixture trainer record the step outcome synchronously before hanging, or assert on the checkpoint and events on disk that the test already checks, or use a timeout that cannot race). Do not weaken it to "either message". Show it no longer races by running the file repeatedly under nice -n 19 with a CPU hog.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions