Distinguish router and non-router replicas - #4101
Open
jvstme wants to merge 2 commits into
Open
Conversation
**Problem**: Non-router replicas in a service with a router should not be registered on the gateway. To determine whether a particular replica should be registered, one has to parse its job spec and run spec and see if it is a router replica or not. This does not work well for the upcoming gateway state sync mechanism, which needs to quickly fetch all jobs to be registered from the database, without parsing their specs. **Solution**: - Do **not** set `JobModel.registered` for non-router replicas in services with a router. That way, `JobModel.registered` can be reliably used to identify the jobs that should be registered on the gateway, without parsing the job spec. - Since the router worker sync pipeline needs both router and non-router replicas with successful probes, introduce `JobModel.ready`, set for any replica once probes pass. ```shell $ dstack event --within-run prefill-decode | grep -E "ready|register" [2026-08-05 10:30:29] [job prefill-decode-0-0] Service replica ready to receive requests [2026-08-05 10:30:29] [job prefill-decode-0-0] Service replica registered to receive requests [2026-08-05 10:44:45] [job prefill-decode-0-1] Service replica ready to receive requests [2026-08-05 10:45:00] [job prefill-decode-0-2] Service replica ready to receive requests [2026-08-05 11:15:22] [job prefill-decode-0-0] Service replica unregistered from receiving requests ```
…ish_router_and_non_router_replicas
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem: Non-router replicas in a service with
a router should not be registered on the gateway.
To determine whether a particular replica should
be registered, one has to parse its job spec and
run spec and see if it is a router replica or not.
This does not work well for the upcoming gateway
state sync mechanism, which needs to quickly fetch
all jobs to be registered from the database,
without parsing their specs.
Solution:
JobModel.registeredfornon-router replicas in services with a router.
That way,
JobModel.registeredcan be reliablyused to identify the jobs that should be
registered on the gateway, without parsing the
job spec.
router and non-router replicas with successful
probes, introduce
JobModel.ready, set for anyreplica once probes pass.
#3959