Yimo/feat/hosted - #3692
Conversation
github-actions
Bot
added
the
first-time-contributor
There was a problem hiding this comment.
we decided against multi language docs for now (since it adds maitenance burden and translation tools are so good)
There was a problem hiding this comment.
I'll keep that in mind for next time. This PR isn't built for merge so the Chinese documentation was just there to help me think things through in Chinese.I will make sure this doesn't happen again.
|
|
||
| The system has three related but separate concerns. | ||
|
|
||
| ### Discovery plane |
There was a problem hiding this comment.
keep in mind, next to RPC, zenoh also offers distributed key/value store, but this is risky/complex so I think rpc for initial implementation is a responsible choice
There was a problem hiding this comment.
Yes! I noticed that some of the transports in the dimos are implemented using Zenoh as well.
|
|
||
| This means the application can continue moving data if the controller exits | ||
| after startup, although the MVP may keep it alive to aggregate status and | ||
| coordinate shutdown. |
There was a problem hiding this comment.
I think keepalive packets will be neccessary between coordinator and Host service(s)
if someone runs kill -9 on a laptop - remote system should notice that owning blueprint is down, and shutdown it's modules
There was a problem hiding this comment.
Got it. I'll add that in just a bit
| - Host presence disappears when its Zenoh liveliness token is lost; | ||
| - a descriptor is fetched from the live Host rather than trusted from a stale | ||
| retained value; | ||
| - discovery works through routers and gossip, not only local multicast; |
There was a problem hiding this comment.
start from here (about to get merged) makes zenoh default across dimos
| | `protocol_version`, `plan_schema_version` | Control and deployment-plan compatibility. | | ||
| | `dimos_version`, `application_revision` | Runtime and code compatibility. | | ||
| | `active_runs`, `leases` | Current occupancy and pending reservations. | | ||
| | `health`, `last_error` | Whether the Host can accept work and why not. | |
There was a problem hiding this comment.
great, later we can add even system metrics etc
| MarkerDetectionStreamModule.blueprint( | ||
| marker_length_m=0.1, | ||
| camera_info=GO2Connection.camera_info_static, | ||
| ).placement(tags={"gpu"}), |
There was a problem hiding this comment.
.placement is a weird word, maybe .remote ?
There was a problem hiding this comment.
That makes a lot of sense. The reason I used "placement" is because local execution can also serve as a placement location. But "remote" is definitely clearer semantically, so I'll change it to "remote".
There was a problem hiding this comment.
Maybe placement is a bad term, but remote is worse. How about "on_host" or "remote_host"?
| | Cross-Host stream using shared memory | Rejected. | | ||
| | Other machine-local transports | Supported only when both endpoints are on the same Host. | | ||
|
|
||
| Keeping explicit shared memory for local image or point-cloud paths avoids a |
There was a problem hiding this comment.
yeah now that we merge zenoh #3617 shm will be used automatically between local modules, so all blueprints will just be on zenoh and you won't have these issues
|
|
||
| # Select one G1 Host and one GPU Host; navigation remains local. | ||
| multi_host_g1 = autoconnect( | ||
| unitree_g1.blueprint().placement(tags={"g1"}), |
There was a problem hiding this comment.
nitpick but dont like the word placement. maybe .hosted()
| This proposal adds a thin distributed orchestration layer: | ||
|
|
||
| - Each execution device runs a small, persistent **Host service**. | ||
| - A Host joins a Zenoh fabric and advertises its generated identity, discoverable |
There was a problem hiding this comment.
Good Question. "Fabric" is a term in zenoh. It can be understood as a "logical data network" composed of Zenoh nodes, connections, routing, and key-expression namespaces.
|
|
||
| This proposal adds a thin distributed orchestration layer: | ||
|
|
||
| - Each execution device runs a small, persistent **Host service**. |
There was a problem hiding this comment.
Presumably this would be a standalone Rust binary, right?
There was a problem hiding this comment.
It would be a standalone binary. I haven't decided on the language yet, but Rust sounds like a good choice!
| 5. Reuse the current blueprint compiler, `ModuleCoordinator`, workers, module | ||
| lifecycle, and stream implementations on each machine. | ||
| 6. Use Zenoh for streams that cross machine boundaries. |
There was a problem hiding this comment.
Are you saying that LCM would be supported on a machine, but for cross machine streams, Zenoh would be enforced?
I think it would be easier to just mandate Zenoh for everything for blueprints containing remote modules.
There was a problem hiding this comment.
Sorry, my document might not have been very clear. I actually mentioned elsewhere in it that LCM isn't enabled on hosted, meaning it doesn't reuse the existing LCM. All RPCs and streams go through Zenoh instead.
| 10. Establish a foundation for discovering robot embodiments and other typed | ||
| hardware capabilities without making fleet replication part of the MVP. |
There was a problem hiding this comment.
Why would this matter? I assume distributed modules are completely independent of what a particular module does (control a robot or not).
There was a problem hiding this comment.
Under the current design, machine capabilities rely entirely on manual tag annotation. For instance, if I tag a machine with GPU, it doesn't actually mean the hardware is there. This kind of manual convention isn't really the best practice, but during the MVP stage, we can go ahead and do it that way for now.
Later on, we will definitely need automated hardware discovery. Otherwise, if someone applies a GPU tag to a machine that doesn't actually have one, it's going to cause issues.
|
|
||
| | Term | Meaning | | ||
| | --- | --- | | ||
| | Application | One logical blueprint graph that may span several Hosts. | |
There was a problem hiding this comment.
Wouldn't this just be Blueprint?
| | Term | Meaning | | ||
| | --- | --- | | ||
| | Application | One logical blueprint graph that may span several Hosts. | | ||
| | Controller | The `dimos run` process that compiles, places, and coordinates an application. | |
There was a problem hiding this comment.
How about Orchestrator?
Controller is a much more generic term and could be easily confused with many other things.
There was a problem hiding this comment.
You are right. It's a good idea.
| | --- | --- | | ||
| | Application | One logical blueprint graph that may span several Hosts. | | ||
| | Controller | The `dimos run` process that compiles, places, and coordinates an application. | | ||
| | Host | One advertised execution service. Usually one per machine, although several may sit behind one router or even share a machine. | |
There was a problem hiding this comment.
Host is an incredibly overloaded term. I much prefer rarer terms because we can imbue them with precise meanings for our system.
I'm not sure what to propose as an alternative though. Node or runner are too generic as well...
There was a problem hiding this comment.
How about "outpost"? :)
It definitely gives out the meaning of "something that's lightweight and waits for you to use it".
| runtime; it is a discoverable supervisor that creates and controls an existing | ||
| local runtime. | ||
|
|
||
| ## Terminology |
There was a problem hiding this comment.
You haven't defined the most important term: fabric. :)
| ### Discovery plane | ||
|
|
||
| Hosts advertise live execution capacity on Zenoh. The controller first joins a | ||
| fabric, observes live Host identities, and queries their descriptors. A Zenoh |
There was a problem hiding this comment.
Can two different controllers place modules on the same host? How do we deal with this? Or should it be left for later?
| | `host_id`, `name`, `epoch` | Stable identity, display, and current process incarnation. | | ||
| | `tags` | Opaque placement labels configured by the user or a capability provider. | | ||
| | `capabilities` | Structured hardware facts, such as robot model/serial, GPU type, architecture, and attached devices. | | ||
| | `resources_total`, `resources_available` | CPU, memory, accelerators, deployment slots, and other schedulable capacity. | |
There was a problem hiding this comment.
I don't think we need this for the first version. We don't even have detailed resource usage requirements for the modules yet (though we've talked about it).
There was a problem hiding this comment.
yeah this is just very easy to log and nice to have but optional
| - A module with no placement metadata has a soft preference for the controlling | ||
| machine, preserving current behavior. | ||
| - `.placement(local=True)` can express a hard local constraint when needed. |
There was a problem hiding this comment.
I don't think this is needed. Local should be the default.
| Unconstrained modules stay local unless merging a module-reference co-location | ||
| unit with an explicitly placed module forces them onto that module's Host. The | ||
| placement explanation should make this inheritance visible. |
There was a problem hiding this comment.
I don't understand the unless part. Unconstrained modules should always be local, right?
| > A module and every module reference it consumes must belong to the same | ||
| > placement unit and resolve to the same Host. |
There was a problem hiding this comment.
Hmm, this is quite limiting. I would have assumed doing RPC across hosts would be easy.
There was a problem hiding this comment.
true this is a good catch, Zenoh gives us RPC
4 participants
This PR introduces the basic API design for hosted support. It is an initial implementation intended to establish a foundation and is open for public discussion and further iteration. @spomichter @leshy