record: add --record mcap alongside sqlite

[jeff-hykin]

add --record mcap

[codecov]

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
4820	2	4818	83

View the top 1 failed test(s) by shortest run time

dimos.e2e_tests.test_manipulation_planning_groups::test_dual_arm_plans_and_dispatches_both_arms_through_control_coordinator

Stack Traces | 120s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x781d349875c0>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x781d16d1bb00>

    def test_dual_arm_plans_and_dispatches_both_arms_through_control_coordinator(
        lcm_spy: LcmSpy,
        start_blueprint: Callable[..., DimosCliCall],
    ) -> None:
        """Plan one generated plan over both arms and dispatch through one trajectory task."""
>       _start_openarm_mock_planner(start_blueprint, lcm_spy)

lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x781d349875c0>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x781d16d1bb00>

dimos/e2e_tests/test_manipulation_planning_groups.py:190: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/e2e_tests/test_manipulation_planning_groups.py:151: in _start_openarm_mock_planner
    lcm_spy.wait_for_saved_topic(JOINT_STATE_TOPIC, timeout=120.0)
        lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x781d349875c0>
        start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x781d16d1bb00>
dimos/e2e_tests/lcm_spy.py:97: in wait_for_saved_topic
    wait_until(
        condition  = <function LcmSpy.wait_for_saved_topic.<locals>.condition at 0x781d16cd44a0>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x781d349875c0>
        timeout    = 120.0
        topic      = '/coordinator_joint_state#sensor_msgs.JointState'
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

predicate = <function LcmSpy.wait_for_saved_topic.<locals>.condition at 0x781d16cd44a0>

    def wait_until(
        predicate: Callable[[], bool],
        *,
        timeout: float,
        interval: float = 0.1,
        message: str | None = None,
    ) -> None:
        """Poll ``predicate`` until it returns truthy or ``timeout`` elapses."""
        deadline = time.monotonic() + timeout
        while time.monotonic() < deadline:
            if predicate():
                return
            time.sleep(interval)
>       raise TimeoutError(message or f"Timed out after {timeout}s waiting for condition")
E       TimeoutError: Timeout waiting for topic /coordinator_joint_state#sensor_msgs.JointState

deadline   = 4287501.2834463
interval   = 0.1
message    = 'Timeout waiting for topic /coordinator_joint_state#sensor_msgs.JointState'
predicate  = <function LcmSpy.wait_for_saved_topic.<locals>.condition at 0x781d16cd44a0>
timeout    = 120.0

.../utils/testing/waiting.py:35: TimeoutError

View the full list of 1 ❄️ flaky test(s)

dimos.e2e_tests.test_manipulation_planning_groups::test_single_arm_plans_and_executes_through_control_coordinator

Flake rate in main: 7.69% (Passed 12 times, Failed 1 times)

Stack Traces | 120s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x781e378db500>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x781d16d1b4c0>

    def test_single_arm_plans_and_executes_through_control_coordinator(
        lcm_spy: LcmSpy,
        start_blueprint: Callable[..., DimosCliCall],
    ) -> None:
        """Plan with one arm and execute through its trajectory task."""
>       _start_openarm_mock_planner(start_blueprint, lcm_spy)

lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x781e378db500>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x781d16d1b4c0>

dimos/e2e_tests/test_manipulation_planning_groups.py:159: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/e2e_tests/test_manipulation_planning_groups.py:151: in _start_openarm_mock_planner
    lcm_spy.wait_for_saved_topic(JOINT_STATE_TOPIC, timeout=120.0)
        lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x781e378db500>
        start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x781d16d1b4c0>
dimos/e2e_tests/lcm_spy.py:97: in wait_for_saved_topic
    wait_until(
        condition  = <function LcmSpy.wait_for_saved_topic.<locals>.condition at 0x781d16d19c60>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x781e378db500>
        timeout    = 120.0
        topic      = '/coordinator_joint_state#sensor_msgs.JointState'
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

predicate = <function LcmSpy.wait_for_saved_topic.<locals>.condition at 0x781d16d19c60>

    def wait_until(
        predicate: Callable[[], bool],
        *,
        timeout: float,
        interval: float = 0.1,
        message: str | None = None,
    ) -> None:
        """Poll ``predicate`` until it returns truthy or ``timeout`` elapses."""
        deadline = time.monotonic() + timeout
        while time.monotonic() < deadline:
            if predicate():
                return
            time.sleep(interval)
>       raise TimeoutError(message or f"Timed out after {timeout}s waiting for condition")
E       TimeoutError: Timeout waiting for topic /coordinator_joint_state#sensor_msgs.JointState

deadline   = 4287381.115174528
interval   = 0.1
message    = 'Timeout waiting for topic /coordinator_joint_state#sensor_msgs.JointState'
predicate  = <function LcmSpy.wait_for_saved_topic.<locals>.condition at 0x781d16d19c60>
timeout    = 120.0

.../utils/testing/waiting.py:35: TimeoutError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[greptile-apps]

Greptile Summary

This change adds MCAP recording and replay support for DIMOS streams. Two verified failures prevent recordings from meeting the intended interoperability and data-preservation behavior: Foxglove cannot directly decode the emitted LCM payloads, and distinct MCAP topics such as foo/bar and foo_bar collapse into a single stream after reopening. The affected MCAP writer and reader need correction before merge.

Confidence Score: 3/5

Unsafe to merge because recordings can be unusable in the advertised viewer and can silently lose access to a stream after reopening.

Two independent non-security correctness failures were reproduced through the MCAP write and reopen paths.

Files Needing Attention: dimos/memory/store/mcap.py needs interoperable schema and message encoding output plus collision-safe stream identity handling.

T-Rex Logs

What T-Rex did

• T-Rex produced a P1 finding proof and attached a focused DIMOS MCAP compatibility harness source to support the validation.
• T-Rex captured and validated the MCAP behavior by observing the DIMOS-generated MCAP output and the Foxglove-compatible MCAP contrast output.
• T-Rex produced a second P1 finding proof including a focused MCAP slash-versus-underscore collision validation source and a successful collision validation result.
• T-Rex produced a third P1 finding proof, but no artifacts were attached.
• A general contract validation run completed with EXIT_CODE 0, showing two written topics, one reopened stream, the _stream_topic mapping, and a KeyError for foo/bar.

_{Ran code and verified through T-Rex}

Comments Outside Diff (2)

1. dimos/memory/store/mcap.py, line 97-103 (link)

   Topic identities collide after reopening

   The writer preserves MCAP topics such as foo/bar, but the reader converts / to _ before using the value as its stream key. A recording containing both foo/bar and foo_bar therefore reopens with only one visible foo_bar stream: the later channel overwrites the earlier metadata and foo/bar cannot be retrieved. Preserve the original topic as the stream identity, or generate collision-safe names and reject duplicate resolved names.

   Artifacts

   
   

   Focused MCAP slash-versus-underscore collision validation source

   • The executed script writes both conflicting topics, reopens the MCAP store, and checks metadata plus retrieval behavior, with the takeaway that the collision is exercised end to end.

   Successful MCAP topic collision validation output

   • Captured output from the focused write-reopen-read execution shows one surviving stream and a failed `foo/bar` lookup, with the takeaway that the identity collision is confirmed.

   

   _{Ran code and verified through T-Rex}

2. General comment

   McapStore silently aliases slash and underscore MCAP topics

   • Bug
     • After writing distinct topics foo/bar and foo_bar, reopening exposes only foo_bar; the original foo/bar stream cannot be retrieved and its payload is absent from the visible backend.
   • Cause
     • _slug() replaces / with _ (mcap.py lines 97–103), and the reader stores channel metadata in dictionaries keyed by this derived name without detecting duplicate keys (lines 203–215). The writer preserves the original name as the MCAP topic (lines 322–324).
   • Fix
     • Preserve topic identity as the default stream key, or introduce collision-safe generated names and raise a clear error when two channels map to the same friendly name; validate explicit streams overrides for duplicate names too.

   _{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "record: add `--record mcap` alongside sq..." | Re-trigger Greptile}

[greptile-apps]

Foxglove decoding is unavailable

McapWriteStore writes opaque DIMOS LCM payloads with the custom lcm message encoding while registering an empty schema encoding and zero schema bytes. Foxglove cannot decode this MCAP encoding pair or infer payload fields from the DIMOS-only schema name, so users need conversion or a custom extension instead of opening and visualizing the recording directly. Emit a Foxglove-supported self-describing representation and schema, such as JSON plus JSON Schema, Protobuf plus a descriptor set, or CDR plus IDL data; otherwise document the required conversion workflow.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Artifacts

Focused DIMOS MCAP compatibility harness source

• This authored harness executes the changed DIMOS writer, parses its MCAP summary with mcap 1.4.0, and compares the emitted pair against Foxglove's documented decoder pairs, confirming the unsupported LCM metadata.

Observed DIMOS-generated MCAP output

• This executed runtime capture shows the actual DIMOS writer emitted `lcm` with empty schema encoding and zero schema bytes, confirming direct Foxglove visualization is unsupported.

Observed Foxglove-compatible MCAP contrast output

• This executed contrast capture shows the same harness recognizes the documented JSON plus JSON Schema encoding pair as supported, confirming the required interoperable alternative.

_{Ran code and verified through T-Rex}

[spomichter]

I have a PR for mcap support that i will merge on top