feat(perception): add configurable OSR scan backends

[ruthwikdasyam]

Summary

• add opt-in OWLv2 detection and EdgeTAM segmentation backends to OSR
• add the request-driven aligned RGB-D scan_scene API and expose it through ObjectSceneRegistrationSpec
• return pending ObjectDB entries from a scan so a newly observed object is immediately addressable by stable ID and point cloud
• retain the existing YOLOE/YOLO defaults and permanent-object stream behavior

Validation

• focused OSR temporal tests: 10 passed
• mypy on the changed OSR files: passed
• Ruff format and check on changed files: passed

Scope

• excludes SigLIP/Dan temporal memory, PnP integration, GraspGenX changes, filtering, and visualization.
• public method descriptions intentionally deferred for API review.

Checklist

• I have read and approved the CLA.

[codecov]

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
4891	2	4889	82

View the full list of 2 ❄️ flaky test(s)

dimos.e2e_tests.test_manipulation_planning_groups::test_dual_arm_plans_and_dispatches_both_arms_through_control_coordinator

Flake rate in main: 100.00% (Passed 0 times, Failed 3 times)

Stack Traces | 120s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x743bb1465c70>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x743bb0625da0>

    def test_dual_arm_plans_and_dispatches_both_arms_through_control_coordinator(
        lcm_spy: LcmSpy,
        start_blueprint: Callable[..., DimosCliCall],
    ) -> None:
        """Plan one generated plan over both arms and dispatch through one trajectory task."""
>       _start_openarm_mock_planner(start_blueprint, lcm_spy)

lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x743bb1465c70>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x743bb0625da0>

dimos/e2e_tests/test_manipulation_planning_groups.py:190: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/e2e_tests/test_manipulation_planning_groups.py:151: in _start_openarm_mock_planner
    lcm_spy.wait_for_saved_topic(JOINT_STATE_TOPIC, timeout=120.0)
        lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x743bb1465c70>
        start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x743bb0625da0>
dimos/e2e_tests/lcm_spy.py:97: in wait_for_saved_topic
    wait_until(
        condition  = <function LcmSpy.wait_for_saved_topic.<locals>.condition at 0x743bb0626340>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x743bb1465c70>
        timeout    = 120.0
        topic      = '/coordinator_joint_state#sensor_msgs.JointState'
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

predicate = <function LcmSpy.wait_for_saved_topic.<locals>.condition at 0x743bb0626340>

    def wait_until(
        predicate: Callable[[], bool],
        *,
        timeout: float,
        interval: float = 0.1,
        message: str | None = None,
    ) -> None:
        """Poll ``predicate`` until it returns truthy or ``timeout`` elapses."""
        deadline = time.monotonic() + timeout
        while time.monotonic() < deadline:
            if predicate():
                return
            time.sleep(interval)
>       raise TimeoutError(message or f"Timed out after {timeout}s waiting for condition")
E       TimeoutError: Timeout waiting for topic /coordinator_joint_state#sensor_msgs.JointState

deadline   = 4420507.987952065
interval   = 0.1
message    = 'Timeout waiting for topic /coordinator_joint_state#sensor_msgs.JointState'
predicate  = <function LcmSpy.wait_for_saved_topic.<locals>.condition at 0x743bb0626340>
timeout    = 120.0

.../utils/testing/waiting.py:35: TimeoutError

dimos.e2e_tests.test_manipulation_planning_groups::test_single_arm_plans_and_executes_through_control_coordinator

Flake rate in main: 25.00% (Passed 12 times, Failed 4 times)

Stack Traces | 120s run time

lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x743cb7650890>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x743bb0625800>

    def test_single_arm_plans_and_executes_through_control_coordinator(
        lcm_spy: LcmSpy,
        start_blueprint: Callable[..., DimosCliCall],
    ) -> None:
        """Plan with one arm and execute through its trajectory task."""
>       _start_openarm_mock_planner(start_blueprint, lcm_spy)

lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x743cb7650890>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x743bb0625800>

dimos/e2e_tests/test_manipulation_planning_groups.py:159: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/e2e_tests/test_manipulation_planning_groups.py:151: in _start_openarm_mock_planner
    lcm_spy.wait_for_saved_topic(JOINT_STATE_TOPIC, timeout=120.0)
        lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x743cb7650890>
        start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x743bb0625800>
dimos/e2e_tests/lcm_spy.py:97: in wait_for_saved_topic
    wait_until(
        condition  = <function LcmSpy.wait_for_saved_topic.<locals>.condition at 0x743bb0625a80>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x743cb7650890>
        timeout    = 120.0
        topic      = '/coordinator_joint_state#sensor_msgs.JointState'
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

predicate = <function LcmSpy.wait_for_saved_topic.<locals>.condition at 0x743bb0625a80>

    def wait_until(
        predicate: Callable[[], bool],
        *,
        timeout: float,
        interval: float = 0.1,
        message: str | None = None,
    ) -> None:
        """Poll ``predicate`` until it returns truthy or ``timeout`` elapses."""
        deadline = time.monotonic() + timeout
        while time.monotonic() < deadline:
            if predicate():
                return
            time.sleep(interval)
>       raise TimeoutError(message or f"Timed out after {timeout}s waiting for condition")
E       TimeoutError: Timeout waiting for topic /coordinator_joint_state#sensor_msgs.JointState

deadline   = 4420387.799218471
interval   = 0.1
message    = 'Timeout waiting for topic /coordinator_joint_state#sensor_msgs.JointState'
predicate  = <function LcmSpy.wait_for_saved_topic.<locals>.condition at 0x743bb0625a80>
timeout    = 120.0

.../utils/testing/waiting.py:35: TimeoutError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[greptile-apps]

Greptile Summary

This change adds configurable OWLv2 and EdgeTAM detection backends, request-driven RGB-D scene scans, prompt RPCs, and wall-clock recency tracking for pending objects.

Repeated request scans can process the same cached aligned RGB-D frame more than once. That can allow one camera observation to increase an object's confirmation count and add duplicate geometry, potentially promoting it before it has been observed in separate frames.

T-Rex validation blocked

The focused reproduction could not complete because the prepared Python tool is unavailable: .venv/bin/python points to a missing interpreter. The fallback Python installation also lacks the required numpy package, so the production module could not load to execute the one-frame, repeated-request path.

Confidence Score: 4/5

The change should not merge until repeated scans are prevented from treating an unchanged cached frame as a new observation.

The cached-frame processing path and the matching ObjectDB update path show that successive requests can reuse the same observation. Runtime confirmation could not be completed because the available Python environments could not import the production module.

Files Needing Attention: dimos/perception/experimental/object_scene_registration.py needs a consumed-frame or frame-version guard for request-driven scans; the associated update behavior in dimos/perception/experimental/objectDB.py makes repeated processing consequential.

T-Rex Logs

What T-Rex did

• Attempted the repeated-frame validation using one-scan and two-scan harnesses, but the run could not start because the Python interpreter at .venv/bin/python was missing and the system Python fallback lacked NumPy, so the harnesses halted before producing observation-count or point-cloud results.
• A P2 finding was posted and a validation note points to the corresponding review comment for details.
• A P1 finding was posted and a validation note points to the corresponding review comment for details.
• No completed runtime proof is available; the retained logs show the harness commands and their import/stub failures, and neither one-scan nor two-scan output was produced.

_{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "fix(perception): serialize on-demand OSR..." | Re-trigger Greptile}