danvi/dim1486/generalize dan perception stack - #3723
Conversation
…he window's cell range - one fancy-index lookup per snapshot's points and one rectangle memset after it
❌ 3 Tests Failed:
View the top 2 failed test(s) by shortest run time
View the full list of 1 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
Greptile SummaryThis change broadens perception-memory localization across depth-camera and registered point-cloud rigs, adds persistent object identities, batches detector scoring, and expands recording CLI configuration. Two localization reliability problems remain. Short embedded recordings can yield no semantic frame to inspect, so objects in one- or two-frame windows are never sent to detection. Separately, a frame that temporarily lacks usable 3D geometry is permanently skipped when the same identity store is reused, even after geometry becomes available. T-Rex validation blockedThe short-recording check could not run to completion because the Python runtime was missing repository dependencies. The final import failure was for the Confidence Score: 3/5Not safe to merge without addressing the two localization failure paths, because valid objects can be omitted from results. One failure was reproduced by executing two localization calls against a reused identity store and demonstrating recovery after delaying ingestion. The short-recording failure is directly supported by the selection path, but its runtime harness was blocked by missing Python packages. Files Needing Attention:
 T-Rex LogsWhat T-Rex did
Reviews (1): Last reviewed commit: "ship go2_short.db.rig.json inside the go..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| .transform(peaks(key=_similarity, distance=1.0)) | ||
| .materialize() | ||
| ) | ||
| logger.info( | ||
| f"localize {q!r}: {label_peaks.count()} semantic peaks of {index.count()} embedded" |
There was a problem hiding this comment.
When an embedded index contains only one or two observations, routing the complete search result through peaks() produces no peaks, so detection never runs and localization returns an empty result even when an available frame contains the requested object.
| key = (obs.ts, queries[j], floor) | ||
| cache.move_to_end(key) | ||
| rows_per_label.append((j, cache[key])) | ||
| ingested[j].add(obs.ts) |
There was a problem hiding this comment.
A frame is added to the persistent ingested set before segmentation and 3D lifting have succeeded. If pose or geometry is temporarily unavailable, a later call using the same IdentityStore skips the frame entirely, so the object cannot be recovered once geometry becomes available. Mark the timestamp ingested only after the frame has produced durable identity evidence, or retain failed frames for retry.
Artifacts
Focused two-call harness source
- Authored narrow harness that executes localize() twice with one reused IdentityStore, toggling geometry availability between calls and comparing the current source with delayed ingestion.
Current-source two-call output
- Executed output (exit 0): first=[]; ingested_after_first=[7.0]; second=[]; score_calls=1; segment_calls=1; lift_geometry_states=[False].
Delayed-ingestion comparison output
- Executed output (exit 0): first=[]; ingested_after_first=[]; second=['recovered']; score_calls=1; segment_calls=2; lift_geometry_states=[False, True].
leshy
left a comment
leshy
left a comment
There was a problem hiding this comment.
basically escaped mem2 immediately, first comments first view to push back on this, might have more stuff
| index.search(query_embedding) | ||
| .order_by("ts") | ||
| .transform(peaks(key=_similarity, distance=1.0)) | ||
| .materialize() |
There was a problem hiding this comment.
why are we ordering by ts? we can order by peak height (this is a default ordering) - why are you materializing the full stream, this is a very expensive operation, that loads all data in memory. we should be able to have a seeker head that loads observation by observation until peak becomes irrelevant. very low mem footprint
| logger.info( | ||
| f"localize {q!r}: {label_peaks.count()} semantic peaks of {index.count()} embedded" | ||
| ) | ||
| peaks_per_label.append(label_peaks) |
There was a problem hiding this comment.
you escaped memory system and streams immediately and are building lists. why isn't this a mem stream? you should always operate on streams of observations, not on raw lists containing all data, we are angling towards parsing an hour, a day of data etc
| expanded.add(peak.ts) | ||
| nearby: Stream[Any, Any] = index.near( | ||
| peak.pose_stamped, radius=policy.verify_radius_m | ||
| ).transform(QualityWindow(lambda img: img.sharpness, window=0.5)) |
There was a problem hiding this comment.
embeddings themselves should already be filtered by the qualitywindow, all these matches will be clean, you don't need to filter downstream
| ).transform(QualityWindow(lambda img: img.sharpness, window=0.5)) | ||
| for obs in nearby: | ||
| frames.setdefault(obs.ts, obs) | ||
| ordered = sorted(frames.values(), key=lambda obs: obs.ts) |
| logger.info(f"detection: {len(ordered)} candidate frames for {len(queries)} labels") | ||
|
|
||
| if ordered: | ||
| from dimos.perception.memory.support_plane import fit_support_plane |
There was a problem hiding this comment.
why are not these following lines a nice mem2 transform?
2 participants
Generealize perception stack
This PR builds, improves, and adds new functionality based on the foundation of these previous PRs:
#3496
#3422
How to test and what to expect
refference machine:
Lenovo Legion laptop, AMD Ruyzen AI 7, 32GB RAM, NVIDIA RTX 5070 Laptop edition (8 GB)
xArm
This uses the deault location for xArm dataset, already on main, xarm6_worldbelief_20260729_203624_161992.db.
You do not need to specify the --dataset <location_to_recording_bag_.db>
time uv run python -m dimos.perception.memory.tool_localize "roll of black tape" "book" "pen" "red marker" "yellow sticky notes" pc_xArm_all_multi.rrd --from 427 --duration 76 --multi && uv run dimos-viewer pc_xArm_all_multi.rrdG1
You will need a g1 recording for this, check this PR, it's on main already, #3527
time uv run python -m dimos.perception.memory.tool_localize "paper box" "plant in a pot" "shoes" "wooden crate" "office chair" "bag" g1_sf_120_180_localize.rrd --dataset data/g1_sf_office.db --from 120 --duration 60 --multi && uv run dimos-viewer g1_sf_120_180_localize.rrdGo2
Run on the default go2_short.db
time uv run python -m dimos.perception.memory.tool_localize "table" "white robot" "potted plants" "wooden doors" "coke bottle" "fanta bottle" "sprite bottle" "lays chips" "pillow" "green plant" "plant pot" go2_short_00_60.rrd --dataset data/go2_short.db --from 0 --duration 60 --multi && uv run dimos-viewer go2_short_00_60.rrdImportant
Use localize() API for testing and building