HDDS-16250. Improve RocksDB read performance by reducing unnecessary iterator positioning

[smengcl]

What changes were proposed in this pull request?

Ozone's RocksDB-backed table iterators currently position themselves at the table or prefix start during construction. Callers that then explicitly seek to another key perform two native positioning operations:

1. Seek to the table or prefix start during iterator construction.
2. Seek to the caller's requested key.

The first operation is redundant. A RocksDB seek to the logical beginning is not merely assigning a cursor to the beginning of one file: RocksDB must merge its internal sources and find the first visible key, which can require skipping a large run of uncompacted tombstones.

This PR defers initial positioning until the first iterator operation:

• seek(), seekToFirst(), and seekToLast() position the iterator directly without an earlier seek.
• hasNext(), direct next(), and removeFromDB() preserve the existing behavior by initializing at the table or prefix start on first use.
• The behavior is implemented in RDBStoreAbstractIterator and applies consistently to the byte-array and CodecBuffer iterators.
• The native RocksDB iterator is still created immediately, preserving its database view and lifetime.

The change does not modify any public API, wire format, RocksDB schema, metadata layout, or caller-visible iterator semantics.

Expected benefit

The change eliminates one native RocksDB positioning operation when a caller creates an iterator and explicitly repositions it. The absolute latency reduction equals the cost of the eliminated table-start or prefix seek.

Production paths that can benefit include:

• listKeys and S3 ListObjects: OM creates a key-table iterator and seeks to the requested bucket, prefix, or continuation marker. First pages can benefit when the bucket sorts after a tombstone-heavy region, while later pages can benefit when the continuation marker is beyond deleted keys in the bucket.
• OzoneFS listStatus: FSO listings use directory and file prefix iterators that may seek to a later start key. LEGACY and OBJECT_STORE listings also use a table iterator followed by a start-key seek.
• listOpenFiles: OM seeks an open-key-table iterator to the requested path or continuation token.
• ListMultipartUploads: paginated requests seek a bucket-prefix iterator to the key and upload-ID marker.
• Key lifecycle scans: resumed scans can seek directly to the saved last-scanned key after suspension, restart, or leadership transfer.
• Snapshot and Recon pagination: listings and endpoints using start-key or previous-key markers avoid the same redundant positioning.

A standalone synthetic benchmark used RocksDB JNI 10.10.1.1, a warm cache, one thread, disabled automatic compaction, and consecutive point tombstones before a live target. The benchmark measures RocksDB positioning, not end-to-end Ozone RPC latency.

When the continuation target was beyond the tombstone band:

Tombstones	Double seek	Single seek	Positioning speedup
10,000	819.5 µs	1.1 µs	approximately 756×
100,000	8.27 ms	1.2 µs	approximately 6,850×
500,000	96.7 ms	1.5 µs	approximately 66,000×

When both positioning operations crossed the tombstone region, removing one of them produced approximately a 2× positioning improvement.

Actual request-level improvement depends on tombstone distribution, compaction state, cache state, result decoding, OM cache merging, storage latency, and RPC processing. Little improvement is expected for compacted tables, low tombstone counts, or callers that begin ordinary iteration without explicitly repositioning the iterator.

The standalone RocksTombstoneBench.java reproducer can be provided as a separate patch. Its header contains the exact compilation and execution commands and documents the benchmark conditions.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16250

How was this patch tested?

Regression tests verify that:

• Iterator construction performs no native seek.
• hasNext() and direct next() still initialize at the table or prefix beginning.
• Repeated hasNext() calls do not repeat initialization.
• An explicit seek on a prefixed iterator performs exactly one native seek to the requested key.
• seekToLast() does not first seek to the beginning.
• removeFromDB() preserves its implicit-start behavior.
• Byte-array and CodecBuffer iterators have identical positioning behavior.

Validation results:

• Targeted iterator tests: 29 passed with no failures or errors.
• Broader hdds-server-framework suite excluding TestDU: 596 passed and 1 skipped.
• TestDU.testExcludePattern fails with the same error on clean master on macOS and is unrelated to this change.
• Repository-wide checkstyle: all 58 modules passed.
• The standalone RocksDB synthetic benchmark compiled and completed successfully under JDK 21.

A wall-clock performance assertion is intentionally excluded from the unit suite because it would depend on host timing, cache state, storage, and RocksDB compaction state. The regression tests instead verify the number and order of native positioning operations deterministically.

Generated-by: Codex (GPT-5.6 Sol)

[Copilot]

Pull request overview

This PR optimizes RocksDB-backed table iteration in HDDS by deferring initial iterator positioning until the first iterator operation, avoiding redundant native seeks (especially costly when scanning across tombstone-heavy regions). It also introduces a fast path in RDBTable#getRangeKVs that uses a point Get for exact single-entry range requests when the contract requires the start key to exist and the request is within the given prefix.

Changes:

• Deferred iterator positioning to first use in RDBStoreAbstractIterator, eliminating construction-time seeks while preserving existing semantics for ordinary iteration (hasNext()/next()/removeFromDB()).
• Added a point-Get optimization for exact count == 1 range requests in RDBTable#getRangeKVs when conditions allow, avoiding iterator creation/positioning.
• Expanded and adjusted unit tests to validate seek counts/order, prefix-boundary semantics, and the new point-Get behavior (including aliasing expectations for returned keys).

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreAbstractIterator.java	Defers initial positioning until first iterator operation; ensures explicit seeks don’t incur an extra initial seek.
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreByteArrayIterator.java	Removes construction-time seekToFirst() to align with deferred initialization semantics.
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreCodecBufferIterator.java	Removes construction-time seekToFirst() to align with deferred initialization semantics.
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBTable.java	Adds point-Get fast path for eligible single-entry exact ranges; retains iterator-based behavior otherwise.
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStoreByteArrayIterator.java	Updates iterator tests to assert no construction seek and correct seek ordering/counts.
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStoreCodecBufferIterator.java	Updates CodecBuffer iterator tests for deferred initialization and exact seek argument verification.
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBTableStore.java	Adds/adjusts prefixed range test coverage for prefix-boundary and filtered/single-entry behaviors.
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBTable.java	Adds tests ensuring eligible getRangeKVs calls use point Get and avoid iterator creation; validates key aliasing expectations.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jojochuang]

lgtm

[rich7420]

lgtm!

[smengcl]

Thanks @jojochuang @rich7420 for the reviews.