Skip to content

HDDS-16229. Add RocksDB multiGetSkipCache for batched table reads. - #11068

Draft
SaketaChalamchala wants to merge 2 commits into
apache:masterfrom
SaketaChalamchala:HDDS-16229
Draft

HDDS-16229. Add RocksDB multiGetSkipCache for batched table reads.#11068
SaketaChalamchala wants to merge 2 commits into
apache:masterfrom
SaketaChalamchala:HDDS-16229

Conversation

@SaketaChalamchala

@SaketaChalamchala SaketaChalamchala commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

RocksDB MultiGet support through RocksDatabase, RDBTable, and TypedTable on a skip-cache path. Callers can batch point lookups in one RocksDB call instead of looping get() / getSkipCache().
Added multiGet with CodecBuffer in TypedTable.

Missing keys return null at the matching list index; order matches the input key list.

Out of scope (follow-ups): cache-aware Table.multiGet, batch metrics, call-site adoption.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16229

How was this patch tested?

Unit Tests

Plumb RocksDB multiGetAsList through RocksDatabase, RDBTable, and
TypedTable.multiGetSkipCache for skip-cache batch point lookups.

Co-authored-by: Cursor <cursoragent@cursor.com>

@swamirishi swamirishi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the patch @SaketaChalamchala this is a good improvement to have.

}

@Override
public List<VALUE> multiGetSkipCache(List<KEY> keys)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please implement CodeBuffer based multiGets as well otherwise the GC pressure on memory would be bad

https://github.com/facebook/rocksdb/blob/805ac7c887bbebd7e6124bbc7e8d0803aeed8f07/java/src/main/java/org/rocksdb/RocksDB.java#L2435-L2460

You would need to implement a similar function like

private VALUE getFromTable(KEY key,
CheckedBiFunction<CodecBuffer, CodecBuffer, Integer, RocksDatabaseException> get)
throws RocksDatabaseException, CodecException {
try (CodecBuffer inKey = keyCodec.toDirectCodecBuffer(key)) {
for (; ;) {
final Integer required;
final int initial = -bufferCapacity.get(); // resizable
try (CodecBuffer outValue = CodecBuffer.allocateDirect(initial)) {
required = get.apply(inKey, outValue);
if (required == null) {
// key not found
return null;
} else if (required < 0) {
throw new IllegalStateException("required = " + required + " < 0");
}
for (; ;) {
if (required == outValue.readableBytes()) {
// buffer size is big enough
return valueCodec.fromCodecBuffer(outValue);
}
// buffer size too small, try increasing the capacity.
if (!outValue.setCapacity(required)) {
break;
}
// retry with the new capacity
outValue.clear();
final int retried = get.apply(inKey, outValue);
Preconditions.assertSame(required.intValue(), retried, "required");
}
}
// buffer size too small, reallocate a new buffer.
bufferCapacity.increase(required);
}
}
}

to ensure the value which didn't fit in the first multiget should be called again with a resized CodecBuffer size.
So the first call should do a multiGet with all CodecBuffer keys and based on the status list returned we need to call multiget again for the ones which didn't fit in the initial fit and collate the list.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants