Warn about Zarr layouts that make viewing awkward - #430
Conversation
Check the dataset layout client-side from metadata that is already parsed, and surface the result through the MetadataHint notifications used for broken metadata. All three checks are advisory - nothing is withheld, and viewer links are untouched. Resolution levels: a multiscales group declaring a single dataset provides no downsampled data, so every zoom reads full-resolution chunks. This was the root cause of the production incident behind these checks. Keyed on the dataset count rather than the number of shapes, because a plain zarr array also has exactly one shape while claiming nothing about being multiscale. Chunk size: chunks past the browser's cache entry limit are re-fetched on every access. Sizes are computed from shape and dtype, so they are logical: exact for a raw array, an upper bound for a compressed one. The limit is 10 MB when no compressor is present and 32 MB when one is, since the compression ratio is a property of the pixels and cannot be derived from metadata. Unknown or unfetched codec info counts as compressed, so uncertainty costs a missed warning rather than a false one. Codec classification reuses capability-manifest's classifyCodec, plus a walk over the pipeline it does not do itself: a sharded v3 array lists only the structural sharding_indexed codec at the top level and carries the real compressor in its configuration. Plain arrays now carry codec info too - the v3 branch reads it from the zarr.json it already parsed, the v2 branch fetches .zarray as the group path does. Axis order: OME-Zarr requires T, C, Z, Y, X order, and many tools take the last two axes to be the image plane. A dataset ordered C, X, Y, Z therefore renders as a Y-Z cross-section in those tools while Neuroglancer, which reads the axis names, looks fine - a confusing failure worth naming explicitly. Datasets using custom axis names are left alone, since their intent cannot be judged. No thumbnail check: ome-zarr.js already refuses when the lowest level exceeds its own limit, and modelling its behaviour a second time proved unreliable - it renders from the level nearest the requested size rather than the smallest, and reads a single plane rather than the whole level. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
At some point, we may want to provide virtualization for Zarr datasets like this as well. |
mkitti
left a comment
mkitti
left a comment
There was a problem hiding this comment.
I am not sure how well these warnings are grounded in the spec and perhaps more specifically in Neuroglancer's implementation. I'm not sure where the 10 MiB cache limit comes from. Could we check the cache limits and add citations when possible to clearly state whether these are browser or middleware issues.
I suspect that the dimension order could be relaxed here especially in light of RFC-3. The main relevant ordering is probably the clustering of the spatial dimensions at the end but the order of the spatial dimensions is not really a problem for Neuroglancer.
Another warning I would add here is the dimensionality of the chunks versus the dimensionality of the array. If we have 2D chunks and a 3D spatial array, we will have issues for viewing in some of the orthogonal views. If the array is nD, then we want the dimension of the chunks to have non-singleton dimensions for at leastmin(n, 3) dimensions.
Also chunks that are too small might be an issue as well. In the extreme a chunk that is a single voxel is highly problematic. I would suggest a minimum element count of 256 elements if there are more elements than this in the array.
| return { | ||
| kind: 'warning', | ||
| title: 'Axes are not in the order OME-Zarr specifies', | ||
| description: `The axes are ordered ${variant.axisOrder}, but the spec requires T, C, Z, Y, X order - here that would be ${variant.expectedOrder}. Many tools take the last two axes to be the image plane, so they will show a cross-section rather than the expected view. Rewriting the dataset with the axes in spec order avoids this.` |
There was a problem hiding this comment.
RFC 3 will likely make this advice obsolete with respect to the spec statement:
There was a problem hiding this comment.
I know that, but in the meantime it is creating issues, specifically with ome-zarr.js (and I would bet there are other cases).
For example: https://github.com/BioNGFF/ome-zarr.js/blob/main/src/image.ts#L336-L337
| return { | ||
| kind: 'warning', | ||
| title: 'Chunks may be too large for efficient viewing', | ||
| description: `This dataset uses ${variant.size} chunks ${variant.compressed ? '(before compression)' : '(without compression)'}. Chunk files larger than the browser's cache limit are re-downloaded on every access, which makes viewing slow. A final chunk size of 1-10 MB works best.` |
There was a problem hiding this comment.
We may need to qualify this for shards. Shards are also large top-level chunks.
There was a problem hiding this comment.
I'm not sure I follow. Shards are on the server side, the client side deals with chunks. Why would shard size matter to the client?
| /** | ||
| * Chunks above this defeat the browser cache. Applies when the array is stored | ||
| * without compression, so the size we compute is the size that transfers. | ||
| */ | ||
| export const MAX_CHUNK_BYTES = 10 * 1024 ** 2; | ||
| /** |
There was a problem hiding this comment.
I'm not sure if this limit is the one that would apply to Zarr chunks.
https://claude.ai/share/2c1c2b4f-c9a3-43d3-a24d-9de809b04062
There was a problem hiding this comment.
Some browsers do have a limit on the size of a file they will cache. For example, Firefox will only cache up to 5 MB files in memory (browser.cache.memory.max_entry_size) and 50 MB on disk (browser.cache.disk.max_entry_size). This 10 MB threshold is uncompressed, and assumes 2-3x compression will put the chunk under the 5 MB threshold. It's just a heuristic.
Of course, there are other reasons to favor smaller chunk sizes, but not too small. Maybe there are better heuristics or rules we can use here.
Browsers have limits at 5 MB, as explained in the code comment. This thread suggests that reasonable chunk file sizes are between 1-10MB. What we want to do here is to provide a soft warning to people when they use chunk sizes that are clearly not ideal. Where would you set that threshold without creating too many false positives? I don't expect that this threshold can be a perfect classifier for correct/wrong, since there are many different (and often opposing) forces acting upon it.
But RFC-3 is not a part of any official specification version, and I'm not sure when it would be merged. Even after it is released, it will take time for libraries (such as ome-zarr.js) to catch up with it. In the meantime, it creates issues when the spec is not followed. Although there may be good reasons for deviating from the spec (and maybe some of those inspired RFC-3), I don't see the problem with alerting people that they are deviating.
Yes, but that seems out of scope for this PR, which is trying to address concrete issues that happened in production.
Yes, but also out of scope here. |
2 participants
Adds three advisory warnings to the Zarr preview panel, flagging layout choices that make a dataset expensive to view. They run client-side on metadata fileglancer already parses, so there are no extra requests on the common path.
Motivation is the production incident that occurred on Friday, August 14th. The IO thread pools were starved by a handful of viewing sessions on a 91 GB image published with a single resolution level. Nothing here changes server behaviour — the aim is to tell data creators before they publish.
The checks
Only one resolution level. A multiscales group declaring a single dataset provides no downsampled data, so every zoom reads full resolution. Keyed on the dataset count rather than the number of shapes, because a plain zarr array also has exactly one shape while claiming nothing about being multiscale — an early version keyed on shapes and fired on healthy pyramid levels that users had browsed into. Gated at 1 GB.
Chunks too large. Chunks past the browser's cache entry limit are re-fetched on every access. Sizes come from shape and dtype, so they are logical: exact for a raw array, an upper bound for a compressed one. Limit is 10 MB with no compressor and 32 MB with one, since the compression ratio is a property of the pixels and is not in the metadata. Unknown or unfetched codec info counts as compressed, so uncertainty costs a missed warning rather than a false one.
Codec classification reuses
capability-manifest'sclassifyCodec, plus a walk over the pipeline it does not do itself: a sharded v3 array lists only the structuralsharding_indexedcodec at the top level and carries the real compressor in its configuration, so a flat scan calls every sharded array uncompressed.Axis order. OME-Zarr requires
t, c, z, y, x, and many tools take the last two axes to be the image plane. A dataset orderedx, y, z, crenders as a cross-section in those tools while Neuroglancer, which reads axis names, looks fine — a confusing failure worth naming. Datasets with custom axis names are left alone.Screenshot
Notes for review
ome-zarr.jsalready refuses when the lowest level exceeds its own limit. Modelling its behaviour a second time proved unreliable: it renders from the level nearest the requested size rather than the smallest, and reads a single plane rather than a whole level.zarrQueries.tsnow populates codec info for plain arrays too — the v3 branch reads it from thezarr.jsonit already parsed, the v2 branch fetches.zarrayas the group path does.__tests__/mocks/omezarrHelper.tsis a whole-module mock, so it needs every new export listed or the component tests fail.Testing
21 unit tests covering both compression paths, nested sharding codecs, unknown codecs, each axis-order case, and the plain-array cases that earlier versions got wrong. Full frontend suite passes (321). Verified end to end against a pair of synthetic fixtures that reproduce the original dataset's defects and the corrected version — the broken one returns all three warnings, the fixed one returns none.
@StephanPreibisch @JaneliaSciComp/fileglancer @dchen116