Add particlefile_to_v3_zarr() helper by VeckoTheGecko · Pull Request #2811 · Parcels-code/Parcels

VeckoTheGecko · 2026-08-12T07:20:49Z

Description

Adds particlefile_to_v3_zarr helper, tests, and strategies for property based testing.

Note that this function is eager. This is noted in the docstring for the function.

Checklist

Closes Parquet to CF-trajectory compliant Zarr conversion tooling #2600
Tests added
This PR targets the correct branch (main for normal development, v3-support for v3 support)

AI Disclosure

This PR contains AI-generated content.
- I have tested any AI-generated content in my PR.
- I take responsibility for any AI-generated content in my PR.
- Describe how you used it: Familiarity with polars API.

Parquet metadata is parsed as bytes strings. This decodes them.

for more information, see https://pre-commit.ci

VeckoTheGecko · 2026-08-12T08:07:45Z

I thought that this would be useful for the future as well

VeckoTheGecko · 2026-08-12T08:08:07Z

not sure about the name of this file (also on the name of test_v3.py)

How about compatibility_v3.py? Or compat_v3.py? Or `regression_v3.py?

compat_v3 sounds good

VeckoTheGecko · 2026-08-12T08:09:14Z

+    df = pl.read_parquet(from_parquet)
+    table = pq.read_table(from_parquet)
+
+    # TODO: Check for available memory here and fail as a safeguard?


This is intentionally left here. I think it would be good to discuss (or leave for a future PR)

VeckoTheGecko · 2026-08-12T08:10:35Z

+    return ret
+
+
+def particlefile_to_v3_zarr(from_parquet: str | os.PathLike, to_zarr: str | os.PathLike) -> None:


I think this API (i.e., from file-> file) is significantly more stable and robust than trying to take in/return in-memory objects like dataframes or Xarray Datasets.

VeckoTheGecko · 2026-08-12T08:11:19Z

+    for var in ["lat", "lon", "depth", "time"]:
+        assert var in ds.variables
+
+    assert set(ds.dims) == {"obs", "trajectory"}


I think that's it in terms of checks that we need to do? Do you have anything else in mind @erikvansebille ?

VeckoTheGecko · 2026-08-12T08:12:19Z

+def test_particlefile_to_v3_zarr(df):
+    with tempfile.TemporaryDirectory() as tmpdir:


Can't do

@given(df=pst.particlefile_output()) def test_particlefile_to_v3_zarr(df, tmp_zarr, tmp_parquet)

since hypothesis doesn't allow mixing of strategies and Pytest fixtures

VeckoTheGecko · 2026-08-12T08:15:40Z

+    Notes
+    -----
+    This is not a lazy operation — the entire parquet file is read into memory
+    and pivoted before writing to zarr. For large particle files this may
+    require significant memory. Performance improvements are welcome via PRs.
+    """


Notes

This is not a lazy operation — the entire parquet file is read into memory
and pivoted before writing to zarr. For large particle files this may
require significant memory. Performance improvements are welcome via PRs.

I think that this sort of functionality (i.e., pivoting dataframes to zarr CF convention output) can live outside of Parcels since its also relevant to other tools (e..g, TRACMASS).

@oj-tooth have you guys done any work around pivoting (larger than in-memory) tabular data into Lagrangian Zarr datasets?

Maybe its something that we can collab on across-teams

erikvansebille

Looks good; a few comments/questions

erikvansebille · 2026-08-12T09:33:47Z

Also remove the TODO about the conversion function higher up in the notebook?

And remove "do" from the "We do also have some tooling ..."?

And filter out the ZarrUserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.?

erikvansebille · 2026-08-12T09:34:57Z

+
+
+def _generate_dummy_data(particle: ParticleClass, nparticles=10, nobs=10) -> pd.DataFrame:
+    """Build a pandera DataFrameSchema from a ParticleClass.


Type-o? Auto-correct? Or intentional?

Suggested change

"""Build a pandera DataFrameSchema from a ParticleClass.

"""Build a pandas DataFrameSchema from a ParticleClass.

This was left over from a different implementation I was exploring (using Panderas hypothesis strategies), though it wasn't as flexible as I was needing.

Cleaned up now

erikvansebille · 2026-08-12T09:35:17Z

+
+    Only variables with ``to_write=True`` are included in the schema.
+    Each column is typed with the variable's numpy dtype and carries the
+    variable's ``attrs`` as pandera column-level metadata.


Suggested change

variable's ``attrs`` as pandera column-level metadata.

variable's ``attrs`` as pandas column-level metadata.

erikvansebille · 2026-08-12T09:37:04Z

 from parcels._xarray import open_raw_zarr
 from parcels._core.particleset import ParticleSet
 from parcels._core.particlefile import ParticleFile, read_particlefile
+from ._v3 import particlefile_to_v3_zarr


Are we at the risk of confusing users which 'v3' we mean here? Zarr v3? Parcels v3? Both? Should we clarify?

erikvansebille · 2026-08-12T09:39:37Z

+
+    Notes
+    -----
+    This is not a lazy operation — the entire parquet file is read into memory


Would a relatively simple improvement be to do the loading/pivoting/writing on a per-variable basis? So that only one variable is kept in memory at a time? Or is this not worth the extra code/effort for now?

erikvansebille · 2026-08-12T15:26:57Z

+
+    Reads the parquet file, renames columns to v3 conventions
+    (``particle_id`` -> ``trajectory``, ``t`` -> ``time``, ``x`` -> ``lon``,
+    ``y`` -> ``lat``, ``z`` -> ``depth``), and reshapes the data into a 2D


Actually, the output was already z in v3

Suggested change

``y`` -> ``lat``, ``z`` -> ``depth``), and reshapes the data into a 2D

``y`` -> ``lat``), and reshapes the data into a 2D

erikvansebille · 2026-08-12T15:27:16Z

+    # TODO: Check for available memory here and fail as a safeguard?
+
+    # Rename columns to v3 conventions
+    rename_map = {"particle_id": "trajectory", "t": "time", "x": "lon", "y": "lat", "z": "depth"}


Suggested change

rename_map = {"particle_id": "trajectory", "t": "time", "x": "lon", "y": "lat", "z": "depth"}

rename_map = {"particle_id": "trajectory", "t": "time", "x": "lon", "y": "lat"}

erikvansebille · 2026-08-12T15:52:43Z

I was trying to implement the particlefile_to_v3_zarr() in #2812 (so that I could much easier compare the old v3 and the new v4 output), but ran into an issue when ow all trajectories have the same length (e.g. because some particles are deleted). This leads to a ragged pivot table and then all NaNs in the zarr file. Could you explore a fix, @VeckoTheGecko? Ragged output is quite normal in Parcels

VeckoTheGecko added 4 commits August 11, 2026 11:32

Add parcels/_strategies/particle.py

5f8ddde

Add particlefile_to_v3_zarr alongside particlefile strategy and test

0a87552

Fix serialization of attrs

c039fca

Parquet metadata is parsed as bytes strings. This decodes them.

Update documentation

06f1d58

VeckoTheGecko force-pushed the push-unusxmtrrptn branch from 5d843b7 to 06f1d58 Compare August 12, 2026 08:06

[pre-commit.ci] auto fixes from pre-commit.com hooks

f8e4f0a

for more information, see https://pre-commit.ci

VeckoTheGecko commented Aug 12, 2026

View reviewed changes

VeckoTheGecko requested a review from erikvansebille August 12, 2026 08:12

VeckoTheGecko commented Aug 12, 2026

View reviewed changes

Fix typing

19c03e3

erikvansebille reviewed Aug 12, 2026

View reviewed changes

Comment thread src/parcels/_v3.py

erikvansebille reviewed Aug 12, 2026

View reviewed changes

VeckoTheGecko added 4 commits August 13, 2026 09:29

Update strategy to use bytesIO

d51b592

Review feedback

6bc7636

Drop "z"->"depth" renaming

d9215ed

Set obs as coord

b0118bc

		return ret


		def particlefile_to_v3_zarr(from_parquet: str \| os.PathLike, to_zarr: str \| os.PathLike) -> None:

		def test_particlefile_to_v3_zarr(df):
		with tempfile.TemporaryDirectory() as tmpdir:



		def _generate_dummy_data(particle: ParticleClass, nparticles=10, nobs=10) -> pd.DataFrame:
		"""Build a pandera DataFrameSchema from a ParticleClass.

	"""Build a pandera DataFrameSchema from a ParticleClass.
	"""Build a pandas DataFrameSchema from a ParticleClass.

	variable's ``attrs`` as pandera column-level metadata.
	variable's ``attrs`` as pandas column-level metadata.

	``y`` -> ``lat``, ``z`` -> ``depth``), and reshapes the data into a 2D
	``y`` -> ``lat``), and reshapes the data into a 2D

	rename_map = {"particle_id": "trajectory", "t": "time", "x": "lon", "y": "lat", "z": "depth"}
	rename_map = {"particle_id": "trajectory", "t": "time", "x": "lon", "y": "lat"}

Uh oh!

Conversation

VeckoTheGecko commented Aug 12, 2026

Description

Checklist

AI Disclosure

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erikvansebille Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erikvansebille left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VeckoTheGecko Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

erikvansebille Aug 12, 2026 •

edited

Loading

VeckoTheGecko Aug 13, 2026 •

edited

Loading