PointZip

Pointwise error-bounded compression for NumPy float32 and float64 arrays.

PointZip supports Linux on x86-64 and ARM64. The commands below use Conda so that Python, HDF5, NetCDF and CDO share the same libraries.

Install

git clone https://github.com/spcl/PointZip.git
cd PointZip

conda create -n pointzip -c conda-forge \
  python=3.11 numpy pip 'cmake<4' make c-compiler hdf5 h5py netcdf4 cdo
conda activate pointzip

python -m pip install --no-deps .
cmake -S . -B build-io -DPOINTZIP_BUILD_HDF5=ON \
  -DHDF5_ROOT="$CONDA_PREFIX" -DCMAKE_INSTALL_PREFIX="$CONDA_PREFIX" \
  -DCMAKE_INSTALL_LIBDIR=lib
cmake --build build-io --parallel 4
cmake --install build-io
export HDF5_PLUGIN_PATH="$CONDA_PREFIX/lib/pointzip"

Set HDF5_PLUGIN_PATH again after opening a new shell.

Run the self-checks:

python examples/basic.py
python examples/search.py
python examples/files.py
python examples/netcdf.py

Each script creates its own small data and ends with a success message.

If only the Python and raw-binary interfaces are needed, a C compiler and CMake 3.20+ are sufficient: create a Python 3.10+ environment and run python -m pip install ..

Python API

import numpy as np
import pointzip

x = np.random.default_rng(0).normal(size=(4, 64, 96)).astype(np.float32)
bound = np.full_like(x, 1e-3)

blob = pointzip.compress(x, bound, cratio=25)
y = pointzip.decompress(blob)

# Search returns the selected cratio and the compressed bytes.
best_cratio, blob = pointzip.search(x, bound)

# Preliminary parallel interface for independent chunks.
blobs = pointzip.compress_many([x, x], [bound, bound],
                               cratio=[20, 40], nthreads=2)

The input shape is (H, W) or (..., H, W). A bound is either a nonnegative scalar or an array with the same shape as the input. Data NaNs are preserved; bound NaNs mean unconstrained points. cratio controls JPEG2000, not the final PointZip compression ratio.

See file interfaces for raw binary, HDF5, NetCDF and CDO. See the C/C++ API for native use.