RECAP

Reproducible provenance for high-throughput experiments

RECAP (Reproducible Experiment Capture and Provenance) is a Python framework for organizing the metadata, resources, and workflows generated by large-scale experiments.

High-throughput experiments produce more than data files. They produce samples, plates, detector outputs, processing results, instrument configurations, workflow parameters, and relationships between all of them. RECAP records those relationships so results remain traceable after the experiment is complete.

What RECAP provides

RECAP helps you:

• Define reusable templates for samples, plates, files, instruments, and other experimental resources.
• Record hierarchical resources such as plates containing wells or collections containing samples.
• Define repeatable workflows as process templates with ordered steps, parameters, and resource slots.
• Record each execution as a process run with its actual inputs, outputs, and parameter values.
• Organize experiments with hierarchical namespaces.
• Query resources, process runs, properties, parameters, and provenance relationships.
• Preserve lineage when immutable resources need to be modified.
• Work with local SQLite databases or authenticated remote RECAP servers.

How the data is organized

RECAP separates reusable definitions from experimental instances:

ResourceTemplate
    └── Resource

ProcessTemplate
    └── ProcessRun
            ├── assigned Resources
            ├── ordered Steps
            └── recorded Parameters

Resources and process runs form a provenance graph:

Sample
  └── Sample Preparation
        └── Prepared Sample
              └── Data Collection
                    └── Raw Data File
                          └── Data Processing
                                └── Processed Result

This structure lets you answer questions such as:

• Which sample and preparation conditions produced this result?
• Which resources were used by this process run?
• Which parameters controlled a particular analysis?
• Which results depend on a specific input?
• What metadata belongs to all experiments in a namespace?

Why this is useful for high-throughput science

High-throughput workflows often repeat the same operations across many samples, plates, visits, or processing batches. Manually maintaining relationships in notebooks, filenames, and spreadsheets makes provenance difficult to query and easy to lose. RECAP turns those relationships into structured records. Templates capture the stable parts of a workflow, while process runs record what happened for one specific experiment. Resource hierarchies preserve relationships within physical and digital artifacts, and namespaces provide organization and access boundaries as projects grow. The result is a provenance layer that can support automation, quality control, reproducibility, and downstream analysis without requiring every application to invent its own data model.

Minimal example

from recap.client import RecapClient

with RecapClient.from_sqlite("experiment.db") as client:
    client.create_namespace("beamline")
    client.create_namespace("beamline/amx", metadata={"beamline": "amx"})
    namespace = client.namespace("beamline/amx")

    with namespace.build_resource_template(
        name="Sample Plate",
        type_names=["container", "plate"],
    ) as template:
        template.add_properties({
            "dimensions": [
                {"name": "rows", "type": "int", "default": 8},
                {"name": "columns", "type": "int", "default": 12},
            ]
        })

    with namespace.build_resource(
        name="Plate 001",
        template_name="Sample Plate",
    ) as plate_builder:
        plate = plate_builder.resource

The same namespace-scoped client can define process templates, create process runs, assign resources, record parameters, and query the resulting provenance graph.

Local and remote use

For local workflows, RECAP manages a SQLite database:

client = RecapClient.from_sqlite("experiment.db")

For shared or service-based deployments, clients connect to an authenticated RECAP server:

client = RecapClient.from_url(
    "https://recap.example.org",
    api_key="your-api-key",
)

Remote clients use authenticated REST for queries, reads, creates, updates, and resource copies. Applications do not need direct access to the server's database filesystem.

Remote queries use ordinary JSON request and result envelopes. Local and remote clients share the same QueryDSL, canonical entity identity, and load-aware model behavior. Query results default to full models with relationships unloaded; explicit include(...) or load="eager" controls hydration. Builders collect drafts and submit one aggregate command at save(), locally or through REST.

Query results can be passed to query.export(format, destination) through the registered exporter extension point. RECAP does not prescribe a built-in export format.

What RECAP does not do

RECAP records experimental data and provenance. It is not:

• An electronic lab notebook.
• A laboratory inventory management system.
• An instrument-control system.
• A scientific analysis or computation engine.

Applications can build those capabilities on top of RECAP's provenance model.

Learn more

• Getting started
• Complete provenance workflow
• Quick start: create and store data locally
• Model a process workflow
• How-to guides
• Reference
• Explanations
• Tutorials

Install

pip install pyrecap

Install server support with:

pip install "pyrecap[server]"

RECAP is released under the 3-clause BSD license.