FairStream

FairStream is a research-oriented streaming fairness-aware AutoML system. It combines online model selection, streaming fairness measurement, a dynamic fairness envelope, drift monitoring, explainability, topology-based bias diagnostics, and an interactive dashboard for running experiments on tabular data streams.

What the project does

The system is built for settings where:

• data arrives as a stream rather than a single static batch
• predictive quality and fairness both need to be monitored continuously
• fairness may drift over time even when raw accuracy looks stable
• the acceptable fairness boundary should adapt to evidence quality and recent drift

At a high level, the runtime loop is:

1. ingest a stream window
2. measure fairness on the window
3. smooth fairness and estimate drift/uncertainty
4. update a dynamic fairness envelope
5. evaluate candidate models against performance and fairness constraints
6. let the OSMAC-style optimizer propose better configurations
7. expose live metrics, diagnostics, and benchmark views through the API and Streamlit UI

Core ideas

The current design centers on three pieces:

• SFM (Streaming Fairness Measurement): produces fm_smooth, fm_drift, and fm_uncertainty
• DFE (Dynamic Fairness Envelope): maintains a time-varying allowable fairness region, typically as an upper bound on a fairness gap
• constrained online optimization: the optimizer ranks candidates using predictive utility, feasibility probability, fairness risk, and support/reliability penalties

Main capabilities

• streaming fairness-aware AutoML over tabular datasets
• rolling-window fairness monitoring and drift detection
• dynamic fairness envelope updates from SFM telemetry
• online candidate evaluation and OSMAC-style constrained optimization
• fairness scope management for protected attributes and intersections
• topology-aware bias analysis
• explainability and LLM-backed narrative auditing
• optional CO2/emissions tracking through CodeCarbon
• Streamlit dashboard for uploads, live monitoring, benchmark views, and debugging
• built-in benchmark helpers for Adult, German Credit, and COMPAS-style datasets

Repository layout

FairStream/
|- backend/
|  |- api.py
|  |- candidate_evaluator.py
|  |- dynamic_fairness_envelope.py
|  |- fairness_engine.py
|  |- fairness_scope_manager.py
|  |- osmac_optimizer.py
|  |- topological_bias_engine.py
|- Dataset
|- experiments/
|  |- synthetic_streams.py
|- forntend/
|  |- app.py
`- README.md

Notes:

• the UI folder is named forntend in the repository; the README keeps that spelling when referencing paths or commands
• Dataset/ contains local benchmark datasets and stream variants
• experiments/reports/ contains generated HTML and JSON report artifacts from prior runs

Architecture

Backend services

• backend/api.py: Flask API, experiment registry, polling loop, and payload assembly
• backend/candidate_evaluator.py: window-level candidate scoring for accuracy, fairness, feasibility, and constrained utility
• backend/dynamic_fairness_envelope.py: adaptive upper/lower fairness bounds driven by SFM telemetry
• backend/osmac_optimizer.py: streaming constrained optimizer with feasibility-first and CEI-style acquisition
• backend/fairness_engine.py: fairness metric computation and fairness snapshots
• backend/fairness_scope_manager.py: attribute/intersection scope selection
• backend/topological_bias_engine.py: topology and cluster-based bias diagnostics
• backend/explainability_engine.py: model and state explanations
• backend/llm_auditor.py: narrative audit generation over Ollama-compatible LLM endpoints
• backend/offline_benchmarks: reference search baselines and dataset loaders

Frontend

• forntend/app.py is a Streamlit application
• default backend target: http://127.0.0.1:5000
• the UI exposes dataset upload, fairness scope controls, live metrics, optimizer views, fairness diagnostics, benchmark views, comparison views, LLM output, and raw payload debugging

Research workflow

The intended research path in the current code is:

1. upload or stream a dataset
2. choose the target and protected attribute(s)
3. configure fairness scope, window size, and optimizer behavior
4. run the streaming experiment
5. monitor fairness gaps, drift, uncertainty, envelope pressure, and model accuracy
6. compare live optimization with reference benchmark behavior
7. inspect generated figures and reports

Data assumptions

The backend expects tabular data represented as records. In the Streamlit UI this is uploaded as a CSV and then sent to the API as JSON rows.

Important assumptions:

• the target column must be binary or coercible to binary
• at least one sensitive attribute should be supplied for fairness tracking
• benchmark helpers assume binary sensitive grouping for the current streaming FairBO-style reference flow
• some built-in loaders normalize common benchmark conventions automatically

Examples handled in code:

• Adult: income target with sex as a sensitive attribute
• German Credit: one of class, credit_risk, risk, target, or label, with gender/sex information
• COMPAS: two_year_recid or equivalent, with race/ethnicity information

Supported benchmark datasets

The benchmark loader registry in backend/offline_benchmarks/datasets.py currently exposes:

• adult
• german
• compas

The repository also contains other local CSVs and generated stream variants under Dataset/, which can be uploaded directly through the UI even when they are not formal registry entries.

Setup

Prerequisites

Recommended:

• Python 3.10
• pip
• a virtual environment

Optional:

• Ollama for local LLM auditing
• CodeCarbon if you want emissions tracking

Install dependencies

Use the included requirements.txt:

pip install -r requirements.txt

The file includes the main runtime, plotting, benchmark, and optional integration packages already used by the repository.

Optional extras beyond that:

pip install jupyter

If you plan to use the LLM auditor, run an Ollama server separately and make sure the configured model is available.

Running the project

Open two terminals from the repository root.

1. Start the backend

python backend/api.py

This starts the Flask server on http://127.0.0.1:5000.

2. Start the frontend

streamlit run forntend/app.py

By default, the Streamlit app points at http://127.0.0.1:5000.

API overview

The backend exposes these main routes:

• GET /health
• GET /experiments
• GET /experiment_meta/<exp_id>
• POST /start_experiment
• GET /poll_experiment/<exp_id>?chunk_size=N
• GET /finalize_experiment/<exp_id>
• POST /reset_experiment/<exp_id>
• POST /explain_mapper_node/<exp_id>

Typical API lifecycle

1. call POST /start_experiment
2. store the returned experiment_id
3. repeatedly call GET /poll_experiment/<exp_id>
4. optionally inspect GET /experiment_meta/<exp_id>
5. finalize with GET /finalize_experiment/<exp_id>

Minimal start_experiment payload shape

The Streamlit app sends a payload like this:

{
  "dataset": [
    {"feature_1": 0.12, "feature_2": "A", "target": 1, "sex": "male"}
  ],
  "dataset_name": "adult",
  "target": "target",
  "sensitive_attr": ["sex"],
  "window_size": 500,
  "sample_budget": 5000
}

Optional payload sections in the current code include configuration for:

• fairness scope
• optimizer runtime
• candidate evaluator behavior
• static or streaming benchmark settings
• mapper behavior

Using the UI

The Streamlit app is the easiest way to work with the project.

Typical flow:

1. upload a CSV
2. pick the target column
3. select one or more sensitive attributes
4. choose the rolling window size and sample budget
5. configure fairness scope and benchmark options if needed
6. start the experiment
7. monitor live plots and diagnostics while polling runs

The UI includes:

• dataset preview and schema inspection
• target mapping preview
• live progress tracking
• accuracy and fairness trend plots
• SFM and DFE views
• optimizer convergence and candidate history views
• benchmark and method comparison views
• LLM auditor output
• raw payload debugging

Method semantics in the current code

The current implementation and accompanying design doc align around these choices:

• primary fairness control is a one-sided upper bound on a fairness gap
• the dynamic fairness envelope tightens when drift, uncertainty, or violations increase
• relaxation is gated by support and confidence
• candidate ranking uses constrained utility rather than raw predictive score alone
• feasible recent candidates are preferred for incumbent selection
• streaming fairness telemetry is tracked explicitly through fm_raw, fm_smooth, fm_drift, and fm_uncertainty

Benchmarks and reports

The streaming reference benchmarking integrated into the experiment flow

• OnlineFairBO (baseline)

The repository also includes a critical-difference benchmark figure generated with the autorank library:

Other files under forntend/generated_figures/ are kept local and ignored by Git.

Generated artifacts commonly produced by local runs include:

• HTML experiment reports under experiments/reports/
• fairness metric figures under forntend/fairness_metric_figures/
• generated comparison figures under forntend/generated_figures/
• validation landscape figures under forntend/validation accuracy/

Configuration through environment variables

The backend reads several FAIRSTREAM_* environment variables. The most important ones currently include:

• FAIRSTREAM_DEFAULT_CHUNK
• FAIRSTREAM_EXPERIMENT_TTL
• FAIRSTREAM_MAX_HISTORY
• FAIRSTREAM_MAX_DATASET_ROWS
• FAIRSTREAM_MAX_DATASET_COLS
• FAIRSTREAM_OSMAC_MIN_WINDOW
• FAIRSTREAM_OSMAC_MAX_WINDOW
• FAIRSTREAM_OPTIMIZER_CANDIDATE_HISTORY
• FAIRSTREAM_STATIC_BENCHMARK_BUDGET
• FAIRSTREAM_STATIC_BENCHMARK_POOL
• FAIRSTREAM_OLLAMA_MODEL
• FAIRSTREAM_OLLAMA_URL
• FAIRSTREAM_OLLAMA_TIMEOUT
• FAIRSTREAM_OLLAMA_MAX_RETRIES

These are defined in backend/api.py.

Optional integrations

Ollama / LLM auditing

If Ollama is running, the backend can request narrative explanations and audit text through the LLM auditor layer.

Relevant defaults:

• model: llama3:latest
• endpoint: http://localhost:11434/api/generate

CodeCarbon

If codecarbon is installed, the backend reports emissions-tracking availability through /health and experiment startup responses.

Current limitations

• there is no packaged project configuration such as pyproject.toml, and requirements.txt is still a hand-curated environment file rather than a fully reproducible lock
• the repository contains a mix of research code, dashboard code, and generated artifacts rather than a packaged library structure
• some naming is historical or inconsistent, such as forntend/
• several dataset and figure directories contain generated outputs alongside source scripts
• the codebase is optimized for experimentation and iteration, not yet for distribution as a polished package

Troubleshooting

Backend imports fail

Run the backend from the repository root:

python backend/api.py

Streamlit cannot reach the API

• make sure the Flask backend is running on port 5000
• verify http://127.0.0.1:5000/health
• check the backend URL in the Streamlit sidebar/session settings

LLM explanations are missing

• confirm Ollama is running
• confirm the configured model exists
• verify the backend can reach FAIRSTREAM_OLLAMA_URL

Benchmark loaders reject a dataset

Check that:

• the target is binary or coercible to binary
• the sensitive attribute contains at least two groups
• the expected benchmark columns are present for Adult, German Credit, or COMPAS normalization

Recommended next improvements

• add a pyproject.toml and/or a fully reproducible lock file
• add reproducible experiment configs
• separate generated artifacts from source directories
• standardize naming, especially forntend/
• add automated tests around API payload contracts and benchmark loaders

License and usage

No license file is currently present in the repository. If this project is going to be shared beyond local research use, add an explicit license before distribution.