OmniBioAI Dev Hub — RAG V6

Production-grade Retrieval-Augmented Generation (RAG) system powering the OmniBioAI ecosystem documentation, architecture search, workflow discovery, and developer assistant APIs.

Features

FAISS-native vector search (IndexFlatIP, 768-dim)
Incremental per-repo indexing with hash-based dedup
Ollama local embeddings (nomic-embed-text) + local LLM inference (llama3)
FastAPI API server
Real token-level SSE streaming via Ollama stream: true
Chunk-level document retrieval with source attribution
Repository-wide multi-project indexing (19 repos)
Fully local execution — no OpenAI dependency
Production-safe embedding normalization
V6 dimension consistency enforcement

Architecture

Repositories  (REPO_BASE/omnibioai-*)
     ↓
Document Loader  (ingestion/doc_loader.py)
     ↓
Chunker  (processing/chunker.py, markdown-structure-aware, header/paragraph/word-boundary splitting, 2000-char max)
     ↓
Ollama Embeddings  (nomic-embed-text, 768-d, normalized)
     ↓
FAISS Vector Index  (IndexFlatIP, data/faiss_index/)
     ↓
RAG Engine  (rag/engine.py)
     ↓
FastAPI API  (api/main.py + api/routes/rag.py)
     ↓
LLM Answer Generation  (llama3 via Ollama, blocking or token-streamed)

V6 Major Improvements

FAISS Native Retrieval

Previous versions used brute-force cosine scanning across vectors.

V6 uses:

faiss.IndexFlatIP

Benefits:

10–50x faster retrieval
scalable search
lower latency
future ANN support

Embedding Consistency Fix

A major issue in previous builds was embedding mismatch.

Old Problem

Stage	Model	Dimension
Indexing	all-MiniLM-L6-v2	384
Querying	nomic-embed-text	768

This caused FAISS assertion failures:

AssertionError: d == self.d

V6 Fix

Both ingestion (scripts/build_index.py) and retrieval (rag/engine.py) now call the same ollama_embed("nomic-embed-text") function — the single source of truth.

embeddings/embedder.py (sentence-transformers, 384-dim) is retained for test coverage only and is not on any live request path.

Repository Structure

omnibioai-dev-hub/
│
├── api/
│   ├── main.py              # FastAPI app, startup, /status, /health
│   ├── auth.py               # JWT dependency for /rag/*, gated by AUTH_ENABLED
│   │                          # — see "Authentication" below
│   ├── middleware/
│   │   └── trace.py           # TraceContext — written but not wired into
│   │                          # main.py or tested; not on any live request path
│   └── routes/
│       └── rag.py           # /rag/query and /rag/stream endpoints (auth-gated)
│
├── rag/
│   ├── engine.py            # RAGEngine: retrieve, build_context, answer, stream_llm
│   ├── control_plane.py     # Singleton lifecycle manager
│   ├── query_router.py      # RAGQueryRouterV4 — superseded agentic router,
│   ├── tool_executor.py     # ToolExecutorV4 — from a prior "V4" architecture,
│   └── memory_store.py      # MemoryStoreV4 — tests only (see tests/test_rag_*),
│                              # not on any live request path in the current V6 engine
│
├── index/
│   ├── vector_store.py      # FAISS wrapper (add, search, save, load)
│   ├── graph_store.py       # In-memory knowledge graph (BFS expansion)
│   └── plugin_index.py      # Plugin doc registry
│
├── embeddings/
│   └── embedder.py          # SentenceTransformer wrapper (tests only, not live)
│
├── retrieval/
│   └── retriever.py         # Retriever class (tests only, not live)
│
├── ingestion/
│   └── doc_loader.py        # Markdown document loader with SKIP_DIRS/SKIP_PATH_SEGMENTS
│
├── processing/
│   └── chunker.py           # 500-word / 2000-char chunker
│
├── scripts/
│   ├── build_index.py       # Index builder entry point
│   ├── ingest.py             # Standalone ingestion helper
│   ├── run_eval.py           # Recall@K eval harness (tests/eval/)
│   └── check_and_reindex.sh  # Rebuilds the index on new Studio releases (hourly cron)
│
├── data/
│   └── faiss_index/         # index.faiss + metadata.pkl (gitignored)
│
├── configs/
│   └── repos.yaml           # Repo list — actual source of truth build_index.py reads
│
├── omnibioai-dev-hub-ui/     # Dev Hub UI (React + TypeScript) — see "Frontend" below
│
└── .env.example             # Environment variable template

Frontend

omnibioai-dev-hub-ui/ is a React + TypeScript UI (Vite, served at /_svc/devhub), built and shipped alongside the API — not a separate repo. The Dockerfile's ui-builder stage builds it and copies the static output into the same image the FastAPI backend runs in (EXPOSE 8082 5173 — API and UI dev server ports both exposed).

cd omnibioai-dev-hub-ui
npm install
npm run dev       # Vite dev server on :5173, proxies /api, /rag, /health to :8082
npm run build      # production build

The dev server's proxy (vite.config.ts) rewrites /api/* to the FastAPI backend at http://127.0.0.1:8082 (stripping the /api prefix) and forwards /rag/* and /health there directly — so AUTH_ENABLED on the backend (see "Authentication" below) applies to UI-issued requests the same as any other client.

Requirements

Python

Recommended:

Python 3.11 or 3.12

Note: Python 3.13 removes numpy.distutils, breaking faiss-cpu. Use 3.11 or 3.12. In Docker the provided image uses Python 3.12.

Ollama

Install:

curl -fsSL https://ollama.com/install.sh | sh

Required Ollama Models

Embedding Model

ollama pull nomic-embed-text

Generation Model

The system uses llama3 by default:

ollama pull llama3

The model name is read from configs/index_config.yaml's llm_model key (see Configuration below) — llama3 is just that file's current value, and also what rag/engine.py falls back to if the config is missing, empty, or unset.

Optional alternatives — pull the model, then set llm_model: in configs/index_config.yaml accordingly:

ollama pull mistral
ollama pull deepseek-coder

Installation

Create Environment

conda create -n omnibioai-dev-hub python=3.12 -y
conda activate omnibioai-dev-hub

Install Dependencies

pip install -r requirements.txt

sentence-transformers is required for cross-encoder reranking (rerank=True in retrieve()). It is also used by the test suite. If reranking is not needed you can omit it — the engine degrades gracefully to FAISS order when the model is unavailable.

requirements.txt alone is enough to run the app — it does not install anything needed to run the test suite.

Running Tests

If you're contributing and want to run pytest locally, also install the dev/test dependencies:

pip install -r requirements.txt -r requirements-dev.txt
pytest

CI (.github/workflows/ci.yml) runs this same install + ruff check . + pytest combination in a Lint & Test job on every push/PR, alongside a separate Frontend Build & Test job (npm ci, npm run build, npm test in omnibioai-dev-hub-ui/) — both must pass before the tag-triggered Docker build job runs.

Configuration

REPO_BASE

REPO_BASE is the only required configuration. It must point to the directory that contains the omnibioai-* repos as immediate children.

export REPO_BASE=/home/manish/Desktop/machine

The indexer exits immediately with a clear error if no repos are found under REPO_BASE:

❌  No repos found under REPO_BASE='/some/wrong/path'
    Set REPO_BASE to the directory that contains the omnibioai-* repos.
    Example:  export REPO_BASE=/home/manish/Desktop/machine
    Docker:   -e REPO_BASE=/repos  (with repos volume mounted at /repos)

In Docker the image sets ENV REPO_BASE=/repos automatically — no action needed.

Copy .env.example to .env for local development:

cp .env.example .env
# edit REPO_BASE as needed

configs/repos.yaml

Lists the repo names scripts/build_index.py indexes (paths are built as ${REPO_BASE}/<name>, same as Supported Repositories below). Edit this file to add, remove, or reorder indexed repos without touching code.

If the file is missing, empty, fails to parse, or its repos: list is empty, the indexer falls back to the same 19-repo list hardcoded in build_index.py — so leaving it untouched changes nothing.

configs/index_config.yaml

Sets llm_model, the Ollama generation model name used by ollama_generate()'s default and RAGEngine.stream_llm() (see Generation Model above). Edit llm_model: here to switch models without touching code.

If the file is missing, empty, or doesn't set llm_model, both call sites fall back to the prior hardcoded value, "llama3" — so leaving it untouched changes nothing.

Build Index

Clean Existing Data

rm -rf data/faiss_index/*

Build V6 Index

python scripts/build_index.py

Expected output:

🚀 Incremental V6 Indexing Starting...
⚠️  2 repos not found, will be skipped: ['omnibioai-security-audit', 'omnibioai-hpc-policy-engine']
📄 Loaded N documents
...
💾 Index saved to .../data/faiss_index (10877 vectors)
✅ V6 Index Complete
{'too_short': 4, 'deduped': 31, 'embed_failed': 4, 'new': 10881, 'chunks_indexed': 10877}

How deduplication works

seen_hashes is reset per repo so cross-repo identical chunks each get their own index entry under their canonical source path. Within a single repo, duplicate chunks (e.g. shared boilerplate across plugin READMEs) are deduplicated.

Chunks shorter than 10 characters are discarded (MIN_CHUNK_CHARS = 10) to filter out low-information fragments (e.g. a lone header with no body).

Excluded paths

ingestion/doc_loader.py skips the following during the walk:

By directory name (SKIP_DIRS):

SKIP_DIRS = {".git", "__pycache__", "node_modules", ".venv", "venv", ".pytest_cache", "obsolete"}

Note: both .venv and bare venv are excluded. .pytest_cache is excluded because it contains auto-generated README.md stubs (present in 17 of the 19 repos) that would otherwise pollute the index with boilerplate.

By path segment (SKIP_PATH_SEGMENTS):

SKIP_PATH_SEGMENTS = {"work"}

This excludes omnibioai/work/ which contains UUID-named runtime copies of workflow bundle READMEs. Without this exclusion, those copies would claim index slots before the canonical omnibioai-workflow-bundles/ paths are processed, causing all 50 bundles to appear indexed under omnibioai/work/ instead.

Current Index Stats

As of 2026-06-14 (Phase 3 — metadata filtering + cross-encoder reranking). Not re-verified since — omnibioai-security-audit and omnibioai-hpc-policy-engine (both listed as "not present on disk" below and in this table's "17 of 19") now exist on disk, so a rebuild today would very likely index closer to 19 of 19 and produce different vector counts than shown here. Regenerate with python scripts/build_index.py for current numbers before relying on this table.

Metric	Value
Total vectors	10,877
Unique source files	~965
Repos indexed	17 of 19
Workflow bundles covered	50 / 50
Chunks filtered (too short)	4
Cross-repo deduped	31
Recall@5 (eval set)	96.8% (30/31)
Eval set size	31 queries

Chunk metadata fields: Each indexed chunk carries text, source, hash, repo (basename of the containing repository directory), and bundle (first subdirectory under the repo root, or None for root-level files). The repo and bundle fields enable post-filtered scoped queries that restrict FAISS candidates to a specific repository or workflow bundle.

Chunking strategy: Markdown-structure-aware — splits on H1/H2/H3 headers as natural section boundaries, never splits inside fenced code blocks, falls back to paragraph boundaries (\n\n) for long sections, and word-boundary splitting for oversized paragraphs. Each chunk is prefixed with its ancestor header breadcrumb (e.g. # ATACseq Pipeline > ## Parameters) so retrieval context carries section identity. Previous fixed 500-word window chunker produced 2,067 vectors; the markdown chunker produces 10,877 (5.3× more granular chunks).

Remaining failure: The one unscoped failure (metagenomics shotgun profiling vs. microbiome/kraken sub-workflows) passes with either bundle="metagenomics" scoped filtering or rerank=True cross-encoder reranking.

At the time these stats were captured, omnibioai-security-audit and omnibioai-hpc-policy-engine were not present on disk and the indexer skipped them with a warning. Both now exist — the "skipped" comments on those two entries in Supported Repositories below are stale; a rebuild today should index them like any other repo, gracefully, no code change needed (the indexer's on-disk check is dynamic, not a hardcoded list).

Note: data/faiss_index/ is excluded from git (see .gitignore). Regenerate with python scripts/build_index.py.

Run API Server

uvicorn api.main:app --host 0.0.0.0 --port 8082 --reload

In Docker the server starts automatically as PID 1.

Authentication

/rag/query and /rag/stream are gated by a JWT bearer-token dependency (api/auth.py), off by default and controlled by AUTH_ENABLED:

export AUTH_ENABLED=true
export JWT_SECRET=...   # HS256 secret, must match omnibioai-auth's SECRET_KEY

AUTH_ENABLED=false (the default for local/uvicorn --reload use) runs both endpoints open, no token required — this is what the curl examples below assume. omnibioai-studio's deployed compose config sets AUTH_ENABLED=true, so against a real deployment both endpoints require:

curl -X POST http://localhost:8082/rag/query \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <jwt-token>" \
  -d '{"query":"..."}'

A missing/invalid/expired token returns 401. This follows the same shared-secret pattern as omnibioai-model-registry's require_auth.

Fail-fast startup check: if AUTH_ENABLED=true and JWT_SECRET is unset or empty, the app now refuses to start — validate_auth_config() (api/auth.py) raises RuntimeError from the FastAPI startup event (api/main.py) before a single request can be served. Previously an unset JWT_SECRET silently fell through to validating tokens against an empty HMAC secret, which PyJWT accepts, so anyone could forge a valid token — this closed that bypass. If you deploy with AUTH_ENABLED=true, make sure JWT_SECRET is actually set or the container/process will not come up.

The Docker image enforces the equivalent check at the shell level before nginx or FastAPI start: if AUTH_ENABLED=true and JWT_SECRET is unset, the container exits immediately with an error instead of generating nginx's devhub.conf with an empty/placeholder value baked into the X-Devhub-Internal header (the internal UI-proxy auth path — see Dockerfile's CMD).

DEBUG_TRACEBACKS (default: unset/false) controls whether /rag/query's error responses include a full stack trace (trace field) — omitted by default, included only when explicitly set to true; keep it off in production.

Test Query API

curl -X POST http://localhost:8082/rag/query \
  -H "Content-Type: application/json" \
  -d '{"query":"What is workflow engine in OmniBioAI?"}'

Add -H "Authorization: Bearer <jwt-token>" if the server you're talking to has AUTH_ENABLED=true — see "Authentication" above.

Example response:

{
  "query": "What is workflow engine in OmniBioAI?",
  "answer": "According to the provided context...",
  "sources": [
    "/repos/omnibioai-workflow-bundles/README.md"
  ],
  "context": [
    {"score": 0.87, "text": "...", "source": "/repos/omnibioai-workflow-bundles/README.md"}
  ],
  "context_used": 5,
  "version": "v6-faiss",
  "api_version": "v6"
}

Scoped Queries

Both /rag/query and /rag/stream accept optional repo and bundle parameters to post-filter FAISS candidates to a specific repository or workflow bundle:

Parameter	Type	Description
`repo`	`string` (optional)	Restrict results to chunks from this repository (e.g. `"omnibioai-model-registry"`)
`bundle`	`string` (optional)	Restrict results to chunks from this workflow bundle subdirectory (e.g. `"metagenomics"`)

When both are provided, bundle takes priority. Omitting both parameters performs a global unscoped search.

curl -X POST http://localhost:8082/rag/query \
  -H "Content-Type: application/json" \
  -d '{"query": "shotgun profiling steps", "bundle": "metagenomics"}'

Streaming API

Endpoint:

POST /rag/stream

Body: {"query": "your question"} — also accepts optional repo and bundle parameters (same semantics as /rag/query).

How it works

The server retrieves the top-5 chunks from FAISS (same path as /rag/query).
It builds the prompt from the retrieved context.
It calls Ollama's /api/generate with "stream": true.

Ollama returns newline-delimited JSON (NDJSON), one object per token:

{"model":"llama3","response":"Based","done":false}
{"model":"llama3","response":" on","done":false}
...
{"model":"llama3","response":"","done":true}

Each token is immediately forwarded as an SSE event:

data: {"type": "token", "content": "Based"}

data: {"type": "token", "content": " on"}

...

data: {"type": "done"}

Client-side consumption

ragStream(
  "your query",
  (token) => appendToUI(token),      // called per token
  () => markComplete(),               // called on done
  (err) => showError(err)            // called on error
);

The UI client (src/api/client.ts) parses the data: envelope and dispatches type: "token" and type: "done" events.

Reranking

Cross-Encoder Reranking

The retrieval pipeline supports optional cross-encoder reranking:

docs = engine.retrieve(query, top_k=5, rerank=True)

Model: cross-encoder/ms-marco-MiniLM-L-6-v2 (via sentence-transformers)

How it works:

FAISS retrieves top_k × 3 candidates (15 for the default top_k=5).
The cross-encoder scores each (query, chunk) pair jointly, capturing fine-grained relevance that the bi-encoder embedding alone misses.
The top top_k by cross-encoder score are returned, each with a ce_score field attached.

Graceful degradation: If sentence-transformers is not installed or the model fails to load, rerank() logs a warning and falls back to the original FAISS order silently. No code change needed.

When to use: Unscoped queries where semantic collision is likely — for example, when a broad term (e.g. "Kraken2") matches many sub-workflow chunks and the pipeline overview you actually want is pushed below rank 5. For narrowly scoped queries (with repo or bundle), the candidate pool is already restricted and reranking adds latency with little benefit.

Supported Repositories

The indexer targets 19 repositories, sourced from configs/repos.yaml (falls back to this hardcoded list if that file is missing/empty). All paths are relative to REPO_BASE:

repos = [
    "omnibioai",                  # main platform repo
    "omnibioai-rag",
    "omnibioai-toolserver",
    "omnibioai-sdk",
    "omnibioai-workflow-bundles",  # 50 workflow bundle subdirectories
    "omnibioai-control-center",
    "omnibioai-lims",
    "omnibioai-model-registry",
    "omnibioai-dev-docker",
    "omnibioai-api-gateway",
    "omnibioai-docs",
    "omnibioai-studio",
    "omnibioai-auth",
    "omnibioai-tool-runtime",
    "omnibioai-iam-client",
    "omnibioai-policy-engine",
    "omnibioai-security-sdk",
    "omnibioai-security-audit",
    "omnibioai-hpc-policy-engine",
]

omnibioai-security-audit and omnibioai-hpc-policy-engine were absent on disk when Current Index Stats above was last captured (2026-06-14) — that table's "skipped with warning" framing reflects that snapshot, not necessarily today's state. Either way, a missing repo is skipped gracefully at build time and needs no code change to pick up once it appears — the indexer checks the filesystem at run time, not a hardcoded present/absent list.

V6 Retrieval Pipeline

Step 1 — Chunking

Documents are split using a markdown-structure-aware chunker (processing/chunker.py). Fenced code blocks (```) are never split. Text is divided at H1/H2/H3 header lines as natural section boundaries; sections longer than 2000 characters (MAX_CHARS) are split at paragraph boundaries (\n\n), and paragraphs still longer than 2000 characters fall back to word-boundary splitting. Each chunk is prefixed with its ancestor header chain (e.g. # H1 > ## H2) so retrieval context carries section identity. Chunks shorter than 10 characters are discarded.

Step 2 — Embedding

Each chunk is embedded using:

nomic-embed-text  (768-dim, L2-normalized)

Step 3 — FAISS Indexing

Vectors are stored in:

faiss.IndexFlatIP

Pre-normalized vectors make inner product equivalent to cosine similarity.

Step 4 — Query Embedding

User query is embedded using the same nomic-embed-text model at query time.

Step 5 — Vector Search

FAISS retrieves the top-5 nearest chunks (configurable via top_k).

Step 6 — Prompt Assembly

Retrieved chunks become the context block in a structured prompt.

Step 7 — LLM Generation

Prompt sent to local Ollama llama3 model, either blocking (/rag/query) or token-streamed (/rag/stream).

Performance

Before V6

brute-force cosine scan
slow retrieval
embedding dimension mismatch
unstable indexing
global dedup silencing canonical paths

After V6

FAISS-native inner product retrieval
stable 768-dim pipeline end-to-end
per-repo dedup with canonical path preservation
real SSE token streaming
local-only execution

Troubleshooting

REPO_BASE not set or wrong path

Symptom:

❌  No repos found under REPO_BASE='/repos'

Cause: REPO_BASE defaults to /home/manish/Desktop/machine locally and /repos in Docker. If neither is correct, set it explicitly:

export REPO_BASE=/path/to/parent/of/omnibioai-repos
python scripts/build_index.py

Index loads but all queries return empty results

Cause: The index on disk was built with a different embedding model or dimension. The loaded vectors won't match query vectors.

Fix: Delete and rebuild:

rm data/faiss_index/index.faiss data/faiss_index/metadata.pkl
python scripts/build_index.py

FAISS Dimension Mismatch

Error:

AssertionError: d == self.d

Cause: Different embedding models used during indexing vs querying (the pre-V6 problem). Rebuild the index — V6 enforces 768-dim at both stages.

Ollama Timeout