Build Your Knowledge Base

Purpose

A governed Python pipeline for turning notes/transcripts into a structured, reviewable knowledge base package.

This repository is being prepared for public GitHub migration. Raw inputs are private by default and are not included in the public repository. Generated outputs are excluded by default.

Current Scope

Governed compact KB release:
input/raw -> inventory -> chunks -> clean notes -> source cards -> KB build -> judge/QA -> publish -> compact ChatGPT package

Use CURRENT_SCOPE.md, DATA_CONTRACTS.md, RUNBOOK.md, and CODEX.md as the active operating contracts. MVP-10 search files remain available as sub-scope documentation when search-specific work is requested.

Repository Structure

Path	Purpose	Publication default
src/notes_to_kb/	Python package code	publish
scripts/	CLI and pipeline entry points	publish
tests/	pytest coverage	publish
prompts/	prompt templates	publish after review
docs/migration/	migration policies and readiness evidence	publish
.github/	GitHub PR, issue, and CI foundation	publish
input/raw/	private raw transcripts	do not publish
workspace/	generated working state	do not publish
publish/	generated release outputs	do not publish by default
publish_assets/	generated/publication support	do not publish by default
release_snapshots/	release evidence snapshots	do not publish by default

See docs/migration/publication_scope.md, docs/migration/raw_input_policy.md, and docs/migration/generated_artifacts_policy.md before preparing a public transfer.

Files

File	Purpose
CURRENT_SCOPE.md	Active scope and source-of-truth routing
DATA_CONTRACTS.md	Current file-first data layout
RUNBOOK.md	Safe setup, validation, and pipeline commands
CODEX.md	Active Codex working guide
SPEC.md	Main MVP-10 specification
PLAN.md	Work packages and execution order
TASKS.md	Implementation checklist
SCOPE_LOCK.md	Allowed and forbidden changes
ACCEPTANCE_CRITERIA.md	Completion gates
MODEL_ROUTING.md	Confirms no model routing for MVP-10
SKILLS_SEQUENCE.md	Minimal workflow sequence

Search CLI

Run lexical search over the default published markdown KB:

python3 scripts/search_kb.py Knowledge --limit 3

Default search file:

publish/markdown_kb/full_kb.md

JSON output:

python3 scripts/search_kb.py Knowledge --limit 1 --json

Search a specific published file:

python3 scripts/search_kb.py "source card" --path publish/chatgpt_project/TRACEABILITY_GUIDE.md --limit 5

The search is deterministic lexical matching only. It does not use LLMs, Ollama, embeddings, a vector DB, or network calls.

Options:

Option	Purpose
positional query	Required search text.
--path	Optional markdown or CSV file to search. Relative paths are resolved from the project root.
--limit	Maximum result count. Default: 10.
--json	Emit deterministic JSON with query, path, and results.

Failure behavior:

• empty query returns a clear error;
• missing search file returns a clear error;
• no matches returns result_count=0 or an empty JSON results list.

Hard Stop

Do not implement:

• embeddings;
• vector DB;
• semantic RAG;
• web UI;
• scheduler;
• Mode B;
• Mode C;
• LLM calls;
• network calls.

This hard stop applies to the lightweight search CLI. The transcript processing pipeline below can use Ollama and Gemini when explicitly run.

Transcript Processing Pipeline

The current supported full route processes .txt transcripts into clean notes, source cards, deterministic KB sections, judge reports, and publish outputs.

Route:

Stage	Engine
Inventory	deterministic Python
Chunking	deterministic Python
Raw transcript -> clean note	Ollama
Clean note + chunks -> source card	Gemini 2.5 Flash-Lite
Source cards -> KB sections	deterministic Python
Final judge	deterministic Python
Publish	deterministic Python
Core tests	pytest

This route does not use embeddings, vector DB, semantic search, web UI, scheduler, Mode B, or Mode C.

Setup

Install local test/runtime dependencies:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt

Create .env from .env.example and fill the Gemini key:

cp .env.example .env

Required for the Gemini route:

GEMINI_API_KEY=...

Common optional settings:

GEMINI_SOURCE_CARD_MODEL=gemini-2.5-flash-lite
GEMINI_TIMEOUT=600
GEMINI_MAX_OUTPUT_TOKENS=4096
GEMINI_THINKING_BUDGET=0
GEMINI_MAX_RETRIES=3
OLLAMA_BASE_URL=http://127.0.0.1:11434
CLEAN_MODEL=devstral-small-2:24b-instruct-2512-q4_K_M

Ollama must be running locally before launch. The command wrapper checks OLLAMA_BASE_URL/api/tags before starting.

Do not commit .env. Keep .env.example limited to placeholder values.

Run

From the project root:

./run_update_compact_ollama_gemini.command

You can also double-click:

run_update_compact_ollama_gemini.command

The wrapper resumes safely:

• existing clean notes are not regenerated;
• existing source cards are not overwritten;
• missing source cards continue from where the previous run stopped;
• transient Gemini timeouts and 429/5xx responses are retried;
• publish is allowed when judge readiness is needs_review;
• publish remains blocked when judge readiness is blocked.

Source card QA defaults to relaxed mode. relaxed sends only empty, broken, unusable, or clearly irrelevant cards to review; formal issues such as section order, extra sections, removed <think> blocks, or incomplete chunk evidence are recorded as warnings. Use --qa-strictness standard or --qa-strictness strict with scripts/run_source_card.py when stricter gating is needed.

Output Files

Main generated working files:

workspace/inventory/files_index.csv
workspace/inventory/sources_index.csv
workspace/chunks/
workspace/clean_notes/
workspace/source_cards/
workspace/knowledge/topics/
workspace/knowledge/concepts/
workspace/knowledge/indexes/
workspace/knowledge/reports/judge_report.md

Ready-to-use publish files:

publish/chatgpt_project/AI_KB_Context_File_v1.0.md
publish/chatgpt_project/INDEX.md
publish/chatgpt_project/CONCEPT_MAP.md
publish/chatgpt_project/WORKFLOW_MAP.md
publish/chatgpt_project/TRACEABILITY_GUIDE.md
publish/chatgpt_project/KB_USAGE_GUIDE.md
publish/chatgpt_project/SMOKE_QUESTIONS.md

publish/markdown_kb/full_kb.md
publish/markdown_kb/sources_index.csv
publish/markdown_kb/concepts_index.csv

publish/obsidian/

How To Use The Publish

For ChatGPT Projects:

1. Open publish/chatgpt_project/.
2. Upload AI_KB_Context_File_v1.0.md as the main context file.
3. Upload the helper files from the same folder when you want navigation, traceability, workflow, and smoke-question support.
4. Use SMOKE_QUESTIONS.md to check whether the project answers from the KB rather than inventing.

For a smaller ChatGPT Project upload package, build the compact seven-file package:

python3 scripts/build_chatgpt_compact_kb.py

Output:

publish/chatgpt_project_compact/

Upload all files from publish/chatgpt_project_compact/ into the same ChatGPT Project. Do not upload raw transcripts, clean notes, source cards, chunks, temp files, logs, embeddings, or vector database files.

To add the deterministic synthesis layer with canonical concepts, frameworks, patterns, anti-patterns, and Sergey-focused use cases:

python3 scripts/build_synthesis_layer.py

This adds:

publish/chatgpt_project_compact/KB__05_CANONICAL_CONCEPTS.md
publish/chatgpt_project_compact/KB__06_OPERATIONAL_FRAMEWORKS.md
publish/chatgpt_project_compact/KB__07_PATTERNS_AND_FAILURES.md
publish/chatgpt_project_compact/KB__08_USE_CASES_FOR_SERGEY.md
publish/chatgpt_project_compact/SYNTHESIS_MANIFEST.md

The synthesis layer is deterministic by default. Weak evidence is marked explicitly instead of being promoted to finished knowledge.

For local markdown reading or search:

python3 scripts/search_kb.py "variance analysis" --path publish/markdown_kb/full_kb.md --limit 5

For Obsidian:

1. Copy or open publish/obsidian/ as a vault or inside an existing vault.
2. Use publish/obsidian/INDEX.md as the entry point.
3. Topic files are under topics/, concepts under concepts/, and per-source cards under sources/.

Judge output:

workspace/knowledge/reports/judge_report.md

needs_review means the publish was created, but some source cards are marked for human review. blocked means publish should not be used without fixing the reported issues or explicitly overriding the publish command.

Core Validation

The Gemini wrapper runs the core pipeline tests:

python3 -m pytest \
  tests/test_inventory.py \
  tests/test_chunking.py \
  tests/test_clean_note.py \
  tests/test_source_card.py \
  tests/test_kb_build.py \
  tests/test_judge.py \
  tests/test_publish.py \
  tests/test_ollama_client.py

The broader python3 -m pytest tests suite also includes legacy/manual QA release artifact checks. Those are not part of the current .txt processing route.

For public GitHub migration checks, run:

python3 -m pytest tests -q
git diff --check

Data and Privacy Policy

input/raw/ is private by default. Raw transcripts must not be published unless they are explicitly reviewed and sanitized. Sanitized samples must remove credentials, personal data, private server details, API keys, tokens, financial/private operational details, and copyrighted/private source material.

Do not read, print, commit, or upload .env.

Generated Artifacts Policy

workspace/, publish/, publish_assets/, and release_snapshots/ are generated or release-output folders. They are ignored by default for public repository migration. Selected release artifacts can be published later only after explicit review, preferably through GitHub Releases rather than source commits.

GitHub Migration Status

Phase 1 migration preparation is local-only. Do not push, force-push, add remotes, or overwrite the destination repository in this phase. The recommended transfer path is documented in docs/migration/github_transfer_plan.md.

Current readiness is partial until dirty runtime/test changes, raw transcript publication decisions, and selected generated artifact decisions are resolved.

Limitations

• The default search CLI is deterministic lexical search, not semantic RAG.
• Embeddings, vector DB, web UI, scheduler, and autonomous retrieval are out of scope unless explicitly approved.
• LLM/Ollama/Gemini routes require explicit approval and local/API configuration.
• Generated outputs may reflect private raw inputs and should not be treated as public-safe without review.

Ollama-Only Pipeline

Clean note and source card generation can use a local Ollama server instead of the deterministic mock provider:

python3 scripts/run_clean_note.py --all --provider ollama --model <MODEL> --overwrite
python3 scripts/run_source_card.py --all --provider ollama --model <MODEL> --overwrite
python3 scripts/run_kb_build.py --all --overwrite
python3 scripts/run_judge.py --all
python3 scripts/run_publish.py --mode all

The wrapper runs the current supported pipeline in the same order:

python3 scripts/run_ollama_pipeline.py --model <MODEL>

By default, Ollama requests use http://127.0.0.1:11434. Override it with --ollama-base-url or OLLAMA_BASE_URL.