Skip to content

fix(core): handle legacy pgvector storage status - #1196

Open
phernandez wants to merge 3 commits into
mainfrom
codex-1195-pgvector-status-schema
Open

fix(core): handle legacy pgvector storage status#1196
phernandez wants to merge 3 commits into
mainfrom
codex-1195-pgvector-status-schema

Conversation

@phernandez

Copy link
Copy Markdown
Member

Why

Production Logfire issue #2173 shows project-info requests failing with asyncpg UndefinedColumnError because a legacy search_vector_embeddings table predates the source_hash column used by current count queries.

The physical table can exist before PgVectorIndex.initialize() has run its existing schema repair. Project status currently treats table existence as schema readiness and executes the invalid join, turning a recoverable reindex state into a customer-visible HTTP 500.

Closes #1195.

What Changed

  • inspect the Postgres pgvector embeddings columns before running physical count queries
  • treat storage without source_hash as outdated and return the existing reindex recommendation contract
  • add a Postgres regression that creates the production-shaped legacy table and proves status does not query the missing column

Implementation Details

ProjectService.get_embedding_status() now distinguishes physical-table existence from schema readiness for the built-in pgvector adapter. When the legacy table is detected, it returns vector_tables_exist=False with Vector storage schema is outdated — run: bm reindex --embeddings.

The status path remains read-only. It does not drop, migrate, or rebuild the table; PgVectorIndex.initialize() remains the single repair owner and will recreate the incompatible physical table when indexing runs. External adapters remain manifest-only, and SQLite behavior is unchanged.

Testing

Automated

  • LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -p pytest_mock --no-cov -q tests/services/test_project_service_embedding_status.py: 15 passed, 1 skipped
  • BASIC_MEMORY_TEST_POSTGRES=1 LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -p pytest_mock --no-cov -q tests/services/test_project_service_embedding_status.py: 14 passed, 2 skipped
  • just fast-check: passed after installing configured optional dependencies with uv sync --all-extras
  • git diff --check: passed

Manual

  • Inspected the production trace and verified the failing SQL joins on the legacy missing column before any pgvector initialization repair runs.

Risks / Follow-ups

  • This change only converts legacy physical storage into an explicit reindex status; it does not perform repair from the read endpoint.
  • After this Core PR merges, Basic Memory Cloud must pin the resulting Core commit, deploy it, and observe a non-recurrence window before Logfire #2173 can be closed.

Signed-off-by: phernandez <paul@basicmachines.co>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2516219be4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/basic_memory/services/project_service.py Outdated
Signed-off-by: phernandez <paul@basicmachines.co>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f56b42e791

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/basic_memory/services/project_service.py Outdated
Signed-off-by: phernandez <paul@basicmachines.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Project info fails on legacy pgvector embeddings schema

1 participant