Skip to content

chore(file-search): retire legacy index storage #8014

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
icecrasher321 wants to merge 1 commit into
base: staging
Choose a base branch
from
Open

chore(file-search): retire legacy index storage #8014

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions apps/sim/lib/workspace-files/search/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,11 @@ Search results retain `fileId`, 1-based `lineNumber`, and bounded `text` preview
1. Deploy the additive migration and new application/Trigger worker versions. Legacy index tables remain readable for the old deployment. The file trigger queues current revisions in the new table; this cutover intentionally allows temporary search unavailability while the new index builds.
2. The dispatcher uses a separate `workspace-file-search-chunks-v2` backfill cursor. It seeds at most 1,000 active files per pass under a shared file lock, with idempotent inserts. Normal dispatch caps remain two outstanding jobs per workspace, 100 outstanding globally, and ten running workers. Reconciliation repeats hourly after a complete pass to repair missing metadata. Failed revisions remain visible as failed; they are not silently declared covered.
3. Before retiring legacy storage, verify the new app and Trigger workers are fully deployed, old runs/retries have drained, the backfill cursor has completed, and scoped coverage is ready or explicitly excluded. Investigate failed or stale pending revisions. Check cleanup backlog and run representative exact/regex searches, including long lines and folder scopes.
4. After the rollback window, ship a separate contract PR removing the legacy schema and dropping `workspace_file_search_segment` / `workspace_file_search_index` with a short lock timeout. Do not delete the entire old index row-by-row or backfill it inside the schema migration. Dropping obsolete tables reclaims their heap, indexes, and TOAST together. The `contract-pending` marker in `packages/db/schema.ts` tracks this step.
4. After the rollback window and the checks above, deploy `0368_retire_legacy_file_search.sql`. It drops only `workspace_file_search_segment` and `workspace_file_search_index`, atomically, with a two-second lock timeout and without `CASCADE`. An unexpected dependency or lock conflict aborts the migration instead of removing dependent objects. Replay tolerates tables already retired. Dropping the tables reclaims their heap, indexes, and TOAST together; it does not delete current chunks or rebuild search.

Until that contract deploy, legacy foreign-key cascades can still make a hard file/workspace deletion expensive. New-index cleanup is bounded; retaining the old schema cannot erase that legacy cost. The earlier timestamp-repair script detects the chunk schema and leaves obsolete legacy text for this contract step instead of deleting it in bulk. Legacy-table retirement remains a separate contract migration.
The status enum, dispatch queue, and v2 backfill cursor remain in use. The timestamp-repair script still preserves provenance and repairs current revisions; its legacy deletion branch has been removed. No bulk deletion or replacement backfill runs in the contract migration.

Rollback before retirement requires restoring the old trigger function as well as the old app/worker version, and reconciling legacy revisions written during the cutover. Do not assume retained tables are automatically up to date. Canonical revision joins prevent stale content from being returned.
After retirement, rollback must stay on a chunk-compatible app and worker release. Restoring the old segment-based implementation requires recreating and rebuilding its retired storage; reverting application code alone is insufficient.

### Direct GIN writes

Expand Down
42 changes: 41 additions & 1 deletion apps/sim/lib/workspace-files/search/chunks.integration.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,7 @@ describe('chunked workspace file search on PostgreSQL', () => {
'0358_workspace_file_content_version_precision.sql',
'0359_workspace_file_search_chunks.sql',
ginWriteMigration,
'0368_retire_legacy_file_search.sql',
]) {
await applyMigration(migration)
}
Expand All @@ -208,7 +209,7 @@ describe('chunked workspace file search on PostgreSQL', () => {
})
beforeEach(async () => {
await connection`TRUNCATE workspace, workspace_files, workspace_file_search_revision, workspace_file_search_build,
workspace_file_search_chunk, workspace_file_search_index, workspace_file_search_segment, workspace_file_search_dispatch_queue, workspace_file_search_backfill`
workspace_file_search_chunk, workspace_file_search_dispatch_queue, workspace_file_search_backfill`
await connection`INSERT INTO workspace_file_search_backfill (id, completed_at) VALUES ('workspace-file-search-chunks-v2', now())`
await addFile('file-1')
})
Expand All @@ -222,6 +223,45 @@ describe('chunked workspace file search on PostgreSQL', () => {
}
})

it('retires legacy tables atomically, preserves current search, and safely replays', async () => {
await index('heading\nretirement needle\ntail')
await connection`CREATE TABLE workspace_file_search_index (file_id text PRIMARY KEY)`
await connection`CREATE TABLE workspace_file_search_segment (file_id text, content text)`
await connection`INSERT INTO workspace_file_search_index VALUES ('retired-file')`
await connection`INSERT INTO workspace_file_search_segment VALUES ('retired-file', 'retired text')`
await connection`CREATE VIEW legacy_dependency AS SELECT * FROM workspace_file_search_index`
try {
await expect(applyMigration('0368_retire_legacy_file_search.sql')).rejects.toMatchObject({
code: '2BP01',
})
expect(
(await connection`SELECT count(*)::int AS count FROM workspace_file_search_segment`)[0]
.count
).toBe(1)
await connection`DROP VIEW legacy_dependency`
for (let attempt = 0; attempt < 2; attempt++) {
await applyMigration('0368_retire_legacy_file_search.sql')
expect(
(
await connection`SELECT to_regclass('workspace_file_search_index') AS legacy_index,
to_regclass('workspace_file_search_segment') AS legacy_segment`
)[0]
).toEqual({ legacy_index: null, legacy_segment: null })
expect((await search('retirement needle')).results).toMatchObject([
{ fileId: 'file-1', lineNumber: 2 },
])
expect((await search('^retirement.*needle$', 'regex')).results).toMatchObject([
{ fileId: 'file-1', lineNumber: 2 },
])
}
} finally {
await connection`DROP VIEW IF EXISTS legacy_dependency`
await applyMigration('0368_retire_legacy_file_search.sql')
}
await connection`DELETE FROM workspace_files WHERE id = 'file-1'`
expect((await search('needle')).results).toEqual([])
})

it('preserves search through disabling, draining, and replaying GIN pending-list maintenance', async () => {
await connection`ALTER INDEX workspace_file_search_chunk_content_idx SET (fastupdate = on)`
try {
Expand Down
7 changes: 7 additions & 0 deletions packages/db/migrations/0368_retire_legacy_file_search.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
SET LOCAL lock_timeout = '2s';

@greptile-apps greptile-apps Bot Sep 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Retirement Is Not Atomic

On fresh installs or deployments migrating from before 0365, earlier pending migrations execute COMMIT, ending Drizzle’s transaction without starting another. This makes SET LOCAL ineffective and causes the two table drops to commit separately. If the second drop fails because of a dependency or lock conflict, the segment table remains permanently dropped instead of both changes rolling back atomically. Ensure this retirement runs inside an explicit transaction.

@cubic-dev-ai cubic-dev-ai Bot Sep 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: When the real runner applies this post-COMMIT migration, SET LOCAL runs outside a transaction and is ignored, so the drops use the runner's 5-second timeout instead of 2 seconds. Use a session-level SET or wrap the drops in an explicit transaction.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At packages/db/migrations/0368_retire_legacy_file_search.sql, line 1:

<comment>When the real runner applies this post-`COMMIT` migration, `SET LOCAL` runs outside a transaction and is ignored, so the drops use the runner's 5-second timeout instead of 2 seconds. Use a session-level `SET` or wrap the drops in an explicit transaction.</comment>

<file context>
@@ -0,0 +1,7 @@
+SET LOCAL lock_timeout = '2s';
+--> statement-breakpoint
+-- migration-safe: contract of #7947; deployed app, workers, and revision triggers use chunk storage. Retire only after the rollback window and completed backfill verification.
</file context>
Suggested change
SET LOCAL lock_timeout = '2s';
SET lock_timeout = '2s';
Fix with cubic

--> statement-breakpoint
-- migration-safe: contract of #7947; deployed app, workers, and revision triggers use chunk storage. Retire only after the rollback window and completed backfill verification.
DROP TABLE IF EXISTS "workspace_file_search_segment";
--> statement-breakpoint
-- migration-safe: contract of #7947; current revision metadata lives in workspace_file_search_revision, with no remaining runtime reader of this legacy table.
DROP TABLE IF EXISTS "workspace_file_search_index";
Loading
Loading