fix(api): job search status refresh

[Gitdowski]

Situation

• Asking chatbot if jobs are completed surfaced the following issue: jobs search could report stale running states, while per-job status checks already showed completed.

Changes

• Added refresh_status to POST /jobs:search (default: true) to refresh running jobs before returning matches.
• Re-applied statuses filtering after refresh so filtered results stay consistent with final status.
• Clarified search_jobs semantics (snapshot vs refresh, tag AND behavior) in API schema/docstrings.
• Updated MCP guidance to use search_jobs for listing/filtering and get_job_status for live status checks, without implying automatic polling.

Validation

• Added regression tests for default refresh, refresh opt-out, and post-refresh status filtering.

[codecov]

Codecov Report

❌ Patch coverage is 7.14286% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...morphouspy_api/src/amorphouspy_api/routers/jobs.py	0.00%	13 Missing ⚠️

📢 Thoughts on this report? Let us know!

[ltalirz]

Hm yeah, this can get expensive if there are many jobs in the query (and many of them to be refreshed)

I think in terms of pattern we should move away entirely on doing these implicit updates on user read requests and have a background task take care of it.
Here is a brief summary of the proposal. You could start to use this as a basis for a new PR or I could work on it next week

---

Move ingestion out of the request path. GET /jobs/{id} becomes a pure SELECT; a single background process is the only writer of job state and the only component aware of both executorlib and the DB.

• Mechanism: reconciliation loop scanning non-terminal jobs (durable, restart-safe), optionally plus add_done_callback() for lower latency.
• Contract: the worker watches executorlib's cache_directory for completed *_o.h5 outputs, keyed by cache_key.
• States: SUBMITTED → RUNNING → FINISHED → INGESTING → INGESTED (+ failure states). Separating "executor finished" from "results in DB" is what fixes the wrong-status queries.
• Safety: claim via compare-and-swap or [SKIP LOCKED](https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE), idempotent upserts, retry limit.
• Deployment: separate process, or FastAPI [lifespan](https://fastapi.tiangolo.com/advanced/events/) behind a pg_advisory_lock (otherwise it runs once per uvicorn worker).

Trade-off: status is stale by at most one poll interval — correct-but-delayed instead of fast-but-wrong.

[Gitdowski]

So we scratch this PR, or do you want to approve/merge this in the meantime?

[ltalirz]

you can merge it if you want, we just need to make sure to remove this logic again since it can potentially stall the process for quite a while