fix(DATA-004): stop re-downloading stocks that listed after the window opened

[DoRmAmMu1997]

Stacked on #112 (DATA-003), which introduced _cache_covers_range. This PR targets fix/data-003-vendor-dedupe, not main — merge #112 first.

Why

Both cache-coverage checks require first_date <= requested_start, where requested_start is today minus ten years. A stock that listed after that date can never satisfy it, because DhanHQ has nothing earlier to give.

200 of 577 cached symbols were in that state — DMART (listed 2017-03-21), RBLBANK (2016-08-31), LTTS, COHANCE and ~196 more. Every prefetch and every scan re-downloaded their full history, wrote the same short frame back, and did it again next time. Forever.

The distinction that had to be preserved

The two cases look identical from the cached file alone, and conflating them would be a correctness bug:

Case	Vendor has earlier data?	Correct action
Interrupted prefetch left a partial file	yes	must still refetch
Stock listed after the window opened	no	refetching is waste forever

What changed

A new .firstbar sidecar records what the vendor actually served, following the precedent DATA-002 established with .checked (an empty tail) and .repaired (a repair cooldown):

{"requested_from": "2016-08-24", "earliest_available": "2017-03-21", "recorded_on": "2026-08-24"}

A cache then reaches back far enough when it either literally covers the requested start or already begins at the vendor's earliest known bar.

Three properties stop the marker ever hiding genuinely missing history:

• It only counts when the recorded probe reached at least as far back as the current request. Learning that nothing exists before 2021 when you only asked from 2021 says nothing about 2016, so a shallow probe cannot suppress a deeper refetch.
• It expires after 30 days (VENDOR_EARLIEST_RECHECK_DAYS), because vendors do occasionally backfill history. One request per affected symbol per month is negligible; a permanent belief is not.
• Any read or parse failure returns None and the strict rule applies, so a corrupt marker can only ever cost an extra request.

Suppressing the backfill deliberately falls through to the normal freshness and incremental logic rather than returning early — a later listing still needs its daily top-up. The point is to stop the pointless ten-year refetch, not to freeze the symbol. There's a test for exactly that.

Measured on a copy of the real 577-file cache

nifty_500, 500 rows, same data for both sides:

Prefetch	Dhan requests	Full-window backfills
pass 1 (learns)	500	176
pass 2	176	0
pass 3	0	0

Scan path	Cache hits	Misses	Files rewritten
before (DATA-003)	324	176	176
after	500	0	0

Each of those 176 misses was a full ten-year re-download that also rewrote the parquet — the bulk of the six minutes run_id=4 took.

Gates

1,951 tests pass, coverage 89.82% (floor 87%); ruff, mypy, bandit, compileall, and pre-commit validate-config clean. No dependency changes.

Housekeeping: .firstbar joins .checked and .repaired in cleanup_stale_cache_files, so an orphan is removed with its parquet.

🤖 Generated with Claude Code

[DoRmAmMu1997]

@codex can you review this?

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5a9e5eff69

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Use wall-clock time for marker expiry

When get_daily_history is called with a fixed historical or future end_date, this stores that request boundary as recorded_on rather than the date the vendor was actually queried; _vendor_earliest_for likewise evaluates age against the same request boundary. Repeating a historical request therefore always calculates an age of zero, while a future-dated request can remain fresh for far longer than 30 days, so vendor backfills may be hidden indefinitely behind a partial cache. Record and compare against the actual current date (ideally via an injectable clock) independently of the requested candle range.

Useful? React with 👍 / 👎.

[DoRmAmMu1997]

Confirmed and fixed in efe240c.

get_daily_history was passing the request's own end_date where a wall clock belongs — both when stamping recorded_on and when judging the marker's age. Exactly as you describe: a repeated historical request always computed an age of zero so the 30-day expiry never fired, and a future-dated end_date kept the marker fresh well beyond it, either way hiding a vendor backfill behind a partial cache indefinitely. ensure_daily_history had it right; this path did not.

DailyDataLoader now takes an injectable today_func, alongside the existing sleep_func injection, and both _record_vendor_earliest and _vendor_earliest_for use it. The two concepts are now explicitly separate in the code and the docstrings: "the date I am asking about" is not "the date it is now", and conflating them is precisely what caused this. ensure_daily_history keeps its today argument, which describes the data window only.

Two regression tests, both failing on the previous commit:

• a marker recorded through a historical request is stamped from the clock, and is expired when the clock advances 31 days;
• a future-dated request does not extend a 45-day-old marker's life.

Plus a positive case so the expiry tests cannot pass vacuously.

This branch also merges the updated DATA-003 base, which replaced the blanket four-day tolerance with weekday arithmetic plus the .checked marker. _cache_covers_range now carries both kinds of evidence, judged independently: the front by vendor_earliest (how far back the vendor's history goes) and the back by which recent days the market actually published.

Re-measured on a copy of the real 577-file cache after that stricter rule — both earlier wins survive:

	Result
prefetch pass 3	0 requests, 0 backfills
scan path	500 hits / 0 misses / 0 files rewritten

Gates: 1,964 tests, 89.81% coverage, ruff/mypy/bandit/compileall clean.