feat(optimizer): flatten cost-neutral grid peaks and report service shortfalls

[frahlg]

Inspired by evcc-io/optimizer: take the two things that are worth stealing, and leave the rest.

evcc’s planner is a cloud MILP that currently only visualises a schedule. Ours is already on the dispatch path. Their useful ideas are (1) a cost-neutral second solve that flattens grid peaks among equally priced schedules, and (2) telling the caller when a goal was missed. Soft grid limits, EV-as-battery, CBC/PuLP, and cloud-by-default stay out.

Accepted text proposal

This work was requested in the agent thread: compare evcc-io/optimizer with FTW, take the useful bits, and implement them. Maintainer (Fredrik) asked to take the comparison into a build plan and then implement.

What changed

After the economic solve, a preference stage keeps that cost (plus numerical slack, not 0.1 öre) and minimises the horizon’s import and export peaks. Site fuse limits stay hard. If the extra solve is late, fractional, or fails, the economic schedule is kept. Both the direct HiGHS champion and the CVXPY fallback do this. A one-slot horizon skips the second solve (preference_stage: single_slot) because there is no tie to break across time.

The solver payload now carries preference_stage, import_peak_w, export_peak_w, and an optional service_report with remaining flex/storage/thermal shortfall. The HiGHS champion path reports storage-target shortfall; flex and thermal still go through CVXPY (direct HiGHS is ineligible for those assets). Core unmarshals those onto SolverInfo so /api already exposes them on active_solver. No UI copy in this PR.

settings.flatten_peaks defaults to true. false turns the stage off. Handshake feature: preference_flatten_peaks. Protocol version is unchanged.

Why this implementation

Putting a tiny peak weight in the same objective is what evcc had to abandon: the MIP gap swallows it. They freeze money, then break the tie. Same here, as a third stage after the existing service-then-economic split. That is the right place: service stays lexicographic, money stays lexicographic, peak shape is only a tie-break.

0.1 öre of cost slack was real money: at 50 öre/kWh it buys 2 W of peak reduction. That failed unique-export and arbitrage-spread tests. Slack is now 1e-8 öre (or 1e-12 of |cost|), which is solver noise, not a watt budget.

Boundaries and safety

• Hard import/export limits are not softened.
• Recourse and multistage challengers are unchanged (diagnostic only).
• Go-DP fallback does not flatten.
• No operator YAML knob yet.
• Planner output is still validated in core before dispatch.
• fix(mpc): surplus-only EV can take leftover PV while the home battery grid-charges #957 (leftover PV + battery grid-charge) still owns the surplus-only constraint in model.py. This PR’s model.py delta is the post-economic hook only; rebase onto fix(mpc): surplus-only EV can take leftover PV while the home battery grid-charges #957 if they land together. Both also touch go/internal/mpc/mpc.go / external_optimizer_test.go. fix(mpc): surplus-only EV can take leftover PV while the home battery grid-charges #957 has no reviews yet.

Dual release

Old optimizer images omit the new fields; Core ignores unknown JSON today and understands the fields after this. Flattening only happens once the optimizer image includes this commit. Ship a new ftw-optimizer image with this Core.

Verification

optimizer/.venv/bin/pytest -q optimizer/tests
go test ./internal/mpc -count=1

• 185 optimizer tests passed locally
• two equal-price cheap hours then an expensive load: flatten charges both cheap hours instead of one spike
• flattening does not spend money versus flatten_peaks: false (abs 1e-3 öre)
• profitable fuse-limited export is not nibbled to lower the peak
• one-slot horizons report single_slot and keep the economic plan
• an expired preference deadline reports no_time and does not start the second solve
• an impossible EV deadline still returns a plan and names the kWh shortfall (CVXPY)
• an impossible storage energy target is named on both HiGHS and CVXPY

Checklist

• The diff implements one accepted scope and does not add follow-on work.
• I checked open pull requests that touch the same files. fix(mpc): surplus-only EV can take leftover PV while the home battery grid-charges #957 has right of way on leftover-PV in model.py; this hook is post-economic only.
• Tests cover the changed behaviour and its failure path.
• A human reviewed every changed web/UI view in a browser, or no UI changed.
• A Changeset is included, or the change is exempt.
• Every commit has a DCO sign-off.

  

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_8ba54eec-280a-4a11-90f6-70ab3df9b05b)

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-11T05:06:12.534906Z	80eeedb	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[miravoss26]

What it does: after the economic MPC solve, runs a second cost-neutral "preference" solve that minimises the horizon's grid import/export peaks, and reports unmet EV/storage/thermal service in the solver payload so the UI can explain a missed deadline.

Correctness

• Cost-neutrality is enforced twice: the preference solve has the cost bound (cost_objective <= cost_value + slack) baked in as a hard constraint, and both backends re-verify the actual spend against economic_objective + slack + COST_BOUND_TOLERANCE_ORE after solving before accepting the candidate — good defense against a solver returning a technically-feasible-but-off-by-epsilon result.
• Fuse limits stay hard because the preference problem reuses the same constraints list as the economic solve (HiGHS: same live model instance with rows added, not rebuilt) — test_flatten_does_not_cut_profitable_fuse_export explicitly proves the flatten stage can't push export past max_export_w, which is the claim in the changeset.
• Every failure path (deadline exceeded, solver error, non-integral MIP solution, non-finite values, cost bound violated) falls back to the original economic incumbent (STAGE_KEPT) rather than silently returning a worse or infeasible plan. test_expired_deadline_skips_flatten_and_keeps_peaks confirms the preference solve doesn't even get invoked once the deadline's gone.
• New SolverInfo/ServiceReport fields are omitempty on the Go side and only populated when computed on the Python side — genuinely dual-release, an older optimizer image omitting them doesn't break Core.
• Test coverage is thorough and matches the changeset's claims point for point (cost-neutral, hard limits, deadline-safe, both HiGHS and cvxpy backends, EV/storage shortfall reporting).

Security: no secrets, no new deps, no new network destinations, no authz surface — pure solver/numerics change.

One blocker, unrelated to the code: gh pr view shows mergeable: CONFLICTING — this needs a rebase against main before it can land, independent of the review.

Nothing wrong with the logic from my read — safe to merge once rebased.

[frahlg]

Closing under the owner-approved FTW direction. Core retired the Python optimizer in #1083, so this implementation will not be rebased or restored. Reporting unmet household goals and evaluating cost-neutral peak reduction remain useful requirements for the current Energyplan integration; the selected roadmap in #1214 records them. The PR and its commits remain available as reference.