Feature: Automated Solar Clipping Mitigation (using native clear-sky models)

[rholligan]

Introduction:

This PR introduces a native clipping buffer to Predbat. It prevents solar energy loss on high-generation days by creating dynamic headroom in the battery SoC, specifically targeting periods where PV generation is forecast to exceed inverter AC capacity or DNO export limits.

This implementation integrates with the existing cloud model. It extends SolarAPI to compute and store new pv_clearsky arrays (supporting multiple native upstream sources like ha_solcast_clearsky, solcast_api, and openmeteo). Clear-sky arrays are used alongside a lightweight auto-tuner to safely estimate peak generation and align it with hardware limits.

Resolves issue #1206, supersedes PR #4036.

Key Features:

• Dynamic Buffer Calculation: Calculates exactly how much spare capacity the battery needs to safely absorb any solar generation that exceeds the inverter limits.
• Cost-Optimized Headroom Creation: Integrates directly into Predbat's financial solver by adding a cost penalty whenever the battery doesn't have enough headroom. This naturally forces Predbat to prioritize exporting or deferring charge just before a solar peak hits, guaranteeing the necessary spare capacity.
• Clear-Sky Cloud Modelling: Introduces ha_solcast_clearsky integration alongside a robust clipping_auto_tune mechanism. This explicitly accounts for those short "break in the clouds" spikes, ensuring the buffer is safely sized even when the standard hourly forecast is smoothed out.
• Defensive HA Polling: Hardened solcast.py with safe float(val or 0.0) casting to gracefully handle missing HA clear-sky sensor history without crashing.

Configuration Parameters:

• clipping_buffer_enable: Master toggle for the clipping buffer feature.
• clipping_clearsky_source: Selection of clear-sky forecast source (recommends ha_solcast_clearsky). This setting is fully exposed in the Home Assistant configuration panel.
• clipping_use_clearsky_peaks: Toggle to use Clear-Sky generation arrays rather than standard PV forecast arrays to size the buffer.
• clipping_auto_tune: Automatically learns the scale difference between standard forecasts and hardware limits based on past clipping behavior.
• clipping_amplification: Manual multiplier for the standard solar forecast (used if Auto-Tune is disabled).
• clipping_cost_weight: Multiplier to add a harsher financial penalty for clipping, forcing the optimizer to prioritize creating headroom.
• clipping_limit_override: Manual configuration of the inverter AC ceiling in Watts.
• clipping_buffer_max_kwh / start_time / end_time: Manual settings to force a fixed static buffer size and time window instead of using the dynamic calculation.

Developed with Gemini ✦

[chalfontchubby]

Claude here, on chalfontchubby's behalf. Closing #4036 and continuing here sounds right — this branch is on current main, mergeable, and it takes on the prediction_kernel.cpp / prediction_kernel.py parity work that was the real cost of reviving the old one. That was the thing I flagged as unavoidable, and you've gone at it rather than around it.

Three things that might save you some time, offered as findings rather than as anything you need to adopt.

#4871 changes the code next door. It fixes an operator-precedence bug that left the cloud divergence model inert for every array except pv10 — so the central PV and all three load series were never modulated at all. It also now makes the subtract respect the load_baseline floor, skips a horizon-edge borrow that leaked into the total, and keys the phase off the minutes_now parameter rather than self.minutes_now. I ran git merge-tree between the two branches: no conflict, your fetch.py hunks are all in fetch_pv_forecast's source selection and none touch step_data_history's modulation block. But it does change what the central forecast looks like once merged, which matters if you are sizing anything off it — the curves gain intra-slot variance they did not have before.

A local clear-sky fallback, if you want one. All three of your sources are external — the Solcast forecast_clearsky attribute, the Solcast API, or Open-Meteo's clear_sky_gti. Anyone on Forecast.Solar, or an older Solcast integration that does not publish the attribute, has no clear-sky curve at all. Clear-sky GHI is small enough to compute locally: solar position is standard arithmetic, Haurwitz is one line (1098 · cos z · exp(−0.059 / cos z)), and it transposes to plane-of-array using the tilt, azimuth and kWp Predbat already collects for the Open-Meteo path. solar_model.py already has the second half — pvwatts_cell_temperature and gti_hourly_to_period_kwh. Roughly 60–80 lines, no new dependency, and it would work for every forecast source. Happy to write it up as a patch against this branch or just hand over the maths — say which, or ignore if the external sources cover enough of the fleet.

Three findings from a prototype. We built a plan-side version before you picked this up, to understand the problem rather than to compete with this — it is on feat/clipping-buffer-plan-side if any of it is useful, and it is not going anywhere near a PR.

• Applying the buffer as a ceiling on the charge-limit search in optimise_charge_limit(), rather than mutating charge_limit inside run_prediction(), keeps the plan that gets scored identical to the plan that gets executed — and leaves the kernel-mirrored hot loop alone entirely.
• The requirement wants scoping per clipping episode, not as a running maximum over the whole horizon. Otherwise a second sunny day pins the first day's headroom all night. The subtlety that caught us out: the overnight gap must still carry the next episode's requirement, because the overnight charge window ends inside that gap and is precisely the one that has to leave room. Releasing the buffer through the gap looked right on the profile and silently reverted the plan to charging full.
• Sizing to absorb all forecast clipping is the wrong objective. On the debug file in PV clipping protection: proactively create battery headroom using PV forecast/PV90 #4865 (9.4 kW array behind a 5.5 kW limit) reserving the full forecast requirement of 4.37 kWh scored worse than reserving nothing; about 10% of the battery scored best. Marginal value meets marginal cost well before the clipping is eliminated.

Supporting data from that issue, simulating a buffer against 8 days of 1-minute history:

day	generated	clipped, battery full	buffer for full recovery	recycled
broken cloud	36.6 kWh	3.50 kWh	2.5 kWh	1.3x
clear sky	57.2 kWh	7.96 kWh	7.5 kWh	1.0x
other 6 days	8–29 kWh	≤0.03 kWh	—	—

Which is the argument for clear-sky sizing over a percentile: Solcast's pv90 for that array sat about 18% below its measured clear-sky ceiling, so percentile-derived sizing undershoots. Your default looks right.

Shout if any of it is useful. Otherwise we will stay out of the way and keep an eye on how it is going.

---

Written by Claude, on chalfontchubby's behalf.

[rholligan]

Thanks @chalfontchubby, these insights are really helpful!

I'm going to have a go at implementing the plan-side buffer and episode scoping from your branch. Applying the ceiling directly to the search limits instead of mutating the simulation is definitely a cleaner approach. Would be great to have your eyes on that, and give it a test run if possible.

@springfall2008 - what are your thoughts on adding the clearsky calcs nateively to predbat?

Also, just a quick check on how you'd like to proceed with this PR. It's now passing pre-commit CI, so let me know if you are happy reviewing it as-is, or if you'd still prefer I split it out into the 3 smaller PRs proposed above.

[chalfontchubby]

The cloud model fix is merged. You might want to pull that in.