fix(piecewise): declare ragged breakpoints with mask= instead of inferring NaN by FBumann · Pull Request #885 · PyPSA/linopy

FBumann · 2026-08-09T14:55:12Z

This PR fixes #884. It takes the simplest, strict path, disallowing nan like everywhere else, except if a matching mask is passed. This works without adding a new primitive like Breaks, which would be another option but a bigger api change and less flexibility (or a complex, dataarray-like class).

Note

The content below this line was generated by AI.

Stacked on #717 (feat/arithmetic-convention). Fixes #884, which blocks PyPSA/PyPSA#1829.

What this changes

Ragged curves — entities with different numbers of breakpoints — are stored densely along _breakpoint with the surplus slots left absent. Under v1 that tripped the §5 user-NaN guard from deep inside the formulation's arithmetic:

ValueError: NaN found in a user-supplied constant. ... if you meant *absent at this slot*,
mark it on the variable instead (mask=, .where(cond), .reindex(...), .shift(...))

— advice that does not apply to breakpoint data, raised from (delta_var * steps).sum(...) rather than from the input.

§5 is right to refuse the input. A shorter curve and a stray NaN look identical, and linopy trusts NaN only from its own structural operations (§4). What was missing is a way to declare the absence. So:

add_piecewise_formulation(..., mask=...) — §4's mechanism, already the vocabulary of add_variables/add_constraints. It declares which (entity, breakpoint) slots hold a real breakpoint and becomes the authoritative breakpoint mask, replacing the isnull() inference. For NaN-padded input the declaration is mask=x_pts.notnull().
Legacy is unchanged except for a LinopySemanticsWarning: it keeps inferring from NaN placement, so nothing breaks before opt-in.
Padding must still not reach the arithmetic as a constant. Provenance is gone by the time a breakpoint table multiplies a variable, so §5 applies to it there. Those coefficients are zeroed — the absence is already carried by the masked variable and propagates on its own (§6).
tangent_lines has no mask to declare with, so it can only report; it now names raggedness instead of surfacing the generic user-NaN message.

Scope: five paths, not three

The issue lists lp, incremental, and incremental + active. sos2 and disjunctive fail identically — they reach the padded slot through (lambda_var * links.eq_bp) instead. All five are covered.

Resulting behaviour

	legacy	v1
ragged, undeclared	builds (+ deprecation warning)	raises at the boundary, naming `mask=`
ragged, `mask=` declared	builds	builds

Verified across lp / sos2 / incremental / incremental + active / disjunctive.

Verification detail

Objectives match legacy exactly on all three solvable cases; the lp path's LP file is byte-identical.
The incremental LP file differs from legacy only by three redundant rows (-1.0 x5 <= -0.0) that legacy emits for fully-masked delta_bound / fill_order constraints and v1 correctly drops. Pre-existing v1 improvement, not from this PR.
Analytic check: on the short curve y: 0 → 60 over x: 0 → 100, x=50 gives y=30.
test_declared_mask_matches_legacy_oracle reuses the oracle from the existing legacy-only padding tests (f_b(10) = 12.5) to assert the declared model is the model legacy inferred.
Full suite: 5074 passed, 446 skipped. ruff clean; mypy clean (the one solvers.py:297 error pre-exists on the base branch).

Notes for review

The existing tests already anticipated this. test_lp_per_entity_nan_padding and test_sos2_per_entity_nan_padding were marked @pytest.mark.legacy with the comment "Legacy-only: NaN-as-mask in user input (see convention.md §5)". Those models now run under v1 too, via mask=.
No spec change. This is an application of §4/§5, not a new rule, so doc/design/convention.rst is untouched.
Slopes(align="leading") is arguably inconsistent with §5 — it requires values[0] to be NaN as a positional sentinel in user data. Out of scope here; worth a separate issue if you agree it is one.
PyPSA impact: the migration becomes "declare the absent breakpoints" (mask=x_pts.notnull()), not "linopy tolerates the NaN".

Alternatives considered

Accept trailing NaN as a documented positional contract — smallest change, but has linopy infer meaning from a user NaN, which is what §5 exists to prevent.
A Breaks carrier returned by breakpoints()/segments() — carries provenance so ragged dict input stays undeclared, but changes what those public factories return. Rejected in favour of zero new public types; the cost is that ragged dict/list input must now be declared too.

…rring NaN Ragged curves — entities with different numbers of breakpoints — are stored densely along `_breakpoint` with the surplus slots left absent. Under v1 that tripped the §5 user-NaN guard from deep inside the formulation's arithmetic, with a message pointing at remedies that do not apply to breakpoint data. All formulation paths were affected: `lp` and `sos2` and `incremental` (with and without `active=`) and disjunctive. §5 is right to refuse the input: a shorter curve and a data error look identical, and linopy trusts NaN only from its own structural operations (§4). What was missing is a way to declare the absence. Add `mask=` to `add_piecewise_formulation` — §4's mechanism, already the vocabulary of `add_variables`/`add_constraints` — and make it the authoritative breakpoint mask. Legacy keeps inferring from NaN placement, now with a LinopySemanticsWarning. Once declared, the padding still must not reach the arithmetic as a *constant*: provenance is gone by the time a breakpoint table multiplies a variable, so §5 applies to it there. Zero those coefficients — the absence is already carried by the masked variable and propagates on its own (§6). `tangent_lines` has no mask to declare with, so it can only report; it now names raggedness rather than surfacing the generic user-NaN message. Closes #884 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

FabianHofmann

wonderful and faster than allowed. There is some references in the code (docstrings & comments) which point to the paragraphs of the convention. I think we should keep those out and all these would need to be updated once we change the convention. let's keep this information absolute

FabianHofmann · 2026-08-10T06:52:46Z

+    # No mask to declare absence on this low-level helper, so a ragged curve
+    # can only be reported (§5), not resolved — say so here rather than let
+    # the generic user-NaN message surface from the chord arithmetic.


FabianHofmann · 2026-08-10T06:53:52Z

+    """
+    Make a breakpoint table safe to use as a *constant* operand.
+
+    Absent slots are marked with ``NaN`` (§2), but as soon as the table


reference to convention in code.

make the docstring a one-liner

FabianHofmann

some more things

FabianHofmann · 2026-08-10T06:56:51Z


 * Default internal integer labels to ``int32``, cutting memory ~25% and speeding up model build 10-35%. Models exceeding the int32 maximum (~2.1 billion labels) widen to ``int64`` automatically with a ``UserWarning``; pass ``Model(dtypes={"labels": np.int64})`` upfront to avoid the mid-build upcast (exposed read-only via ``Model.dtypes``).
 * ``add_variables(binary=True, ...)`` now accepts ``lower``/``upper`` bounds, as long as they are 0 or 1. Previously binary bounds could only be set via the ``.lower``/``.upper`` setters after creation. (https://github.com/PyPSA/linopy/issues/776)
+* ``add_piecewise_formulation`` gained a ``mask`` parameter declaring which breakpoint slots hold a real breakpoint. It is needed for **ragged** curves — entities with different numbers of breakpoints — which are stored densely with the surplus slots left absent. Under v1 that absence must be declared (``mask=x_pts.notnull()``) rather than read off the NaN padding, since §5 does not let linopy tell a shorter curve from a data error; legacy keeps inferring it, with a ``LinopySemanticsWarning``. Previously ragged curves failed under v1 with a generic "NaN found in a user-supplied constant" raised from deep inside the formulation's arithmetic, on all of the ``lp``, ``sos2``, ``incremental`` and disjunctive paths. (https://github.com/PyPSA/linopy/issues/884)


Suggested change

* ``add_piecewise_formulation`` gained a ``mask`` parameter declaring which breakpoint slots hold a real breakpoint. It is needed for **ragged** curves — entities with different numbers of breakpoints — which are stored densely with the surplus slots left absent. Under v1 that absence must be declared (``mask=x_pts.notnull()``) rather than read off the NaN padding, since §5 does not let linopy tell a shorter curve from a data error; legacy keeps inferring it, with a ``LinopySemanticsWarning``. Previously ragged curves failed under v1 with a generic "NaN found in a user-supplied constant" raised from deep inside the formulation's arithmetic, on all of the ``lp``, ``sos2``, ``incremental`` and disjunctive paths. (https://github.com/PyPSA/linopy/issues/884)

* ``add_piecewise_formulation`` gained a ``mask`` parameter declaring which breakpoint slots hold a real breakpoint. It is needed for **ragged** curves — entities with different numbers of breakpoints — which are stored densely with the surplus slots left absent. Under v1 that absence must be declared (``mask=x_pts.notnull()``) rather than read off the NaN padding. ```

FabianHofmann · 2026-08-10T06:57:30Z

+    """
+    Make a breakpoint table safe to use as a *constant* operand.
+
+    Absent slots are marked with ``NaN`` (§2), but as soon as the table


make the docstring a one-liner

FabianHofmann · 2026-08-10T06:59:03Z

+        Which breakpoint slots hold a real breakpoint — ``True`` where one
+        exists, ``False`` where it is absent.  Shaped like the breakpoint
+        arrays (entity dims × ``_breakpoint``), or anything that broadcasts
+        against them.
+
+        Needed only for **ragged** curves, where entities have different
+        numbers of breakpoints.  These are stored densely with the surplus
+        slots left absent, and under v1 that absence has to be declared
+        rather than read off the data: a shorter curve and a stray NaN look
+        identical, so linopy refuses to guess (see the convention, §4/§5).
+        For breakpoints that are already NaN-padded, the declaration is
+        ``mask=x_pts.notnull()``.
+
+        Slots marked absent are excluded from the formulation: no auxiliary
+        variable is created for them and no constraint references them.
+        Passing a mask that hides a *present* value is allowed and drops
+        that breakpoint; a NaN at a slot the mask calls present is a data
+        error and raises.


should be more compact

Per review on #885: keep docstrings and comments absolute instead of citing convention paragraphs, shorten the _drop_absent docstring to a one-liner, compact the mask parameter docs, and apply the suggested trim of the release-note entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

FabianHofmann · 2026-08-10T07:13:25Z

wonderful! let's merge this

FBumann marked this pull request as ready for review August 9, 2026 15:05

FBumann requested a review from FabianHofmann August 9, 2026 15:05

FabianHofmann reviewed Aug 10, 2026

View reviewed changes

FabianHofmann requested changes Aug 10, 2026

View reviewed changes

FBumann requested a review from FabianHofmann August 10, 2026 07:10

FabianHofmann approved these changes Aug 10, 2026

View reviewed changes

FabianHofmann merged commit 1badc76 into feat/arithmetic-convention Aug 10, 2026
3 checks passed

FabianHofmann deleted the fix/piecewise-ragged-mask branch August 10, 2026 07:13

Navigation Menu

Uh oh!

fix(piecewise): declare ragged breakpoints with mask= instead of inferring NaN - #885

fix(piecewise): declare ragged breakpoints with mask= instead of inferring NaN#885
FabianHofmann merged 2 commits into
feat/arithmetic-conventionfrom
fix/piecewise-ragged-mask

FBumann commented Aug 9, 2026 •

edited

Loading

Uh oh!

FabianHofmann left a comment

Uh oh!

FabianHofmann Aug 10, 2026

Uh oh!

FabianHofmann Aug 10, 2026

Uh oh!

FabianHofmann Aug 10, 2026

Uh oh!

FabianHofmann left a comment

Uh oh!

FabianHofmann Aug 10, 2026

Uh oh!

FabianHofmann Aug 10, 2026

Uh oh!

FabianHofmann Aug 10, 2026

Uh oh!

FabianHofmann commented Aug 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

FBumann commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Scope: five paths, not three

Resulting behaviour

Notes for review

Alternatives considered

Uh oh!

FabianHofmann left a comment

Choose a reason for hiding this comment

Uh oh!

FabianHofmann Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

FabianHofmann Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

FabianHofmann Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

FabianHofmann left a comment

Choose a reason for hiding this comment

Uh oh!

FabianHofmann Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

FabianHofmann Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

FabianHofmann Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

FabianHofmann commented Aug 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

FBumann commented Aug 9, 2026 •

edited

Loading