FIBERALLOC-62: Add a HiGHS solver backend - #32
Open
monodera wants to merge 9 commits into
Open
Conversation
HighsProblem(LPProblem), reached through highspy, as an open-source alternative to Gurobi that needs no licence. Benchmarked on 22 real target lists from the PFS target uploader: HiGHS finished the same 20 of them Gurobi did, at 1.08x the total runtime, with pointing counts agreeing to within the spread a single solver shows across repeated runs of the same input. Additive throughout. buildProblem() gains solver= and solverOptions=, both defaulting to None, and the existing `gurobi` flag keeps selecting between Gurobi and PuLP whenever solver is not given -- so callers that do not pass it take exactly the path they took before. GurobiProblem and PulpProblem are untouched. Two implementation notes, both measured rather than assumed: Columns are created in one batch. Adding them individually through highspy costs ~50 us each, minutes of overhead on the million-variable problems this module builds; addCols takes the batch at once and measures ~180x faster. buildProblem() creates every variable before its first constraint, so a single deferred flush catches all of them. Solutions are read from one cached vector. Highs.val() recomputes per call at O(numCol) -- 295 us per variable on a 20k-column model, 1083 us on an 80k one -- so reading a solution back variable by variable is quadratic, and needs about an hour on a 500k-column problem whose solve takes 20 s. getSolution() costs a millisecond, once. HiGHS has no lazy-constraint hint, so add_lazy_constraint() adds an ordinary constraint. That costs nothing here: the collision constraints are built up front rather than generated in a callback, and marking them lazy for a backend that does support it left both runtime and objective unchanged on a 1.7M-variable instance. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YKQTw7ZXAZDjgKoaigTGQA
A Gurobi variable has no value to read when the solve failed, so the caller notices at once. HiGHS instead hands back an all-zero column vector for an infeasible or unsolved model, which is indistinguishable from a feasible solution that assigns nothing -- callers reading the solution back would silently treat the failure as an empty assignment. Check getModelStatus() in solve() and raise unless the model was solved to optimality or a limit stopped the search after an incumbent had been found, matching what PulpProblem already does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comment described a binary/integer split the code does not make: every column is created integral, and a 0/1 range is simply bounded to [0, 1]. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- _cacheSolution's try/except never caught anything: HiGHS returns an all-zero vector rather than raising when there is no solution, and the status check added to solve() is what rules that case out. Drop it and say so, instead of claiming value() will fall back to val(). - varBounds() now reports floats whatever the caller passed in, matching what the Gurobi and PuLP backends return. - Note that `name` is accepted only for interface parity: highspy exposes no model-name API to set it on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # ets_fiber_assigner/netflow.py
The prerequisites still described the solver choice as PuLP versus Gurobi. Add HiGHS to that list so the third backend is discoverable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016hLuRYpDagE8nVbkGMm96k
HighsProblem.add_constraint used to hand every row to Highs.addConstr, about 19 us per row (highspy's unique_elements 9.5, addRow 4.8, passRowName 0.9, wrapper 3), i.e. 10-17 s per million rows on the real instances. Rows are now collected in flat typed arrays (_RowBuffer) and handed to HiGHS in one addRows call at the next flush, ordered within each row like unique_elements does, so the model HiGHS stores and the .lp/.mps it writes are byte-identical to the one-row-at-a-time path. About 2 us per row; a 625k-row synthetic model builds its rows in 1.2 s instead of 12.5 s, a 147k-row real instance in 18.8 s instead of 21.6 s. Column and row names are passed to HiGHS in dump() only (HiGHS's own log refers to rows and columns by index either way). Two behaviours now follow GurobiProblem: value() raises when there is no valid solution -- before solve(), after a failed solve(), or once a change (new columns or rows, changeVarBounds) has been applied by update(), dump() or solve(); a still-pending change leaves the previous solution readable, as a pending Gurobi modification leaves var.X. changeVarBounds itself is pending until the next flush, like a Gurobi bound change until update(). dump() before solve() writes an empty objective, as GurobiProblem.dump() does. Docstring lists the HiGHS options relevant here with their defaults, notes that varBounds/changeVarBounds are instance methods on this backend, and that highspy >= 1.15 is required. tests/test_highs_problem.py (pytest) checks the behaviour, identity with highspy's own addVariable/addConstr path, known optima, and agreement with GurobiProblem (skipped without gurobipy). misc/bench_highs_problem.py is the synthetic benchmark; misc/compare_highs_real_instance.py compares solver="gurobi" and solver="highs" through buildProblem on a real instance. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://pfs-jira.naoj.org/jira/browse/FIBERALLOC-62
Adds HiGHS (through
highspy) as a third solver backendfor the netflow MILP, alongside Gurobi and PuLP — an open-source option that
does not require a Gurobi licence.
Changes
HighsProblembackend (ets_fiber_assigner/netflow.py), following thesame
LPProbleminterface asGurobiProblemandPulpProblem. Columns arereserved on
addVar()and created in a single_flush()batch, becauseadding them one at a time through
highspycosts ~50 us each.solve()raises when no solution was produced. Unlike Gurobi, HiGHShands back an all-zero column vector for an infeasible or unsolved model,
which is indistinguishable from a feasible solution that assigns nothing;
the model status is now checked so callers cannot silently read a failure
as an empty assignment.
buildProblem()parameterssolverandsolverOptions.solver="gurobi" | "pulp" | "highs"selects the backend, andsolverOptionspasses options in that backend's own parameter names.When
solverisNonethe existinggurobiflag behaves exactly asbefore, so current callers are unaffected.
highspyis imported lazily insideHighsProblem.__init__, the same waygurobipyandpulpare, so it is only needed when that backend is used andno new hard dependency is introduced.
Update (commit 8043410): batched rows, Gurobi-consistent read rules, tests
add_constraint()used to hand every rowto
Highs.addConstr, ~19 us per row (highspy'sunique_elements9.5,addRow4.8,passRowName0.9, wrapper 3), i.e. 10-17 s per million rows.Rows are now collected in flat typed arrays (
_RowBuffer) and passed toHiGHS in one
addRowscall at the next flush, ordered within each row theway
unique_elementsorders them, so the stored matrix and the written.lp/.mpsare byte-identical to the one-row-at-a-time path. ~2 us perrow: a 625k-row synthetic model builds its rows in 1.2 s instead of 12.5 s,
a 147k-row real instance (20000 targets, 4 pointings, cobraOps Bench) in
18.8 s instead of 21.6 s, with identical objective and assignment.
dump(). HiGHS's own log refers to rows andcolumns by index whether or not they are named, so nothing is lost; to see
names, read the file
dump()writes.GurobiProblem.value()raises when there is novalid solution: before
solve(), after a failedsolve(), or once a change(new columns or rows,
changeVarBounds) has been applied byupdate(),dump()orsolve(); a still-pending change leaves the previous solutionreadable, as a pending Gurobi modification leaves
var.X.changeVarBounds()itself is pending until the next flush, like a Gurobibound change until
update().dump()beforesolve()writes an emptyobjective, as
GurobiProblem.dump()does.(
mip_rel_gap1e-4 equals Gurobi'sMIPGapdefault), notes thatvarBounds/changeVarBoundsare instance methods on this backend (callingthem through the class does not work), and that highspy >= 1.15 is required
(module-level
highs_var).tests/test_highs_problem.py(pytest, 27 tests, < 1 s):backend behaviour, identity with highspy's own
addVariable/addConstrpath, known optima, and agreement with
GurobiProblemon the same models(skipped without gurobipy; the pip wheel's size-limited licence suffices).
misc/bench_highs_problem.pyis the synthetic benchmark used for thenumbers above;
misc/compare_highs_real_instance.pybuilds one realinstance with
solver="gurobi"andsolver="highs"throughbuildProblemand compares model content, objective and assignment (identical on a
300-target instance).
Benchmark
Benchmarked on 22 real target lists: HiGHS finished the same 20 that Gurobi
did, at 1.08x the total runtime, with pointing counts agreeing to within the
spread a single solver shows across repeated runs of the same input.
Notes
separately on release.
values with
> 0, which is not tolerance-safe for the near-integral doublesa MIP solver returns. The demo scripts are considered obsolete, so no fix is
planned for now.
🤖 Generated with Claude Code