Attempt to reducing CI strain through compile time

[borisdevos]

Description

I first considered splitting symmetric and non-symmetric code, but then many files would be super tiny. The next best thing I thought of was the finite-infinite split. This is in no way definitive, just wanted to get the ball rolling on considering this, especially before merging #509 which is ridiculously expensive.

Summary:

• Split the test files according to finite, infinite, window, mixed, gpu or misc testing. The workers will then work in phases, so whatever they compile within a phase is useful when they're done with a file and move on to the next file within the same phase.
• Introduced fast tests, skipping certain symmetries and scalar types. I deliberately didn't skip at the algorithmic level, but this can be considered.

None of the content or hyperparameters of tests have been changed, it's really just copy-paste, introducing some lists and truncating them depending on fast_tests, or skipping certain test sets based on this.

The downside to this approach is that all checks will take longer to finish, but at least they'll finish instead of time out. And who knows, maybe the time saved on compiling might be enough to be comparable with full parallelisation.

Checklist

• Tests pass locally (julia --project=test test/runtests.jl, or the relevant subset)
• Documentation updated, if this PR changes public API (docstrings, docs/src/)
• Runic formatter is run
• Changelog entry added under [Unreleased] in docs/src/changelog.md, if this PR is user-facing (new feature, behavior change, bug fix, deprecation, or removal)

[lkdvos]

I'd be somewhat surprised if this helps too much, it might be worth it to measure where the actual compilation is happening but I'd expect most of it to be in the tensor kernels, which are largely shared between finite and infinite implementations. A different thing could be to just split up the tests into more groups, i.e. just use more runners, for example splitting off time evolution algorithms from groundstate/leading boundary algorithms etc

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[borisdevos]

I'd be somewhat surprised if this helps too much, it might be worth it to measure where the actual compilation is happening but I'd expect most of it to be in the tensor kernels, which are largely shared between finite and infinite implementations. A different thing could be to just split up the tests into more groups, i.e. just use more runners, for example splitting off time evolution algorithms from groundstate/leading boundary algorithms etc

Yeah, testing both fast and normal mode, it seems that there's no global speedup this way. I'm looking into --trace-compile and SnoopCompile.jl to see if I can more literally catch what's taking up compile times. I'll also check how beneficial QuantumKitHub/TensorKit.jl#487 is here afterwards.

[lkdvos]

Good point, I almost forgot about that PR. I'll try and take that back up, since I do think that should help out quite severely.

[borisdevos]

I haven't tested yet with TensorKit precompilation, but at least within MPSKit the compilation bottlenecks are (unsurprisingly) eigsolve/linsolves specialised to some operator types, followed by the actual construction of some of these operators like JordanMPOTensor and JordanMPO_AC(2)_Hamiltonian. Other constructors like (in)finite hamiltonians or states also contribute, but comparatively less. I'll look now at how the tests perform with TensorKit precompilation.

[borisdevos]

So I have 3 data points on running 2 of the compile-heaviest tests (MPOHamiltonian and changebonds) with and without TensorKit precompilation. I did this on the MPSKit main branch, so whatever (minimal or no) effects this branch currently has plays no role. The run-to-run variance is massive, but TensorKit precompilation does help. This being a statistical coincidence is (1/2)^6 so I think it's safe to say that it's beneficial to precompile.

It is fairly modest on my machine though. It fluctuates between 3-17% speedup, on average ~9% which is about 4 minutes per file. Given that, I believe that the best thing to do is to set up precompilation within MPSKit as well. Afterwards we can see if test splitting can fill in remaining gaps.

[lkdvos]

I'm not entirely sure what you measured of course (did you include the tensorkit precompilation time?), but there is the additional effect that sometimes the CI can actually cache the TensorKit precompilation entirely, in which case this is more or less a free speedup.

[borisdevos]

No, indeed my first run did have an additional cost in precompiling TK (15 minutes on my machine, though I think it was under strain from other tasks), so with the small amount of tests I did it's more or less break-even. But it's clear when running all the tests multiple times, its effects are more and more prominent.

I ran algorithms/changebonds and operators/mpohamiltonian 3 times each, the first with --jobs=3 and the last with 1. The first had more jobs because I ran the entire testsuite, but that took the entire night so I cut it down to whatever 2 files that first run took the longest to complete.