perf: improved bench settings performance

[alejandro-vaz]

this PR customizes the compile-time profiles in Cargo.toml to improve performance

on release

it sets opt-level to 3 by default

this doesn't increase compile-time noticeably and yields fairly consistent 2-10% performance improvements in benchmarks

on bench

it enables LTO, sets codegen units to one, strips debug info, and sets opt-level to three

letting the compiler do actual optimization allows us to judge whether codegen has really improved and smallvec is getting faster rather than performance changes being a byproduct of how the compiler decided to transform the code

closes #515

[fereidani]

Hey, opt-level 3 is default for release, there is no need to explicitly set it.

[alejandro-vaz]

alright I did not know that, I thought it was 2

but it's really weird, I'm getting this now that I removed it

bench_insert_push_vec_small
                        time:   [35.907 ns 36.200 ns 36.682 ns]
                        change: [+11.324% +15.800% +24.016%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
  3 (3.00%) low mild
  2 (2.00%) high mild
  6 (6.00%) high severe

bench_insert_vec        time:   [892.71 ns 918.92 ns 953.64 ns]
                        change: [+10.098% +14.846% +20.985%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
  3 (3.00%) high mild
  8 (8.00%) high severe

bench_insert_vec_small  time:   [96.646 ns 97.135 ns 97.700 ns]
                        change: [+1.4180% +2.3982% +3.3770%] (p = 0.00 < 0.05)
                        Performance has regressed.

bench_remove_vec        time:   [766.88 ns 778.96 ns 790.59 ns]
                        change: [−1.1449% +0.7903% +2.6792%] (p = 0.42 > 0.05)
                        No change in performance detected.

bench_remove_vec_small  time:   [65.976 ns 66.783 ns 67.506 ns]
                        change: [+3.3733% +4.7989% +6.1877%] (p = 0.00 < 0.05)
                        Performance has regressed.

bench_extend_vec        time:   [52.294 ns 52.495 ns 52.676 ns]
                        change: [+4.6924% +5.5028% +6.2511%] (p = 0.00 < 0.05)
                        Performance has regressed.

bench_extend_vec_small  time:   [12.284 ns 12.329 ns 12.394 ns]
                        change: [+13.594% +13.909% +14.261%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)

and more

bench_push_vec          time:   [233.56 ns 234.56 ns 235.83 ns]
                        change: [+10.001% +10.450% +10.893%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe

bench_push_vec_small    time:   [28.277 ns 28.393 ns 28.534 ns]
                        change: [+7.6177% +8.1634% +8.7005%] (p = 0.00 < 0.05)
                        Performance has regressed.

bench_insert_push_vec   time:   [284.12 ns 284.61 ns 285.20 ns]
                        change: [+14.130% +14.698% +15.245%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

bench_insert_push_vec_small
                        time:   [35.907 ns 36.200 ns 36.682 ns]
                        change: [+11.324% +15.800% +24.016%] (p = 0.00 < 0.05)
                        Performance has regressed.

I haven't touched anything else

[alejandro-vaz]

honestly I'm rerunning benchmarks now a bit better at it seems that without modifying any code I get 2-10% magic jumps either for better or worse

[alejandro-vaz]

yeah it's weird, I'm going to remove the release one

[alejandro-vaz]

I've been rerunning benchmarks without modifying absolutely anything a few times, and benchmark precision doesn't seem to be great

bench_push              time:   [315.24 ns 315.65 ns 316.10 ns]
                        change: [+5.3084% +5.7077% +6.0741%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 20 outliers among 100 measurements (20.00%)
  15 (15.00%) low mild
  3 (3.00%) high mild
  2 (2.00%) high severe

bench_push_small        time:   [44.366 ns 44.872 ns 45.341 ns]
                        change: [−4.6883% −3.9200% −3.0034%] (p = 0.00 < 0.05)
                        Performance has improved.

bench_insert_push       time:   [323.82 ns 324.59 ns 325.49 ns]
                        change: [+0.3717% +0.5977% +0.8277%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 16 outliers among 100 measurements (16.00%)
  4 (4.00%) high mild
  12 (12.00%) high severe

bench_insert_push_small time:   [45.726 ns 45.780 ns 45.834 ns]
                        change: [+0.2306% +0.4858% +0.7626%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

that's after repeated runs of the same code with zero changes nor anything

it's wild

it looks like it depends on codegen or who knows what

not sure I can trust this

I've also gotten

bench_remove_vec        time:   [812.58 ns 814.49 ns 816.26 ns]
                        change: [+13.551% +14.556% +15.510%] (p = 0.00 < 0.05)
                        Performance has regressed.

with no changes. it's totally unreliable or I'm doing something very wrong

[fereidani]

Increase benchmark time to 1500-5000ms and retest.

.measurement_time(Duration::from_millis(1500));

• cargo clean before every run