Performance regression checks

TABASCAL’s runtime and memory use are guarded by ReFrame checks in ci/reframe/tabascal_perf_check.py. The checks themselves are system-generic — they declare a generic:default partition and run anywhere ReFrame does, a dev machine included (see Running the checks directly); it is the reference values that belong to a node type. In this project’s CI they run on, and their references were measured on, the CSCS Daint GH200 partition defined in ci/cscs.yml.

They complement the pipeline tests: the pipeline tests catch a change in what TABASCAL infers, these catch a change in how long it takes.

What is checked

Two test classes, each parameterised over variant (Riemann, RiemannFFI) and precision (single, double):

TabascalPerfCheck — single GPU.
TabascalMultiGpuPerfCheck — every GPU on the node. It unsets the CUDA_VISIBLE_DEVICES=0 default that the rest of the pipeline pins, so it exercises the sharded solve.

The metrics compared against references are total_runtime, optimizer_runtime and memory_usage. Metrics without a reference entry are reported but not asserted.

The Riemann references describe a deliberate trade rather than a target: that component bounds its peak memory by scanning the baseline axis and recomputing each block in the backward pass (see baseline_block_size), which on the GH200 references above is roughly an order of magnitude less memory for an optimiser step 1.2 to 1.7 times longer. A run that comes out faster and hungrier than these numbers has probably lost the scan, not gained a speed-up.

References are keyed by (variant, precision) and are only meaningful on the node type they were measured on. Sanity therefore requires the run to have used exactly the expected number of GPUs of exactly the expected kind (GH200) — anything else fails rather than being silently compared against timings it cannot be compared against. That gate is on the comparison, not on running the checks: elsewhere, run them with -S strict_check=0 (below) to get the metrics reported without being asserted.

Reproducing a regression locally

When CI reports a regression, reproduce it on a smaller dataset with 8 antennas. This is fast enough to simulate and run on any dev machine, so you do not need GH200 access to bisect the cause:

# Generate an 8-antenna simulation from the standard 96A config
sim-vis -c ci/reframe/data/sim_target_96A.yaml -a 8

# Run tabascal with timing output against the generated dataset
tabascal run -c ci/reframe/data/tab_target.yaml -od ci/reframe/data/data/pnt_src_obs_08A_090T-0000-0890_001I_001F-1.500e+08-1.500e+08_050PAST_000GAST_000EAST_32SAT_0GRD_1.0e+00RFI -t

The -t flag prints a per-function timing table identical to the CI output.

The absolute numbers from an 8-antenna run on a laptop are not comparable to the GH200 references — use it to find which function got slower, then confirm the magnitude on the real configuration.

Running the checks directly

With the dev environment installed (see Developer install), ReFrame is available as reframe:

reframe -C ci/reframe/settings.py \
        -c ci/reframe/tabascal_perf_check.py \
        --system=generic:default \
        --exec-policy=serial \
        --run --performance-report

Add -S strict_check=0 to report the metrics without failing on the reference comparison, which is what the nightly cron job does.

Updating the references

A reference change and a regression look identical in the diff, so treat the references the same way as the pipeline test references: when a change intentionally alters performance, re-measure rather than widening the tolerances, and record in the commit message why the numbers moved, with the before/after values.

The comments in tabascal_perf_check.py are part of that record. For example the current values note that #103 replaced the real-space rfi_signal:ComplexRFI with the scanned Fourier ComplexRFIVarAnt, a deliberate accuracy-for-runtime trade (see issue #107) that made the optimiser slower where the RFI-signal component dominates the step and slightly faster where RFI-vis does. Those older, faster numbers describe a component that no longer exists and must not be “restored”.

Benchmark history

The nightly cron run converts the ReFrame report with ci/reframe/reframe_to_bmf.py and uploads it to bencher.dev under the tabascal project, testbed cscs-daint-gh200. The gpu_mode variable keeps the single- and multi-GPU series apart: anything other than single is appended to the benchmark name, so the existing single-GPU series keep their historical names.