Invariant: When the Tests Pass but the ML Evaluation Is Wrong

Your code compiles. Your tests pass. Your results can still be wrong.

During the IBM Bob 2.0 Hackathon in September 2026, I built Invariant, a code-review tool for a particular kind of machine-learning error: an evaluation that looks valid in code but breaks when the data is actually split. Its first check measures whether the same patient appears in both the training and test sets. It records the result at a specific commit, blocks a pull request when it finds leakage, and helps the reviewer verify the repair.

I built it around a fully synthetic, NSCLC-inspired imaging benchmark. Every patient ID, label, and paired CT-like/PET-like 2D image is procedurally generated. This is an engineering demonstration, not a clinical model or a result about real patients.

The failure I wanted to make visible

Medical-imaging datasets often contain several images from each patient. If we randomly assign images to training and test sets, images of the same patient can land in both. A model may then exploit patient-specific patterns, and a high test score no longer tells us how well it generalizes to unseen patients.

In code, the mistake is one reasonable-looking line:

# Splits rows (images), not patients. Balances labels, but not patients.
train_df, test_df = train_test_split(df, test_size=0.2, stratify=df["label"])

The uncomfortable part is that nothing necessarily crashes. A row-level split produces the expected number of training and test samples. Data-loading tests pass. Training completes. A metric appears in a report. None of those facts establishes patient independence, and a reviewer reading the diff has no way to see the problem.

In my sample project, the existing baseline tests were green on the buggy pull request. When Invariant ran the split against the project's own metadata, it found 166 patients in both training and test: all 166 test patients. That concrete measurement, rather than a model's opinion about suspicious code, became the basis for the review.

What Invariant does

Invariant joins four checks into a pull-request workflow:

1. Inspect the split: locate the split operation and flag static risk indicators, such as assigning rows without a patient group key.
2. Measure overlap: execute the configured split on the project's metadata, validate the partition, and count patients present in both sets.
3. Assess the guard: look for a recognized patient-disjointness assertion in the tests.
4. Run required tests: execute the regression tests selected by the project's invariant.toml.

Static inspection gives the reviewer a place to look. The executed split supplies the finding. A separate, deterministic report builder runs the checks itself and derives one of three statuses—blocked, review_required, or no_findings—from the current commit's results. An invalid partition cannot masquerade as a zero-overlap pass.

The builder writes a Validity Report tied to the target commit, the analyzer commit, and hashes of the protocol, metadata, configuration, and image manifest. It refuses an authoritative report when the target or analyzer working tree is dirty, and each run gets its own report directory. A later fix therefore cannot overwrite the evidence of the original failure. Historical training runs remain visible (11 runs were confirmed affected by the leak), but they do not decide the status of a fixed commit.

This boundary is intentional: Bob explains and fixes; the checks measure; the builder decides the report status. Bob cannot supply the authoritative overlap count, commit SHA, input hashes, or status.

Where IBM Bob fits

I installed a custom ML Reviewer mode and a data-leakage skill into the sample repository with a single install script. Through an MCP server, Bob can read the study protocol, call Invariant's analyzers, and request a report. It distinguishes the invariant this version checks (patient-level independence) from protocol requirements it does not yet check.

The review then moves through investigation and repair:

• Parallel subagents. Two focused explore subagents run side by side: one traces patient identity through the pipeline, while the other assesses whether existing tests guard patient disjointness.
• Fix and tests. Bob changes the split to assign whole patients to training or test, and adds four regression tests, including a Hypothesis property test that generates many small datasets.
• Agent mode. Bob commits the change, pushes it, and opens the fix pull request.

Report creation is never auto-approved, so a human signs off on every report, and Bob's text is labelled in the report as explanation, not evidence. The whole build used 9.52 of my 40 allocated Bobcoins across six Bob tasks. I checked each task against the real repository with expected results written down in advance, and I kept every prompt and every defect I found in docs/bob_tasks.md. My first review attempt failed; its report is still in the repository.

The same report builder runs in the required GitHub Actions check. That matters because a local demonstration would be much less useful if CI made a different decision about the same commit.

The evidence chain: fail, fix, challenge

I structured the demonstration as a three-stage lifecycle across separate pull requests, producing an append-only evidence trail:

Here is why this sequence matters:

• Report A preserves the failure evidence at the buggy head commit, built before any code was changed, in a report directory that later runs never overwrite.
• Report B records that, at the new commit, the configured checks measured zero patient overlap and all regression tests passed.
• The Challenge PR provides the decisive stress test: it verifies that the guard catches the defect if reintroduced. When the faulty split returned, Invariant re-measured 166 overlapping patients and Bob's regression tests failed—halting CI and blocking the merge even while standard baseline tests remained completely green.

What the leak did to the metric

A perfect score was hiding a broken evaluation. These are results from an engineered synthetic benchmark, in which each fictional patient deliberately has a persistent texture a model can memorize. The two split methods also hold out different patients. So the difference illustrates this benchmark's leakage effect; it is not a clinical performance estimate or a controlled causal estimate for a real model.

The reports and a separate comparison artifact link every number to its commit and run source. I wanted a reader to be able to move from a headline number to the specific check and execution that produced it.

What I learned from building it

Keep risk indicators, measured findings and reviewer prose separate. A static pattern can point to a likely leak but cannot prove overlap. A language model can explain why a finding matters but should not invent an evidence value. The executed check establishes what happened on this dataset and commit; the deterministic builder makes the gate decision. Keeping the AI away from the verdict made the system easier to trust, including for me.

Test the gate in both directions. A red report on a deliberately buggy branch is easy to show. A stronger demonstration preserves the red report, verifies a green fix on a new commit, and then proves the check turns red again when the bug returns.

Decide what "correct" looks like before asking the AI. Writing expected results down before each Bob task caught real defects in its output that I might otherwise have accepted.

State scope plainly. Invariant currently checks the study protocol's patient-independence requirement. It lists train-only preprocessing and inference consistency as not_checked in every report. Its test-guard recognition is pattern-based and does not understand every valid assertion style. For example, it does not yet follow two variables that each hold a patient set (a.isdisjoint(b)), which is exactly the form Bob's first assertion used. A no_findings result means the configured checks found no issue on that commit; it is not a declaration that an entire ML system is correct, clinically valid, or regulatorily compliant.

What comes next

The next step is to make the evidence easier to explore. I want an A/B view that opens on the current finding, lets a reviewer inspect the source check and commit bindings, and then shows exactly what changed in the fix. I also want to run Invariant against a second, independently structured ML repository to test how well its configuration and analyzers generalize beyond this benchmark, and to make guard detection follow patient sets held in variables.

Further measured checks could cover train-only preprocessing and evaluation-versus-inference consistency. Each would need its own executable failure case, regression tests, and explicit report coverage. The goal is to expand what Invariant can prove without letting a polished explanation outrun the evidence.

For now, the hackathon prototype demonstrates a complete loop: measure the leak, block the merge, have Bob repair it, verify the new commit, and catch the regression.

Project repositories: Invariant · Synthetic NSCLC-inspired demo

All results in this article come from the synthetic demonstration. No real patient data was used.