Engineering

Mutation Testing in CI Without Burning the Budget

Ravinder··8 min read
EngineeringTestingMutation TestingCI/CD
Share:
Mutation Testing in CI Without Burning the Budget

Code coverage tells you which lines your tests execute. It does not tell you whether your tests would catch a bug on those lines. A test that calls a function and asserts nothing contributes 100% line coverage while doing exactly zero verification. This is not a contrived edge case — I've reviewed codebases with 85% coverage where the tests were structured this way, not out of malice but out of a misunderstanding of what coverage measures.

Mutation testing measures something different: whether your tests would detect a change in behavior. It introduces deliberate, small bugs — mutations — into your code and checks whether the test suite fails. A test suite that detects most mutations is genuinely verifying behavior. One that misses most mutations is asserting nothing useful. The problem is that mutation testing is computationally expensive. Run it naively on every commit and you're looking at hours of CI time per PR. This post is about running it selectively enough to be affordable while keeping it meaningful enough to matter.

How Mutation Testing Works

A mutation testing framework applies a set of operators to your production code, creating thousands of "mutants" — program variants that each contain one small bug. Common mutation operators:

  • Replace > with >= or <
  • Negate a boolean condition: if (isValid) → if (!isValid)
  • Remove a method call or statement
  • Replace an arithmetic operator: + → -, * → /
  • Change a constant: 0 → 1, "" → null

Each mutant is compiled and run against your test suite. If the tests pass against the mutant, the mutant "survives" — your tests didn't catch the bug. If the tests fail, the mutant is "killed." The mutation score is the ratio of killed to total mutants.

flowchart TD A[Source code] --> B[Mutation framework] B --> C[Generate N mutants] C --> D{For each mutant} D --> E[Run test suite] E --> F{Tests fail?} F -->|Yes| G[Mutant killed\n✓ tests effective] F -->|No| H[Mutant survived\n✗ tests weak] G --> I[Increment killed count] H --> J[Log surviving mutant\nfor review] I --> D J --> D D --> K[Mutation score =\nkilled / total] K --> L{Score ≥ threshold?} L -->|Yes| M[CI passes] L -->|No| N[CI fails\nReport surviving mutants]

A mutation score of 80% is a reasonable starting threshold for well-tested business logic. Below 60% and you should question whether the tests provide any real protection.

Tooling: PIT for JVM, Stryker for JS/TS

PITest (PIT) is the standard for JVM languages. It's fast by JVM standards — it runs mutants in memory, not as separate processes.

<!-- pom.xml -->
<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.16.1</version>
    <configuration>
        <targetClasses>
            <param>com.example.payments.*</param>
            <param>com.example.pricing.*</param>
        </targetClasses>
        <targetTests>
            <param>com.example.*Test</param>
        </targetTests>
        <mutators>
            <mutator>STRONGER</mutator>
        </mutators>
        <outputFormats>
            <outputFormat>HTML</outputFormat>
            <outputFormat>XML</outputFormat>
        </outputFormats>
        <timestampedReports>false</timestampedReports>
        <mutationThreshold>80</mutationThreshold>
        <coverageThreshold>85</coverageThreshold>
    </configuration>
</plugin>

Run PIT:

mvn test-compile org.pitest:pitest-maven:mutationCoverage

Stryker handles JavaScript, TypeScript, and several other ecosystems:

// stryker.config.json
{
  "packageManager": "npm",
  "reporters": ["html", "clear-text", "json"],
  "testRunner": "jest",
  "coverageAnalysis": "perTest",
  "mutate": [
    "src/payments/**/*.ts",
    "src/pricing/**/*.ts",
    "!src/**/*.spec.ts",
    "!src/**/*.test.ts"
  ],
  "thresholds": {
    "high": 80,
    "low": 60,
    "break": 50
  },
  "timeoutMS": 30000,
  "concurrency": 4
}
npx stryker run

The Problem: Naive Mutation Testing Is Slow

A medium-sized Java service with 50 KLOC and 2,000 tests might generate 15,000 mutants. Running each against the full test suite at 30 seconds per run: 125 hours. Even with parallelism and PIT's in-process optimization, you're looking at 30–60 minutes for a full run on useful hardware. That's not CI-compatible.

The solution is incremental, selective, and risk-weighted mutation testing.

Incremental: Only Test Changed Code

The most impactful optimization: only mutate the code that changed in the current PR. If you changed PaymentProcessor.java, only generate mutants for that file and its direct dependents.

PIT supports this natively via the targetClasses parameter. Combined with a script that derives the changed classes from git:

#!/bin/bash
# scripts/changed-classes.sh
# Returns comma-separated PIT class patterns for files changed vs main
 
git diff --name-only origin/main...HEAD \
  | grep '\.java$' \
  | sed 's|src/main/java/||' \
  | sed 's|/|.|g' \
  | sed 's|\.java$|.*|' \
  | tr '\n' ',' \
  | sed 's/,$//'
# .github/workflows/mutation.yml
jobs:
  mutation-incremental:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # needed for git diff
 
      - name: Determine changed classes
        id: classes
        run: |
          CLASSES=$(bash scripts/changed-classes.sh)
          echo "classes=$CLASSES" >> $GITHUB_OUTPUT
 
      - name: Run incremental mutation testing
        if: steps.classes.outputs.classes != ''
        run: |
          mvn org.pitest:pitest-maven:mutationCoverage \
            -DtargetClasses="${{ steps.classes.outputs.classes }}" \
            -DmutationThreshold=75

For a typical PR touching 3–5 files, this generates 200–500 mutants instead of 15,000. Runtime drops from 45 minutes to under 3 minutes.

For Stryker, the equivalent:

# Get changed files from git
CHANGED=$(git diff --name-only origin/main...HEAD | grep -E '\.(ts|js)$' | grep -v '\.spec\.' | grep -v '\.test\.')
 
# Write to stryker config override
cat > stryker-incremental.config.json <<EOF
{
  "extends": "./stryker.config.json",
  "mutate": [$(echo "$CHANGED" | sed 's/^/"/;s/$/"/' | paste -sd,)]
}
EOF
 
npx stryker run --config stryker-incremental.config.json

Risk-Weighted Scoring

Not all code is equally critical. A mutation in your payment processing logic that survives is far more alarming than one in your admin dashboard formatting code. Risk-weighting means applying different thresholds to different modules.

In PIT, configure multiple executions with module-specific thresholds:

<executions>
    <execution>
        <id>payment-critical</id>
        <goals><goal>mutationCoverage</goal></goals>
        <configuration>
            <targetClasses>
                <param>com.example.payments.*</param>
                <param>com.example.billing.*</param>
            </targetClasses>
            <mutationThreshold>85</mutationThreshold>
        </configuration>
    </execution>
    <execution>
        <id>admin-standard</id>
        <goals><goal>mutationCoverage</goal></goals>
        <configuration>
            <targetClasses>
                <param>com.example.admin.*</param>
                <param>com.example.reporting.*</param>
            </targetClasses>
            <mutationThreshold>65</mutationThreshold>
        </configuration>
    </execution>
</executions>

For a more dynamic approach, maintain a risk registry in code:

# tools/mutation_thresholds.py
RISK_THRESHOLDS = {
    "src/payments": 85,
    "src/auth": 85,
    "src/billing": 80,
    "src/inventory": 75,
    "src/reporting": 65,
    "src/admin": 60,
}
 
def get_threshold(file_path: str) -> int:
    for prefix, threshold in RISK_THRESHOLDS.items():
        if file_path.startswith(prefix):
            return threshold
    return 70  # default

Baseline Scores and Score Ratcheting

A score threshold on a fresh codebase is simple. On a legacy codebase with existing weak tests, you can't suddenly require 80% — you'll fail every PR. Instead, establish a baseline and ratchet up over time.

Store the current mutation score in a file committed to the repo:

// .mutation-baseline.json
{
  "overall": 67.3,
  "by_module": {
    "src/payments": 82.1,
    "src/billing": 74.5,
    "src/reporting": 58.2
  },
  "recorded_at": "2025-11-01",
  "target_date": "2026-03-01",
  "target_score": 80.0
}

CI enforces that the score never regresses below the baseline:

#!/usr/bin/env python3
# scripts/check_mutation_score.py
import json
import sys
 
with open(".mutation-baseline.json") as f:
    baseline = json.load(f)
 
with open("target/pit-reports/mutations.json") as f:
    results = json.load(f)
 
current_score = results["mutationScore"]
baseline_score = baseline["overall"]
 
if current_score < baseline_score - 1.0:  # 1% tolerance for statistical noise
    print(f"Mutation score regressed: {current_score:.1f}% < baseline {baseline_score:.1f}%")
    sys.exit(1)
 
print(f"Mutation score: {current_score:.1f}% (baseline: {baseline_score:.1f}%)")

Update the baseline quarterly via a deliberate PR, not automatically — the score should only move up intentionally.

Debugging Surviving Mutants

When a mutant survives, the report tells you exactly what was changed and which tests ran against it. That's your diagnostic:

Survived mutation in PaymentProcessor.java line 87:
  Original:  if (amount.compareTo(MINIMUM) >= 0) {
  Mutant:    if (amount.compareTo(MINIMUM) > 0) {
  Tests run: PaymentProcessorTest#testValidAmount, PaymentProcessorTest#testNegativeAmount
  Tests that would catch it: NONE

The surviving mutant on line 87 tells you: no test covers the case where amount equals MINIMUM exactly. Add that test:

@Test
void testAmountAtMinimumBoundaryIsValid() {
    Money minimumAmount = Money.of(MINIMUM_AMOUNT, "USD");
    assertTrue(processor.isValidAmount(minimumAmount));
}
 
@Test
void testAmountBelowMinimumIsInvalid() {
    Money belowMinimum = Money.of(MINIMUM_AMOUNT.subtract(ONE_CENT), "USD");
    assertFalse(processor.isValidAmount(belowMinimum));
}

The surviving mutant guided you to exactly the boundary condition you'd missed. This is mutation testing's highest value: it doesn't just tell you that tests are weak, it tells you precisely what to test next.

Full vs. Incremental: When to Run Each

# Full run on scheduled basis (weekly, not per-commit)
name: Full Mutation Score
on:
  schedule:
    - cron: '0 2 * * 0'  # Sunday 2am
 
jobs:
  full-mutation:
    steps:
      - run: mvn org.pitest:pitest-maven:mutationCoverage
      - name: Update baseline if improved
        run: python scripts/update_baseline.py
 
# Incremental run on every PR touching source files
name: Incremental Mutation
on:
  pull_request:
    paths:
      - 'src/main/**'
 
jobs:
  incremental-mutation:
    steps:
      - run: bash scripts/incremental_mutation.sh

The weekly full run catches systemic drift. The incremental PR run catches regressions in the code being changed. Together they give you the coverage of full mutation testing at a fraction of the cost.

Key Takeaways

  • Mutation testing measures test effectiveness, not test presence — a 90% coverage suite with weak assertions may have a 40% mutation score, and that 40% is closer to the truth.
  • Never run full mutation testing on every PR in a mid-size or larger codebase; run it incrementally on changed code and schedule full runs weekly or per-release.
  • Apply risk-weighted thresholds so critical modules (payments, auth) require higher mutation scores than low-stakes modules; a uniform threshold creates the wrong incentives.
  • Use surviving mutants as a prioritized list of missing tests — each surviving mutant pinpoints a specific condition your test suite ignores.
  • Establish a baseline score on legacy codebases and enforce ratcheting (never regress) rather than an absolute threshold that would fail every existing PR.
  • PIT for JVM and Stryker for JS/TS are production-ready; the in-process execution model of PIT makes it practical for CI where an out-of-process approach would be too slow.
Share: