The compiler needs a test stand

Say you have built a language. The compiler compiles, the demos run, and now you want a number you could show a stranger without flinching. A tight loop will not get you one — every optimizer looks good in a loop — because the question a stranger actually cares about is what happens when strings, allocation, error paths, separate compilation, and library policy all arrive in the same executable.

So treat it as candidate elimination. Five familiar programs go into the filter, and four come out labeled with the specific way they would ruin the measurement.

Eliminate the projects around the compiler#

The eliminations teach more than the winner does. Port SQLite and you are testing whether a young language can express a storage engine — a fine question, and years too early. zstd carries a decade of hand tuning that would dominate anything the compiler contributes, ripgrep leans on a regex and filesystem stack nobody has rewritten yet, and jq smuggles in an entire second runtime. A slow port of any of them indicts the replacement libraries louder than the code generator.

cmark survives because its edges are unusually clean: standard C, no required external library, a document tree with multiple renderers, and an official specification suite that judges observable output. It still exercises none of floating point, concurrency, sockets, or service lifetime, so a good score here settles less than it will be tempting to claim.

The specification suite is what keeps everyone honest. A candidate cannot get "faster" by accepting a smaller Markdown dialect or emitting approximately similar HTML, because the CommonMark examples and byte-for-byte renderer comparison settle correctness before any clock starts. A crash is a failed run, never a fast sample.

Freeze the cmark contract#

Pin cmark commit 9562204. Its benchmark document already ships a 1,106 KB corpus assembled from the first edition of Pro Git, and the same commit's pathological suite includes 25,000 nested opening brackets with their matching closers. Steady-state parsing and hostile structure, both in the repository — nobody gets to invent a private fixture.

One more line has to be drawn before timing anything: the faithful port keeps the C source's algorithms, data structures, and observable behavior. An idiomatic rewrite is worth measuring too — I suspect it flatters the language, and that suspicion is exactly what a second scoreboard exists to test — but it answers a different question and must not inherit the faithful port's label.

The line matters because good language features make excellent benchmark cheats. Swap cmark's manual allocation for a tracing collector and you have changed the program's memory behavior. A library parser goes further and replaces the measured program outright. Record those as their own candidates, proudly, on their own row.

set -eureference=./bin/cmark-ccandidate=./bin/cmark-portfor markdown in corpus/*.md; do  "$reference" --to html "$markdown" > /tmp/reference.html  "$candidate" --to html "$markdown" > /tmp/candidate.html  cmp /tmp/reference.html /tmp/candidate.htmldonetaskset -c 2 hyperfine --warmup 3 --runs 10 \  "$reference --to html corpus/progit.md" \  "$candidate --to html corpus/progit.md"size "$reference" "$candidate"

The record that survives names everything that could explain a difference: both commits, compiler revision and flags, the CPU and its pinning, allocator, warm-up count, raw samples, output hashes, and the tools that measured memory and size. LLVM's test-suite guide keeps runtime, compile time, code size, and reference output as separate measurements for the same reason — an aggregate score is where the regression you came to find goes to hide.

No port has been timed for this essay, which is the deliberate anticlimax: the contract is the deliverable. Meanwhile the least glamorous input sits in test/pathological_tests.py — 25,000 left brackets, one a, 25,000 right brackets — waiting for every implementation that copies a substring on each split.