MEP 23. Cross-language Baseline Benchmarks

Field	Value
MEP	23
Title	Cross-language Baseline Benchmarks
Author	Mochi core
Status	Draft
Type	Process
Created	2026-05-16

Abstract

MEP 17 gates per-PR performance with Mochi-vs-Mochi regression budgets. It does not answer the question contributors and prospective users actually ask: how does the Mochi VM compare to other small dynamic runtimes. This MEP defines a separate, periodic cross-language baseline that runs hand-written, idiomatic implementations of the same workloads in Mochi, Python, and Lua, and publishes the numbers. It does not gate merges. It tells everyone where the floor is and how far the floor has moved.

The split from MEP 17 is deliberate. MEP 17 measures the VM against itself so optimizations have a target. MEP 23 measures the VM against the world so the project has a story.

Motivation

MEP 17 §Open questions says comparative numbers are "mostly noise" and explicitly leaves them out. That position was right for the per-PR gate (cross-language numbers are too sensitive to host CPU, GC pauses, and idiom drift to merge against), but wrong as a permanent stance on cross-language reporting. Two concrete problems argue for owning a cross-language suite:

• No baseline at all today. A new contributor asking "is the Mochi VM in a reasonable place" has nothing to point at. The first numbers anyone collects are ad-hoc and unreviewed.
• Optimization MEPs have no destination. MEPs 18 through 22 (dispatch, inline caches, value representation, JIT) target an unstated goal. "Close enough to Lua on the math suite" is a goal; "30% faster than today" is a budget. Both matter.

The math benchmarks already shipped under bench/template/math/ are useful for this. Today the runner transpiles Mochi to Python, TS, Go, and C, which measures the transpilers, not Mochi-the-runtime against peer runtimes. To measure peer runtimes we need hand-written, idiomatic implementations in each language that solve the same problem the same way. Those existed for Python (unused by the runner); they are added for Lua in this MEP.

Specification

Scope

The baseline suite is hand-written, idiomatic, problem-equivalent programs. It is not:

• transpiled output (that measures the transpiler)
• micro-benchmarks of single opcodes (those belong to the MEP 18 dispatch work)
• the per-PR gate (that is MEP 17)

The suite covers Mochi (mochi run, VM backend), CPython 3 (python3), and Lua 5.x (lua). Other runtimes (PyPy, LuaJIT, Node, Deno-compiled-TS) may be added as separate columns when there is a story to tell.

Programs

The first cut reuses the existing bench/template/math/ programs. Each program directory holds three files: <name>.mochi, <name>.py, <name>.lua. All three:

• accept the same {{ .N }} template parameter
• perform the same computation with the same control flow
• emit one line of JSON to stdout: {"duration_us": <int>, "output": <value>}
• exclude all I/O and parsing from the timed window; only the inner repeats are timed

Program	What it stresses
fact_rec	recursive call, integer multiply
fib_iter	tight numeric loop, integer add
fib_rec	deep recursion, return-heavy call frames
mul_loop	counted loop with integer accumulator
sum_loop	counted loop with integer accumulator
prime_count	nested loop with branch and modulo
matrix_mul	nested loop over allocated 2D lists

Workload fidelity is enforced by output match: a run that produces a different output field from Mochi for the same N is a workload bug in one of the three implementations, not a perf result. The runner asserts equality of output across the three columns before reporting timing.

Runner integration

The existing bench/runner.go registers two new template kinds:

• native_py, points at <dir>/<name>.py, renders {{ .N }}, executes with python3
• native_lua, points at <dir>/<name>.lua, renders {{ .N }}, executes with lua

These are siblings of the existing mochi_py and mochi_ts rows, which transpile Mochi to those languages and remain in place for the transpiler-focused story. The two perspectives report side-by-side: the transpiled row says "how well does our codegen for language X do", and the native row says "how well does language X do on this problem".

Reporting cadence

This is a periodic baseline, not a per-merge gate. The expected cadence:

• One full run per minor release, attached to the release notes table.
• One full run when an optimization MEP (18 through 22) lands a tier-2 milestone, attached to that PR for context.
• Ad-hoc runs by anyone, anytime; the harness is reproducible by design.

Geometric mean is reported alongside the per-program numbers so the headline number is one ratio per peer language ("Mochi VM is currently 14.7x CPython, 45.2x Lua across the math suite"). Per-program numbers prevent that headline from hiding a pathology in one workload.

Reproducibility

The harness inherits MEP 17's reproducibility rules: pinned MOCHI_NOW_SEED, MOCHI_BENCH=1, CPU model and OS recorded in the run header. Cross-language runs additionally record:

• python3 --version
• lua -v
• go version (for the Mochi VM build)
• the binary path of the Mochi runtime under test (release tag or commit SHA)

Cross-machine comparison is not meaningful and the runner refuses to merge results from different hosts into a single table.

Failure mode

A baseline that drifts is worse than no baseline. The runner treats these as hard failures, not soft warnings:

• A native program that fails to parse or exits non-zero.
• An output field that disagrees with Mochi's output for the same N.
• A missing peer file (<name>.py or <name>.lua not present where <name>.mochi is).

The first failure ends the run with a non-zero exit. Silent skipping is what produced the "we used to be faster than Lua" problem at LuaJIT-Wars-era languages; the project does not repeat that.

Initial baseline

Recorded on the author's machine (Apple Silicon M4, macOS Darwin 24.6.0). Numbers are wall-clock microseconds for the inner repeat loop only. Lower is better. Mochi runtimes: runtime/vm is the long-standing stack VM; runtime/vm2 is the from-scratch register VM landed under MEP-21 v2. Python is CPython 3.14.5. Lua is 5.5.0.

runtime/vm baseline (original)

The original MEP-23 baseline, taken against runtime/vm at the time this MEP was registered.

Program	N	Mochi VM (µs)	CPython (µs)	Lua (µs)	Mochi / CPython	Mochi / Lua
fact_rec	15	8,350	475	177	17.6x	47.2x
fib_iter	30	7,291	445	191	16.4x	38.2x
fib_rec	25	116,925	5,896	2,695	19.8x	43.4x
mul_loop	20	4,866	423	79	11.5x	61.6x
sum_loop	10,000	2,121,548	142,502	31,316	14.9x	67.7x
prime_count	300	213,004	13,966	4,612	15.3x	46.2x
matrix_mul	20	- (broken)	3,570	1,142	-	-

Geometric mean across the six working programs: Mochi (runtime/vm) is 15.7x CPython and 49.7x Lua. matrix_mul is excluded from the headline because the current Mochi runtime produces output: null for it.

runtime/vm2 baseline (MEP-21 v2)

Taken with the cross-language sweep at bench/crosslang (build: go run ./bench/crosslang). Programs fact_rec and mul_loop are capped at n ≤ 16 (resp. n ≤ 17) because larger factorials overflow vm2's 48-bit signed Cell payload; lifting that cap needs boxed-int support in vm2. The remaining programs (fib_iter, fib_rec, prime_count, sum_loop) match Python's and Lua's outputs exactly at every sampled n.

Program	N	vm2 (µs)	CPython (µs)	Lua (µs)	vm2 / CPython	vm2 / Lua	vm2 RSS	CPython RSS	Lua RSS
fact_rec	10	1,514	948	396	1.60x	3.82x	4.5 MB	13.4 MB	2.0 MB
fact_rec	13	1,937	1,203	543	1.61x	3.57x	4.3 MB	13.3 MB	2.3 MB
fact_rec	16	2,146	1,763	592	1.22x	3.62x	4.5 MB	13.5 MB	1.8 MB
fib_iter	10	896	514	210	1.74x	4.27x	4.5 MB	13.5 MB	1.8 MB
fib_iter	20	1,625	996	383	1.63x	4.24x	4.4 MB	13.4 MB	1.9 MB
fib_iter	30	1,920	1,011	420	1.90x	4.57x	4.4 MB	12.3 MB	1.5 MB
fib_rec	15	160	105	53	1.52x	3.02x	4.2 MB	12.2 MB	1.5 MB
fib_rec	20	1,767	1,169	546	1.51x	3.24x	4.1 MB	11.8 MB	1.5 MB
fib_rec	25	17,772	12,605	5,892	1.41x	3.02x	4.1 MB	11.7 MB	1.5 MB
mul_loop	10	391	464	98	0.84x	3.99x	4.1 MB	12.0 MB	1.5 MB
mul_loop	13	420	465	109	0.90x	3.85x	4.1 MB	12.2 MB	1.6 MB
mul_loop	17	549	635	130	0.86x	4.22x	4.1 MB	11.6 MB	1.5 MB
prime_count	50	1,893	1,710	486	1.11x	3.90x	4.3 MB	12.0 MB	1.5 MB
prime_count	100	5,361	4,644	1,438	1.15x	3.73x	4.2 MB	11.9 MB	1.5 MB
prime_count	300	31,416	22,528	7,630	1.39x	4.12x	4.2 MB	11.7 MB	1.5 MB
sum_loop	1,000	23,757	23,498	5,204	1.01x	4.57x	4.1 MB	11.6 MB	1.5 MB
sum_loop	10,000	235,813	240,137	52,325	0.98x	4.51x	4.2 MB	12.2 MB	1.5 MB
sum_loop	100,000	2,200,486	2,387,934	430,488	0.92x	5.11x	4.1 MB	11.6 MB	1.5 MB

Headline: vm2 is 0.84x – 1.90x CPython (often faster on tight integer loops, e.g. mul_loop and sum_loop at large N) and 3.0x – 5.1x Lua. Resident-set size is ~4 MB across all programs, roughly 1/3 of CPython's ~12 MB; Lua holds onto only ~1.5 MB.

The original MEP-23 baseline above measured runtime/vm against the same Python and Lua programs; vm2 cuts the geometric-mean gap to CPython from 15.7x to ~1.3x and the gap to Lua from 49.7x to ~3.9x. The CPython gap is essentially closed; the remaining headline target is Lua. Closing that requires interpreter-loop micro-optimizations (computed-goto-equivalent dispatch, superinstructions for the hot loop bigram, frame-reuse on cross-fn tailcalls) plus boxed-int support so the suite is not capped at 48-bit results.

Methodology note. Two of the original native baselines (bench/template/math/sum_loop/sum_loop.py and mul_loop.py) iterated range(1, n+1) while the Mochi and Lua templates iterated 1..n (exclusive). Both Python files were corrected to range(1, n) in the same PR as this baseline update so all three runtimes now compute identical outputs and the numbers are directly comparable.

The numbers above are the floor on the day vm2 entered the suite. Every subsequent optimization MEP that lands inherits an obligation to move them.

Optimization deltas

Recorded as PRs land. The floor table above is frozen; this section tracks movement against it. Same harness (bench/crosslang), same machine, same day-of-measurement Python and Lua. Per-program N matches the floor's smallest two sizes per program (largest N was dropped per maintainer feedback after the floor was set, since the big rows ran for tens of seconds without changing the relative ranking).

Program	N	vm2 (µs)	CPython (µs)	Lua (µs)	vm2 / CPython	vm2 / Lua
fact_rec	10	773	588	272	1.31x	2.84x
fact_rec	13	921	834	325	1.10x	2.83x
fib_iter	10	424	369	189	1.15x	2.24x
fib_iter	20	552	631	321	0.87x	1.72x
fib_rec	15	152	107	56	1.41x	2.71x
fib_rec	20	1,476	980	460	1.51x	3.21x
mul_loop	10	278	423	98	0.66x	2.84x
mul_loop	13	342	580	125	0.59x	2.74x
prime_count	50	1,616	1,940	475	0.83x	3.40x
prime_count	100	4,266	4,940	1,569	0.86x	2.72x
sum_loop	1,000	21,663	30,658	6,875	0.71x	3.15x
sum_loop	10,000	211,156	313,199	61,128	0.67x	3.45x

Landed in this delta:

• OpTailCall: dropped the argument snapshot. emit.go always stages tailcall args at B = np + ra.NumRegs, which is ≥ np, so the source slice [B..B+n) and the destination param slice [0..n) are guaranteed disjoint. The defensive snapshot in OpTailCall was therefore copying twice for no reason; removing it gave ~28% on sum_loop / fib_iter.
• Fused i64 compare + branch superinstructions. Six new opcodes (OpJumpIfLessI64, OpJumpIfLessEqI64, OpJumpIfGreaterI64, OpJumpIfGreaterEqI64, OpJumpIfEqualI64, OpJumpIfNotEqualI64) collapse the OpLessI64 → OpJumpIfFalse → OpJump triple into one instruction. The emitter picks the form whose fallthrough matches the next layout block so the trailing OpJump is dropped too: 3 instructions → 1 per loop test, plus one fewer bool-register round-trip per iteration.

Headline movement vs the floor: vm2 vs Lua tightened from 3.0x – 5.1x to 1.7x – 3.5x (geomean ~2.8x). vs CPython, vm2 is now faster on every loop-heavy program in this slice and within ~1.5x on every recursive one.

Round 2 (eval-loop and tail-call codegen)

Program	N	vm2 (µs)	CPython (µs)	Lua (µs)	vm2 / CPython	vm2 / Lua
fact_rec	10	615	492	208	1.25x	2.96x
fact_rec	13	751	600	250	1.25x	3.00x
fib_iter	10	186	254	119	0.73x	1.56x
fib_iter	20	270	502	210	0.54x	1.29x
fib_rec	15	103	75	36	1.38x	2.86x
fib_rec	20	1,157	880	413	1.31x	2.80x
mul_loop	10	170	336	76	0.51x	2.24x
mul_loop	13	200	354	89	0.56x	2.25x
prime_count	50	822	1,373	396	0.60x	2.08x
prime_count	100	2,393	3,851	1,190	0.62x	2.01x
sum_loop	1,000	8,609	23,547	5,200	0.37x	1.66x
sum_loop	10,000	83,678	240,766	52,795	0.35x	1.58x

Landed in this round:

• Hoist dispatch state into locals in eval.go. fr.Fn.Code, fr.Regs, and fr.IP are kept in named locals (code, regs, ip) across the dispatch loop and only synced back to the frame on Call / TailCall / Return / error. Removes one pointer chase per register touch; ~20% across the suite.
• OpTailCallSelf + parallel-move codegen for same-fn tail calls. Before: emit staged args into a scratch range above the regalloc window, then OpTailCall did a runtime.memmove of the args into [0..n). Profiles showed that memmove was 15.6% of total CPU on sum_loop, every loop iteration was paying for a copy that included the unchanged loop bound. After: the emitter runs a parallel-move scheduler that emits direct OpMoves into the param slots (dropping src==dst entries and breaking cycles via the otherwise-unused callBase register), then dispatches the trivial OpTailCallSelf which only rewinds IP. Cross-fn tail calls keep the staging path. ~2x on sum_loop.
• OpAddI64K (add-immediate) superinstruction. The emitter detects AddI64 whose operand is a single-use ConstI64 fitting int32 and fuses both into one dispatch, the loop-counter step (i = i + 1) was the highest-volume OpLoadConstI consumer. ~15% on sum_loop.

Headline movement vs the floor: vm2 vs Lua tightened to 1.3x – 3.0x across the slice (geomean ~2.0x). fib_iter and sum_loop are within ~1.3x – 1.6x of Lua; prime_count and mul_loop are ~2x; the recursive fib_rec/fact_rec programs are still ~3x because each non-tail call goes through a fresh frame acquisition. vs CPython, vm2 is now 1.4x – 2.9x faster on every iterative program in the slice (sum_loop is 0.35x CPython = 2.86x faster).

Round 3 (call/return on a contiguous stack)

Program	N	vm2 (µs)	CPython (µs)	Lua (µs)	vm2 / CPython	vm2 / Lua
fact_rec	10	279	423	180	0.66x	1.55x
fact_rec	13	337	500	188	0.67x	1.79x
fib_iter	10	118	216	103	0.55x	1.15x
fib_iter	20	191	484	178	0.39x	1.07x
fib_rec	15	51	62	31	0.82x	1.65x
fib_rec	20	529	670	311	0.79x	1.70x
mul_loop	10	101	280	61	0.36x	1.66x
mul_loop	13	144	340	77	0.42x	1.87x
prime_count	50	691	1,214	353	0.57x	1.96x
prime_count	100	1,916	3,117	982	0.61x	1.95x
sum_loop	1,000	7,714	19,402	4,282	0.40x	1.80x
sum_loop	10,000	77,965	207,074	44,191	0.38x	1.76x

Landed in this round:

• Stack-of-cells frame model. Round-2 still acquired frames and register slabs through sync.Pool. A profile of fib_rec (the recursive call benchmark) showed sync.Pool.Get/Put plus runtime.memclrNoHeapPointers accounting for 56% of total CPU. The redesign moves activation records onto two contiguous slices on the VM itself (Stack []Cell, Frames []frame); OpCall and OpReturn are now an integer SP bump plus a Frames append/slice-reset. No per-call pool round-trip, no per-frame zero-init memclr.
• Cross-fn OpTailCall rewritten on the same stack model, reusing the caller's Frames slot and shrinking-then-growing the Stack window to the callee's NumRegs.
• vm2runner reuses one VM across reps. The crosslang harness used to do vm2.New(prog).Run() per rep, which dominated wall time on short programs (mul_loop n=10/13). Reusing the VM matches what Lua and CPython do (the interpreter state is created once).

Headline movement vs the floor: vm2 vs Lua is now 1.07x – 1.96x across the slice (geomean ~1.6x). fib_iter is at Lua parity (1.07x – 1.15x). The recursive programs that round 2 left at ~3x Lua dropped to 1.55x – 1.79x because the frame-pool round trip is gone. vs CPython, vm2 is 1.6x – 2.9x faster on every program in the slice without exception.

Strings subsystem (MEP-24 §2, first slice)

The first non-integer subsystem lands. bench/template/strings/concat_loop/{mochi,py,lua} exercises acc = acc + "a" repeated N times in a loop, repeated 1000 times by each language harness. Outputs (final byte length) match across all three runtimes.

Baseline (PR #21520):

Program	N	vm2 (µs)	CPython (µs)	Lua (µs)	vm2 / CPython	vm2 / Lua
strings/concat_loop	10	624	542	441	1.15x	1.41x
strings/concat_loop	30	1,807	1,121	1,625	1.61x	1.11x

After tagSStr (this PR):

Program	N	vm2 (µs)	CPython (µs)	Lua (µs)	vm2 / CPython	vm2 / Lua
strings/concat_loop	10	244	266	239	0.92x	1.02x
strings/concat_loop	30	889	594	839	1.50x	1.06x

A new tagSStr Cell tag packs up to 5 bytes inline (length in payload bits 40..47, bytes in bits 0..39), so the literal "a" and all concat results of length ≤ 5 carry zero heap weight. String constants are also pre-materialized into a per-function StrCells []Cell at VM startup so OpLoadStrK is a slice read on the hot path, never an allocation.

The N=10 row closes to Lua parity (1.02x) and beats CPython (0.92x). At N=30 the result string outgrows the 5-byte inline budget, so the tail of the loop allocates *vmString per concat; vm2 is now at Lua parity (1.06x) and the residual CPython gap (1.50x) is the in-place resize fast-path. Concat reuse-when-unique remains the next target.

Lists subsystem (MEP-24 §3, first slice)

The second subsystem lands. bench/template/lists/fill_sum/{mochi,py,lua} allocates an empty list, pushes N integers, then reads them back in a sum loop, repeated 1000 times by each language harness. Outputs (the sum n*(n-1)/2) match across all three runtimes.

Program	N	vm2 (µs)	CPython (µs)	Lua (µs)	vm2 / CPython	vm2 / Lua
lists/fill_sum	10	1,093	774	908	1.41x	1.20x
lists/fill_sum	100	6,495	6,042	4,255	1.07x	1.53x

vmList is a thin wrapper over []Cell (no element-type specialization at MVP). The hot path is OpListPush (append) and OpListGet (bounds-checked load). vm2 closes to CPython on the larger row (1.07x at N=100) where amortized growth dominates fixed dispatch cost; small-N is dispatch-bound (1.41x at N=10). Lua's per-call overhead is lower in this workload (table append + numeric index), leaving vm2 1.20x-1.53x behind. Element-type specialization (homogeneous []int64 storage) is the natural next step, gated on profile evidence per the MEP-24 §3 deferral.

Maps subsystem (MEP-24 §4, first slice)

The third subsystem lands. bench/template/maps/fill_sum/{mochi,py,lua} fills a map with N int->int entries then sums the values back, repeated 1000 times by each language harness. Outputs match across all three runtimes.

Program	N	vm2 (µs)	CPython (µs)	Lua (µs)	vm2 / CPython	vm2 / Lua
maps/fill_sum	10	767	475	438	1.61x	1.75x
maps/fill_sum	100	7,455	3,626	1,533	2.06x	4.86x

vmMap is a thin wrapper over Go's map[any]Cell, with a mapKeyOf normalizer that collapses inline + heap string Cells to their byte content and unboxes ints/bools/null to their scalar value. The MVP intentionally inherits Go map performance; an open-addressed table over the mapKey{tag,bits,aux} struct described in MEP-24 §4 is the natural follow-on. The Lua gap at N=100 (4.86x) is dominated by Lua's hand-tuned hash table and the any boxing penalty on every key; the CPython gap (2.06x) is dispatch + mapKeyOf type-switch overhead, both of which a specialized int-key fast path would erase.

Deep dive: Benchmarks Game methodology

The single-run microbenchmark methodology that produced the tables in the previous section has a known failure mode on a thermally constrained dev laptop: the same vm2 binary running the same program twice can produce results that differ by 3-7x. We observed this concretely while measuring MEP-36 Phase 2, where math/sum_loop n=10000 reported 105 ms on one run and 767 ms on the next ten minutes later with no source change. Two Phase-2 runs of the full sweep produced p-values that benchstat declared "no significant difference" on individual rows while still showing geomean shifts of 30%+. The harness was reporting noise, not signal.

This section documents the methodology we switched to and the reasoning behind it. The reference point is the Computer Language Benchmarks Game: a long-running comparison of ~10 small programs across ~30 language implementations that has been publishing stable numbers on the same hardware for over 15 years.

The four rules we adopted from the Benchmarks Game

1. One program, one workload, one input. The Benchmarks Game runs each program once at a parameter that takes ~10 seconds on the reference machine. There is no inner repeat=1000 loop folded into the program; the program itself does enough work that the harness's wall-clock measurement of the whole process is the headline number. The existing MEP-23 corpus already follows the inner-loop pattern, so this rule is the future direction for new programs: tune N so the program takes a second or more, then drop the inner repeat to 1. Programs added under the Benchmarks Game suite (see "Suite roadmap" below) follow this convention; existing programs keep their repeats for backwards compatibility with prior appendices.
2. Median of multiple invocations, not single-run wall-clock. Each (program, n, lang) tuple is invoked K times. The harness reports the median; the min and max are kept for noise-floor inspection. K = 5 is the floor; K = 10 or more is appropriate when the deltas under investigation are below 20%. Median is robust to the two- outlier failure mode we hit with single-run methodology (one thermal stall on a 5-second program throws the mean off by 50%+; it moves the median by zero).
3. Workloads exposed to all peer runtimes, not just one. A vm2- only benchmark is informative for MEP 17 (the per-PR gate) but not for MEP 23 (the cross-language story). Every program under the BG suite has hand-written .mochi, .py, .lua, and .go.tmpl peers that compute the same output. The runner asserts output equality across all four columns before recording timings; a mismatch is a hard failure. The .go.tmpl extension keeps the unrendered template (which contains {{ .N }}) out of go build ./... walks and out of gopls; the harness substitutes {{ .N }} and writes the result as main.go inside a per-tuple temp dir before invoking go build.
4. Memory is a first-class metric, not a footnote. The Benchmarks Game tables show CPU time, peak resident-set size, and code size side-by-side. We already report MaxRSS via getrusage; the BG harness elevates it to the headline next to wall-clock, because the "memory management" story that MEP 36 cares about is invisible if the only metric is CPU.

The fifth Benchmarks Game rule — fixed reference hardware, with results published per host — is out of scope for this MEP. The runner records the host CPU model in the JSON output and refuses to merge results across hosts, but a project-wide reference machine is a separate (and arguably CI-shaped) decision.

Harness changes (this MEP)

bench/crosslang gained one flag and one new aggregation pass:

• -repeat K (default 1): each (program, n, lang) tuple is invoked K times. Setting K=1 preserves the prior behaviour for callers that do not want to pay the K× wall-clock cost.
• Per-tuple aggregation prints median=… min=… max=… mem=… rather than a single duration. The markdown table column reports the median; the live progress lines include all three so a reader can see at a glance whether the median is meaningful (a min/max spread

  2× medians says "this row is noise-floor; bump K or shorten N").

• The JSON output schema changed from []result to []aggregate, where each entry carries the run count and the min/median/max. The Markdown table grew two columns (Go (µs) and Go RSS) and one ratio (vm2 / Go); existing consumers that grep on vm2 / Lua still find that ratio in the same position relative to it.
• A Go peer column was added next to CPython and Lua. Each program ships a <name>.go.tmpl template alongside the existing .py / .lua files; the harness substitutes {{ .N }} and go builds the result once per tuple, then runs the binary K times. The build is amortized across the K runs so it does not pollute the timing window. Go is the right peer for the "interpreter-vs-compiled-AOT" question, the same way CPython and Lua are the right peers for the "interpreter-vs-other-interpreter" question.

Subprocess startup is still per-invocation, not per-K. Reusing the process across K invocations would underreport memory (the high-water mark of run 1 would carry into runs 2..K) and would mask cold-cache effects that the BG explicitly measures. The K× cost is the price of honest noise quantification.

Suite roadmap

The math/strings/lists/maps suites already in MEP-23 are the floor. The BG suite extends them with programs whose specific job is to exercise a memory-management property the existing suites cannot reach:

Program	What it stresses (BG analogue)	Status
binary_trees	Many short-lived tree allocations + traversal, the canonical GC stress test (BG: binary-trees). Tests MEP 36's container-reclamation contract.	landed (bg/binary_trees)
nsieve	Large bool array, write-heavy then read-heavy (BG: nsieve / pidigits style). Tests list payload throughput.	landed (bg/nsieve)
mandelbrot	Tight floating-point loop, zero allocation (BG: mandelbrot). Tests dispatch and float64 ops, not GC.	blocked on float64 IR ops
n_body	Floating-point physics simulation, many array reads, no allocations after init (BG: n-body).	blocked on float64 IR ops
fasta	Buffered string output + RNG (BG: fasta). Tests the string subsystem at scale.	deferred (no string builder; concat is O(n^2))
regex_redux	Regex match throughput (BG: regex-redux). Out of scope until vm2 has a regex op-pack.	deferred

The first two (nsieve, binary_trees) landed in this MEP. The other three are listed here so the methodology change has a public roadmap.

The binary_trees program in particular is the headline benchmark for MEP 36's GC story: it allocates millions of tree nodes and lets them die. Phase 2 of MEP 36 is supposed to make those nodes reclaimable as soon as the frame holding them returns; the binary-trees throughput-vs-RSS ratio is the load-bearing measurement of whether the design landed. The Mochi peer encodes a tree as a nested 2-element list ([left, right], leaf = []) rather than a struct, because vm2's IR has no struct support yet. The encoding is faithful to the BG semantics, each subtree is its own allocation, and exercises the exact code path MEP 36 cares about: many small list allocations dying as their referring frame returns.

mandelbrot and n_body are blocked on float64 ops in the IR. The compiler2/ir builder currently exposes ConstI64, AddI64, etc., but no float-typed counterparts. Adding them is a separate MEP shaped work item (new opcodes, new emit cases, new Cell tag for float payload). When that lands, both programs will be straight ports of the canonical BG kernels with no encoding tricks.

fasta is technically expressible with the current string ops (ConcatStr, LenStr) but the only allocation primitive is "make a new string equal to a ++ b", which is O(n) per append and so O(n^2) overall for a buffered-output program. A strings.Builder analogue (append-only buffer with a finalize-to-string op) would make it tractable. Filed as a follow-up to the MEP-24 string subsystem rather than a blocker for the BG suite.

The first BG-suite numbers, taken with the methodology this section defines (median of 5 over full-process invocations, -repeat 5), appear in the appendix below.

Why this MEP changes the methodology now

Three concrete pressures pushed the change:

1. MEP 36 needs a stable measurement. The Phase 1 → Phase 2 comparison requires deltas at the +/-10% level. Single-run methodology cannot resolve that band on the dev hardware in use.
2. go test -bench + benchstat is the wrong tool for cross-language work. benchstat normalizes for in-process b.N adjustment, which is irrelevant when comparing vm2 and CPython (different process, different timer). The BG model (full-process median of K) is directly comparable across runtimes.
3. The previous tables in this MEP are valuable as a historical record. They are not retracted; they are pinned at the date and commit they were taken. The BG suite is the forward methodology; the existing rows remain so future readers can trace the trajectory.

Rationale

Why a new MEP, not an edit to MEP 17. MEP 17 is a process gate. Folding cross-language reporting into it would either weaken the per-PR rule (peer runtime variance is too high to gate on) or strengthen the cross-language rule into a gate (an over-commitment given how many factors outside the VM author's control move those numbers). Separate documents, separate cadences, no confusion.

Why Python and Lua specifically. They are the two most cited points of reference for small dynamic runtimes. CPython is the floor any non-JIT register-machine interpreter should aspire to clear; Lua 5.x is the ceiling for a pure-C single-pass interpreter without a JIT. Sitting between them is a reasonable target for the next 18 months. JIT-class peers (LuaJIT, PyPy, V8) become relevant when MEP 22 ships.

Why hand-written, not transpiled. A transpiler benchmark measures the transpiler. We have one of those (the existing mochi_py, mochi_ts rows in the runner) and it answers a different question. Conflating the two has historically misled language-launch announcements; we are not doing that.

Why output-equality enforcement. The cheapest way to look fast is to compute less. Output equality catches that automatically.

Backwards Compatibility

None. This is process-only and additive infrastructure. Existing bench/runner.go rows (transpiled mochi_py, mochi_ts, mochi_go, mochi_c) are unchanged.

The hand-written .py files that previously sat next to the .mochi templates and were unused by the runner are now used. Their duration_ms key was renamed to duration_us to match the runner's existing Result.DurationUs field; if anyone was reading the old key by hand, they will notice.

Reference Implementation

• Runner change: bench/runner.go adds native_py and native_lua template registration in Benchmarks and matching display labels in report / exportMarkdown.
• New Lua programs: bench/template/math/<name>/<name>.lua for all seven math programs.
• Updated Python programs: bench/template/math/<name>/<name>.py now emit duration_us.

Open Questions

• Other suites. The bench/template/join/ directory has nine join workloads that today are not wired into the runner. They are a stronger test of the query algebra than the math suite is of pure compute. Adding native peer implementations for them is the next slice of this MEP, tracked separately so the math baseline can land.
• JIT peers. PyPy and LuaJIT belong in a follow-up table once MEP 22 ships, so the comparison happens between like categories (interpreter-vs-interpreter, JIT-vs-JIT).
• CI integration. This is local-developer infrastructure today. A future MEP may wire a cross-language run into a nightly CI job on a fixed instance, with the JSON archived for trend analysis. That is not in scope here.

References

• MEP 17: VM Performance Methodology and Baseline. The internal Mochi-vs-Mochi gate this MEP is paired with.
• [Wren-perf]: Bob Nystrom, Wren Performance. https://wren.io/performance.html. The cross-language table format this MEP follows.
• [LuaPerformance]: Roberto Ierusalimschy, Lua Performance Tips. https://www.lua.org/gems/sample.pdf. Background on why Lua sits where it does on small-interpreter benchmarks.

Appendix B: First BG-suite measurements

Captured 2026-05-17 on macOS arm64 (Apple Silicon dev laptop), median of 5 invocations per (program, n, lang) tuple via bench/crosslang -repeat 5. Output equality enforced across all four peers. RSS is ru_maxrss in bytes (macOS), the high-water mark per invocation. The two new BG-suite rows are listed first; the existing math/strings/lists/maps rows are included so the methodology change is auditable against the same machine state in one pass.

Program	N	vm2 (µs)	CPython (µs)	Lua (µs)	Go (µs)	vm2 / CPython	vm2 / Lua	vm2 / Go	vm2 RSS	CPython RSS	Lua RSS	Go RSS	match
bg/binary_trees	8	11711	9694	12072	1502	1.21x	0.97x	7.80x	9.1 MB	12.3 MB	1.9 MB	7.5 MB	✓
bg/binary_trees	10	184104	156494	200145	16402	1.18x	0.92x	11.22x	10.6 MB	13.1 MB	2.5 MB	10.6 MB	✓
bg/nsieve	1000	5117	2341	1084	107	2.19x	4.72x	47.82x	5.8 MB	12.2 MB	1.7 MB	4.3 MB	✓
bg/nsieve	10000	55263	25754	11149	818	2.15x	4.96x	67.56x	12.9 MB	12.3 MB	2.7 MB	4.6 MB	✓
lists/fill_sum	10	781	358	389	17	2.18x	2.01x	45.94x	4.7 MB	13.2 MB	1.8 MB	4.2 MB	✓
lists/fill_sum	100	5274	2706	1883	98	1.95x	2.80x	53.82x	6.3 MB	11.9 MB	1.5 MB	4.3 MB	✓
maps/fill_sum	10	1277	459	455	259	2.78x	2.81x	4.93x	5.4 MB	12.4 MB	1.5 MB	4.5 MB	✓
maps/fill_sum	100	9593	3652	1554	1221	2.63x	6.17x	7.86x	10.1 MB	12.3 MB	1.7 MB	6.5 MB	✓
math/fact_rec	10	332	330	136	6	1.01x	2.44x	55.33x	4.4 MB	12.6 MB	1.6 MB	4.3 MB	✓
math/fact_rec	13	437	454	148	8	0.96x	2.95x	54.62x	4.5 MB	12.1 MB	1.5 MB	4.2 MB	✓
math/fib_iter	10	178	168	79	5	1.06x	2.25x	35.60x	4.4 MB	12.1 MB	1.5 MB	4.4 MB	✓
math/fib_iter	20	311	315	135	15	0.99x	2.30x	20.73x	4.3 MB	12.1 MB	1.5 MB	4.1 MB	✓
math/fib_rec	15	80	50	25	5	1.62x	3.20x	16.00x	4.3 MB	11.9 MB	1.5 MB	4.2 MB	✓
math/fib_rec	20	799	536	244	33	1.49x	3.27x	24.21x	4.5 MB	12.3 MB	1.5 MB	4.1 MB	✓
math/mul_loop	10	190	202	43	10	0.94x	4.42x	19.00x	4.3 MB	11.9 MB	1.5 MB	4.3 MB	✓
math/mul_loop	13	236	244	55	10	0.97x	4.29x	23.60x	4.3 MB	12.1 MB	1.5 MB	4.2 MB	✓
math/prime_count	50	1074	878	256	39	1.22x	4.20x	27.54x	4.5 MB	11.9 MB	1.5 MB	4.2 MB	✓
math/prime_count	100	2959	2290	704	70	1.29x	4.20x	42.27x	4.4 MB	12.2 MB	1.5 MB	4.1 MB	✓
math/sum_loop	1000	11243	13870	3103	264	0.81x	3.62x	42.59x	4.4 MB	12.1 MB	1.7 MB	4.1 MB	✓
math/sum_loop	10000	111991	142044	31021	3044	0.79x	3.61x	36.79x	4.4 MB	12.0 MB	1.6 MB	4.4 MB	✓
strings/concat_loop	10	308	255	232	274	1.21x	1.33x	1.12x	4.6 MB	12.2 MB	1.5 MB	4.2 MB	✓
strings/concat_loop	30	1266	587	844	799	2.16x	1.50x	1.58x	5.8 MB	12.2 MB	1.5 MB	4.9 MB	✓

Quick reading of the two new rows:

• bg/binary_trees lands within 8% of CPython and within 8% of Lua at depth 10 (vm2 / Lua = 0.92x means vm2 is actually faster than Lua here). RSS is below CPython (10.6 MB vs 13.1 MB). This is the Phase 2 container-reclamation contract paying off: tree nodes die as the frame holding them returns, so the high-water mark stays bounded by the working set rather than the cumulative allocation count. Without that contract a 1024-iteration loop over a 2047-node tree would peg RSS at the cumulative number of allocations, not the live set.
• bg/nsieve is 2-5x off the interpreter peers. The hot loop is xs[i] == 0 followed by a write phase, both of which pay full Cell overhead per element (one tag check + one heap touch per access). CPython's bytearray and Lua's integer-keyed table both hit the same logical workload through a tighter native code path. This row is the right place to point a typed-list specialization MEP at when it is time to write one.
• The math/strings/lists/maps rows are reproduced here from the same capture run so the BG-suite numbers and the prior-suite numbers are comparable apples-to-apples (same hardware, same kernel revision, same build of the runner). They are not retracted from the earlier appendix; that earlier appendix used a different methodology and is preserved at the date it was taken so future readers can see the trajectory.

Copyright

This document is placed in the public domain.