XEN Mint Waves and Client newPayload Divergence

Question

On 2026-07-09, geth, nethermind, reth and besu all slowed down 2-3x on the EIP-7870 reference nodes while ethrex stayed completely flat. Was it hardware, broken metrics, or something in the blocks?

Background

The EIP-7870 fleet runs every major execution client in three independently operated clusters: utility, sigma and berlin. The clusters sit in different datacenters on different hardware, which makes cross-cluster comparison a clean hardware control. Each node pairs an EL with tysm, which times every engine_newPayload call and ships the measurement to Xatu (consensus_engine_api_new_payload). An engine snooper sitting between CL and EL independently records the same calls from the EL side (execution_engine_new_payload).

A report came in that reth, nethermind and geth were underperforming on the 7870 deep-dive dashboard, but ethrex wasn't. Three candidate explanations, in rough order of prior probability:

1. Hardware: noisy neighbors, disk trouble, one bad datacenter
2. Broken metrics: tysm not actually measuring execution time
3. Workload: the blocks themselves changed in some way that hurts clients unevenly

Data range: 2026-07-08 00:00 to 2026-07-10 00:00 UTC, mainnet, blocks 25,493,987-25,494,583 for the per-block analysis.

Investigation

When It Happened

The anomaly is a two-hour window on 2026-07-09 from roughly 08:45 to 10:45 UTC. Four clients (geth, nethermind, reth, besu) roughly doubled-to-tripled their median newPayload duration. Ethrex did not move. Erigon barely moved.

Ruling Out Hardware

The three clusters are independent machines in independent datacenters. If this were hardware, the spike would live in one cluster. It doesn't: the same client spikes in all three at the same minute and recovers at the same minute.

Hardware telemetry from the hosts says the same thing. Each utility client runs on its own dedicated host, and none of them broke a sweat: CPU temperatures sat at 55-62°C through the incident window, marginally cooler than the daily average, and NVMe temperatures stayed between 28°C and 70°C across both days. No thermal throttling, no overheating drive.

Ruling Out the Metrics

tysm measures from the CL side of the Engine API. The snooper measures from the EL side. If tysm were misreporting, the two would disagree. They don't:

Both capture points agree to within a few milliseconds: ethrex 32ms in both periods on both sides, everyone else elevated during the spike. The status mix was also clean: no burst of SYNCING or ERROR responses hiding slow calls from the VALID-only median.

Splitting Blocks by Content

The spike-window blocks divide cleanly into two populations: blocks containing transactions to three XEN batch-mint contracts, and blocks without them. The contracts are CoinTool: XEN Batch Minter (0x0de8bf...), MCT XENFT (0x000000...) and MCT-XEN Batch Minter (0x2f8489...), at 6.8 to 15.4 Mgas per transaction, from about five sender addresses.

During the same minutes, clean blocks executed at normal speed on every client. Only blocks carrying XEN mint transactions were slow, and only for four of the six clients.

The per-block scatter makes the mechanism visible. Pick a client to compare against ethrex; nethermind and reth are the interesting cases, since on clean blocks they run neck-and-neck with ethrex. For every client except ethrex, XEN blocks sit far above the gas trend line, and a small block with one mint tx costs as much as a full 60 Mgas block. For ethrex, XEN blocks sit exactly on the trend:

All six clients on the same XEN-carrying blocks, one color per client. The vertical spread at each gas level is the client divergence; ethrex hugs the bottom:

What a XEN Mint Actually Does

Execution traces of the 30 largest CoinTool mint transactions show where the gas goes. Per ~6.8 Mgas transaction: roughly 1,900 SSTOREs, 2,100 SLOADs (about 700 of them cold), DELEGATECALL fan-out to ~600 minimal proxies, and millions of cheap interpreter opcodes.

This is a state-stress workload: gas pays linearly for storage touches, but the real cost (cold trie reads, dirty-node accumulation, state-root recomputation) scales with how scattered those touches are. Thirty Mgas of XEN mints is far more expensive to merkleize than thirty Mgas of swaps, and clients whose state layout amortizes that work (flat state layouts, different trie-commit strategies) barely notice. Ethrex processed XEN blocks at its normal ~1.05 ms/Mgas; erigon's penalty was also small. The others paid heavily:

One deferral check for ethrex: if it were postponing the trie work (returning VALID fast and paying later), the next block after a XEN block would be slow. Splitting every spike-window block by what preceded it shows no such cost; clean blocks run at the same median whether they follow a XEN block or not:

The Fingerprint Replicates

The mint campaign has run for weeks, and it peaked on June 28. If the July 9 result is real client behavior rather than a quirk of that morning, the same split should appear there. It does. During the heaviest two hours of June 28, ethrex again paid almost nothing for XEN blocks (30ms clean, 43ms with XEN) while geth went 58ms to 151ms and reth 25ms to 61ms:

One difference is worth flagging to client teams: nethermind's penalty was as mild as ethrex's in June (30ms to 42ms) but grew to nearly 3x by July 9. Both the nethermind build (1.40.0+166efc69 to +32e36e05) and the mint mix (July added 13-15 Mgas MCT transactions) changed between the two windows, so the cause needs a bisect rather than a guess.

Takeaways

• The 2026-07-09 divergence was real client behavior under a XEN batch-mint spam wave (08:45-10:45 UTC): not hardware (the same spike in three clusters on different hardware in different datacenters) and not metrics (CL-side and EL-side measurements agree exactly).
• Total gas explains nothing here: blocks averaged ~30 Mgas throughout. Block content is what changed: each mint tx does ~1,900 SSTOREs and ~700 cold SLOADs via proxy fan-out, a worst-case state-access pattern per unit of gas.
• A single 6.8 Mgas mint transaction added 60-100ms to geth, nethermind and reth, and ~5ms to ethrex. Gas-normalized throughput dropped 23-38% for geth/nethermind/reth/besu while ethrex was unchanged.
• ethrex and erigon are architecturally resilient to this workload; the result is worth flagging to client teams on both sides of the gap; it is exactly the divergence that matters for gas-limit-increase discussions, since gas is supposed to price this work.
• The fingerprint replicates: at the June 28 campaign peak, ethrex showed the same near-zero penalty while geth and reth slowed 2.4-2.6x. Per-block content splits (this page's method) separate workload effects from genuine client regressions; nethermind's penalty growing from 1.4x (June build) to 2.8x (July build) is worth a client-side bisect.