Ethereum Client Trace Comparisons
This report compares debug_traceTransaction outputs across Ethereum execution clients to identify behavioral differences and formatting quirks.
Source: This analysis was generated using ethereum-trace-compare
Overview
- Total transactions: 501
- Skipped (client errors): 1
- Analyzed: 500
- Clients: besu, erigon, geth, nethermind, reth
Difference Types Found
These are semantic differences detected in trace outputs. High counts indicate systematic differences in how clients report EVM execution.
| Type | Count | Description |
|---|---|---|
| missing_key | 1,627,819 | Field present in some clients but not others |
| value_mismatch | 782,075 | Same field has different values |
| type_mismatch | 6,211 | Same field has different types |
| top_level_mismatch | 61 | Top-level trace fields differ |
| structlog_count_mismatch | 30 | Different number of structLog entries |
Client Quirk Summary
This table shows where each client differs from the majority behavior. Lower numbers indicate better conformance.
| Client | Missing Keys | Extra Keys | Type Mismatches | Value Mismatches | Gas Diffs | ReturnValue Diffs |
|---|---|---|---|---|---|---|
| besu | 1,159,759 | 468,060 | 6,211 | 782,069 | 57 | 2 |
| erigon | 849,258 | 778,561 | 0 | 0 | 0 | 0 |
| geth | 472,379 | 1,155,440 | 0 | 0 | 0 | 0 |
| nethermind | 1,148,859 | 478,960 | 0 | 1,673 | 3 | 0 |
| reth | 979,808 | 648,011 | 0 | 1,993 | 0 | 0 |
Per-Client Reports
Detailed reports for each client:
Format Quirks (Not Bugs)
These are formatting differences that don't affect trace correctness. The comparison tool normalizes these before semantic comparison.
Return Value Encoding
How empty return values are represented. These are semantically equivalent.
| Client | Format | Count |
|---|---|---|
| besu | empty_as_empty | 424 |
| besu | no_0x | 74 |
| erigon | empty_as_0x | 424 |
| erigon | with_0x | 74 |
| geth | empty_as_0x | 424 |
| geth | with_0x | 74 |
| nethermind | empty_as_0x | 424 |
| nethermind | with_0x | 74 |
| reth | empty_as_0x | 424 |
| reth | with_0x | 74 |
Example (block 24281427, tx 0xbe3839523703cc914573c2a304367bff26fd6d08738389fc8d2bf857cdf3391c):
| Client | Value |
|---|---|
| besu | (empty) |
| erigon | 0x |
| geth | 0x |
| nethermind | 0x |
| reth | 0x |
Memory Format
How memory words are formatted in structLogs.
| Client | Format | Count |
|---|---|---|
| besu | short_hex_with_0x | 212 |
| erigon | 64char_no_prefix | 212 |
| geth | 64char_no_prefix | 212 |
| nethermind | 64char_no_prefix | 212 |
| reth | 64char_no_prefix | 212 |
Example (block 24281427, tx 0x584b891391ad58c565ea1bc9933b2f9b098e34d693ac8d77c465ba951ac2373a):
| Client | Value |
|---|---|
| besu | 0x0 |
| erigon | 0000000000000000000000000000000000000000000000000000000000000000 |
| geth | 0000000000000000000000000000000000000000000000000000000000000000 |
| nethermind | 0000000000000000000000000000000000000000000000000000000000000000 |
| reth | 0000000000000000000000000000000000000000000000000000000000000000 |
Storage Format
How storage keys and values are formatted. Clients differ in zero-padding.
| Client | Keys | Values | Observations |
|---|---|---|---|
| besu | compact (39), 64-char (160) | compact (189), 64-char (10) | 199 txs |
| erigon | 64-char (199) | 64-char (199) | 199 txs |
| geth | 64-char (199) | 64-char (199) | 199 txs |
| reth | 64-char (199) | 64-char (199) | 199 txs |
Example (block 24281427, tx 0x584b891391ad58c565ea1bc9933b2f9b098e34d693ac8d77c465ba951ac2373a):
| Client | Key | Value |
|---|---|---|
| besu | ef7498a2a84dafd941a59743b477cd1cfda22d923c58854eac2728d8ec554431 | 0 |
| erigon | ef7498a2a84dafd941a59743b477cd1cfda22d923c58854eac2728d8ec554431 | 0000000000000000000000000000000000000000000000000000000000000000 |
| geth | ef7498a2a84dafd941a59743b477cd1cfda22d923c58854eac2728d8ec554431 | 0000000000000000000000000000000000000000000000000000000000000000 |
| reth | ef7498a2a84dafd941a59743b477cd1cfda22d923c58854eac2728d8ec554431 | 0000000000000000000000000000000000000000000000000000000000000000 |
Optional Fields
Fields some clients include that others may omit.
| Client | Field | Count | Description |
|---|---|---|---|
| besu | structLog_for_simple_transfer | 288 | Emits structLog for simple ETH transfers (others emit 0 entries) |
| nethermind | error | 1,524,536 | Includes error: null on every step |
| nethermind | storage | 1,052,519 | Includes storage: {} on every step |
| reth | returnData | 1,139,197 | Includes returnData field on every step |
error example:
Example (block 24281427, tx 0x584b891391ad58c565ea1bc9933b2f9b098e34d693ac8d77c465ba951ac2373a):
| Client | Value |
|---|---|
| besu | missing |
| erigon | missing |
| geth | missing |
| nethermind | present: null |
| reth | missing |
returnData example:
Example (block 24281427, tx 0x584b891391ad58c565ea1bc9933b2f9b098e34d693ac8d77c465ba951ac2373a):
| Client | Value |
|---|---|
| besu | missing |
| erigon | missing |
| geth | missing |
| nethermind | missing |
| reth | present: '0x' |
storage example:
Example (block 24281427, tx 0x584b891391ad58c565ea1bc9933b2f9b098e34d693ac8d77c465ba951ac2373a):
| Client | Value |
|---|---|
| besu | missing |
| erigon | missing |
| geth | missing |
| nethermind | present: {} |
| reth | missing |
structLog_for_simple_transfer example:
Example (block 24281427, tx 0xbe3839523703cc914573c2a304367bff26fd6d08738389fc8d2bf857cdf3391c):
| Client | Value |
|---|---|
| besu | 1 structLog entries |
| erigon | 0 structLog entries |
| geth | 0 structLog entries |
| nethermind | 0 structLog entries |
| reth | 0 structLog entries |
AI Analysis of Client Differences
Ethereum Client debug_traceTransaction Analysis
Executive Summary
Across 501 transactions, there are 2.4M+ differences detected, but the vast majority are formatting variations rather than semantic bugs. The data reveals three distinct categories:
- Format-only differences (cosmetic, no semantic impact)
- Field presence differences (when/whether optional fields appear)
- Potential semantic differences (actual behavioral divergence)
1. Key Behavioral Differences by Client
Besu
| Issue | Severity | Details |
|---|---|---|
| Type mismatches | High | stack: returns bytes_array vs others' list; op: returns string vs hex_string; error: returns list vs string |
| Value mismatches | High | 782,069 mismatches across memory, storage, gasCost, gas, stack, pc, op, depth |
| Return value discrepancy | High | 2 transactions return empty string instead of actual revert data (e.g., "Min return not reached" error lost) |
| Extra structLog entries | Medium | Adds 1 structLog entry for simple ETH transfers (288 cases) where others return 0 |
| Gas differences | Medium | 57 transactions with different gas values |
Geth
| Issue | Severity | Details |
|---|---|---|
| Clean trace output | N/A | Zero value mismatches, zero type mismatches |
| Consistent behavior | N/A | No structlog count mismatches, no gas differences |
Erigon
| Issue | Severity | Details |
|---|---|---|
| Clean trace output | N/A | Zero value mismatches, zero type mismatches |
| Consistent with Geth | N/A | Field presence/absence aligns with Geth |
Nethermind
| Issue | Severity | Details |
|---|---|---|
| Storage value mismatches | Medium | 1,673 storage value differences |
| Gas differences | Medium | 3 transactions with different gas values |
| Extra empty fields | Low | Includes storage: {} and error: null on every step (1M+ extra fields) |
Reth
| Issue | Severity | Details |
|---|---|---|
| Value mismatches | Medium | 1,993 mismatches in storage, memory, gasCost |
| Extra returnData field | Low | Includes returnData: '0x' on every step (1.1M extra fields) |
2. Separating Semantic Differences from Format Noise
Format-Only (Safe to Normalize)
| Field | Variation | Clients |
|---|---|---|
| returnValue encoding | "" vs "0x" for empty | Besu uses "", others use "0x" |
| returnValue prefix | With/without 0x | Besu omits 0x prefix |
| memory format | "0x0" vs 64-char padded | Besu uses short hex, others use zero-padded |
| storage values | Short vs 64-char padded | Besu uses minimal encoding ("0" vs "00...00") |
| storage keys | Occasional short keys | Besu sometimes uses 63-char keys |
Actual Semantic Differences
| Issue | Evidence | Affected Client(s) |
|---|---|---|
| Lost revert reason | Besu returns '' when others return ABI-encoded error | Besu |
| Type representation | stack as bytes vs list fundamentally changes parsing | Besu |
| structLog for transfers | 1 entry vs 0 for simple ETH transfers changes trace structure | Besu |
| Storage state divergence | Different storage key/value pairs at same execution point | Nethermind, Reth |
| Gas accounting | 60 total gas differences across Besu (57), Nethermind (3) | Besu, Nethermind |
3. Patterns Suggesting Bugs or Spec Violations
[BUG] Besu: Return Data Loss
Impact: Debugging tools lose revert reasons, making troubleshooting significantly harder.
[BUG] Besu: Inconsistent Type Serialization
stack:bytes_arrayinstead oflist- breaks standard JSON array iterationop:stringinstead ofhex_string- inconsistent with other hex fieldserror:listinstead ofstring- unexpected structure for error messages
[BUG] Besu: Phantom structLog for Transfers
Simple ETH transfers (no contract code) produce 1 structLog entry in Besu, 0 in all others. This could indicate:
- Besu traces the implicit "stop" at empty code
- Others skip tracing entirely for EOA-to-EOA transfers
[WARNING] Storage Reporting Point Divergence
The storage field shows massive differences in which opcode reports storage changes:
- Geth/Erigon: Report storage on
SLOAD/SSTOREonly - Besu: Missing storage on
SLOAD/SSTORE, but reports on unrelated opcodes (PUSH, DUP, etc.) - Nethermind/Reth: Report storage on many non-storage opcodes
This suggests different interpretations of "when to snapshot storage state."
[WARNING] Refund Counter Reporting
- Geth/Erigon/Reth: Include
refundfield throughout execution - Besu/Nethermind: Missing
refundon most opcodes
4. What Each Client Does (Without Assuming Reference)
Storage Field Timing
| Client | Reports storage on SLOAD/SSTORE | Reports storage on other opcodes |
|---|---|---|
| Geth | No | Yes (after state changes) |
| Erigon | Yes | No |
| Besu | No | Yes (on many opcodes) |
| Nethermind | Yes | Yes (always includes {}) |
| Reth | Yes | Yes |
Memory Initialization Reporting
| Client | Reports memory on MSTORE/CALLDATACOPY | Reports memory on PUSH/stack ops |
|---|---|---|
| Geth | Yes | No |
| Erigon | Yes | Yes |
| Besu | Yes (different) | No |
| Nethermind | Yes | Yes |
| Reth | Yes | Yes |
Refund Counter Availability
| Client | Includes refund field |
|---|---|
| Geth | Yes |
| Erigon | Yes |
| Besu | No |
| Nethermind | No |
| Reth | Partial (different values) |
5. Notable Quirks by Client
Besu
- Most divergent client with 782K+ value mismatches
- Uses compact encoding (no zero-padding) - saves bytes but breaks simple string comparison
- Missing
0xprefix on hex values in several places - Returns empty revert data (data loss bug)
- Adds trace entries for EOA transfers
- Different type representations break API compatibility
Geth
- Cleanest output - zero semantic mismatches in this dataset
- De facto reference for field presence patterns
- Consistent 64-char zero-padded hex encoding
Erigon
- Closest to Geth in behavior
- Identical field presence/absence patterns
- Same encoding conventions
Nethermind
- Verbose output - includes
storage: {}anderror: nulleven when empty - 1,673 storage value differences suggest different snapshot timing
- 3 gas calculation differences worth investigating
Reth
- Extra returnData field on every step (1.1M occurrences)
- Different refund counter behavior vs majority
- 1,993 value differences in storage/memory/gasCost
- Storage changes reported at slightly different execution points
Recommendations
For Besu team: Investigate return data loss (high severity); consider aligning type serialization with majority
For tooling developers: Normalize these formats before comparison:
- Strip/add
0xprefix - Zero-pad hex values to expected length
- Treat
""and"0x"as equivalent for empty data
- Strip/add
For spec authors: Clarify:
- When storage field should be populated
- Whether refund is required/optional
- Expected behavior for EOA-to-EOA transfer traces
- Canonical encoding for empty values