diff --git a/_config.yml b/_config.yml index 7885de85a5..32ec2d4dc9 100644 --- a/_config.yml +++ b/_config.yml @@ -66,9 +66,15 @@ collections: caption: Clef sidebar_index: 7 frontpage: _clef/Introduction.md + evm-tracing: + output: true + permalink: docs/:collection/:slug + caption: EVM tracing + sidebar_index: 8 + frontpage: _evm-tracing/index.md vulnerabilities: output: true permalink: docs/:collection/:slug caption: Vulnerabilities - sidebar_index: 8 + sidebar_index: 9 frontpage: _vulnerabilies/vulnerabilities.md diff --git a/docs/_dapp/builtin-tracers.md b/docs/_dapp/builtin-tracers.md deleted file mode 100644 index 4499a96982..0000000000 --- a/docs/_dapp/builtin-tracers.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: Built-in tracers -sort_key: C ---- - -Geth comes bundled with a choice of tracers ready for usage through the [tracing API](/docs/rpc/ns-debug). Some of them are implemented natively in Go, and others in JS. In this page a summary of each of these will be outlined. They have to be specified by name when sending a request. The only exception is the opcode logger (otherwise known as struct logger) which is the default tracer for all the methods and cannot be specified by name. - -* TOC -{:toc} - -## Struct logger - -Struct logger or opcode logger is a native Go tracer which executes a transaction and emits the opcode and execution context at every step. This is the tracer that will be used when no name is passed to the API, e.g. `debug.traceTransaction()`. The following information is emitted at each step: - -| field | type | description | -|------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------| -| pc | uint64 | program counter | -| op | byte | opcode to be executed | -| gas | uint64 | remaining gas | -| gasCost | uint64 | cost for executing op | -| memory | []byte | EVM memory. Enabled via `enableMemory` | -| memSize | int | Size of memory | -| stack | []uint256 | EVM stack. Disabled via `disableStack` | -| returnData | []byte | Last call's return data. Enabled via `enableReturnData` | -| storage | map[hash]hash | Storage slots of current contract read from and written to. Only emitted for `SLOAD` and `SSTORE`. Disabled via `disableStorage` | -| depth | int | Current call depth | -| refund | uint64 | Refund counter | -| error | string | Error message if any | - -Note that the fields `memory`, `stack`, `returnData`, and `storage` have dynamic size and depending on the exact transaction they could grow large in size. This is specially true for `memory` which could blow up the trace size. It is recommended to keep them disabled unless they are explicitly required for a given use-case. - -## Native tracers - -The following tracers are implement in Go and as such have offer good performance. They are selected by their name when invoking a tracing API method, e.g. `debug.traceTransaction(, { tracer: 'callTracer' })`. - -### 4byteTracer - -Solidity contract functions are [addressed](https://docs.soliditylang.org/en/develop/abi-spec.html#function-selector) by the first four four byte of the Keccak-256 hash of their signature. Therefore when calling the function of a contract, the caller must send this function selector as well as the ABI-encoded arguments as call data. - -The `4byteTracer` collects the function selectors of every function executed in the lifetime of a transaction, along with the size of the supplied call data. The result is a `map[string]int` where the keys are `SELECTOR-CALLDATASIZE` and the values are number of occurances of this key. E.g.: - -```terminal -> debug.traceTransaction( "0x214e597e35da083692f5386141e69f47e973b2c56e7a8073b1ea08fd7571e9de", {tracer: "4byteTracer"}) -{ - "0x27dc297e-128": 1, - "0x38cc4831-0": 2, - "0x524f3889-96": 1, - "0xadf59f99-288": 1, - "0xc281d19e-0": 1 -} -``` - -### callTracer - -The `callTracer` tracks all the call frames executed during a transaction, including depth 0. The result will be a nested list of call frames, resembling how EVM works. They form a tree with the top-level call at root and sub-calls as children of the higher levels. Each call frame has the following fields: - -| field | type | description | -|---------|-------------|-------------------------------------------| -| type | string | CALL or CREATE | -| from | string | address | -| to | string | address | -| value | string | hex-encoded amount of value transfer | -| gas | string | hex-encoded gas provided for call | -| gasUsed | string | hex-encoded gas used during call | -| input | string | call data | -| output | string | return data | -| error | string | error, if any | -| revertReason | string | Solidity revert reason, if any | -| calls | []callframe | list of sub-calls | - -Things to note about the call tracer: - -- Calls to precompiles are also included in the result -- In case a frame reverts, the field `output` will contain the raw return data -- In case the top level frame reverts, its `revertReason` field will contain the parsed reason of revert as returned by the Solidity contract - -`callTracer` has an option to only trace the main (top-level) call and none of the sub-calls. This avoids extra processing for each call frame if only the top-level call info are required. Here's how it can be configured: - -```terminal -> debug.traceTransaction('0xc73e70f6d60e63a71dabf90b9983f2cdd56b0cb7bcf1a205f638d630a95bba73', { tracer: 'callTracer', tracerConfig: { onlyTopCall: true } }) -``` - -### noopTracer - -This tracer is noop. It returns an empty object and is only meant for testing the setup. - -### prestateTracer - -Executing a transaction requires the prior state, including account of sender and recipient, contracts that are called during execution, etc. The `prestateTracer` replays the tx and tracks every part of state that is touched. This is similar to the concept of a [stateless witness](https://ethresear.ch/t/the-stateless-client-concept/172), the difference being this tracer doesn't return any cryptographic proof, rather only the trie leaves. The result is an object. The keys are addresses of accounts. The value is an object with the following fields: - -| field | type | description | -|---------|-------------------|-------------------------------| -| balance | string | balance in Wei | -| nonce | uint64 | nonce | -| code | string | hex-encoded bytecode | -| storage | map[string]string | storage slots of the contract | - -## JS tracers - -The following are a list of tracers written in JS that come as part of Geth: - -- `bigramTracer`: Counts the opcode bigrams, i.e. how many times 2 opcodes were executed one after the other -- `evmdisTracer`: Returns sufficient information from a trace to perform [evmdis](https://github.com/Arachnid/evmdis)-style disassembly -- `opcountTracer` Counts the total number of opcodes executed -- `trigramTracer`: Counts the opcode trigrams -- `unigramTracer`: Counts the occurances of each opcode diff --git a/docs/_dapp/custom-tracer.md b/docs/_dapp/custom-tracer.md deleted file mode 100644 index c680b03677..0000000000 --- a/docs/_dapp/custom-tracer.md +++ /dev/null @@ -1,460 +0,0 @@ ---- -title: Custom EVM tracer -sort_key: B ---- - -In addition to the default opcode tracer and the built-in tracers, Geth offers the possibility to write custom code -that hook to events in the EVM to process and return the data in a consumable format. Custom tracers can be -written either in Javascript or Go. JS tracers are good for quick prototyping and experimentation as well as for -less intensive applications. Go tracers are performant but require the tracer to be compiled together with the Geth source code. - -* TOC -{:toc} - -## Custom Javascript tracing - -Transaction traces include the complete status of the EVM at every point during the transaction execution, which -can be a very large amount of data. Often, users are only interested in a small subset of that data. Javascript trace -filters are available to isolate the useful information. Detailed information about `debug_traceTransaction` and its -component parts is available in the [reference documentation](/docs/rpc/ns-debug#debug_tracetransaction). - -### A simple filter - -Filters are Javascript functions that select information from the trace to persist and discard based on some -conditions. The following Javascript function returns only the sequence of opcodes executed by the transaction as a -comma-separated list. The function could be written directly in the Javascript console, but it is cleaner to -write it in a separate re-usable file and load it into the console. - -1. Create a file, `filterTrace_1.js`, with this content: - - ```javascript - - tracer = function(tx) { - return debug.traceTransaction(tx, {tracer: - '{' + - 'retVal: [],' + - 'step: function(log,db) {this.retVal.push(log.getPC() + ":" + log.op.toString())},' + - 'fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))},' + - 'result: function(ctx,db) {return this.retVal}' + - '}' - }) // return debug.traceTransaction ... - } // tracer = function ... - - ``` - -2. Run the [JavaScript console](https://geth.ethereum.org/docs/interface/javascript-console). - -3. Get the hash of a recent transaction from a node or block explorer. - -4. Run this command to run the script: - - ```javascript - loadScript("filterTrace_1.js") - ``` - -5. Run the tracer from the script. Be patient, it could take a long time. - - ```javascript - tracer("") - ``` - - The bottom of the output looks similar to: - ```sh - "3366:POP", "3367:JUMP", "1355:JUMPDEST", "1356:PUSH1", "1358:MLOAD", "1359:DUP1", "1360:DUP3", "1361:ISZERO", "1362:ISZERO", - "1363:ISZERO", "1364:ISZERO", "1365:DUP2", "1366:MSTORE", "1367:PUSH1", "1369:ADD", "1370:SWAP2", "1371:POP", "1372:POP", "1373:PUSH1", - "1375:MLOAD", "1376:DUP1", "1377:SWAP2", "1378:SUB", "1379:SWAP1", "1380:RETURN" - ``` - -6. Run this line to get a more readable output with each string in its own line. - - ```javascript - console.log(JSON.stringify(tracer(""), null, 2)) - ``` - -More information about the `JSON.stringify` function is available -[here](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify). - -The commands above worked by calling the same `debug.traceTransaction` function that was previously -explained in [basic traces](https://geth.ethereum.org/docs/dapp/tracing), but with a new parameter, `tracer`. -This parameter takes the JavaScript object formated as a string. In the case of the trace above, it is: - -```javascript -{ - retVal: [], - step: function(log,db) {this.retVal.push(log.getPC() + ":" + log.op.toString())}, - fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))}, - result: function(ctx,db) {return this.retVal} -} -``` -This object has three member functions: - -- `step`, called for each opcode. -- `fault`, called if there is a problem in the execution. -- `result`, called to produce the results that are returned by `debug.traceTransaction` after the execution is done. - -In this case, `retVal` is used to store the list of strings to return in `result`. - -The `step` function adds to `retVal` the program counter and the name of the opcode there. Then, in `result`, this -list is returned to be sent to the caller. - - -### Filtering with conditions - -For actual filtered tracing we need an `if` statement to only log relevant information. For example, to isolate -the transaction's interaction with storage, the following tracer could be used: - -```javascript -tracer = function(tx) { - return debug.traceTransaction(tx, {tracer: - '{' + - 'retVal: [],' + - 'step: function(log,db) {' + - ' if(log.op.toNumber() == 0x54) ' + - ' this.retVal.push(log.getPC() + ": SLOAD");' + - ' if(log.op.toNumber() == 0x55) ' + - ' this.retVal.push(log.getPC() + ": SSTORE");' + - '},' + - 'fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))},' + - 'result: function(ctx,db) {return this.retVal}' + - '}' - }) // return debug.traceTransaction ... -} // tracer = function ... -``` - -The `step` function here looks at the opcode number of the op, and only pushes an entry if the opcode is -`SLOAD` or `SSTORE` ([here is a list of EVM opcodes and their numbers](https://github.com/wolflo/evm-opcodes)). -We could have used `log.op.toString()` instead, but it is faster to compare numbers rather than strings. - -The output looks similar to this: - -```javascript -[ - "5921: SLOAD", - . - . - . - "2413: SSTORE", - "2420: SLOAD", - "2475: SSTORE", - "6094: SSTORE" -] -``` - - -### Stack Information - -The trace above reports the program counter (PC) and whether the program read from storage or wrote to it. -That alone isn't particularly useful. To know more, the `log.stack.peek` function can be used to peek -into the stack. `log.stack.peek(0)` is the stack top, `log.stack.peek(1)` the entry below it, etc. - -The values returned by `log.stack.peek` are Go `big.Int` objects. By default they are converted to JavaScript -floating point numbers, so you need `toString(16)` to get them as hexadecimals, which is how 256-bit values such as -storage cells and their content are normally represented. - -#### Storage Information - -The function below provides a trace of all the storage operations and their parameters. This gives -a more complete picture of the program's interaction with storage. - -```javascript -tracer = function(tx) { - return debug.traceTransaction(tx, {tracer: - '{' + - 'retVal: [],' + - 'step: function(log,db) {' + - ' if(log.op.toNumber() == 0x54) ' + - ' this.retVal.push(log.getPC() + ": SLOAD " + ' + - ' log.stack.peek(0).toString(16));' + - ' if(log.op.toNumber() == 0x55) ' + - ' this.retVal.push(log.getPC() + ": SSTORE " +' + - ' log.stack.peek(0).toString(16) + " <- " +' + - ' log.stack.peek(1).toString(16));' + - '},' + - 'fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))},' + - 'result: function(ctx,db) {return this.retVal}' + - '}' - }) // return debug.traceTransaction ... -} // tracer = function ... - -``` - -The output is similar to: - -```javascript -[ - "5921: SLOAD 0", - . - . - . - "2413: SSTORE 3f0af0a7a3ed17f5ba6a93e0a2a05e766ed67bf82195d2dd15feead3749a575d <- fb8629ad13d9a12456", - "2420: SLOAD cc39b177dd3a7f50d4c09527584048378a692aed24d31d2eabeddb7f3c041870", - "2475: SSTORE cc39b177dd3a7f50d4c09527584048378a692aed24d31d2eabeddb7f3c041870 <- 358c3de691bd19", - "6094: SSTORE 0 <- 1" -] -``` - -#### Operation Results - -One piece of information missing from the function above is the result on an `SLOAD` operation. The -state we get inside `log` is the state prior to the execution of the opcode, so that value is not -known yet. For more operations we can figure it out for ourselves, but we don't have access to the -storage, so here we can't. - -The solution is to have a flag, `afterSload`, which is only true in the opcode right after an -`SLOAD`, when we can see the result at the top of the stack. - -```javascript -tracer = function(tx) { - return debug.traceTransaction(tx, {tracer: - '{' + - 'retVal: [],' + - 'afterSload: false,' + - 'step: function(log,db) {' + - ' if(this.afterSload) {' + - ' this.retVal.push(" Result: " + ' + - ' log.stack.peek(0).toString(16)); ' + - ' this.afterSload = false; ' + - ' } ' + - ' if(log.op.toNumber() == 0x54) {' + - ' this.retVal.push(log.getPC() + ": SLOAD " + ' + - ' log.stack.peek(0).toString(16));' + - ' this.afterSload = true; ' + - ' } ' + - ' if(log.op.toNumber() == 0x55) ' + - ' this.retVal.push(log.getPC() + ": SSTORE " +' + - ' log.stack.peek(0).toString(16) + " <- " +' + - ' log.stack.peek(1).toString(16));' + - '},' + - 'fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))},' + - 'result: function(ctx,db) {return this.retVal}' + - '}' - }) // return debug.traceTransaction ... -} // tracer = function ... -``` - -The output now contains the result in the line that follows the `SLOAD`. - -```javascript -[ - "5921: SLOAD 0", - " Result: 1", - . - . - . - "2413: SSTORE 3f0af0a7a3ed17f5ba6a93e0a2a05e766ed67bf82195d2dd15feead3749a575d <- fb8629ad13d9a12456", - "2420: SLOAD cc39b177dd3a7f50d4c09527584048378a692aed24d31d2eabeddb7f3c041870", - " Result: 0", - "2475: SSTORE cc39b177dd3a7f50d4c09527584048378a692aed24d31d2eabeddb7f3c041870 <- 358c3de691bd19", - "6094: SSTORE 0 <- 1" -] -``` - -### Dealing With Calls Between Contracts - -So the storage has been treated as if there are only 2256 cells. However, that is not true. -Contracts can call other contracts, and then the storage involved is the storage of the other contract. -We can see the address of the current contract in `log.contract.getAddress()`. This value is the execution -context - the contract whose storage we are using - even when code from another contract is executed (by using -[`CALLCODE` or `DELEGATECALL`][solidity-delcall]). - -However, `log.contract.getAddress()` returns an array of bytes. To convert this to the familiar hexadecimal -representation of Ethereum addresses, `this.byteHex()` and `array2Hex()` can be used. - -```javascript -tracer = function(tx) { - return debug.traceTransaction(tx, {tracer: - '{' + - 'retVal: [],' + - 'afterSload: false,' + - 'callStack: [],' + - - 'byte2Hex: function(byte) {' + - ' if (byte < 0x10) ' + - ' return "0" + byte.toString(16); ' + - ' return byte.toString(16); ' + - '},' + - - 'array2Hex: function(arr) {' + - ' var retVal = ""; ' + - ' for (var i=0; i 0 { - t.env.Cancel() - return - } - - name := op.String() - if _, ok := t.counts[name]; !ok { - t.counts[name] = 0 - } - t.counts[name]++ -} - -// CaptureEnter is called when EVM enters a new scope (via call, create or selfdestruct). -func (t *opcounter) CaptureEnter(op vm.OpCode, from common.Address, to common.Address, input []byte, gas uint64, value *big.Int) {} - -// CaptureExit is called when EVM exits a scope, even if the scope didn't -// execute any code. -func (t *opcounter) CaptureExit(output []byte, gasUsed uint64, err error) {} - -// CaptureFault implements the EVMLogger interface to trace an execution fault. -func (t *opcounter) CaptureFault(pc uint64, op vm.OpCode, gas, cost uint64, scope *vm.ScopeContext, depth int, err error) {} - -// CaptureEnd is called after the call finishes to finalize the tracing. -func (t *opcounter) CaptureEnd(output []byte, gasUsed uint64, _ time.Duration, err error) {} - -func (*opcounter) CaptureTxStart(gasLimit uint64) {} - -func (*opcounter) CaptureTxEnd(restGas uint64) {} - -// GetResult returns the json-encoded nested list of call traces, and any -// error arising from the encoding or forceful termination (via `Stop`). -func (t *opcounter) GetResult() (json.RawMessage, error) { - res, err := json.Marshal(t.counts) - if err != nil { - return nil, err - } - return res, t.reason -} - -// Stop terminates execution of the tracer at the first opportune moment. -func (t *opcounter) Stop(err error) { - t.reason = err - atomic.StoreUint32(&t.interrupt, 1) -} -``` - -As can be seen every method of the [EVMLogger interface](https://pkg.go.dev/github.com/ethereum/go-ethereum/core/vm#EVMLogger) needs to be implemented (even if empty). Key parts to notice are the `init()` function which registers the tracer in Geth, the `CaptureState` hook where the opcode counts are incremented and `GetResult` where the result is serialized and delivered. Note that the constructor takes in a `cfg json.RawMessage`. This will be filled with a JSON object that user provides to the tracer to pass in optional config fields. - -To test out this tracer the source is first compiled with `make geth`. Then in the console it can be invoked through the usual API methods by passing in the name it was registered under: - -```console -> debug.traceTransaction('0x7ae446a7897c056023a8104d254237a8d97783a92900a7b0f7db668a9432f384', { tracer: 'opcounter' }) -{ - ADD: 4, - AND: 3, - CALLDATALOAD: 2, - ... -} -``` - -[solidity-delcall]:https://docs.soliditylang.org/en/v0.8.14/introduction-to-smart-contracts.html#delegatecall-callcode-and-libraries -[debug-docs]: /docs/rpc/ns-debug diff --git a/docs/_dapp/mobile.md b/docs/_dapp/mobile.md index 72b0026a99..17cecbd3fa 100644 --- a/docs/_dapp/mobile.md +++ b/docs/_dapp/mobile.md @@ -1,6 +1,6 @@ --- title: Geth for Mobile -sort_key: F +sort_key: D --- Embedding clients into mobile devices is an important part of Ethereum's decentralization vision. diff --git a/docs/_dapp/native-accounts.md b/docs/_dapp/native-accounts.md index 867a438018..504fa7dade 100644 --- a/docs/_dapp/native-accounts.md +++ b/docs/_dapp/native-accounts.md @@ -1,6 +1,6 @@ --- title: Go Account Management -sort_key: D +sort_key: B --- Geth provides a simple, yet thorough accounts package that includes all the tools developers diff --git a/docs/_dapp/native-bindings.md b/docs/_dapp/native-bindings.md index cbc1fee7d6..466e048677 100644 --- a/docs/_dapp/native-bindings.md +++ b/docs/_dapp/native-bindings.md @@ -1,6 +1,6 @@ --- title: Go Contract Bindings -sort_key: E +sort_key: C --- This page introduces the concept of server-side native dapps. Geth provides the tools required diff --git a/docs/_dapp/native.md b/docs/_dapp/native.md index 43a3bc2cc0..9d52b27015 100644 --- a/docs/_dapp/native.md +++ b/docs/_dapp/native.md @@ -1,6 +1,6 @@ --- title: Go API -sort_key: D +sort_key: A --- Ethereum was originally conceptualized to be the base layer for [Web3][web3-link], providing diff --git a/docs/_dapp/tracing.md b/docs/_dapp/tracing.md deleted file mode 100644 index 74dca40f83..0000000000 --- a/docs/_dapp/tracing.md +++ /dev/null @@ -1,232 +0,0 @@ ---- -title: EVM Tracing -sort_key: A ---- - -There are two different types of [transactions][transactions] -in Ethereum: simple value transfers and contract executions. A value transfer just -moves Ether from one account to another. If however the recipient of a transaction is -a contract account with associated [EVM][evm] (Ethereum Virtual Machine) bytecode - beside -transferring any Ether - the code will also be executed as part of the transaction. - -Having code associated with Ethereum accounts permits transactions to do arbitrarily -complex data storage and enables them to act on the previously stored data by further -transacting internally with outside accounts and contracts. This creates an interlinked -ecosystem of contracts, where a single transaction can interact with tens or hundreds of -accounts. - -The downside of contract execution is that it is very hard to say what a transaction -actually did. A transaction receipt does contain a status code to check whether execution -succeeded or not, but there is no way to see what data was modified, nor what external -contracts where invoked. Geth resolves this by re-running transactions locally and collecting -data about precisely what was executed by the EVM. This is known as "tracing" the transaction. - - -* TOC -{:toc} - - -## Tracing prerequisites - -In its simplest form, tracing a transaction entails requesting the Ethereum node to -reexecute the desired transaction with varying degrees of data collection and have it -return the aggregated summary for post processing. Reexecuting a transaction however has a -few prerequisites to be met. - -In order for an Ethereum node to reexecute a transaction, all historical state accessed -by the transaction must be available. This includes: - -* Balance, nonce, bytecode and storage of both the recipient as well as all internally invoked contracts. -* Block metadata referenced during execution of both the outer as well as all internally created transactions. -* Intermediate state generated by all preceding transactions contained in the same block as the one being traced. - -This means there are limits on the transactions that can be traced imposed by the synchronization and -pruning configuration of a node. - -* An **archive** node retains **all historical data** back to genesis. It can therefore -trace arbitrary transactions at any point in the history of the chain. Tracing a single -transaction requires reexecuting all preceding transactions in the same block. - -* A **full synced** node retains the most recent 128 blocks in memory, so transactions in -that range are always accessible. Full nodes also store occasional checkpoints back to genesis -that can be used to rebuild the state at any point on-the-fly. This means older transactions -can be traced but if there is a large distance between the requested transaction and the most -recent checkpoint rebuilding the state can take a long time. Tracing a single -transaction requires reexecuting all preceding transactions in the same block -**and** all preceding blocks until the previous stored snapshot. - -* A **snap synced** node holds the most recent 128 blocks in memory, so transactions in that -range are always accessible. However, snap-sync only starts processing from a relatively recent -block (as opposed to genesis for a full node). Between the initial sync block and the 128 most -recent blocks, the node stores occasional checkpoints that can be used to rebuild the state on-the-fly. -This means transactions can be traced back as far as the block that was used for the initial sync. -Tracing a single transaction requires reexecuting all preceding transactions in the same block, -**and** all preceding blocks until the previous stored snapshot. - -* A **light synced** node retrieving data **on demand** can in theory trace transactions -for which all required historical state is readily available in the network. This is because the data -required to generate the trace is requested from an les-serving full node. In practice, data -availability **cannot** be reasonably assumed. - -*There are exceptions to the above rules when running batch traces of entire blocks or -chain segments. Those will be detailed later.* - -## Basic traces - -The simplest type of transaction trace that Geth can generate are raw EVM opcode -traces. For every VM instruction the transaction executes, a structured log entry is -emitted, containing all contextual metadata deemed useful. This includes the *program -counter*, *opcode name*, *opcode cost*, *remaining gas*, *execution depth* and any -*occurred error*. The structured logs can optionally also contain the content of the -*execution stack*, *execution memory* and *contract storage*. - -The entire output of a raw EVM opcode trace is a JSON object having a few metadata -fields: *consumed gas*, *failure status*, *return value*; and a list of *opcode entries*: - -```json -{ - "gas": 25523, - "failed": false, - "returnValue": "", - "structLogs": [] -} -``` - -An example log for a single opcode entry has the following format: - -```json -{ - "pc": 48, - "op": "DIV", - "gasCost": 5, - "gas": 64532, - "depth": 1, - "error": null, - "stack": [ - "00000000000000000000000000000000000000000000000000000000ffffffff", - "0000000100000000000000000000000000000000000000000000000000000000", - "2df07fbaabbe40e3244445af30759352e348ec8bebd4dd75467a9f29ec55d98d" - ], - "memory": [ - "0000000000000000000000000000000000000000000000000000000000000000", - "0000000000000000000000000000000000000000000000000000000000000000", - "0000000000000000000000000000000000000000000000000000000000000060" - ], - "storage": { - } -} -``` - -### Generating basic traces - -To generate a raw EVM opcode trace, Geth provides a few [RPC API endpoints](/docs/rpc/ns-debug). -The most commonly used is [`debug_traceTransaction`](/docs/rpc/ns-debug#debug_tracetransaction). - -In its simplest form, `traceTransaction` accepts a transaction hash as its only argument. It then -traces the transaction, aggregates all the generated data and returns it as a **large** -JSON object. A sample invocation from the Geth console would be: - -```js -debug.traceTransaction("0xfc9359e49278b7ba99f59edac0e3de49956e46e530a53c15aa71226b7aa92c6f") -``` - -The same call can also be invoked from outside the node too via HTTP RPC (e.g. using Curl). In this -case, the HTTP endpoint must be enabled in Geth using the `--http` command and the `debug` API -namespace must be exposed using `--http.api=debug`. - -``` -$ curl -H "Content-Type: application/json" -d '{"id": 1, "method": "debug_traceTransaction", "params": ["0xfc9359e49278b7ba99f59edac0e3de49956e46e530a53c15aa71226b7aa92c6f"]}' localhost:8545 -``` - -To follow along with this tutorial, transaction hashes can be found from a local Geth node (e.g. by -attaching a [Javascript console](/docs/interface/javascript-console) and running `eth.getBlock('latest')` -then passing a transaction hash from the returned block to `debug.traceTransaction()`) or from a block -explorer (for [Mainnet](https://etherscan.io/) or a [testnet](https://goerli.etherscan.io/)). - -It is also possible to configure the trace by passing Boolean (true/false) values for four parameters -that tweak the verbosity of the trace. By default, the *EVM memory* and *Return data* are not reported -but the *EVM stack* and *EVM storage* are. To report the maximum amount of data: - -```shell -enableMemory: true -disableStack: false -disableStorage: false -enableReturnData: true -``` - -An example call, made in the Geth Javascript console, configured to report the maximum amount of data -looks as follows: - -```js -debug.traceTransaction("0xfc9359e49278b7ba99f59edac0e3de49956e46e530a53c15aa71226b7aa92c6f",{enableMemory: true, disableStack: false, disableStorage: false, enableReturnData: true}) -``` - -Running the above operation on the Rinkeby network (with a node retaining enough history) -will result in this [trace dump](https://gist.github.com/karalabe/c91f95ac57f5e57f8b950ec65ecc697f). - -Alternatively, disabling *EVM Stack*, *EVM Memory*, *Storage* and *Return data* (as demonstrated in the Curl request below) -results in the following, much shorter, [trace dump](https://gist.github.com/karalabe/d74a7cb33a70f2af75e7824fc772c5b4). - -``` -$ curl -H "Content-Type: application/json" -d '{"id": 1, "method": "debug_traceTransaction", "params": ["0xfc9359e49278b7ba99f59edac0e3de49956e46e530a53c15aa71226b7aa92c6f", {"disableStack": true, "disableStorage": true}]}' localhost:8545 -``` - -### Limits of basic traces - -Although the raw opcode traces generated above are useful, having an individual log entry for every single -opcode is too low level for most use cases, and will require developers to create additional tools to -post-process the traces. Additionally, a full opcode trace can easily go into the hundreds of -megabytes, making them very resource intensive to get out of the node and process externally. - -To avoid those issues, Geth supports running custom JavaScript tracers *within* the Ethereum node, -which have full access to the EVM stack, memory and contract storage. This means developers only have to -gather the data they actually need, and do any processing at the source. - -## Pruning - -Geth does in-memory state-pruning by default, discarding state entries that it deems -no longer necessary to maintain. This is configured via the `--gcmode` command. An error -message alerting the user that the necessary state is not available is common in EVM tracing on -anything other than an archive node. - -```sh -Error: required historical state unavailable (reexec=128) - at web3.js:6365:37(47) - at send (web3,js:5099:62(35)) - at :1:23(13) -``` - -The pruning behaviour, and consequently the state availability and tracing capability of -a node depends on its sync and pruning configuration. The 'oldest' block after which -state is immediately available, and before which state is not immediately available, -is known as the "pivot block". There are then several possible cases for a trace request -on a Geth node. - -For tracing a transaction in block `B` where the pivot block is `P` can regenerate the desired -state by replaying blocks from the last : - -1. a fast-sync'd node can regenerate the desired state by replaying blocks from the most recent -checkpoint between `P` and `B` as long as `P` < `B`. If `P` > `B` there is no available checkpoint -and the state cannot be regenerated without replying the chain from genesis. - -2. a fully sync'd node can regenerate the desired state by replaying blocks from the last available -full state before `B`. A fully sync'd node re-executes all blocks from genesis, so checkpoints are available -across the entire history of the chain. However, database pruning discards older data, moving `P` to a more -recent position in the chain. If `P` > `B` there is no available checkpoint and the state cannot be -regenerated without replaying the chain from genesis. - -3. A fully-sync'd node without pruning (i.e. an archive node configured with `--gcmode=archive`) -does not need to replay anything, it can immediately load up any state and serve the request for any `B`. - -The time taken to regenerate a specific state increases with the distance between `P` and `B`. If the distance -between `P` and `B` is large, the regeneration time can be substantial. - -## Summary - -This page covered the concept of EVM tracing and how to generate traces with the default opcode-based tracers using RPC. -More advanced usage is possible, including using other built-in tracers as well as writing [custom tracing](/docs/dapp/custom-tracer) code in Javascript -and Go. The API as well as the JS tracing hooks are defined in [the reference](/docs/rpc/ns-debug#debug_traceTransaction). - - -[transactions]: https://ethereum.org/en/developers/docs/transactions -[evm]: https://ethereum.org/en/developers/docs/evm diff --git a/docs/_evm-tracing/builtin-tracers.md b/docs/_evm-tracing/builtin-tracers.md new file mode 100644 index 0000000000..e68089d911 --- /dev/null +++ b/docs/_evm-tracing/builtin-tracers.md @@ -0,0 +1,521 @@ +--- +title: Built-in tracers +sort_key: B +--- + +Geth comes bundled with a choice of tracers that can be invoked via the +[tracing API](/docs/rpc/ns-debug). Some of these built-in tracers are implemented +natively in Go, and others in Javascript. The default tracer is the opcode logger +(otherwise known as struct logger) which is the default tracer for all the methods. +Other tracers have to be specified by passing their name to the `tracer` parameter +in the API call. + +* TOC +{:toc} + +## Struct/opcode logger + +The struct logger (aka opcode logger) is a native Go tracer which executes a +transaction and emits the opcode and execution context at every step. This is +the tracer that will be used when no name is passed to the API, e.g. +`debug.traceTransaction()`. The following information is emitted at each step: + +| field | type | description | +| ---------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| pc | uint64 | program counter | +| op | byte | opcode to be executed | +| gas | uint64 | remaining gas | +| gasCost | uint64 | cost for executing op | +| memory | []byte | EVM memory. Enabled via `enableMemory` | +| memSize | int | Size of memory | +| stack | []uint256 | EVM stack. Disabled via `disableStack` | +| returnData | []byte | Last call's return data. Enabled via `enableReturnData` | +| storage | map[hash]hash | Storage slots of current contract read from and written to. Only emitted for `SLOAD` and `SSTORE`. Disabled via `disableStorage` | +| depth | int | Current call depth | +| refund | uint64 | Refund counter | +| error | string | Error message if any | + +Note that the fields `memory`, `stack`, `returnData`, and `storage` have dynamic +size and depending on the exact transaction they could grow large in size. This +is specially true for `memory` which could blow up the trace size. It is recommended +to keep them disabled unless they are explicitly required for a given use-case. + +It is also possible to configure the trace by passing Boolean (true/false) values +for four parameters that tweak the verbosity of the trace. By default, the +_EVM memory_ and _Return data_ are not reported but the _EVM stack_ and +_EVM storage_ are. To report the maximum amount of data: + +```shell +enableMemory: true +disableStack: false +disableStorage: false +enableReturnData: true +``` + +An example call: + +```js +debug.traceTransaction('0xfc9359e49278b7ba99f59edac0e3de49956e46e530a53c15aa71226b7aa92c6f', { + enableMemory: true, + disableStack: false, + disableStorage: false, + enableReturnData: true +}); +``` + +Return: + +```terminal +{ + "gas":25523, + "failed":false, + "returnValue":"", + "structLogs":[ + { + "pc":0, + "op":"PUSH1", + "gas":64580, + "gasCost":3, + "depth":1, + "error":null, + "stack":[ + + ], + "memory":null, + "storage":{ + + } + }, + { + "pc":2, + "op":"PUSH1", + "gas":64577, + "gasCost":3, + "depth":1, + "error":null, + "stack":[ + "0000000000000000000000000000000000000000000000000000000000000060" + ], + "memory":null, + "storage":{ + + } + }, + + ... + +``` + +## Native tracers + +The following tracers are implement in Go. This means they are much more performant +than other tracers that are written in Javascript. The tracers are selected by +passing their name to the `tracer` parameter when invoking a tracing API method, +e.g. `debug.traceTransaction(, { tracer: 'callTracer' })`. + + +### 4byteTracer + +Solidity contract functions are +[addressed](https://docs.soliditylang.org/en/develop/abi-spec.html#function-selector) +using the first four four byte of the Keccak-256 hash of their signature. Therefore +when calling the function of a contract, the caller must send this function selector +as well as the ABI-encoded arguments as call data. + +The `4byteTracer` collects the function selectors of every function executed in the +lifetime of a transaction, along with the size of the supplied call data. The result +is a `map[string]int` where the keys are `SELECTOR-CALLDATASIZE` and the values +are number of occurances of this key. For example: + +Example call: + +```sh +debug.traceTransaction( "0x214e597e35da083692f5386141e69f47e973b2c56e7a8073b1ea08fd7571e9de", {tracer: "4byteTracer"}) +``` + +Return: + +```terminal +{ + "0x27dc297e-128": 1, + "0x38cc4831-0": 2, + "0x524f3889-96": 1, + "0xadf59f99-288": 1, + "0xc281d19e-0": 1 +} +``` + +### callTracer + +The `callTracer` tracks all the call frames executed during a transaction, including depth 0. +The result will be a nested list of call frames, resembling how EVM works. They form a tree +with the top-level call at root and sub-calls as children of the higher levels. Each call +frame has the following fields: + +| field | type | description | +| ------- | ----------- | ------------------------------------ | +| type | string | CALL or CREATE | +| from | string | address | +| to | string | address | +| value | string | hex-encoded amount of value transfer | +| gas | string | hex-encoded gas provided for call | +| gasUsed | string | hex-encoded gas used during call | +| input | string | call data | +| output | string | return data | +| error | string | error, if any | +| calls | []callframe | list of sub-calls | + + +Example Call: + +```sh +> debug.traceTransaction("0x44bed3dc0f584b2a2ab32f5e2948abaaca13917eeae7ae3b959de3371a6e9a95", {tracer: 'callTracer'}) +``` + +Return: + +```terminal +{ + calls: [{ + from: "0xc8ba32cab1757528daf49033e3673fae77dcf05d", + gas: "0x18461", + gasUsed: "0x60", + input: "0x000000204895cd480cc8412691a880028a25aec86786f1ed2aa5562bc400000000000000c6403c14f35be1da6f433eadbb6e9178a47fbc7c6c1d568d2f2b876e929089c8d8db646304fd001a187dc8a6", + output: "0x557904b74478f8810cc02198544a030d1829bb491e14fe1dd0354e933c5e87bd", + to: "0x0000000000000000000000000000000000000002", + type: "STATICCALL" + }, { + from: "0xc8ba32cab1757528daf49033e3673fae77dcf05d", + gas: "0x181db", + gasUsed: "0x48", + input: "0x557904b74478f8810cc02198544a030d1829bb491e14fe1dd0354e933c5e87bd", + output: "0x5fb393023b12544491a5b8fb057943b4ebf5b1401e88e44a7800000000000000", + to: "0x0000000000000000000000000000000000000002", + type: "STATICCALL" + }], + from: "0x35a9f94af726f07b5162df7e828cc9dc8439e7d0", + gas: "0x1a310", + gasUsed: "0xfcb6", + input: "0xd1a2eab2000000000000000000000000000000000000000000000000000000000024aea100000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000050000000204895cd480cc8412691a880028a25aec86786f1ed2aa5562bc400000000000000c6403c14f35be1da6f433eadbb6e9178a47fbc7c6c1d568d2f2b876e929089c8d8db646304fd001a187dc8a600000000000000000000000000000000", + to: "0xc8ba32cab1757528daf49033e3673fae77dcf05d", + type: "CALL", + value: "0x0" +} +``` + +Things to note about the call tracer: + +- Calls to precompiles are also included in the result +- In case a frame reverts, the field `output` will contain the raw return data, +- unlike [revertReasonTracer](#revertreasontracer) which parses the data and +- returns the revert message + + +### prestateTracer + +The prestate tracer has two modes: `prestate` and `diff`. The `prestate` mode returns +the accounts necessary to execute a given transaction. `diff` mode returns the differences +between the transaction's pre and post-state (i.e. what changed because the transaction +happened). The `prestateTracer` defaults to `prestate` mode. It reexecutes the given +transaction and tracks every part of state that is touched. This is similar to the +concept of a [stateless witness](https://ethresear.ch/t/the-stateless-client-concept/172), +the difference being this tracer doesn't return any cryptographic proof, rather only the +trie leaves. The result is an object. The keys are addresses of accounts. The value is an +object with the following fields: + +| field | type | description | +| ------- | ----------------- | ----------------------------- | +| balance | string | balance in Wei | +| nonce | uint64 | nonce | +| code | string | hex-encoded bytecode | +| storage | map[string]string | storage slots of the contract | + +To run this tracer in `diff` mode, pass `tracerConfig: {diffMode: true}` in the APi call. + +Example: + +```js +debug.traceCall({from: "0x35a9f94af726f07b5162df7e828cc9dc8439e7d0", to: "0xc8ba32cab1757528daf49033e3673fae77dcf05d", data: "0xd1a2eab2000000000000000000000000000000000000000000000000000000000024aea100000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000050000000204895cd480cc8412691a880028a25aec86786f1ed2aa5562bc400000000000000c6403c14f35be1da6f433eadbb6e9178a47fbc7c6c1d568d2f2b876e929089c8d8db646304fd001a187dc8a600000000000000000000000000000000"}, 'latest', {tracer: 'prestateTracer'}) +``` + +Return: + +```terminal +{ + 0x0000000000000000000000000000000000000002: { + balance: "0x0" + }, + 0x008b3b2f992c0e14edaa6e2c662bec549caa8df1: { + balance: "0x2638035a26d133809" + }, + 0x35a9f94af726f07b5162df7e828cc9dc8439e7d0: { + balance: "0x7a48734599f7284", + nonce: 1133 + }, + 0xc8ba32cab1757528daf49033e3673fae77dcf05d: { + balance: "0x0", + code: "0x608060405234801561001057600080fd5b50600436106100885760003560e01c8063a9c2d... + nonce: 1, + storage: { + 0x0000000000000000000000000000000000000000000000000000000000000000: "0x000000000000000000000000000000000000000000000000000000000024aea6", + 0x59fb7853eb21f604d010b94c123acbeae621f09ce15ee5d7616485b1e78a72e9: "0x00000000000000c42b56a52aedf18667c8ae258a0280a8912641c80c48cd9548", + 0x8d8ebb65ec00cb973d4fe086a607728fd1b9de14aa48208381eed9592f0dee9a: "0x00000000000000784ae4881e40b1f5ebb4437905fbb8a5914454123b0293b35f", + 0xff896b09014882056009dedb136458f017fcef9a4729467d0d00b4fd413fb1f1: "0x000000000000000e78ac39cb1c20e9edc753623b153705d0ccc487e31f9d6749" + } + } +} +``` + +Return (same call with `{diffMode: True}`): + +```terminal +{ + post: { + 0x35a9f94af726f07b5162df7e828cc9dc8439e7d0: { + nonce: 1135 + } + }, + pre: { + 0x35a9f94af726f07b5162df7e828cc9dc8439e7d0: { + balance: "0x7a48429e177130a", + nonce: 1134 + } + } +} +``` + + +### revertReasonTracer + +The `revertReasonTracer` is useful for analyzing failed transactions. If the transaction +reverted, the reason for the revert (according to the Solidity contract) is returned. +For any other failure, the error message is returned. + +Example: + +```js +> debug.traceTransaction('0x97695ffb034be7e1faeb372a564bb951ba4ebf4fee4caff2f9d1702497bb2b8b', { tracer: 'revertReasonTracer' }) +``` + +Returns: + +```terminal +"execution reverted: tokensMintedPerAddress exceed MAX_TOKENS_MINTED_PER_ADDRESS" +``` + +### noopTracer + +This tracer is noop. It returns an empty object and is only meant for testing the setup. + + +## Javascript tracers + +There are also a set of tracers written in Javascript. These are less performant than +the Go native tracers because of overheads associated with interpreting the Javascript +in Geth's Go environment. + +### bigram + +`bigramTracer` counts the opcode bigrams, i.e. how many times 2 opcodes were executed +one after the other. + +Example: + +```js +debug.traceCall({from: "0x35a9f94af726f07b5162df7e828cc9dc8439e7d0", to: "0xc8ba32cab1757528daf49033e3673fae77dcf05d", data: "0xd1a2eab2000000000000000000000000000000000000000000000000000000000024aea100000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000050000000204895cd480cc8412691a880028a25aec86786f1ed2aa5562bc400000000000000c6403c14f35be1da6f433eadbb6e9178a47fbc7c6c1d568d2f2b876e929089c8d8db646304fd001a187dc8a600000000000000000000000000000000"}, 'latest', {tracer: 'bigramTracer'}) +``` + +Returns: + +```terminal +{ + ADD-ADD: 1, + ADD-AND: 2, + ADD-CALLDATALOAD: 1, + ADD-DUP1: 2, + ADD-DUP2: 2, + ADD-GT: 1, + ADD-MLOAD: 1, + ADD-MSTORE: 4, + ADD-PUSH1: 1, + ADD-PUSH2: 4, + ADD-SLT: 1, + ADD-SWAP1: 10, + ADD-SWAP2: 1, + ADD-SWAP3: 1, + ADD-SWAP4: 3, + ADD-SWAP5: 1, + AND-DUP3: 2, + AND-ISZERO: 4, + ... + } + +``` + +### evmdis + +`evmdisTracer` returns sufficient information from a trace to perform +[evmdis](https://github.com/Arachnid/evmdis)-style disassembly + +Example: + +```js +> debug.traceCall({from: "0x35a9f94af726f07b5162df7e828cc9dc8439e7d0", to: "0xc8ba32cab1757528daf49033e3673fae77dcf05d", data: "0xd1a2eab2000000000000000000000000000000000000000000000000000000000024aea100000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000050000000204895cd480cc8412691a880028a25aec86786f1ed2aa5562bc400000000000000c6403c14f35be1da6f433eadbb6e9178a47fbc7c6c1d568d2f2b876e929089c8d8db646304fd001a187dc8a600000000000000000000000000000000"}, 'latest', {tracer: 'evmdisTracer'}) +``` + +Returns: + +```terminal +[{ + depth: 1, + len: 2, + op: 96, + result: ["80"] +}, { + depth: 1, + len: 2, + op: 96, + result: ["40"] +}, { + depth: 1, + op: 82, + result: [] +}, { + depth: 1, + op: 52, + result: ["0"] +}, { + depth: 1, + op: 128, + result: ["0", "0"] +}, { + depth: 1, + op: 21, + result: ["1"] +}, { + depth: 1, + len: 3, + op: 97, + result: ["10"] +}, { + depth: 1, + op: 87, + result: [] +}, { + depth: 1, + op: 91, + pc: 16, + result: [] +}, +... +``` + +### opcount + +`opcountTracer` counts the total number of opcodes executed and simply +returns the number. + +Example: + +```js +debug.traceCall({from: "0x35a9f94af726f07b5162df7e828cc9dc8439e7d0", to: "0xc8ba32cab1757528daf49033e3673fae77dcf05d", data: "0xd1a2eab2000000000000000000000000000000000000000000000000000000000024aea100000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000050000000204895cd480cc8412691a880028a25aec86786f1ed2aa5562bc400000000000000c6403c14f35be1da6f433eadbb6e9178a47fbc7c6c1d568d2f2b876e929089c8d8db646304fd001a187dc8a600000000000000000000000000000000"}, 'latest', {tracer: 'opcountTracer'}) +``` + +Returns: + +```terminal +1384 +``` + +### trigram +`trigramTracer` counts the opcode trigrams. Trigrams are the possible combinations +of three opcodes this tracer reports how many times each combination is seen during +execution. + +Example: + +```js +debug.traceCall({from: "0x35a9f94af726f07b5162df7e828cc9dc8439e7d0", to: "0xc8ba32cab1757528daf49033e3673fae77dcf05d", data: "0xd1a2eab2000000000000000000000000000000000000000000000000000000000024aea100000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000050000000204895cd480cc8412691a880028a25aec86786f1ed2aa5562bc400000000000000c6403c14f35be1da6f433eadbb6e9178a47fbc7c6c1d568d2f2b876e929089c8d8db646304fd001a187dc8a600000000000000000000000000000000"}, 'latest', {tracer: 'trigramTracer'}) +``` + +Returns: +```terminal +{ + --PUSH1: 1, + -PUSH1-MSTORE: 1, + ADD-ADD-GT: 1, + ADD-AND-DUP3: 2, + ADD-CALLDATALOAD-PUSH8: 1, + ADD-DUP1-PUSH1: 2, + ADD-DUP2-ADD: 1, + ADD-DUP2-MSTORE: 1, + ADD-GT-ISZERO: 1, + ADD-MLOAD-DUP6: 1, + ADD-MSTORE-ADD: 1, + ADD-MSTORE-PUSH1: 2, + ADD-MSTORE-PUSH32: 1, + ADD-PUSH1-KECCAK256: 1, + ADD-PUSH2-JUMP: 2, + ADD-PUSH2-JUMPI: 1, + ADD-PUSH2-SWAP2: 1, + ADD-SLT-PUSH2: 1, +... +} +``` + + +### unigram + +`unigramTracer` counts the frequency of occurrance of each opcode. + +Example: +```js +> debug.traceCall({from: "0x35a9f94af726f07b5162df7e828cc9dc8439e7d0", to: "0xc8ba32cab1757528daf49033e3673fae77dcf05d", data: "0xd1a2eab2000000000000000000000000000000000000000000000000000000000024aea100000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000050000000204895cd480cc8412691a880028a25aec86786f1ed2aa5562bc400000000000000c6403c14f35be1da6f433eadbb6e9178a47fbc7c6c1d568d2f2b876e929089c8d8db646304fd001a187dc8a600000000000000000000000000000000"}, 'latest', {tracer: 'unigramTracer'}) +``` + +Returns: +```terminal +{ + ADD: 36, + AND: 23, + BYTE: 4, + CALLDATACOPY: 1, + CALLDATALOAD: 6, + CALLDATASIZE: 2, + CALLVALUE: 1, + DIV: 9, + DUP1: 29, + DUP10: 2, + DUP11: 1, + DUP12: 3, + DUP13: 2, + ... + } + +``` + +## State overrides + +It is possible to give temporary state modifications to Geth in order to simulate +the effects of `eth_call`. For example, some new byetcode could be deployed to +some address *temporarily just for the duration of the execution* and then a +transaction interacting with that address canm be traced. This can be used for +scenario testing or determining the outcome of some hypothetical transaction +before executing for real. + + +To do this, the tracer is written as normal, but the parameter `stateOverrides` is +passed an address and some bytecode. + +```js +var code = //contract bytecode +var tracer = //tracer name +debug.traceCall({from: , to: , input: }, 'latest', {stateOverrides: {'0x...': {code: code}}, tracer: tracer}) +``` + +## Summary + +This page showed how to use the tracers that come bundled with Geth. There are a set +written in Go and a set written in Javascript. They are invoked by passing their names +when calling an API method. State overrides can be used in combination with tracers to +examine precisely what the EVM will do in some hypothetical scenario. \ No newline at end of file diff --git a/docs/_evm-tracing/custom-tracer.md b/docs/_evm-tracing/custom-tracer.md new file mode 100644 index 0000000000..84c0c3ebf2 --- /dev/null +++ b/docs/_evm-tracing/custom-tracer.md @@ -0,0 +1,478 @@ +--- +title: Custom EVM tracer +sort_key: C +--- + +In addition to the default opcode tracer and the built-in tracers, Geth offers the +possibility to write custom code that hook to events in the EVM to process and return +the data in a consumable format. Custom tracers can be written either in Javascript +or Go. JS tracers are good for quick prototyping and experimentation as well as for +less intensive applications. Go tracers are performant but require the tracer to be +compiled together with the Geth source code. + +* TOC +{:toc} + +## Custom Javascript tracing + +Transaction traces include the complete status of the EVM at every point during the +transaction execution, which can be a very large amount of data. Often, users are +only interested in a small subset of that data. Javascript trace filters are available +to isolate the useful information. Detailed information about `debug_traceTransaction` +and its component parts is available in the +[reference documentation](/content/docs/developers/interacting-with-geth/rpc/ns-debug#debug_tracetransaction). + +### A simple filter + +Filters are Javascript functions that select information from the trace to persist +and discard based on some conditions. The following Javascript function returns +only the sequence of opcodes executed by the transaction as a comma-separated list. +The function could be written directly in the Javascript console, but it is cleaner +to write it in a separate re-usable file and load it into the console. + +1. Create a file, `filterTrace_1.js`, with this content: + + ```javascript + tracer = function (tx) { + return debug.traceTransaction(tx, { + tracer: + '{' + + 'retVal: [],' + + 'step: function(log,db) {this.retVal.push(log.getPC() + ":" + log.op.toString())},' + + 'fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))},' + + 'result: function(ctx,db) {return this.retVal}' + + '}' + }); // return debug.traceTransaction ... + }; // tracer = function ... + ``` + +2. Run the [JavaScript console](https://geth.ethereum.org/docs/interface/javascript-console). +3. Get the hash of a recent transaction from a node or block explorer. + +4. Run this command to run the script: + + ```javascript + loadScript('filterTrace_1.js'); + ``` + +5. Run the tracer from the script. Be patient, it could take a long time. + + ```javascript + tracer(''); + ``` + + The bottom of the output looks similar to: + + ```sh + "3366:POP", "3367:JUMP", "1355:JUMPDEST", "1356:PUSH1", "1358:MLOAD", "1359:DUP1", "1360:DUP3", "1361:ISZERO", "1362:ISZERO", + "1363:ISZERO", "1364:ISZERO", "1365:DUP2", "1366:MSTORE", "1367:PUSH1", "1369:ADD", "1370:SWAP2", "1371:POP", "1372:POP", "1373:PUSH1", + "1375:MLOAD", "1376:DUP1", "1377:SWAP2", "1378:SUB", "1379:SWAP1", "1380:RETURN" + ``` + +6. Run this line to get a more readable output with each string in its own line. + + ```javascript + console.log(JSON.stringify(tracer(''), null, 2)); + ``` + +More information about the `JSON.stringify` function is available +[here](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify). + +The commands above worked by calling the same `debug.traceTransaction` function that +was previously explained in [basic traces](https://geth.ethereum.org/docs/dapp/tracing), +but with a new parameter, `tracer`. This parameter takes the JavaScript object formated +as a string. In the case of the trace above, it is: + +```javascript +{ + retVal: [], + step: function(log,db) {this.retVal.push(log.getPC() + ":" + log.op.toString())}, + fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))}, + result: function(ctx,db) {return this.retVal} +} +``` + +This object has three member functions: + +- `step`, called for each opcode. +- `fault`, called if there is a problem in the execution. +- `result`, called to produce the results that are returned by `debug.traceTransaction` +- after the execution is done. + +In this case, `retVal` is used to store the list of strings to return in `result`. + +The `step` function adds to `retVal` the program counter and the name of the opcode there. +Then, in `result`, this list is returned to be sent to the caller. + +### Filtering with conditions + +For actual filtered tracing we need an `if` statement to only log relevant information. +For example, to isolate the transaction's interaction with storage, the following tracer +could be used: + +```javascript +tracer = function (tx) { + return debug.traceTransaction(tx, { + tracer: + '{' + + 'retVal: [],' + + 'step: function(log,db) {' + + ' if(log.op.toNumber() == 0x54) ' + + ' this.retVal.push(log.getPC() + ": SLOAD");' + + ' if(log.op.toNumber() == 0x55) ' + + ' this.retVal.push(log.getPC() + ": SSTORE");' + + '},' + + 'fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))},' + + 'result: function(ctx,db) {return this.retVal}' + + '}' + }); // return debug.traceTransaction ... +}; // tracer = function ... +``` + +The `step` function here looks at the opcode number of the op, and only pushes +an entry if the opcode is `SLOAD` or `SSTORE` ([here is a list of EVM opcodes and +their numbers](https://github.com/wolflo/evm-opcodes)). We could have used +`log.op.toString()` instead, but it is faster to compare numbers rather than strings. + +The output looks similar to this: + +```javascript +[ + "5921: SLOAD", + . + . + . + "2413: SSTORE", + "2420: SLOAD", + "2475: SSTORE", + "6094: SSTORE" +] +``` + +### Stack Information + +The trace above reports the program counter (PC) and whether the program read from storage +or wrote to it. That alone isn't particularly useful. To know more, the `log.stack.peek` +function can be used to peek into the stack. `log.stack.peek(0)` is the stack top, +`log.stack.peek(1)` the entry below it, etc. + +The values returned by `log.stack.peek` are Go `big.Int` objects. By default they are +converted to JavaScript floating point numbers, so you need `toString(16)` to get them +as hexadecimals, which is how 256-bit values such as storage cells and their content +are normally represented. + +#### Storage Information + +The function below provides a trace of all the storage operations and their parameters. +This gives a more complete picture of the program's interaction with storage. + +```javascript +tracer = function (tx) { + return debug.traceTransaction(tx, { + tracer: + '{' + + 'retVal: [],' + + 'step: function(log,db) {' + + ' if(log.op.toNumber() == 0x54) ' + + ' this.retVal.push(log.getPC() + ": SLOAD " + ' + + ' log.stack.peek(0).toString(16));' + + ' if(log.op.toNumber() == 0x55) ' + + ' this.retVal.push(log.getPC() + ": SSTORE " +' + + ' log.stack.peek(0).toString(16) + " <- " +' + + ' log.stack.peek(1).toString(16));' + + '},' + + 'fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))},' + + 'result: function(ctx,db) {return this.retVal}' + + '}' + }); // return debug.traceTransaction ... +}; // tracer = function ... +``` + +The output is similar to: + +```javascript +[ + "5921: SLOAD 0", + . + . + . + "2413: SSTORE 3f0af0a7a3ed17f5ba6a93e0a2a05e766ed67bf82195d2dd15feead3749a575d <- fb8629ad13d9a12456", + "2420: SLOAD cc39b177dd3a7f50d4c09527584048378a692aed24d31d2eabeddb7f3c041870", + "2475: SSTORE cc39b177dd3a7f50d4c09527584048378a692aed24d31d2eabeddb7f3c041870 <- 358c3de691bd19", + "6094: SSTORE 0 <- 1" +] +``` + +#### Operation Results + +One piece of information missing from the function above is the result on an `SLOAD` +operation. The state we get inside `log` is the state prior to the execution of the +opcode, so that value is not known yet. For more operations we can figure it out for +ourselves, but we don't have access to the +storage, so here we can't. + +The solution is to have a flag, `afterSload`, which is only true in the opcode right +after an `SLOAD`, when we can see the result at the top of the stack. + +```javascript +tracer = function (tx) { + return debug.traceTransaction(tx, { + tracer: + '{' + + 'retVal: [],' + + 'afterSload: false,' + + 'step: function(log,db) {' + + ' if(this.afterSload) {' + + ' this.retVal.push(" Result: " + ' + + ' log.stack.peek(0).toString(16)); ' + + ' this.afterSload = false; ' + + ' } ' + + ' if(log.op.toNumber() == 0x54) {' + + ' this.retVal.push(log.getPC() + ": SLOAD " + ' + + ' log.stack.peek(0).toString(16));' + + ' this.afterSload = true; ' + + ' } ' + + ' if(log.op.toNumber() == 0x55) ' + + ' this.retVal.push(log.getPC() + ": SSTORE " +' + + ' log.stack.peek(0).toString(16) + " <- " +' + + ' log.stack.peek(1).toString(16));' + + '},' + + 'fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))},' + + 'result: function(ctx,db) {return this.retVal}' + + '}' + }); // return debug.traceTransaction ... +}; // tracer = function ... +``` + +The output now contains the result in the line that follows the `SLOAD`. + +```javascript +[ + "5921: SLOAD 0", + " Result: 1", + . + . + . + "2413: SSTORE 3f0af0a7a3ed17f5ba6a93e0a2a05e766ed67bf82195d2dd15feead3749a575d <- fb8629ad13d9a12456", + "2420: SLOAD cc39b177dd3a7f50d4c09527584048378a692aed24d31d2eabeddb7f3c041870", + " Result: 0", + "2475: SSTORE cc39b177dd3a7f50d4c09527584048378a692aed24d31d2eabeddb7f3c041870 <- 358c3de691bd19", + "6094: SSTORE 0 <- 1" +] +``` + +### Dealing With Calls Between Contracts + +So the storage has been treated as if there are only 2256 cells. However, +that is not true. Contracts can call other contracts, and then the storage involved +is the storage of the other contract. We can see the address of the current contract +in `log.contract.getAddress()`. This value is the execution context - the contract +whose storage we are using - even when code from another contract is executed (by +using +[`CALLCODE` or `DELEGATECALL`](https://docs.soliditylang.org/en/v0.8.14/introduction-to-smart-contracts.html#delegatecall-callcode-and-libraries)). + +However, `log.contract.getAddress()` returns an array of bytes. To convert this to +the familiar hexadecimal representation of Ethereum addresses, `this.byteHex()` +and `array2Hex()` can be used. + +```javascript +tracer = function (tx) { + return debug.traceTransaction(tx, { + tracer: + '{' + + 'retVal: [],' + + 'afterSload: false,' + + 'callStack: [],' + + 'byte2Hex: function(byte) {' + + ' if (byte < 0x10) ' + + ' return "0" + byte.toString(16); ' + + ' return byte.toString(16); ' + + '},' + + 'array2Hex: function(arr) {' + + ' var retVal = ""; ' + + ' for (var i=0; i 0 { + t.env.Cancel() + return + } + + name := op.String() + if _, ok := t.counts[name]; !ok { + t.counts[name] = 0 + } + t.counts[name]++ +} + +// CaptureEnter is called when EVM enters a new scope (via call, create or selfdestruct). +func (t *opcounter) CaptureEnter(op vm.OpCode, from common.Address, to common.Address, input []byte, gas uint64, value *big.Int) {} + +// CaptureExit is called when EVM exits a scope, even if the scope didn't +// execute any code. +func (t *opcounter) CaptureExit(output []byte, gasUsed uint64, err error) {} + +// CaptureFault implements the EVMLogger interface to trace an execution fault. +func (t *opcounter) CaptureFault(pc uint64, op vm.OpCode, gas, cost uint64, scope *vm.ScopeContext, depth int, err error) {} + +// CaptureEnd is called after the call finishes to finalize the tracing. +func (t *opcounter) CaptureEnd(output []byte, gasUsed uint64, _ time.Duration, err error) {} + +func (*opcounter) CaptureTxStart(gasLimit uint64) {} + +func (*opcounter) CaptureTxEnd(restGas uint64) {} + +// GetResult returns the json-encoded nested list of call traces, and any +// error arising from the encoding or forceful termination (via `Stop`). +func (t *opcounter) GetResult() (json.RawMessage, error) { + res, err := json.Marshal(t.counts) + if err != nil { + return nil, err + } + return res, t.reason +} + +// Stop terminates execution of the tracer at the first opportune moment. +func (t *opcounter) Stop(err error) { + t.reason = err + atomic.StoreUint32(&t.interrupt, 1) +} +``` + +Every method of the +[EVMLogger interface](https://pkg.go.dev/github.com/ethereum/go-ethereum/core/vm#EVMLogger) +needs to be implemented (even if empty). Key parts to notice are the `init()` function +which registers the tracer in Geth, the `CaptureState` hook where the opcode counts are +incremented and `GetResult` where the result is serialized and delivered. To test this, +the source is first compiled with `make geth`. Then in the console it can be invoked +through the usual API methods by passing in the name it was registered under: + +```console +> debug.traceTransaction('0x7ae446a7897c056023a8104d254237a8d97783a92900a7b0f7db668a9432f384', { tracer: 'opcounter' }) +{ + ADD: 4, + AND: 3, + CALLDATALOAD: 2, + ... +} +``` + +## Summary + +This page described how to write custom tracers for Geth. Custom tracers can be written in Javascript or Go. \ No newline at end of file diff --git a/docs/_evm-tracing/index.md b/docs/_evm-tracing/index.md new file mode 100644 index 0000000000..0c7cb84b8d --- /dev/null +++ b/docs/_evm-tracing/index.md @@ -0,0 +1,84 @@ +--- +title: EVM Tracing +sort-key: A +--- + +Tracing allows users to examine precisely what was executed by the EVM during some specific transaction or set of transactions. There are two different types of [transactions](https://ethereum.org/en/developers/docs/transactions) in Ethereum: value transfers and contract executions. A value transfer just moves ETH from one account to another. A contract interaction executes some code stored at a contract address which can include altering stored data and transacting multiple times with other contracts and externally-owned accounts. A contract execution transaction can therefore be a complicated web of interactions that can be difficult to unpick. The transaction receipt contains a status code that shows whether the transaction succeeded or failed, but more detailed information is not readily available, meaning it is very difficult to know what a contract execution actually did, what data was modified and which addresses were touched. This is the problem that EVM tracing solves. Geth traces transactions by re-running them locally and collecting data about precisely what was executed by the EVM. + +Also see this [Devcon 2022 talk](https://www.youtube.com/watch?v=b8RdmGsilfU) on tracing in Geth. + +## State availability + +In its simplest form, tracing a transaction entails requesting the Ethereum node to reexecute the desired transaction with varying degrees of data collection and have it return an aggregated summary. In order for a Geth node to reexecute a transaction, all historical state accessed by the transaction must be available. This includes: + +- Balance, nonce, bytecode and storage of both the recipient as well as all internally invoked contracts. +- Block metadata referenced during execution of both the outer as well as all internally created transactions. +- Intermediate state generated by all preceding transactions contained in the same block as the one being traced. + +This means there are limits on the transactions that can be traced imposed by the synchronization and pruning configuration of a node: + +- An **archive** node retains **all historical data** back to genesis. It can therefore trace arbitrary transactions at any point in the history of the chain. Tracing a single transaction requires reexecuting all preceding transactions in the same block. + +- A **node synced from genesis** node only retains the most recent 128 block states in memory. Older states are represented by a sequence of occasional checkpoints that intermediate states can be regenerated from. This means that states within the msot recent 128 blocks are immediately available, older states have to be regenerated from snapshots "on-the-fly". If the distance between the requested transaction and the most recent checkpoint is large, rebuilding the state can take a long time. Tracing a single transaction requires reexecuting all preceding transactions in the same block **and** all preceding blocks until the previous stored snapshot. + +- A **snap synced** node holds the most recent 128 blocks in memory, so transactions in that range are always accessible. However, snap-sync only starts processing from a relatively recent block (as opposed to genesis for a full node). Between the initial sync block and the 128 most recent blocks, the node stores occasional checkpoints that can be used to rebuild the state on-the-fly. This means transactions can be traced back as far as the block that was used for the initial sync. Tracing a single transaction requires reexecuting all preceding transactions in the same block, + **and** all preceding blocks until the previous stored snapshot. + +- A **light synced** node retrieving data **on demand** can in theory trace transactions for which all required historical state is readily available in the network. This is because the data required to generate the trace is requested from an les-serving full node. In practice, data + availability **cannot** be reasonably assumed. + +![state pruning options](/static/images/state-pruning.png) + +*This image shows the state stored by each sync-mode - red indicates stored state. The full width of each line represents origin to present head* + + +More detailed information about syncing is available on the [sync modes page](/pages/docs/fundamentals/sync-modes.md). + +When a trace of a specific transaction is executed, the state is prepared by fetching the state of the parent block from the database. If it is not available, Geth will crawl backwards in time to find the next available state but only up to a limit defined in the `reexec` parameter which defaults to 128 blocks. If no state is available within the `reexec` window then the trace fails with `Error: required historical state unavailable` and the `reexec` parameter must be increased. If a valid state *is* found in the `reexec` window, then Geth sequentially re-executes the transcations in each block between the last available state and the target block. The greater the value of `reexec` the longer the tracing will take because more blocks have to be re-executed to regenerate the target state. + +The `debug_getAccessibleStates` endpoint is a useful tool for estimating a suitable value for `reexec`. Passing the number of the block that contains the target transaction and a search distance to this endpoint will return the number of blocks behind the current head where the most recent available state exists. This value can be passed to the tracer as `re-exec`. + +It is also possible to force Geth to store the state for specific sequences of block by stopping Geth, running again with `--gcmode archive` for some period - this prevents state prunign for blocks that arrive while Geth is running with `--gcmode archive`. + +_There are exceptions to the above rules when running batch traces of entire blocks or chain segments. Those will be detailed later._ + +## Types of trace + +### Built-in tracers + +The tracing API accepts an optional `tracer` parameter that defines how the data +returned to the API call should be processed. If this parameter is ommitted the +default tracer is used. The default is the struct (or 'opcode') logger. These raw +opcode traces are sometimes useful, but the returned data is very low level and +can be too extensive and awkward to read for many use-cases. A full opcode trace +can easily go into the hundreds of megabytes, making them very resource intensive +to get out of the node and process externally. For these reasons, there are a set +of non-default built-in tracers that can be named in the API call to return +different data from the method. Under the hood, these tracers are Go or Javascript +functions that do some specific preprocessing on the trace data before it is returned. + +More information about Geth's built-in tracers is available on the +[built-in tracers](/pages/docs/developers/dapp-developer/evm-tracing/built-in-tracers.md) +page. + + +### Custom tracers + +In addition to built-in tracers, it is possible to provide custom code that hooks +to events in the EVM to process and return data in a consumable format. Custom +tracers can be written either in Javascript or Go. JS tracers are good for quick +prototyping and experimentation as well as for less intensive applications. Go +tracers are performant but require the tracer to be compiled together with the +Geth source code. This means developers only have to gather the data they actually +need, and do any processing at the source. + +More information about custom tracers is available on the +[custom tracers](/pages/docs/developers/dapp-developer/evm-tracing/custom-tracers.md) +page. + + +## Summary + +This page gave an introduction to the concept of tracing and explained issues around +state availability. More detailed information on Geth's built-in and custom tracers +can be found on their dedicated pages. \ No newline at end of file diff --git a/static/images/state-pruning.png b/static/images/state-pruning.png new file mode 100644 index 0000000000..c4bfd45558 Binary files /dev/null and b/static/images/state-pruning.png differ diff --git a/static/styles/custom/common.css b/static/styles/custom/common.css index 2ce921806d..401a0d3c99 100644 --- a/static/styles/custom/common.css +++ b/static/styles/custom/common.css @@ -2,7 +2,9 @@ body { padding-top: 50px; } -h1, h2, h3 { +h1, +h2, +h3 { margin-top: 36px; margin-bottom: 24px; } @@ -23,8 +25,13 @@ code { overflow-x: scroll; overflow-y: hidden; } + /* Darken down code highlighting */ -.highlight { color: #4b5a5b } /* Name */ +.highlight { + color: #4b5a5b +} + +/* Name */ a code { color: #18bc9c; @@ -44,9 +51,9 @@ a code { } .emoji { - width: 1.5em; - height: 1.5em; - display: inline-block; + width: 1.5em; + height: 1.5em; + display: inline-block; margin-bottom: -0.25em; } @@ -58,7 +65,9 @@ table { margin-top: 16px; margin-bottom: 16px; } -table thead tr th, table tbody tr td { + +table thead tr th, +table tbody tr td { border-bottom: 1px solid #ddd; padding: 4px; } @@ -68,24 +77,37 @@ table thead tr th, table tbody tr td { } -img{ +img { margin-top: 2rem; margin-bottom: 2rem } -img[alt="Remix"]{ - width: 800px; +img[alt="state pruning options"] { + width: 850px; margin-top: 2rem; margin-bottom: 2rem; } -img[alt="Clef ruleset logic"]{ - width: 800px; +img[alt="Peter's Devcon2 talk"] { + width: 300px; margin-top: 2rem; margin-bottom: 2rem; } -img[alt="Clef signing logic"]{ - width: 800px; + +img[alt="Remix"] { + width: 800px; + margin-top: 2rem; + margin-bottom: 2rem; +} + +img[alt="Clef ruleset logic"] { + width: 800px; + margin-top: 2rem; + margin-bottom: 2rem; +} + +img[alt="Clef signing logic"] { + width: 800px; margin-top: 2rem; margin-bottom: 2rem;