cmd/evm: implement input txs via rlp in t8n tool (#23138)

In many cases, it's desireable to use already-signed transactions as input to the state transition, instead of having the evm sign them internally (for example to use malformed or not-yet-valid transactions). This PR adds support + docs for that feature.

cmd/evm/README.md

@@ -37,6 +37,8 @@ Command line params that has to be supported are
    --output.result result             Determines where to put the result (stateroot, txroot etc) of the post-state.
                                       `stdout` - into the stdout output
                                       `stderr` - into the stderr output
+   --output.body value                If set, the RLP of the transactions (block body) will be written to this file.
+   --input.txs stdin                  stdin or file name of where to find the transactions to apply. If the file prefix is '.rlp', then the data is interpreted as an RLP list of signed transactions.The '.rlp' format is identical to the output.body format. (default: "txs.json")
    --state.fork value                 Name of ruleset to use.
    --state.chainid value              ChainID to use (default: 1)
    --state.reward value               Mining reward. Set to -1 to disable (default: 0)
@@ -110,7 +112,10 @@ Two resulting files:
   }
  ],
  "rejected": [
-  1
+  {
+   "index": 1,
+   "error": "nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+  }
  ]
 }
 ```
@@ -156,7 +161,10 @@ Output:
    }
   ],
   "rejected": [
-   1
+   {
+    "index": 1,
+    "error": "nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+   }
   ]
  }
 }
@@ -231,34 +239,46 @@ The `BLOCKHASH` opcode requires blockhashes to be provided by the caller, inside
 If a required blockhash is not provided, the exit code should be `4`:
 Example where blockhashes are provided:
 ```
-./evm t8n --input.alloc=./testdata/3/alloc.json --input.txs=./testdata/3/txs.json --input.env=./testdata/3/env.json --trace
+./evm --verbosity=1 t8n --input.alloc=./testdata/3/alloc.json --input.txs=./testdata/3/txs.json --input.env=./testdata/3/env.json  --trace
+INFO [07-27|11:53:40.960] Trie dumping started                     root=b7341d..857ea1
+INFO [07-27|11:53:40.960] Trie dumping complete                    accounts=3 elapsed="103.298µs"
+INFO [07-27|11:53:40.960] Wrote file                               file=alloc.json
+INFO [07-27|11:53:40.960] Wrote file                               file=result.json
+
 ```
+
 ```
 cat trace-0-0x72fadbef39cd251a437eea619cfeda752271a5faaaa2147df012e112159ffb81.jsonl | grep BLOCKHASH -C2
 ```
 ```
-{"pc":0,"op":96,"gas":"0x5f58ef8","gasCost":"0x3","memory":"0x","memSize":0,"stack":[],"returnStack":[],"returnData":"0x","depth":1,"refund":0,"opName":"PUSH1","error":""}
+{"pc":0,"op":96,"gas":"0x5f58ef8","gasCost":"0x3","memory":"0x","memSize":0,"stack":[],"returnData":"0x","depth":1,"refund":0,"opName":"PUSH1","error":""}
-{"pc":2,"op":64,"gas":"0x5f58ef5","gasCost":"0x14","memory":"0x","memSize":0,"stack":["0x1"],"returnStack":[],"returnData":"0x","depth":1,"refund":0,"opName":"BLOCKHASH","error":""}
+{"pc":2,"op":64,"gas":"0x5f58ef5","gasCost":"0x14","memory":"0x","memSize":0,"stack":["0x1"],"returnData":"0x","depth":1,"refund":0,"opName":"BLOCKHASH","error":""}
-{"pc":3,"op":0,"gas":"0x5f58ee1","gasCost":"0x0","memory":"0x","memSize":0,"stack":["0xdac58aa524e50956d0c0bae7f3f8bb9d35381365d07804dd5b48a5a297c06af4"],"returnStack":[],"returnData":"0x","depth":1,"refund":0,"opName":"STOP","error":""}
+{"pc":3,"op":0,"gas":"0x5f58ee1","gasCost":"0x0","memory":"0x","memSize":0,"stack":["0xdac58aa524e50956d0c0bae7f3f8bb9d35381365d07804dd5b48a5a297c06af4"],"returnData":"0x","depth":1,"refund":0,"opName":"STOP","error":""}
-{"output":"","gasUsed":"0x17","time":142709}
+{"output":"","gasUsed":"0x17","time":156276}
 ```
 
 In this example, the caller has not provided the required blockhash:
 ```
 ./evm t8n --input.alloc=./testdata/4/alloc.json --input.txs=./testdata/4/txs.json --input.env=./testdata/4/env.json --trace
-```
-```
 ERROR(4): getHash(3) invoked, blockhash for that block not provided
 ```
 Error code: 4
+
 ### Chaining
 
 Another thing that can be done, is to chain invocations:
 ```
 ./evm t8n --input.alloc=./testdata/1/alloc.json --input.txs=./testdata/1/txs.json --input.env=./testdata/1/env.json --output.alloc=stdout | ./evm t8n --input.alloc=stdin --input.env=./testdata/1/env.json --input.txs=./testdata/1/txs.json
-INFO [01-21|22:41:22.963] rejected tx                              index=1 hash=0557ba..18d673 from=0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192 error="nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+INFO [07-27|11:53:41.049] rejected tx                              index=1 hash=0557ba..18d673 from=0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192 error="nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
-INFO [01-21|22:41:22.966] rejected tx                              index=0 hash=0557ba..18d673 from=0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192 error="nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+INFO [07-27|11:53:41.050] Trie dumping started                     root=84208a..ae4e13
-INFO [01-21|22:41:22.967] rejected tx                              index=1 hash=0557ba..18d673 from=0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192 error="nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+INFO [07-27|11:53:41.050] Trie dumping complete                    accounts=3 elapsed="59.412µs"
+INFO [07-27|11:53:41.050] Wrote file                               file=result.json
+INFO [07-27|11:53:41.051] rejected tx                              index=0 hash=0557ba..18d673 from=0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192 error="nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+INFO [07-27|11:53:41.051] rejected tx                              index=1 hash=0557ba..18d673 from=0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192 error="nonce too low: address 0x8A8eAFb1cf62BfBeb1741769DAE1a9dd47996192, tx: 0 state: 1"
+INFO [07-27|11:53:41.052] Trie dumping started                     root=84208a..ae4e13
+INFO [07-27|11:53:41.052] Trie dumping complete                    accounts=3 elapsed="45.734µs"
+INFO [07-27|11:53:41.052] Wrote file                               file=alloc.json
+INFO [07-27|11:53:41.052] Wrote file                               file=result.json
 
 ```
 What happened here, is that we first applied two identical transactions, so the second one was rejected.
@@ -267,3 +287,52 @@ the same two transactions: this time, both failed due to too low nonce.
 
 In order to meaningfully chain invocations, one would need to provide meaningful new `env`, otherwise the
 actual blocknumber (exposed to the EVM) would not increase.
+
+### Transactions in RLP form
+
+It is possible to provide already-signed transactions as input to, using an `input.txs` which ends with the `rlp` suffix.
+The input format for RLP-form transactions is _identical_ to the _output_ format for block bodies. Therefore, it's fully possible
+to use the evm to go from `json` input to `rlp` input.
+
+The following command takes **json** the transactions in `./testdata/13/txs.json` and signs them. After execution, they are output to `signed_txs.rlp`.:
+```
+./evm t8n --state.fork=London --input.alloc=./testdata/13/alloc.json --input.txs=./testdata/13/txs.json --input.env=./testdata/13/env.json --output.result=alloc_jsontx.json --output.body=signed_txs.rlp
+INFO [07-27|11:53:41.124] Trie dumping started                     root=e4b924..6aef61
+INFO [07-27|11:53:41.124] Trie dumping complete                    accounts=3 elapsed="94.284µs"
+INFO [07-27|11:53:41.125] Wrote file                               file=alloc.json
+INFO [07-27|11:53:41.125] Wrote file                               file=alloc_jsontx.json
+INFO [07-27|11:53:41.125] Wrote file                               file=signed_txs.rlp
+
+```
+
+The `output.body` is the rlp-list of transactions, encoded in hex and placed in a string a'la `json` encoding rules:
+```
+cat signed_txs.rlp
+"0xf8d2b86702f864010180820fa08284d09411111111111111111111111111111111111111118080c001a0b7dfab36232379bb3d1497a4f91c1966b1f932eae3ade107bf5d723b9cb474e0a06261c359a10f2132f126d250485b90cf20f30340801244a08ef6142ab33d1904b86702f864010280820fa08284d09411111111111111111111111111111111111111118080c080a0d4ec563b6568cd42d998fc4134b36933c6568d01533b5adf08769270243c6c7fa072bf7c21eac6bbeae5143371eef26d5e279637f3bd73482b55979d76d935b1e9"
+```
+
+We can use `rlpdump` to check what the contents are:
+```
+rlpdump -hex $(cat signed_txs.rlp | jq -r )
+[
+  02f864010180820fa08284d09411111111111111111111111111111111111111118080c001a0b7dfab36232379bb3d1497a4f91c1966b1f932eae3ade107bf5d723b9cb474e0a06261c359a10f2132f126d250485b90cf20f30340801244a08ef6142ab33d1904,
+  02f864010280820fa08284d09411111111111111111111111111111111111111118080c080a0d4ec563b6568cd42d998fc4134b36933c6568d01533b5adf08769270243c6c7fa072bf7c21eac6bbeae5143371eef26d5e279637f3bd73482b55979d76d935b1e9,
+]
+```
+Now, we can now use those (or any other already signed transactions), as input, like so:
+```
+./evm t8n --state.fork=London --input.alloc=./testdata/13/alloc.json --input.txs=./signed_txs.rlp --input.env=./testdata/13/env.json --output.result=alloc_rlptx.json
+INFO [07-27|11:53:41.253] Trie dumping started                     root=e4b924..6aef61
+INFO [07-27|11:53:41.253] Trie dumping complete                    accounts=3 elapsed="128.445µs"
+INFO [07-27|11:53:41.253] Wrote file                               file=alloc.json
+INFO [07-27|11:53:41.255] Wrote file                               file=alloc_rlptx.json
+
+```
+
+You might have noticed that the results from these two invocations were stored in two separate files.
+And we can now finally check that they match.
+```
+cat alloc_jsontx.json | jq .stateRoot && cat alloc_rlptx.json | jq .stateRoot
+"0xe4b924a6adb5959fccf769d5b7bb2f6359e26d1e76a2443c5a91a36d826aef61"
+"0xe4b924a6adb5959fccf769d5b7bb2f6359e26d1e76a2443c5a91a36d826aef61"
+```

cmd/evm/internal/t8ntool/flags.go

@@ -80,7 +80,9 @@ var (
 	}
 	InputTxsFlag = cli.StringFlag{
 		Name: "input.txs",
-		Usage: "`stdin` or file name of where to find the transactions to apply.",
+		Usage: "`stdin` or file name of where to find the transactions to apply. " +
+			"If the file prefix is '.rlp', then the data is interpreted as an RLP list of signed transactions." +
+			"The '.rlp' format is identical to the output.body format.",
 		Value: "txs.json",
 	}
 	RewardFlag = cli.Int64Flag{

cmd/evm/internal/t8ntool/transition.go

@@ -25,6 +25,7 @@ import (
 	"math/big"
 	"os"
 	"path"
+	"strings"
 
 	"github.com/ethereum/go-ethereum/common"
 	"github.com/ethereum/go-ethereum/common/hexutil"
@@ -72,6 +73,7 @@ type input struct {
 	Alloc core.GenesisAlloc `json:"alloc,omitempty"`
 	Env   *stEnv            `json:"env,omitempty"`
 	Txs   []*txWithKey      `json:"txs,omitempty"`
+	TxRlp string            `json:"txsRlp,omitempty"`
 }
 
 func Main(ctx *cli.Context) error {
@@ -199,12 +201,45 @@ func Main(ctx *cli.Context) error {
 		}
 		defer inFile.Close()
 		decoder := json.NewDecoder(inFile)
+		if strings.HasSuffix(txStr, ".rlp") {
+			var body hexutil.Bytes
+			if err := decoder.Decode(&body); err != nil {
+				return err
+			}
+			var txs types.Transactions
+			if err := rlp.DecodeBytes(body, &txs); err != nil {
+				return err
+			}
+			for _, tx := range txs {
+				txsWithKeys = append(txsWithKeys, &txWithKey{
+					key: nil,
+					tx:  tx,
+				})
+			}
+		} else {
 			if err := decoder.Decode(&txsWithKeys); err != nil {
 				return NewError(ErrorJson, fmt.Errorf("failed unmarshaling txs-file: %v", err))
 			}
+		}
+	} else {
+		if len(inputData.TxRlp) > 0 {
+			// Decode the body of already signed transactions
+			body := common.FromHex(inputData.TxRlp)
+			var txs types.Transactions
+			if err := rlp.DecodeBytes(body, &txs); err != nil {
+				return err
+			}
+			for _, tx := range txs {
+				txsWithKeys = append(txsWithKeys, &txWithKey{
+					key: nil,
+					tx:  tx,
+				})
+			}
 		} else {
+			// JSON encoded transactions
 			txsWithKeys = inputData.Txs
 		}
+	}
 	// We may have to sign the transactions.
 	signer := types.MakeSigner(chainConfig, big.NewInt(int64(prestate.Env.Number)))
 
@@ -365,6 +400,7 @@ func dispatchOutput(ctx *cli.Context, baseDir string, result *ExecutionResult, a
 			return NewError(ErrorJson, fmt.Errorf("failed marshalling output: %v", err))
 		}
 		os.Stdout.Write(b)
+		os.Stdout.Write([]byte("\n"))
 	}
 	if len(stdErrObject) > 0 {
 		b, err := json.MarshalIndent(stdErrObject, "", " ")
@@ -372,6 +408,7 @@ func dispatchOutput(ctx *cli.Context, baseDir string, result *ExecutionResult, a
 			return NewError(ErrorJson, fmt.Errorf("failed marshalling output: %v", err))
 		}
 		os.Stderr.Write(b)
+		os.Stderr.Write([]byte("\n"))
 	}
 	return nil
 }

cmd/evm/testdata/13/alloc.json

@@ -0,0 +1,23 @@
+{
+  "0x1111111111111111111111111111111111111111" : {
+    "balance" : "0x010000000000",
+    "code" : "0xfe",
+    "nonce" : "0x01",
+    "storage" : {
+    }
+  },
+  "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b" : {
+    "balance" : "0x010000000000",
+    "code" : "0x",
+    "nonce" : "0x01",
+    "storage" : {
+    }
+  },
+  "0xd02d72e067e77158444ef2020ff2d325f929b363" : {
+    "balance" : "0x01000000000000",
+    "code" : "0x",
+    "nonce" : "0x01",
+    "storage" : {
+    }
+  }
+}

cmd/evm/testdata/13/env.json

@@ -0,0 +1,12 @@
+{
+  "currentCoinbase" : "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba",
+  "currentDifficulty" : "0x020000",
+  "currentNumber" : "0x01",
+  "currentTimestamp" : "0x079e",
+  "previousHash" : "0xcb23ee65a163121f640673b41788ee94633941405f95009999b502eedfbbfd4f",
+  "currentGasLimit" : "0x40000000",
+  "currentBaseFee" : "0x036b",
+  "blockHashes" : {
+    "0" : "0xcb23ee65a163121f640673b41788ee94633941405f95009999b502eedfbbfd4f"
+  }
+}

cmd/evm/testdata/13/readme.md

@@ -0,0 +1,4 @@
+## Input transactions in RLP form
+
+This testdata folder is used to examplify how transaction input can be provided in rlp form. 
+Please see the README in `evm` folder for how this is performed. 

cmd/evm/testdata/13/txs.json

@@ -0,0 +1,34 @@
+[
+  {
+    "input" : "0x",
+    "gas" : "0x84d0",
+    "nonce" : "0x1",
+    "to" : "0x1111111111111111111111111111111111111111",
+    "value" : "0x0",
+    "v" : "0x0",
+    "r" : "0x0",
+    "s" : "0x0",
+    "secretKey" : "0x41f6e321b31e72173f8ff2e292359e1862f24fba42fe6f97efaf641980eff298",
+    "chainId" : "0x1",
+    "type" : "0x2",
+    "maxFeePerGas" : "0xfa0",
+    "maxPriorityFeePerGas" : "0x0",
+    "accessList" : []
+  },
+  {
+    "input" : "0x",
+    "gas" : "0x84d0",
+    "nonce" : "0x2",
+    "to" : "0x1111111111111111111111111111111111111111",
+    "value" : "0x0",
+    "v" : "0x0",
+    "r" : "0x0",
+    "s" : "0x0",
+    "secretKey" : "0x41f6e321b31e72173f8ff2e292359e1862f24fba42fe6f97efaf641980eff298",
+    "chainId" : "0x1",
+    "type" : "0x2",
+    "maxFeePerGas" : "0xfa0",
+    "maxPriorityFeePerGas" : "0x0",
+    "accessList" : []
+  }
+]

cmd/evm/transition-test.sh

@@ -11,6 +11,8 @@ function showjson(){
 function demo(){
   echo "$ticks"
   echo "$1"
+  $1
+  echo ""
   echo "$ticks"
   echo ""
 }
@@ -152,9 +154,7 @@ echo ""
 echo "The \`BLOCKHASH\` opcode requires blockhashes to be provided by the caller, inside the \`env\`."
 echo "If a required blockhash is not provided, the exit code should be \`4\`:"
 echo "Example where blockhashes are provided: "
-cmd="./evm t8n --input.alloc=./testdata/3/alloc.json --input.txs=./testdata/3/txs.json --input.env=./testdata/3/env.json  --trace"
+demo "./evm --verbosity=1 t8n --input.alloc=./testdata/3/alloc.json --input.txs=./testdata/3/txs.json --input.env=./testdata/3/env.json  --trace"
-tick && echo $cmd && tick
-$cmd 2>&1 >/dev/null
 cmd="cat trace-0-0x72fadbef39cd251a437eea619cfeda752271a5faaaa2147df012e112159ffb81.jsonl | grep BLOCKHASH -C2"
 tick && echo $cmd && tick
 echo "$ticks"
@@ -164,13 +164,11 @@ echo ""
 
 echo "In this example, the caller has not provided the required blockhash:"
 cmd="./evm t8n --input.alloc=./testdata/4/alloc.json --input.txs=./testdata/4/txs.json --input.env=./testdata/4/env.json  --trace"
-tick && echo $cmd && tick
+tick && echo $cmd && $cmd
-tick
-$cmd
 errc=$?
 tick
 echo "Error code: $errc"
-
+echo ""
 
 echo "### Chaining"
 echo ""
@@ -189,3 +187,28 @@ echo ""
 echo "In order to meaningfully chain invocations, one would need to provide meaningful new \`env\`, otherwise the"
 echo "actual blocknumber (exposed to the EVM) would not increase."
 echo ""
+
+echo "### Transactions in RLP form"
+echo ""
+echo "It is possible to provide already-signed transactions as input to, using an \`input.txs\` which ends with the \`rlp\` suffix."
+echo "The input format for RLP-form transactions is _identical_ to the _output_ format for block bodies. Therefore, it's fully possible"
+echo "to use the evm to go from \`json\` input to \`rlp\` input."
+echo ""
+echo "The following command takes **json** the transactions in \`./testdata/13/txs.json\` and signs them. After execution, they are output to \`signed_txs.rlp\`.:"
+demo "./evm t8n --state.fork=London --input.alloc=./testdata/13/alloc.json --input.txs=./testdata/13/txs.json --input.env=./testdata/13/env.json --output.result=alloc_jsontx.json --output.body=signed_txs.rlp"
+echo "The \`output.body\` is the rlp-list of transactions, encoded in hex and placed in a string a'la \`json\` encoding rules:"
+demo "cat signed_txs.rlp"
+echo "We can use \`rlpdump\` to check what the contents are: "
+echo "$ticks"
+echo "rlpdump -hex \$(cat signed_txs.rlp | jq -r )"
+rlpdump -hex $(cat signed_txs.rlp | jq -r )
+echo "$ticks"
+echo "Now, we can now use those (or any other already signed transactions), as input, like so: "
+demo "./evm t8n --state.fork=London --input.alloc=./testdata/13/alloc.json --input.txs=./signed_txs.rlp --input.env=./testdata/13/env.json --output.result=alloc_rlptx.json"
+
+echo "You might have noticed that the results from these two invocations were stored in two separate files. "
+echo "And we can now finally check that they match."
+echo "$ticks"
+echo "cat alloc_jsontx.json | jq .stateRoot && cat alloc_rlptx.json | jq .stateRoot"
+cat alloc_jsontx.json | jq .stateRoot && cat alloc_rlptx.json | jq .stateRoot
+echo "$ticks"