7088f1e814
core, eth: faster snapshot generation ( #22504 )
...
* eth/protocols: persist received state segments
* core: initial implementation
* core/state/snapshot: add tests
* core, eth: updates
* eth/protocols/snapshot: count flat state size
* core/state: add metrics
* core/state/snapshot: skip unnecessary deletion
* core/state/snapshot: rename
* core/state/snapshot: use the global batch
* core/state/snapshot: add logs and fix wiping
* core/state/snapshot: fix
* core/state/snapshot: save generation progress even if the batch is empty
* core/state/snapshot: fixes
* core/state/snapshot: fix initial account range length
* core/state/snapshot: fix initial account range
* eth/protocols/snap: store flat states during the healing
* eth/protocols/snap: print logs
* core/state/snapshot: refactor (#4 )
* core/state/snapshot: refactor
* core/state/snapshot: tiny fix and polish
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* core, eth: fixes
* core, eth: fix healing writer
* core, trie, eth: fix paths
* eth/protocols/snap: fix encoding
* eth, core: add debug log
* core/state/generate: release iterator asap (#5 )
core/state/snapshot: less copy
core/state/snapshot: revert split loop
core/state/snapshot: handle storage becoming empty, improve test robustness
core/state: test modified codehash
core/state/snapshot: polish
* core/state/snapshot: optimize stats counter
* core, eth: add metric
* core/state/snapshot: update comments
* core/state/snapshot: improve tests
* core/state/snapshot: replace secure trie with standard trie
* core/state/snapshot: wrap return as the struct
* core/state/snapshot: skip wiping correct states
* core/state/snapshot: updates
* core/state/snapshot: fixes
* core/state/snapshot: fix panic due to reference flaw in closure
* core/state/snapshot: fix errors in state generation logic + fix log output
* core/state/snapshot: remove an error case
* core/state/snapshot: fix condition-check for exhausted snap state
* core/state/snapshot: use stackTrie for small tries
* core/state/snapshot: don't resolve small storage tries in vain
* core/state/snapshot: properly clean up storage of deleted accounts
* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks
* core/state/snapshot: fix error (+testcase)
* core/state/snapshot: clean up tests a bit
* core/state/snapshot: work in progress on better tests
* core/state/snapshot: polish code
* core/state/snapshot: fix trie iteration abortion trigger
* core/state/snapshot: fixes flaws
* core/state/snapshot: remove panic
* core/state/snapshot: fix abort
* core/state/snapshot: more tests (plus failing testcase)
* core/state/snapshot: more testcases + fix for failing test
* core/state/snapshot: testcase for malformed data
* core/state/snapshot: some test nitpicks
* core/state/snapshot: improvements to logging
* core/state/snapshot: testcase to demo error in abortion
* core/state/snapshot: fix abortion
* cmd/geth: make verify-state report the root
* trie: fix failing test
* core/state/snapshot: add timer metrics
* core/state/snapshot: fix metrics
* core/state/snapshot: udpate tests
* eth/protocols/snap: write snapshot account even if code or state is needed
* core/state/snapshot: fix diskmore check
* core/state/snapshot: review fixes
* core/state/snapshot: improve error message
* cmd/geth: rename 'error' to 'err' in logs
* core/state/snapshot: fix some review concerns
* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync
* core: add error log
* core/state/snapshot: use proper timers for metrics collection
* core/state/snapshot: address some review concerns
* eth/protocols/snap: improved log message
* eth/protocols/snap: fix heal logs to condense infos
* core/state/snapshot: wait for generator termination before restarting
* core/state/snapshot: revert timers to counters to track total time
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
4 years ago
0c70b83e00
all: add read-only option to database ( #22407 )
...
* all: add read-only option to database
* all: fixes tests
* cmd/geth: migrate flags
* cmd/geth: fix the compact
* cmd/geth: fix the format
* cmd/geth: fix log
* cmd: add chain-readonly
* core: add readonly notion to freezer
* core/rawdb: add log
* core/rawdb: fix freezer close
* cmd: fix
* cmd, core: construct db
* core: update tests
4 years ago
bbfb1e4008
all: add support for EIP-2718, EIP-2930 transactions ( #21502 )
...
This adds support for EIP-2718 typed transactions as well as EIP-2930
access list transactions (tx type 1). These EIPs are scheduled for the
Berlin fork.
There very few changes to existing APIs in core/types, and several new APIs
to deal with access list transactions. In particular, there are two new
constructor functions for transactions: types.NewTx and types.SignNewTx.
Since the canonical encoding of typed transactions is not RLP-compatible,
Transaction now has new methods for encoding and decoding: MarshalBinary
and UnmarshalBinary.
The existing EIP-155 signer does not support the new transaction types.
All code dealing with transaction signatures should be updated to use the
newer EIP-2930 signer. To make this easier for future updates, we have
added new constructor functions for types.Signer: types.LatestSigner and
types.LatestSignerForChainID.
This change also adds support for the YoloV3 testnet.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Ryan Schneider <ryanleeschneider@gmail.com>
4 years ago
e991bdae24
trie: fix bloom crash on fast sync restart
4 years ago
f566dd305e
all: bloom-filter based pruning mechanism ( #21724 )
...
* cmd, core, tests: initial state pruner
core: fix db inspector
cmd/geth: add verify-state
cmd/geth: add verification tool
core/rawdb: implement flatdb
cmd, core: fix rebase
core/state: use new contract code layout
core/state/pruner: avoid deleting genesis state
cmd/geth: add helper function
core, cmd: fix extract genesis
core: minor fixes
contracts: remove useless
core/state/snapshot: plugin stacktrie
core: polish
core/state/snapshot: iterate storage concurrently
core/state/snapshot: fix iteration
core: add comments
core/state/snapshot: polish code
core/state: polish
core/state/snapshot: rebase
core/rawdb: add comments
core/rawdb: fix tests
core/rawdb: improve tests
core/state/snapshot: fix concurrent iteration
core/state: run pruning during the recovery
core, trie: implement martin's idea
core, eth: delete flatdb and polish pruner
trie: fix import
core/state/pruner: add log
core/state/pruner: fix issues
core/state/pruner: don't read back
core/state/pruner: fix contract code write
core/state/pruner: check root node presence
cmd, core: polish log
core/state: use HEAD-127 as the target
core/state/snapshot: improve tests
cmd/geth: fix verification tool
cmd/geth: use HEAD as the verification default target
all: replace the bloomfilter with martin's fork
cmd, core: polish code
core, cmd: forcibly delete state root
core/state/pruner: add hash64
core/state/pruner: fix blacklist
core/state: remove blacklist
cmd, core: delete trie clean cache before pruning
cmd, core: fix lint
cmd, core: fix rebase
core/state: fix the special case for clique networks
core/state/snapshot: remove useless code
core/state/pruner: capping the snapshot after pruning
cmd, core, eth: fixes
core/rawdb: update db inspector
cmd/geth: polish code
core/state/pruner: fsync bloom filter
cmd, core: print warning log
core/state/pruner: adjust the parameters for bloom filter
cmd, core: create the bloom filter by size
core: polish
core/state/pruner: sanitize invalid bloomfilter size
cmd: address comments
cmd/geth: address comments
cmd/geth: address comment
core/state/pruner: address comments
core/state/pruner: rename homedir to datadir
cmd, core: address comments
core/state/pruner: address comment
core/state: address comments
core, cmd, tests: address comments
core: address comments
core/state/pruner: release the iterator after each commit
core/state/pruner: improve pruner
cmd, core: adjust bloom paramters
core/state/pruner: fix lint
core/state/pruner: fix tests
core: fix rebase
core/state/pruner: remove atomic rename
core/state/pruner: address comments
all: run go mod tidy
core/state/pruner: avoid false-positive for the middle state roots
core/state/pruner: add checks for middle roots
cmd/geth: replace crit with error
* core/state/pruner: fix lint
* core: drop legacy bloom filter
* core/state/snapshot: improve pruner
* core/state/snapshot: polish concurrent logs to report ETA vs. hashes
* core/state/pruner: add progress report for pruning and compaction too
* core: fix snapshot test API
* core/state: fix some pruning logs
* core/state/pruner: support recovering from bloom flush fail
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
4 years ago
ef84da8481
all: remove unneeded parentheses ( #21921 )
...
* remove uneeded convertion type
* remove redundant type in composite literal
* omit explicit type where implicit
* remove unused redundant parenthesis
* remove redundant import alias duktape
4 years ago
9e1bd0f367
trie: fix range prover ( #22210 )
...
Fixes a special case when the trie only has a single trie node and the range proof only contains a single element.
4 years ago
93a89b2681
go.mod: use github.com/holiman/bloomfilter/v2 ( #22044 )
...
* deps: use improved bloom filter implementation
* eth/handler, trie: use 4 keys for syncbloom + minor fixes
* eth/protocols, trie: revert change on syncbloom method signature
4 years ago
d2e1b17f18
snapshot, trie: fixed typos, mostly in snapshot pkg ( #22133 )
4 years ago
017831dd5b
core, eth: split eth package, implement snap protocol ( #21482 )
...
This commit splits the eth package, separating the handling of eth and snap protocols. It also includes the capability to run snap sync (https://github.com/ethereum/devp2p/blob/master/caps/snap.md ) , but does not enable it by default.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
4 years ago
b44f24e3e6
core, trie: speed up some tests with quadratic processing flaw ( #21987 )
...
This commit fixes a flaw in two testcases, and brings down the exec-time from ~40s to ~8s for trie/TestIncompleteSync.
The checkConsistency was performed over and over again on the complete set of nodes, not just the recently added, turning it into a quadratic runtime.
4 years ago
810f9e057d
all: remove redundant conversions and import names ( #21903 )
4 years ago
c92faee66e
all: simplify nested complexity and if blocks ending with a return statement ( #21854 )
...
Changes:
Simplify nested complexity
If an if blocks ends with a return statement then remove the else nesting.
Most of the changes has also been reported in golint https://goreportcard.com/report/github.com/ethereum/go-ethereum#golint
4 years ago
6f88d6530a
trie, rpc, cmd/geth: fix tests on 32-bit and windows + minor rpc fixes ( #21871 )
...
* trie: fix tests to work on 32-bit systems
* les: make test work on 32-bit platform
* cmd/geth: fix windows-issues on tests
* trie: improve balance
* cmd/geth: make account tests less verbose + less mem intense
* rpc: make debug-level log output less verbose
* cmd/geth: lint
4 years ago
23524f8900
all: disable recording preimage of trie keys ( #21402 )
...
* cmd, core, eth, light, trie: disable recording preimage by default
* core, eth: fix unit tests
* core: fix import
* all: change to nopreimage
* cmd, core, eth, trie: use cache.preimages flag
* cmd: enable preimages for archive node
* cmd/utils, trie: simplify preimage tracking a bit
* core: fix linter
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
4 years ago
81678971db
trie, tests/fuzzers: implement a stacktrie fuzzer + stacktrie fixes ( #21799 )
...
* trie: fix error in stacktrie not committing small roots
* fuzzers: make trie-fuzzer use correct returnvalues
* trie: improved tests
* tests/fuzzers: fuzzer for stacktrie vs regular trie
* test/fuzzers: make stacktrie fuzzer use 32-byte keys
* trie: fix error in stacktrie with small nodes
* trie: add (skipped) testcase for stacktrie
* tests/fuzzers: address review comments for stacktrie fuzzer
* trie: fix docs in stacktrie
4 years ago
348c3bc47d
trie: fix flaw in stacktrie pool reuse ( #21699 )
4 years ago
86dd005544
trie: polish commit function ( #21692 )
...
* trie: polish commit function
* trie: fix typo
4 years ago
9d1e2027a0
trie: add Commit-sequence tests for stacktrie commit ( #21643 )
4 years ago
053ed9cc84
trie: polishes to trie committer ( #21351 )
...
* trie: update tests to check commit integrity
* trie: polish committer
* trie: fix typo
* trie: remove hasvalue notion
According to the benchmarks, type assertion between the pointer and
interface is extremely fast.
BenchmarkIntmethod-12 1000000000 1.91 ns/op
BenchmarkInterface-12 1000000000 2.13 ns/op
BenchmarkTypeSwitch-12 1000000000 1.81 ns/op
BenchmarkTypeAssertion-12 2000000000 1.78 ns/op
So the overhead for asserting whether the shortnode has "valuenode"
child is super tiny. No necessary to have another field.
* trie: linter nitpicks
Co-authored-by: Martin Holst Swende <martin@swende.se>
4 years ago
6c8310ebb4
trie: use stacktrie for Derivesha operation ( #21407 )
...
core/types: use stacktrie for derivesha
trie: add stacktrie file
trie: fix linter
core/types: use stacktrie for derivesha
rebased: adapt stacktrie to the newer version of DeriveSha
Co-authored-by: Martin Holst Swende <martin@swende.se>
More linter fixes
review feedback: no key offset for nodes converted to hashes
trie: use EncodeRLP for full nodes
core/types: insert txs in order in derivesha
trie: tests for derivesha with stacktrie
trie: make stacktrie use pooled hashers
trie: make stacktrie reuse tmp slice space
trie: minor polishes on stacktrie
trie/stacktrie: less rlp dancing
core/types: explain the contorsions in DeriveSha
ci: fix goimport errors
trie: clear mem on subtrie hashing
squashme: linter fix
stracktrie: use pooling, less allocs (#3 )
trie: in-place hex prefix, reduce allocs and add rawNode.EncodeRLP
Reintroduce the `[]node` method, add the missing `EncodeRLP` implementation for `rawNode` and calculate the hex prefix in place.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Martin Holst Swende <martin@swende.se>
4 years ago
e5defccd58
trie: extend range proof ( #21250 )
...
* trie: support non-existent right proof
* trie: improve test
* trie: minor linter fix
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
4 years ago
25b16085da
trie: support empty range proof ( #21199 )
4 years ago
e1365b2464
trie: fix gaped range proof test case ( #21484 )
4 years ago
eeaf191633
core, eth, trie: prepare trie sync for path based operation
4 years ago
d8da0b3d81
core/state, eth, trie: stabilize memory use, fix memory leak
4 years ago
87c0ba9213
core, eth, les, trie: add a prefix to contract code ( #21080 )
4 years ago
8cbdc8638f
core: define and test chain rewind corner cases ( #21409 )
...
* core: define and test chain reparation cornercases
* core: write up a variety of set-head tests
* core, eth: unify chain rollbacks, handle all the cases
* core: make linter smile
* core: remove commented out legacy code
* core, eth/downloader: fix review comments
* core: revert a removed recovery mechanism
4 years ago
93da0cf8a1
cmd, core, eth, light, trie: dump clean cache periodically ( #20391 )
...
* cmd, core, eth, light, trie: dump clean cache periodically
* eth: update config
* trie: minor fix
* core, trie: address comments
* eth: remove useless
* trie: print clean cache dump start too
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
4 years ago
295693759e
core/vm: less allocations for various call variants ( #21222 )
...
* core/vm/runtime/tests: add more benchmarks
* core/vm: initial work on improving alloc count for calls to precompiles
name old time/op new time/op delta
SimpleLoop/identity-precompile-10M-6 117ms ±75% 43ms ± 1% -63.09% (p=0.008 n=5+5)
SimpleLoop/loop-10M-6 79.6ms ± 4% 70.5ms ± 1% -11.42% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
SimpleLoop/identity-precompile-10M-6 24.4MB ± 0% 4.9MB ± 0% -79.94% (p=0.008 n=5+5)
SimpleLoop/loop-10M-6 13.2kB ± 0% 13.2kB ± 0% ~ (p=0.357 n=5+5)
name old allocs/op new allocs/op delta
SimpleLoop/identity-precompile-10M-6 382k ± 0% 153k ± 0% -59.99% (p=0.000 n=5+4)
SimpleLoop/loop-10M-6 40.0 ± 0% 40.0 ± 0% ~ (all equal)
* core/vm: don't allocate big.int for touch
name old time/op new time/op delta
SimpleLoop/identity-precompile-10M-6 43.3ms ± 1% 42.4ms ± 7% ~ (p=0.151 n=5+5)
SimpleLoop/loop-10M-6 70.5ms ± 1% 76.7ms ± 1% +8.67% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
SimpleLoop/identity-precompile-10M-6 4.90MB ± 0% 2.46MB ± 0% -49.83% (p=0.008 n=5+5)
SimpleLoop/loop-10M-6 13.2kB ± 0% 13.2kB ± 1% ~ (p=0.571 n=5+5)
name old allocs/op new allocs/op delta
SimpleLoop/identity-precompile-10M-6 153k ± 0% 76k ± 0% -49.98% (p=0.029 n=4+4)
SimpleLoop/loop-10M-6 40.0 ± 0% 40.0 ± 0% ~ (all equal)
* core/vm: reduce allocs in staticcall
name old time/op new time/op delta
SimpleLoop/identity-precompile-10M-6 42.4ms ± 7% 37.5ms ± 6% -11.68% (p=0.008 n=5+5)
SimpleLoop/loop-10M-6 76.7ms ± 1% 69.1ms ± 1% -9.82% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
SimpleLoop/identity-precompile-10M-6 2.46MB ± 0% 0.02MB ± 0% -99.35% (p=0.008 n=5+5)
SimpleLoop/loop-10M-6 13.2kB ± 1% 13.2kB ± 0% ~ (p=0.143 n=5+5)
name old allocs/op new allocs/op delta
SimpleLoop/identity-precompile-10M-6 76.4k ± 0% 0.1k ± 0% ~ (p=0.079 n=4+5)
SimpleLoop/loop-10M-6 40.0 ± 0% 40.0 ± 0% ~ (all equal)
* trie: better use of hasher keccakState
* core/state/statedb: reduce allocations in getDeletedStateObject
* core/vm: reduce allocations in all call derivates
* core/vm: reduce allocations in call variants
- Make returnstack `uint32`
- Use a `sync.Pool` of `stack`s
* core/vm: fix tests
* core/vm: goimports
* core/vm: tracer fix + staticcall gas fix
* core/vm: add back snapshot to staticcall
* core/vm: review concerns + make returnstack pooled + enable returndata in traces
* core/vm: fix some test tracer method signatures
* core/vm: run gencodec, minor comment polish
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
4 years ago
240d1851db
trie: quell linter in commiter.go ( #21329 )
4 years ago
6eef141aef
les: historical data garbage collection ( #19570 )
...
This change introduces garbage collection for the light client. Historical
chain data is deleted periodically. If you want to disable the GC, use
the --light.nopruning flag.
4 years ago
ddeea1e0c6
core: types: less allocations when hashing and tx handling ( #21265 )
...
* core, crypto: various allocation savings regarding tx handling
* core: reduce allocs for gas price comparison
This change reduces the allocations needed for comparing different transactions to each other.
A call to `tx.GasPrice()` copies the gas price as it has to be safe against modifications and
also needs to be threadsafe. For comparing and ordering different transactions we don't need
these guarantees
* core: added tx.GasPriceIntCmp for comparison without allocation
adds a method to remove unneeded allocation in comparison to tx.gasPrice
* core/types: pool legacykeccak256 objects in rlpHash
rlpHash is by far the most used function in core that allocates a legacyKeccak256 object on each call.
Since it is so widely used it makes sense to add pooling here so we relieve the GC.
On my machine these changes result in > 100 MILLION less allocations and > 30 GB less allocated memory.
* reverted some changes
* reverted some changes
* trie: use crypto.KeccakState instead of replicating code
Co-authored-by: Martin Holst Swende <martin@swende.se>
4 years ago
b482423e61
trie: reduce allocs in insertPreimage ( #21261 )
4 years ago
389da6aa48
trie: enforce monotonic range in prover and return end marker ( #21130 )
...
* trie: add hasRightElement indicator
* trie: ensure the range is monotonic increasing
* trie: address comment and fix lint
* trie: address comment
* trie: make linter happy
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
5 years ago
070a5e1252
trie: fix for range proof ( #21107 )
...
* trie: fix for range proof
* trie: fix typo
5 years ago
4f2784b38f
all: fix typos in comments ( #21118 )
5 years ago
65ce550b37
trie: extend range proofs with non-existence ( #21000 )
...
* trie: implement range proof with non-existent edge proof
* trie: fix cornercase
* trie: consider empty range
* trie: add singleSide test
* trie: support all-elements range proof
* trie: fix typo
* trie: tiny typos and formulations
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
5 years ago
234cc8e77f
eth/downloader: minor typo fixes in comments ( #21035 )
5 years ago
5cdc2dffda
trie: fix TestBadRangeProof unit test ( #21034 )
5 years ago
44ff3f3dc9
trie: initial implementation for range proof ( #20908 )
...
* trie: initial implementation for range proof
* trie: add benchmark
* trie: fix lint
* trie: fix minor issue
* trie: unset the edge valuenode as well
* trie: unset the edge valuenode as nilValuenode
5 years ago
6402c42b67
all: simplify and fix database iteration with prefix/start ( #20808 )
...
* core/state/snapshot: start fixing disk iterator seek
* ethdb, rawdb, leveldb, memorydb: implement iterators with prefix and start
* les, core/state/snapshot: iterator fixes
* all: remove two iterator methods
* all: rename Iteratee.NewIteratorWith -> NewIterator
* ethdb: fix review concerns
5 years ago
af4080b4b7
trie: fix concurrent usage of secKeyBuf, ref #20920
5 years ago
351a5903b0
core/rawdb, core/state/snapshot: runtime snapshot generation
5 years ago
542df8898e
core: initial version of state snapshots
5 years ago
a1313b5b1e
trie: make hasher parallel when number of changes are large ( #20488 )
...
* trie: make hasher parallel when number of changes are large
* trie: remove unused field dirtyCount
* trie: rename unhashedCount/unhashed
5 years ago
5a9c96454e
trie: separate hashes and committer, collapse on commit
...
* trie: make db insert use size instead of full data
* core/state: minor optimization in state onleaf allocation
* trie: implement dedicated committer and hasher
* trie: use dedicated committer/hasher
* trie: linter nitpicks
* core/state, trie: avoid unnecessary storage trie load+commit
* trie: review feedback, mainly docs + minor changes
* trie: start deprecating old hasher
* trie: fix misspell+lint
* trie: deprecate hasher.go, make proof framework use new hasher
* trie: rename pure_committer/hasher to committer/hasher
* trie, core/state: fix review concerns
* trie: more review concerns
* trie: make commit collapse into hashnode, don't touch dirtyness
* trie: goimports fixes
* trie: remove panics
5 years ago
9b09c0fc83
* trie: utilize callbacks instead of amassing lists in ref/unref ( #20529 )
...
* trie/tests: add benchmarks and update trie tests
* trie: update benchmark tests
* trie: utilize callbacks instead of amassing lists of hashes in database ref/unref
* trie: replace remaining non-callback based accesses
5 years ago
4f7b7f84ae
add node.go unit test file node_test.go ( #20028 )
...
* add node.go unit test file node_test.go
* add node_test.go file license and rollback trie_test.go
* fix unused variable v
* trie: fix license year
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
5 years ago
13ccf6016e
trie: track dirty cache metrics, track clean writes on commit
5 years ago
gary rong
lightclient
Péter Szilágyi
Alex Prut
Martin Holst Swende
Melvin Junhee Woo
Guillaume Ballet
Marius van der Wijden
Martin Michlmayr
me020523