503f1f7ada
all: activate pbss as experimental feature ( #26274 )
...
* all: activate pbss
* core/rawdb: fix compilation error
* cma, core, eth, les, trie: address comments
* cmd, core, eth, trie: polish code
* core, cmd, eth: address comments
* cmd, core, eth, les, light, tests: address comment
* cmd/utils: shorten log message
* trie/triedb/pathdb: limit node buffer size to 1gb
* cmd/utils: fix opening non-existing db
* cmd/utils: rename flag name
* cmd, core: group chain history flags and fix tests
* core, eth, trie: fix memory leak in snapshot generation
* cmd, eth, internal: deprecate flags
* all: enable state tests for pathdb, fixes
* cmd, core: polish code
* trie/triedb/pathdb: limit the node buffer size to 256mb
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
1 year ago
88f3d61468
all: expose block number information to statedb ( #27753 )
...
* core/state: clean up
* all: add block number infomration to statedb
* core, trie: rename blockNumber to block
1 year ago
4b06e4f25e
core/state: value diff tracking in StateDB ( #27349 )
...
This change makes the StateDB track the state key value diff of a block transition.
We already tracked current account and storage values for the purpose of updating
the state snapshot. With this PR, we now also track the original (pre-transition) values
of accounts and storage slots.
1 year ago
c7b099b2ea
trie, core/state: revert error removal in (*state.Trie).Commit ( #27544 )
...
* trie, core/state: revert error removal in (*state.Trie).Commit
* Gary's nitpick :)
Co-Authored-By: rjl493456442 <garyrong0905@gmail.com>
---------
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
1 year ago
0e5d2c7c53
core/state/snapshot, core/types, eth: move account definition to type ( #27323 )
...
* core/state/snapshot, core/types, eth: move account definition to type
* core, eth: revert snapshot Account API change
1 year ago
a14301823e
all: new empty trie with types.EmptyRootHash instead of null ( #27230 )
1 year ago
5021d36d35
all: port boring changes from pbss ( #27176 )
...
* all: port boring changes from pbss
* core, trie: address comments from martin
* trie: minor fixes
* core/rawdb: update comment
* core, eth, tests, trie: address comments
* tests, trie: add extra check when update trie database
* trie/triedb/hashdb: degrade the error to warning
1 year ago
99f81d2724
all: refactor trie API ( #26995 )
...
In this PR, all TryXXX(e.g. TryGet) APIs of trie are renamed to XXX(e.g. Get) with an error returned.
The original XXX(e.g. Get) APIs are renamed to MustXXX(e.g. MustGet) and does not return any error -- they print a log output. A future PR will change the behaviour to panic on errorrs.
1 year ago
fe01a2f63b
all: use unified emptyRootHash and emptyCodeHash ( #26718 )
...
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
2 years ago
22c3ad1d12
core/state, trie: remove unused error-return from trie Commit operation ( #26641 )
2 years ago
9842301376
all: remove database commit callback, rework noderesolver ( #26637 )
...
This change ports some changes from the main PBSS PR:
- get rid of callback function in `trie.Database.Commit` which is not required anymore
- rework the `nodeResolver` in `trie.Iterator` to make it compatible with multiple state scheme
- some other shallow changes in tests and typo-fixes
2 years ago
8860b39754
all: prepare for path-based trie storage ( #26603 )
...
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.
This PR is intended to be a no-functionality-change prep PR for #25963 .
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2 years ago
743e404906
core, eth, les, tests, trie: abstract node scheme ( #25532 )
...
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon.
Apart from that, a few changes are also included which is worth mentioning:
- port the changes in the stacktrie, tracking the path prefix of nodes during commit
- use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
2 years ago
bff84a99fe
cmd, core, eth, les, light: track deleted nodes ( #25757 )
...
* cmd, core, eth, les, light: track deleted nodes
* trie: add docs
* trie: address comments
* cmd, core, eth, les, light, trie: trie id
* trie: add tests
* trie, core: updates
* trie: fix imports
* trie: add utility print-method for nodeset
* trie: import err
* trie: fix go vet warnings
Co-authored-by: Martin Holst Swende <martin@swende.se>
2 years ago
b628d72766
build: upgrade to go 1.19 ( #25726 )
...
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.
In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.
With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
2 years ago
5ddedd2f83
core, light, trie: remove DiskDB function from trie database ( #25690 )
2 years ago
f67e54c92f
core: use TryGetAccount to read what TryUpdateAccount has written ( #25458 )
...
* core: use TryGetAccount to read where TryUpdateAccount has been used to write
* Gary's review feedback
* implement Gary's suggestion
* fix bug + rename NewSecure into NewStateTrie
* trie: add backwards-compatibility aliases for SecureTrie
* Update database.go
* make the linter happy
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2 years ago
8b53b92eb4
core, trie: rework trie committer ( #25320 )
...
* all: rework trie and trie committer
* all: get rid of internal cache in trie
* all: fixes
* trie: polish
* core, trie: address comments
* trie: fix imports
* core/state: address comments
* core/state/snapshot: polish
* trie: remove unused code
* trie: update tests
* trie: don't set db as nil
* trie: address comments
* trie: unskip test
2 years ago
22defa5af7
all: introduce trie owner notion ( #24750 )
...
* cmd, core/state, light, trie, eth: add trie owner notion
* all: refactor
* tests: fix goimports
* core/state/snapshot: fix ineffasigns
Co-authored-by: Martin Holst Swende <martin@swende.se>
2 years ago
c375ee91e9
cmd/geth, core/state/snapshot: rework journal loading, implement account-check ( #24765 )
...
* cmd/geth, core/state/snapshot: rework journal loading, implement account-check
* core/state/snapshot, cmd/geth: polish code (#37 )
* core/state/snapshot: minor nits
* core/state/snapshot: simplify error logic
* cmd/geth: go format
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2 years ago
9244d5cd61
all: update license headers and AUTHORS from git history ( #24947 )
2 years ago
59ac229f87
core/state/snapshot: detect and clean up dangling storage snapshot in generation ( #24811 )
...
* core/state/snapshot: check dangling storages when generating snapshot
* core/state/snapshot: polish
* core/state/snapshot: wipe the last part of the dangling storages
* core/state/snapshot: fix and add tests
* core/state/snapshot: fix comment
* README: remove mentions of fast sync (#24656 )
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* core, cmd: expose dangling storage detector for wider usage
* core/state/snapshot: rename variable
* core, ethdb: use global iterators for snapshot generation
* core/state/snapshot: polish
* cmd, core/state/snapshot: polish
* core/state/snapshot: polish
* Update core/state/snapshot/generate.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
* ethdb: extend db test suite and fix memorydb iterator
* ethdb/dbtest: rollback changes
* ethdb/memorydb: simplify iteration
* core/state/snapshot: update dangling counter
* core/state/snapshot: release iterators
* core/state/snapshot: update metrics
* core/state/snapshot: update time metrics
* metrics/influxdb: temp solution to present counter meaningfully, remove it
* add debug log, revert later
* core/state/snapshot: fix iterator panic
* all: customized snapshot iterator for backward iteration
* core, ethdb: polish
* core/state/snapshot: remove debug log
* core/state/snapshot: address comments from peter
* core/state/snapshot: reopen the iterator at the next position
* ethdb, core/state/snapshot: address comment from peter
* core/state/snapshot: reopen exhausted iterators
Co-authored-by: Tbnoapi <63448616+nuoomnoy02@users.noreply.github.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2 years ago
a5a5237178
core, light, tests, trie: add state metrics ( #23433 )
3 years ago
c73652da0b
core/state/snapshot: fix flaky tests ( #22944 )
...
* core/state/snapshot: fix flaky tests
* core/state/snapshot: fix tests
3 years ago
7088f1e814
core, eth: faster snapshot generation ( #22504 )
...
* eth/protocols: persist received state segments
* core: initial implementation
* core/state/snapshot: add tests
* core, eth: updates
* eth/protocols/snapshot: count flat state size
* core/state: add metrics
* core/state/snapshot: skip unnecessary deletion
* core/state/snapshot: rename
* core/state/snapshot: use the global batch
* core/state/snapshot: add logs and fix wiping
* core/state/snapshot: fix
* core/state/snapshot: save generation progress even if the batch is empty
* core/state/snapshot: fixes
* core/state/snapshot: fix initial account range length
* core/state/snapshot: fix initial account range
* eth/protocols/snap: store flat states during the healing
* eth/protocols/snap: print logs
* core/state/snapshot: refactor (#4 )
* core/state/snapshot: refactor
* core/state/snapshot: tiny fix and polish
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* core, eth: fixes
* core, eth: fix healing writer
* core, trie, eth: fix paths
* eth/protocols/snap: fix encoding
* eth, core: add debug log
* core/state/generate: release iterator asap (#5 )
core/state/snapshot: less copy
core/state/snapshot: revert split loop
core/state/snapshot: handle storage becoming empty, improve test robustness
core/state: test modified codehash
core/state/snapshot: polish
* core/state/snapshot: optimize stats counter
* core, eth: add metric
* core/state/snapshot: update comments
* core/state/snapshot: improve tests
* core/state/snapshot: replace secure trie with standard trie
* core/state/snapshot: wrap return as the struct
* core/state/snapshot: skip wiping correct states
* core/state/snapshot: updates
* core/state/snapshot: fixes
* core/state/snapshot: fix panic due to reference flaw in closure
* core/state/snapshot: fix errors in state generation logic + fix log output
* core/state/snapshot: remove an error case
* core/state/snapshot: fix condition-check for exhausted snap state
* core/state/snapshot: use stackTrie for small tries
* core/state/snapshot: don't resolve small storage tries in vain
* core/state/snapshot: properly clean up storage of deleted accounts
* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks
* core/state/snapshot: fix error (+testcase)
* core/state/snapshot: clean up tests a bit
* core/state/snapshot: work in progress on better tests
* core/state/snapshot: polish code
* core/state/snapshot: fix trie iteration abortion trigger
* core/state/snapshot: fixes flaws
* core/state/snapshot: remove panic
* core/state/snapshot: fix abort
* core/state/snapshot: more tests (plus failing testcase)
* core/state/snapshot: more testcases + fix for failing test
* core/state/snapshot: testcase for malformed data
* core/state/snapshot: some test nitpicks
* core/state/snapshot: improvements to logging
* core/state/snapshot: testcase to demo error in abortion
* core/state/snapshot: fix abortion
* cmd/geth: make verify-state report the root
* trie: fix failing test
* core/state/snapshot: add timer metrics
* core/state/snapshot: fix metrics
* core/state/snapshot: udpate tests
* eth/protocols/snap: write snapshot account even if code or state is needed
* core/state/snapshot: fix diskmore check
* core/state/snapshot: review fixes
* core/state/snapshot: improve error message
* cmd/geth: rename 'error' to 'err' in logs
* core/state/snapshot: fix some review concerns
* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync
* core: add error log
* core/state/snapshot: use proper timers for metrics collection
* core/state/snapshot: address some review concerns
* eth/protocols/snap: improved log message
* eth/protocols/snap: fix heal logs to condense infos
* core/state/snapshot: wait for generator termination before restarting
* core/state/snapshot: revert timers to counters to track total time
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
3 years ago
a15d71a255
core/state/snapshot: stop generator if it hits missing trie nodes ( #21649 )
...
* core/state/snapshot: exit Geth if generator hits missing trie nodes
* core/state/snapshot: error instead of hard die on generator fault
* core/state/snapshot: don't enable logging on the tests
4 years ago
351a5903b0
core/rawdb, core/state/snapshot: runtime snapshot generation
5 years ago
d754091a87
core/state/snapshot: unlink snapshots from blocks, quad->linear cleanup
5 years ago
542df8898e
core: initial version of state snapshots
5 years ago
rjl493456442
Guillaume Ballet
Martin Holst Swende
Felix Lange
Péter Szilágyi