6d2aeb43d5
cmd, core/state, eth, tests, trie: improve state reader ( #27428 )
...
The state availability is checked during the creation of a state reader.
- In hash-based database, if the specified root node does not exist on disk disk, then
the state reader won't be created and an error will be returned.
- In path-based database, if the specified state layer is not available, then the
state reader won't be created and an error will be returned.
This change also contains a stricter semantics regarding the `Commit` operation: once it has been performed, the trie is no longer usable, and certain operations will return an error.
1 year ago
0e5d2c7c53
core/state/snapshot, core/types, eth: move account definition to type ( #27323 )
...
* core/state/snapshot, core/types, eth: move account definition to type
* core, eth: revert snapshot Account API change
1 year ago
fe01a2f63b
all: use unified emptyRootHash and emptyCodeHash ( #26718 )
...
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
2 years ago
9842301376
all: remove database commit callback, rework noderesolver ( #26637 )
...
This change ports some changes from the main PBSS PR:
- get rid of callback function in `trie.Database.Commit` which is not required anymore
- rework the `nodeResolver` in `trie.Iterator` to make it compatible with multiple state scheme
- some other shallow changes in tests and typo-fixes
2 years ago
8860b39754
all: prepare for path-based trie storage ( #26603 )
...
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.
This PR is intended to be a no-functionality-change prep PR for #25963 .
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2 years ago
bff84a99fe
cmd, core, eth, les, light: track deleted nodes ( #25757 )
...
* cmd, core, eth, les, light: track deleted nodes
* trie: add docs
* trie: address comments
* cmd, core, eth, les, light, trie: trie id
* trie: add tests
* trie, core: updates
* trie: fix imports
* trie: add utility print-method for nodeset
* trie: import err
* trie: fix go vet warnings
Co-authored-by: Martin Holst Swende <martin@swende.se>
2 years ago
3da42f85d9
all: clean up the configs for pruner and snapshotter ( #22396 )
...
This PR cleans up the configurations for pruner and snapshotter by passing a config struct.
And also, this PR disables the snapshot background generation if the chain is opened in "read-only" mode. The read-only mode is necessary in some cases. For example, we have a list of commands to open the etheruem node in "read-only" mode, like export-chain. In these cases, the snapshot background generation is non expected and should be banned explicitly.
2 years ago
f67e54c92f
core: use TryGetAccount to read what TryUpdateAccount has written ( #25458 )
...
* core: use TryGetAccount to read where TryUpdateAccount has been used to write
* Gary's review feedback
* implement Gary's suggestion
* fix bug + rename NewSecure into NewStateTrie
* trie: add backwards-compatibility aliases for SecureTrie
* Update database.go
* make the linter happy
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2 years ago
1ed8b7d24f
cmd: use flags.Merge for grouping flags ( #25392 )
2 years ago
52ed3570c4
cmd: migrate to urfave/cli/v2 ( #24751 )
...
This change updates our urfave/cli dependency to the v2 branch of the library.
There are some Go API changes in cli v2:
- Flag values can now be accessed using the methods ctx.Bool,
ctx.Int, ctx.String, ... regardless of whether the flag is 'local' or
'global'.
- v2 has built-in support for flag categories. Our home-grown category
system is removed and the categories of flags are assigned as part of
the flag definition.
For users, there is only one observable difference with cli v2: flags must now
strictly appear before regular arguments. For example, the following command is
now invalid:
geth account import mykey.json --password file.txt
Instead, the command must be invoked as follows:
geth account import --password file.txt mykey.json
2 years ago
22defa5af7
all: introduce trie owner notion ( #24750 )
...
* cmd, core/state, light, trie, eth: add trie owner notion
* all: refactor
* tests: fix goimports
* core/state/snapshot: fix ineffasigns
Co-authored-by: Martin Holst Swende <martin@swende.se>
2 years ago
c375ee91e9
cmd/geth, core/state/snapshot: rework journal loading, implement account-check ( #24765 )
...
* cmd/geth, core/state/snapshot: rework journal loading, implement account-check
* core/state/snapshot, cmd/geth: polish code (#37 )
* core/state/snapshot: minor nits
* core/state/snapshot: simplify error logic
* cmd/geth: go format
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2 years ago
a10660b7f8
cmd/geth: extend traverseRawState command ( #24954 )
...
This PR adds node verification into traverseRawState command, so corrupted trie nodes can also be detected.
2 years ago
9244d5cd61
all: update license headers and AUTHORS from git history ( #24947 )
3 years ago
59ac229f87
core/state/snapshot: detect and clean up dangling storage snapshot in generation ( #24811 )
...
* core/state/snapshot: check dangling storages when generating snapshot
* core/state/snapshot: polish
* core/state/snapshot: wipe the last part of the dangling storages
* core/state/snapshot: fix and add tests
* core/state/snapshot: fix comment
* README: remove mentions of fast sync (#24656 )
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* core, cmd: expose dangling storage detector for wider usage
* core/state/snapshot: rename variable
* core, ethdb: use global iterators for snapshot generation
* core/state/snapshot: polish
* cmd, core/state/snapshot: polish
* core/state/snapshot: polish
* Update core/state/snapshot/generate.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
* ethdb: extend db test suite and fix memorydb iterator
* ethdb/dbtest: rollback changes
* ethdb/memorydb: simplify iteration
* core/state/snapshot: update dangling counter
* core/state/snapshot: release iterators
* core/state/snapshot: update metrics
* core/state/snapshot: update time metrics
* metrics/influxdb: temp solution to present counter meaningfully, remove it
* add debug log, revert later
* core/state/snapshot: fix iterator panic
* all: customized snapshot iterator for backward iteration
* core, ethdb: polish
* core/state/snapshot: remove debug log
* core/state/snapshot: address comments from peter
* core/state/snapshot: reopen the iterator at the next position
* ethdb, core/state/snapshot: address comment from peter
* core/state/snapshot: reopen exhausted iterators
Co-authored-by: Tbnoapi <63448616+nuoomnoy02@users.noreply.github.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
3 years ago
344d6f95cf
cmd: group network and db path flags together ( #24698 )
...
This PR groups all built-in network flags together and list them in the command as a whole.
And all database path flags(datadir, ancient) are also grouped, since usually these two are
used together.
3 years ago
0914234d10
cmd/geth, core/state/snapshot: fix flaw in dangling-storage check + inspect difflayers ( #24677 )
...
This PR fixes the flaw that @rjl493456442 found in https://github.com/ethereum/go-ethereum/pull/#issuecomment-1093817551 , namely, that the snapshot iterator uses the combined (disk + difflayers) 'view', wheres the raw iterator uses only the disk 'view'.
This PR instead splits up the work: one phase is iterating the disk layer data, another phase is loading the journalled difflayers and performing the same check there.
3 years ago
92e3c56e7b
cmd/geth: inspect snapshot dangling storage ( #24643 )
...
* cmd/geth: inspect snapshot dangling storage
* cmd/geth: make verify-state invoke verify-dangling
3 years ago
8401e4277a
core/rawdb: add HasCode, HashTrieNode and use them where possible ( #24454 )
3 years ago
893502e561
trie, core, eth: use db.has over db.get where possible
3 years ago
e1c000b0dd
cmd/geth: add support for sepolia testnet ( #23730 )
...
* cmd/geth: add support for sepolia testnet
* core: last details on sepolia genesis
* params: fix sepolia hash + reduce testing code
* Update params/bootnodes.go
* cmd/geth: fix attach path for sepolia
* params: update bootnodes
* params: fix
* core: fix docstring
* params: add sepolia CHT
3 years ago
443afc975c
core/state: move state account to core/types + abstracted "write account to trie" ( #23567 )
...
* core/state: abstracted "write account to trie" method
* fix appveyor build
* Apply suggestions from code review
Co-authored-by: Martin Holst Swende <martin@swende.se>
* review feedback
* core/state/accounts: move Account to core/types
* core/types: rename Account -> StateAccount
* core/state: restore EncodeRLP for stateObject
* core/types: add the missing file
* more review feedback
* more review feedback
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
3 years ago
b69f5ca7d4
cmd/utils: fix typo in variable name ( #23451 )
3 years ago
addd8824cf
cmd/geth, eth, core: snapshot dump + unify with trie dump ( #22795 )
...
* cmd/geth, eth, core: snapshot dump + unify with trie dump
* cmd/evm: dump API fixes
* cmd/geth, core, eth: fix some remaining errors
* cmd/evm: dump - add limit, support address startkey, address review concerns
* cmd, core/state, eth: minor polishes, fix snap dump crash, unify format
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
4 years ago
7088f1e814
core, eth: faster snapshot generation ( #22504 )
...
* eth/protocols: persist received state segments
* core: initial implementation
* core/state/snapshot: add tests
* core, eth: updates
* eth/protocols/snapshot: count flat state size
* core/state: add metrics
* core/state/snapshot: skip unnecessary deletion
* core/state/snapshot: rename
* core/state/snapshot: use the global batch
* core/state/snapshot: add logs and fix wiping
* core/state/snapshot: fix
* core/state/snapshot: save generation progress even if the batch is empty
* core/state/snapshot: fixes
* core/state/snapshot: fix initial account range length
* core/state/snapshot: fix initial account range
* eth/protocols/snap: store flat states during the healing
* eth/protocols/snap: print logs
* core/state/snapshot: refactor (#4 )
* core/state/snapshot: refactor
* core/state/snapshot: tiny fix and polish
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* core, eth: fixes
* core, eth: fix healing writer
* core, trie, eth: fix paths
* eth/protocols/snap: fix encoding
* eth, core: add debug log
* core/state/generate: release iterator asap (#5 )
core/state/snapshot: less copy
core/state/snapshot: revert split loop
core/state/snapshot: handle storage becoming empty, improve test robustness
core/state: test modified codehash
core/state/snapshot: polish
* core/state/snapshot: optimize stats counter
* core, eth: add metric
* core/state/snapshot: update comments
* core/state/snapshot: improve tests
* core/state/snapshot: replace secure trie with standard trie
* core/state/snapshot: wrap return as the struct
* core/state/snapshot: skip wiping correct states
* core/state/snapshot: updates
* core/state/snapshot: fixes
* core/state/snapshot: fix panic due to reference flaw in closure
* core/state/snapshot: fix errors in state generation logic + fix log output
* core/state/snapshot: remove an error case
* core/state/snapshot: fix condition-check for exhausted snap state
* core/state/snapshot: use stackTrie for small tries
* core/state/snapshot: don't resolve small storage tries in vain
* core/state/snapshot: properly clean up storage of deleted accounts
* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks
* core/state/snapshot: fix error (+testcase)
* core/state/snapshot: clean up tests a bit
* core/state/snapshot: work in progress on better tests
* core/state/snapshot: polish code
* core/state/snapshot: fix trie iteration abortion trigger
* core/state/snapshot: fixes flaws
* core/state/snapshot: remove panic
* core/state/snapshot: fix abort
* core/state/snapshot: more tests (plus failing testcase)
* core/state/snapshot: more testcases + fix for failing test
* core/state/snapshot: testcase for malformed data
* core/state/snapshot: some test nitpicks
* core/state/snapshot: improvements to logging
* core/state/snapshot: testcase to demo error in abortion
* core/state/snapshot: fix abortion
* cmd/geth: make verify-state report the root
* trie: fix failing test
* core/state/snapshot: add timer metrics
* core/state/snapshot: fix metrics
* core/state/snapshot: udpate tests
* eth/protocols/snap: write snapshot account even if code or state is needed
* core/state/snapshot: fix diskmore check
* core/state/snapshot: review fixes
* core/state/snapshot: improve error message
* cmd/geth: rename 'error' to 'err' in logs
* core/state/snapshot: fix some review concerns
* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync
* core: add error log
* core/state/snapshot: use proper timers for metrics collection
* core/state/snapshot: address some review concerns
* eth/protocols/snap: improved log message
* eth/protocols/snap: fix heal logs to condense infos
* core/state/snapshot: wait for generator termination before restarting
* core/state/snapshot: revert timers to counters to track total time
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
4 years ago
0c70b83e00
all: add read-only option to database ( #22407 )
...
* all: add read-only option to database
* all: fixes tests
* cmd/geth: migrate flags
* cmd/geth: fix the compact
* cmd/geth: fix the format
* cmd/geth: fix log
* cmd: add chain-readonly
* core: add readonly notion to freezer
* core/rawdb: add log
* core/rawdb: fix freezer close
* cmd: fix
* cmd, core: construct db
* core: update tests
4 years ago
6387c520b7
cmd/geth: add ancient datadir flag to snapshot subcommands ( #22486 )
4 years ago
8e547eecd5
cmd/utils: remove deprecated command line flags ( #22263 )
...
This removes support for all deprecated flags except --rpc*.
4 years ago
f566dd305e
all: bloom-filter based pruning mechanism ( #21724 )
...
* cmd, core, tests: initial state pruner
core: fix db inspector
cmd/geth: add verify-state
cmd/geth: add verification tool
core/rawdb: implement flatdb
cmd, core: fix rebase
core/state: use new contract code layout
core/state/pruner: avoid deleting genesis state
cmd/geth: add helper function
core, cmd: fix extract genesis
core: minor fixes
contracts: remove useless
core/state/snapshot: plugin stacktrie
core: polish
core/state/snapshot: iterate storage concurrently
core/state/snapshot: fix iteration
core: add comments
core/state/snapshot: polish code
core/state: polish
core/state/snapshot: rebase
core/rawdb: add comments
core/rawdb: fix tests
core/rawdb: improve tests
core/state/snapshot: fix concurrent iteration
core/state: run pruning during the recovery
core, trie: implement martin's idea
core, eth: delete flatdb and polish pruner
trie: fix import
core/state/pruner: add log
core/state/pruner: fix issues
core/state/pruner: don't read back
core/state/pruner: fix contract code write
core/state/pruner: check root node presence
cmd, core: polish log
core/state: use HEAD-127 as the target
core/state/snapshot: improve tests
cmd/geth: fix verification tool
cmd/geth: use HEAD as the verification default target
all: replace the bloomfilter with martin's fork
cmd, core: polish code
core, cmd: forcibly delete state root
core/state/pruner: add hash64
core/state/pruner: fix blacklist
core/state: remove blacklist
cmd, core: delete trie clean cache before pruning
cmd, core: fix lint
cmd, core: fix rebase
core/state: fix the special case for clique networks
core/state/snapshot: remove useless code
core/state/pruner: capping the snapshot after pruning
cmd, core, eth: fixes
core/rawdb: update db inspector
cmd/geth: polish code
core/state/pruner: fsync bloom filter
cmd, core: print warning log
core/state/pruner: adjust the parameters for bloom filter
cmd, core: create the bloom filter by size
core: polish
core/state/pruner: sanitize invalid bloomfilter size
cmd: address comments
cmd/geth: address comments
cmd/geth: address comment
core/state/pruner: address comments
core/state/pruner: rename homedir to datadir
cmd, core: address comments
core/state/pruner: address comment
core/state: address comments
core, cmd, tests: address comments
core: address comments
core/state/pruner: release the iterator after each commit
core/state/pruner: improve pruner
cmd, core: adjust bloom paramters
core/state/pruner: fix lint
core/state/pruner: fix tests
core: fix rebase
core/state/pruner: remove atomic rename
core/state/pruner: address comments
all: run go mod tidy
core/state/pruner: avoid false-positive for the middle state roots
core/state/pruner: add checks for middle roots
cmd/geth: replace crit with error
* core/state/pruner: fix lint
* core: drop legacy bloom filter
* core/state/snapshot: improve pruner
* core/state/snapshot: polish concurrent logs to report ETA vs. hashes
* core/state/pruner: add progress report for pruning and compaction too
* core: fix snapshot test API
* core/state: fix some pruning logs
* core/state/pruner: support recovering from bloom flush fail
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
4 years ago
rjl493456442
Martin Holst Swende
Guillaume Ballet
willian.eth
Felix Lange
Ceyhun Onur
Ivan Bogatyy
michael1011
rene