d468c333a7
cmd/{geth,utils}: add cmd to export preimages in snap enumeration order ( #28256 )
...
Adds a subcommand: `geth snapshot export-preimages`, to export preimages of every hash found during a snapshot enumeration: that is, it exports _only the active state_, and not _all_ preimages that have been used but are no longer part of the state.
This tool is needed for the verkle transition, in order to distribute the preimages needed for the conversion. Since only the 'active' preimages are exported, the output is shrunk from ~70GB to ~4GB.
The order of the output is the order used by the snapshot enumeration, which avoids database thrashing. However, it also means that storage-slot preimages are not deduplicated.
12 months ago
fa8d39807d
cmd, core, trie: verkle-capable `geth init` ( #28270 )
...
This change allows the creation of a genesis block for verkle testnets. This makes for a chunk of code that is easier to review and still touches many discussion points.
1 year ago
f265cc24b4
cmd/geth: remove some whitespace in code and comments ( #28148 )
...
This changes just removes some whitespace
1 year ago
3d297fc2d7
cmd/geth: ensure db is closed before exit ( #28150 )
1 year ago
82ec555d70
cmd: add state.scheme to the database flag group for local flag handling ( #28107 )
1 year ago
d135bafdcb
cmd/geth: print progress logs when iterating large contracts too ( #28179 )
1 year ago
503f1f7ada
all: activate pbss as experimental feature ( #26274 )
...
* all: activate pbss
* core/rawdb: fix compilation error
* cma, core, eth, les, trie: address comments
* cmd, core, eth, trie: polish code
* core, cmd, eth: address comments
* cmd, core, eth, les, light, tests: address comment
* cmd/utils: shorten log message
* trie/triedb/pathdb: limit node buffer size to 1gb
* cmd/utils: fix opening non-existing db
* cmd/utils: rename flag name
* cmd, core: group chain history flags and fix tests
* core, eth, trie: fix memory leak in snapshot generation
* cmd, eth, internal: deprecate flags
* all: enable state tests for pathdb, fixes
* cmd, core: polish code
* trie/triedb/pathdb: limit the node buffer size to 256mb
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
1 year ago
59f7b289c3
cmd, core, eth, graphql, trie: no persisted clean trie cache file ( #27525 )
...
The clean trie cache is persisted periodically, therefore Geth can
quickly warmup the cache in next restart.
However it will reduce the robustness of system. The assumption is
held in Geth that if the parent trie node is present, then the entire
sub-trie associated with the parent are all prensent.
Imagine the scenario that Geth rewinds itself to a past block and
restart, but Geth finds the root node of "future state" in clean
cache then regard this state is present in disk, while is not in fact.
Another example is offline pruning tool. Whenever an offline pruning
is performed, the clean cache file has to be removed to aviod hitting
the root node of "deleted states" in clean cache.
All in all, compare with the minor performance gain, system robustness
is something we care more.
1 year ago
6d2aeb43d5
cmd, core/state, eth, tests, trie: improve state reader ( #27428 )
...
The state availability is checked during the creation of a state reader.
- In hash-based database, if the specified root node does not exist on disk disk, then
the state reader won't be created and an error will be returned.
- In path-based database, if the specified state layer is not available, then the
state reader won't be created and an error will be returned.
This change also contains a stricter semantics regarding the `Commit` operation: once it has been performed, the trie is no longer usable, and certain operations will return an error.
1 year ago
0e5d2c7c53
core/state/snapshot, core/types, eth: move account definition to type ( #27323 )
...
* core/state/snapshot, core/types, eth: move account definition to type
* core, eth: revert snapshot Account API change
1 year ago
fe01a2f63b
all: use unified emptyRootHash and emptyCodeHash ( #26718 )
...
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
2 years ago
9842301376
all: remove database commit callback, rework noderesolver ( #26637 )
...
This change ports some changes from the main PBSS PR:
- get rid of callback function in `trie.Database.Commit` which is not required anymore
- rework the `nodeResolver` in `trie.Iterator` to make it compatible with multiple state scheme
- some other shallow changes in tests and typo-fixes
2 years ago
8860b39754
all: prepare for path-based trie storage ( #26603 )
...
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.
This PR is intended to be a no-functionality-change prep PR for #25963 .
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2 years ago
bff84a99fe
cmd, core, eth, les, light: track deleted nodes ( #25757 )
...
* cmd, core, eth, les, light: track deleted nodes
* trie: add docs
* trie: address comments
* cmd, core, eth, les, light, trie: trie id
* trie: add tests
* trie, core: updates
* trie: fix imports
* trie: add utility print-method for nodeset
* trie: import err
* trie: fix go vet warnings
Co-authored-by: Martin Holst Swende <martin@swende.se>
2 years ago
3da42f85d9
all: clean up the configs for pruner and snapshotter ( #22396 )
...
This PR cleans up the configurations for pruner and snapshotter by passing a config struct.
And also, this PR disables the snapshot background generation if the chain is opened in "read-only" mode. The read-only mode is necessary in some cases. For example, we have a list of commands to open the etheruem node in "read-only" mode, like export-chain. In these cases, the snapshot background generation is non expected and should be banned explicitly.
2 years ago
f67e54c92f
core: use TryGetAccount to read what TryUpdateAccount has written ( #25458 )
...
* core: use TryGetAccount to read where TryUpdateAccount has been used to write
* Gary's review feedback
* implement Gary's suggestion
* fix bug + rename NewSecure into NewStateTrie
* trie: add backwards-compatibility aliases for SecureTrie
* Update database.go
* make the linter happy
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2 years ago
1ed8b7d24f
cmd: use flags.Merge for grouping flags ( #25392 )
2 years ago
52ed3570c4
cmd: migrate to urfave/cli/v2 ( #24751 )
...
This change updates our urfave/cli dependency to the v2 branch of the library.
There are some Go API changes in cli v2:
- Flag values can now be accessed using the methods ctx.Bool,
ctx.Int, ctx.String, ... regardless of whether the flag is 'local' or
'global'.
- v2 has built-in support for flag categories. Our home-grown category
system is removed and the categories of flags are assigned as part of
the flag definition.
For users, there is only one observable difference with cli v2: flags must now
strictly appear before regular arguments. For example, the following command is
now invalid:
geth account import mykey.json --password file.txt
Instead, the command must be invoked as follows:
geth account import --password file.txt mykey.json
2 years ago
22defa5af7
all: introduce trie owner notion ( #24750 )
...
* cmd, core/state, light, trie, eth: add trie owner notion
* all: refactor
* tests: fix goimports
* core/state/snapshot: fix ineffasigns
Co-authored-by: Martin Holst Swende <martin@swende.se>
2 years ago
c375ee91e9
cmd/geth, core/state/snapshot: rework journal loading, implement account-check ( #24765 )
...
* cmd/geth, core/state/snapshot: rework journal loading, implement account-check
* core/state/snapshot, cmd/geth: polish code (#37 )
* core/state/snapshot: minor nits
* core/state/snapshot: simplify error logic
* cmd/geth: go format
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2 years ago
a10660b7f8
cmd/geth: extend traverseRawState command ( #24954 )
...
This PR adds node verification into traverseRawState command, so corrupted trie nodes can also be detected.
2 years ago
9244d5cd61
all: update license headers and AUTHORS from git history ( #24947 )
3 years ago
59ac229f87
core/state/snapshot: detect and clean up dangling storage snapshot in generation ( #24811 )
...
* core/state/snapshot: check dangling storages when generating snapshot
* core/state/snapshot: polish
* core/state/snapshot: wipe the last part of the dangling storages
* core/state/snapshot: fix and add tests
* core/state/snapshot: fix comment
* README: remove mentions of fast sync (#24656 )
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* core, cmd: expose dangling storage detector for wider usage
* core/state/snapshot: rename variable
* core, ethdb: use global iterators for snapshot generation
* core/state/snapshot: polish
* cmd, core/state/snapshot: polish
* core/state/snapshot: polish
* Update core/state/snapshot/generate.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
* ethdb: extend db test suite and fix memorydb iterator
* ethdb/dbtest: rollback changes
* ethdb/memorydb: simplify iteration
* core/state/snapshot: update dangling counter
* core/state/snapshot: release iterators
* core/state/snapshot: update metrics
* core/state/snapshot: update time metrics
* metrics/influxdb: temp solution to present counter meaningfully, remove it
* add debug log, revert later
* core/state/snapshot: fix iterator panic
* all: customized snapshot iterator for backward iteration
* core, ethdb: polish
* core/state/snapshot: remove debug log
* core/state/snapshot: address comments from peter
* core/state/snapshot: reopen the iterator at the next position
* ethdb, core/state/snapshot: address comment from peter
* core/state/snapshot: reopen exhausted iterators
Co-authored-by: Tbnoapi <63448616+nuoomnoy02@users.noreply.github.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
3 years ago
344d6f95cf
cmd: group network and db path flags together ( #24698 )
...
This PR groups all built-in network flags together and list them in the command as a whole.
And all database path flags(datadir, ancient) are also grouped, since usually these two are
used together.
3 years ago
0914234d10
cmd/geth, core/state/snapshot: fix flaw in dangling-storage check + inspect difflayers ( #24677 )
...
This PR fixes the flaw that @rjl493456442 found in https://github.com/ethereum/go-ethereum/pull/#issuecomment-1093817551 , namely, that the snapshot iterator uses the combined (disk + difflayers) 'view', wheres the raw iterator uses only the disk 'view'.
This PR instead splits up the work: one phase is iterating the disk layer data, another phase is loading the journalled difflayers and performing the same check there.
3 years ago
92e3c56e7b
cmd/geth: inspect snapshot dangling storage ( #24643 )
...
* cmd/geth: inspect snapshot dangling storage
* cmd/geth: make verify-state invoke verify-dangling
3 years ago
8401e4277a
core/rawdb: add HasCode, HashTrieNode and use them where possible ( #24454 )
3 years ago
893502e561
trie, core, eth: use db.has over db.get where possible
3 years ago
e1c000b0dd
cmd/geth: add support for sepolia testnet ( #23730 )
...
* cmd/geth: add support for sepolia testnet
* core: last details on sepolia genesis
* params: fix sepolia hash + reduce testing code
* Update params/bootnodes.go
* cmd/geth: fix attach path for sepolia
* params: update bootnodes
* params: fix
* core: fix docstring
* params: add sepolia CHT
3 years ago
443afc975c
core/state: move state account to core/types + abstracted "write account to trie" ( #23567 )
...
* core/state: abstracted "write account to trie" method
* fix appveyor build
* Apply suggestions from code review
Co-authored-by: Martin Holst Swende <martin@swende.se>
* review feedback
* core/state/accounts: move Account to core/types
* core/types: rename Account -> StateAccount
* core/state: restore EncodeRLP for stateObject
* core/types: add the missing file
* more review feedback
* more review feedback
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
3 years ago
b69f5ca7d4
cmd/utils: fix typo in variable name ( #23451 )
3 years ago
addd8824cf
cmd/geth, eth, core: snapshot dump + unify with trie dump ( #22795 )
...
* cmd/geth, eth, core: snapshot dump + unify with trie dump
* cmd/evm: dump API fixes
* cmd/geth, core, eth: fix some remaining errors
* cmd/evm: dump - add limit, support address startkey, address review concerns
* cmd, core/state, eth: minor polishes, fix snap dump crash, unify format
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
4 years ago
7088f1e814
core, eth: faster snapshot generation ( #22504 )
...
* eth/protocols: persist received state segments
* core: initial implementation
* core/state/snapshot: add tests
* core, eth: updates
* eth/protocols/snapshot: count flat state size
* core/state: add metrics
* core/state/snapshot: skip unnecessary deletion
* core/state/snapshot: rename
* core/state/snapshot: use the global batch
* core/state/snapshot: add logs and fix wiping
* core/state/snapshot: fix
* core/state/snapshot: save generation progress even if the batch is empty
* core/state/snapshot: fixes
* core/state/snapshot: fix initial account range length
* core/state/snapshot: fix initial account range
* eth/protocols/snap: store flat states during the healing
* eth/protocols/snap: print logs
* core/state/snapshot: refactor (#4 )
* core/state/snapshot: refactor
* core/state/snapshot: tiny fix and polish
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* core, eth: fixes
* core, eth: fix healing writer
* core, trie, eth: fix paths
* eth/protocols/snap: fix encoding
* eth, core: add debug log
* core/state/generate: release iterator asap (#5 )
core/state/snapshot: less copy
core/state/snapshot: revert split loop
core/state/snapshot: handle storage becoming empty, improve test robustness
core/state: test modified codehash
core/state/snapshot: polish
* core/state/snapshot: optimize stats counter
* core, eth: add metric
* core/state/snapshot: update comments
* core/state/snapshot: improve tests
* core/state/snapshot: replace secure trie with standard trie
* core/state/snapshot: wrap return as the struct
* core/state/snapshot: skip wiping correct states
* core/state/snapshot: updates
* core/state/snapshot: fixes
* core/state/snapshot: fix panic due to reference flaw in closure
* core/state/snapshot: fix errors in state generation logic + fix log output
* core/state/snapshot: remove an error case
* core/state/snapshot: fix condition-check for exhausted snap state
* core/state/snapshot: use stackTrie for small tries
* core/state/snapshot: don't resolve small storage tries in vain
* core/state/snapshot: properly clean up storage of deleted accounts
* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks
* core/state/snapshot: fix error (+testcase)
* core/state/snapshot: clean up tests a bit
* core/state/snapshot: work in progress on better tests
* core/state/snapshot: polish code
* core/state/snapshot: fix trie iteration abortion trigger
* core/state/snapshot: fixes flaws
* core/state/snapshot: remove panic
* core/state/snapshot: fix abort
* core/state/snapshot: more tests (plus failing testcase)
* core/state/snapshot: more testcases + fix for failing test
* core/state/snapshot: testcase for malformed data
* core/state/snapshot: some test nitpicks
* core/state/snapshot: improvements to logging
* core/state/snapshot: testcase to demo error in abortion
* core/state/snapshot: fix abortion
* cmd/geth: make verify-state report the root
* trie: fix failing test
* core/state/snapshot: add timer metrics
* core/state/snapshot: fix metrics
* core/state/snapshot: udpate tests
* eth/protocols/snap: write snapshot account even if code or state is needed
* core/state/snapshot: fix diskmore check
* core/state/snapshot: review fixes
* core/state/snapshot: improve error message
* cmd/geth: rename 'error' to 'err' in logs
* core/state/snapshot: fix some review concerns
* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync
* core: add error log
* core/state/snapshot: use proper timers for metrics collection
* core/state/snapshot: address some review concerns
* eth/protocols/snap: improved log message
* eth/protocols/snap: fix heal logs to condense infos
* core/state/snapshot: wait for generator termination before restarting
* core/state/snapshot: revert timers to counters to track total time
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
4 years ago
0c70b83e00
all: add read-only option to database ( #22407 )
...
* all: add read-only option to database
* all: fixes tests
* cmd/geth: migrate flags
* cmd/geth: fix the compact
* cmd/geth: fix the format
* cmd/geth: fix log
* cmd: add chain-readonly
* core: add readonly notion to freezer
* core/rawdb: add log
* core/rawdb: fix freezer close
* cmd: fix
* cmd, core: construct db
* core: update tests
4 years ago
6387c520b7
cmd/geth: add ancient datadir flag to snapshot subcommands ( #22486 )
4 years ago
8e547eecd5
cmd/utils: remove deprecated command line flags ( #22263 )
...
This removes support for all deprecated flags except --rpc*.
4 years ago
f566dd305e
all: bloom-filter based pruning mechanism ( #21724 )
...
* cmd, core, tests: initial state pruner
core: fix db inspector
cmd/geth: add verify-state
cmd/geth: add verification tool
core/rawdb: implement flatdb
cmd, core: fix rebase
core/state: use new contract code layout
core/state/pruner: avoid deleting genesis state
cmd/geth: add helper function
core, cmd: fix extract genesis
core: minor fixes
contracts: remove useless
core/state/snapshot: plugin stacktrie
core: polish
core/state/snapshot: iterate storage concurrently
core/state/snapshot: fix iteration
core: add comments
core/state/snapshot: polish code
core/state: polish
core/state/snapshot: rebase
core/rawdb: add comments
core/rawdb: fix tests
core/rawdb: improve tests
core/state/snapshot: fix concurrent iteration
core/state: run pruning during the recovery
core, trie: implement martin's idea
core, eth: delete flatdb and polish pruner
trie: fix import
core/state/pruner: add log
core/state/pruner: fix issues
core/state/pruner: don't read back
core/state/pruner: fix contract code write
core/state/pruner: check root node presence
cmd, core: polish log
core/state: use HEAD-127 as the target
core/state/snapshot: improve tests
cmd/geth: fix verification tool
cmd/geth: use HEAD as the verification default target
all: replace the bloomfilter with martin's fork
cmd, core: polish code
core, cmd: forcibly delete state root
core/state/pruner: add hash64
core/state/pruner: fix blacklist
core/state: remove blacklist
cmd, core: delete trie clean cache before pruning
cmd, core: fix lint
cmd, core: fix rebase
core/state: fix the special case for clique networks
core/state/snapshot: remove useless code
core/state/pruner: capping the snapshot after pruning
cmd, core, eth: fixes
core/rawdb: update db inspector
cmd/geth: polish code
core/state/pruner: fsync bloom filter
cmd, core: print warning log
core/state/pruner: adjust the parameters for bloom filter
cmd, core: create the bloom filter by size
core: polish
core/state/pruner: sanitize invalid bloomfilter size
cmd: address comments
cmd/geth: address comments
cmd/geth: address comment
core/state/pruner: address comments
core/state/pruner: rename homedir to datadir
cmd, core: address comments
core/state/pruner: address comment
core/state: address comments
core, cmd, tests: address comments
core: address comments
core/state/pruner: release the iterator after each commit
core/state/pruner: improve pruner
cmd, core: adjust bloom paramters
core/state/pruner: fix lint
core/state/pruner: fix tests
core: fix rebase
core/state/pruner: remove atomic rename
core/state/pruner: address comments
all: run go mod tidy
core/state/pruner: avoid false-positive for the middle state roots
core/state/pruner: add checks for middle roots
cmd/geth: replace crit with error
* core/state/pruner: fix lint
* core: drop legacy bloom filter
* core/state/snapshot: improve pruner
* core/state/snapshot: polish concurrent logs to report ETA vs. hashes
* core/state/pruner: add progress report for pruning and compaction too
* core: fix snapshot test API
* core/state: fix some pruning logs
* core/state/pruner: support recovering from bloom flush fail
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
4 years ago
Guillaume Ballet
Delweng
Péter Szilágyi
rjl493456442
Martin Holst Swende
willian.eth
Felix Lange
Ceyhun Onur
Ivan Bogatyy
michael1011
rene