From 8285cccc7ca4820857d3f678470818fbf29ecb00 Mon Sep 17 00:00:00 2001 From: Joe Date: Thu, 28 Jul 2022 12:04:48 +0100 Subject: [PATCH] update pages to newest aproved versions --- .../{mobile/mobile-accounts.md => mobile.md} | 261 ++++---- .../dapp-developer/mobile/mobile.md | 180 ------ .../fundamentals/Managing-your-accounts.md | 364 ++++------- content/docs/fundamentals/mining.md | 208 ++++++ .../getting-started-with-clef.md | 597 ++++++++++++++++++ content/docs/getting_started/index.md | 492 +++++++-------- content/docs/tools/Clef/Rules.md | 275 +++++--- 7 files changed, 1509 insertions(+), 868 deletions(-) rename content/docs/developers/dapp-developer/{mobile/mobile-accounts.md => mobile.md} (51%) delete mode 100644 content/docs/developers/dapp-developer/mobile/mobile.md create mode 100644 content/docs/fundamentals/mining.md create mode 100644 content/docs/getting_started/getting-started-with-clef.md diff --git a/content/docs/developers/dapp-developer/mobile/mobile-accounts.md b/content/docs/developers/dapp-developer/mobile.md similarity index 51% rename from content/docs/developers/dapp-developer/mobile/mobile-accounts.md rename to content/docs/developers/dapp-developer/mobile.md index 9d4bd39219..6eb3111e9a 100644 --- a/content/docs/developers/dapp-developer/mobile/mobile-accounts.md +++ b/content/docs/developers/dapp-developer/mobile.md @@ -1,148 +1,197 @@ --- -title: Mobile Account Management -sort_key: G +title: Geth for Mobile +sort_key: F --- -To provide Ethereum integration for your mobile applications, the very first thing you -should be interested in doing is account management. +Embedding clients into mobile devices is an important part of Ethereum's decentralization vision. +This is because being able to verify data, follow the chain and submit transactions without +relying on centralized intermediaries is critical for censorship resistant access +to the network. Doing so on a mobile device is the most convenient route for many users. +This relies on Geth running a [light client](/docs/interface/les) on the mobile +device and exposing an API that developers can use to build mobile apps on top of Geth. This +page outlines how to download Geth for mobile and how to get started with managing Ethereum +accounts in mobile applications. Ethereum mobile development is relatively nascent, but there is +an active developer community. For further information on Geth mobile development visit the +#mobile channel in the [Geth discord](https://discord.gg/wQdpS5aA). -Although all current leading Ethereum implementations provide account management built in, -it is ill advised to keep accounts in any location that is shared between multiple -applications and/or multiple people. The same way you do not entrust your ISP (who is -after all your gateway into the internet) with your login credentials; you should not -entrust an Ethereum node (who is your gateway into the Ethereum network) with your -credentials either. +## Download and install -The proper way to handle user accounts in your mobile applications is to do client side -account management, everything self-contained within your own application. This way you -can ensure as fine grained (or as coarse) access permissions to the sensitive data as -deemed necessary, without relying on any third party application's functionality and/or -vulnerabilities. +### Android -To support this, `go-ethereum` provides a simple, yet thorough accounts library that gives -you all the tools to do properly secured account management via encrypted keystores and -passphrase protected accounts. You can leverage all the security of the `go-ethereum` -crypto implementation while at the same time running everything in your own application. +#### Android Studio -## Encrypted keystores +Geth for Mobile bundles can be downloaded directly from [the download page](https://geth.ethereum.org/downloads/) +and inserted into a project in Android Studio via `File -> New -> New module... -> Import .JAR/.AAR Package`. -Although handling your users' accounts locally on their own mobile device does provide -certain security guarantees, access keys to Ethereum accounts should never lay around in -clear-text form. As such, we provide an encrypted keystore that provides the proper -security guarantees for you without requiring a thorough understanding from your part of -the associated cryptographic primitives. +It is also necessary to configure `gradle` to link the mobile library bundle to the +application. This can be done by adding a new entry to the `dependencies` section of the +`build.gradle` script, pointing it to the module that was just added (named `geth` by default). -The important thing to know when using the encrypted keystore is that the cryptographic -primitives used within can operate either in *standard* or *light* mode. The former -provides a higher level of security at the cost of increased computational burden and -resource consumption: +```gradle +dependencies { + // All previous dependencies + compile project(':geth') +} +``` + +#### Manual build + +Geth can also be built it locally using a `make` command. This will create an Android +archive called `geth.aar` in the `build/bin` folder that can be imported into Android +Studio as described above. + +```shell +$ make android +[...] +Done building. +Import "build/bin/geth.aar" to use the library. +``` + +### iOS + +Geth must be downloaded and built locally for IoS. Building locally is achieved using the +`make` command. This will create an iOS XCode framework called `Geth.framework` in the +`build/bin` folder that can be imported into XCode as described above. + +```bash +$ make ios +[...] +Done building. +Import "build/bin/Geth.framework" to use the library. +``` + +## Mobile API - * *standard* needs 256MB memory and 1 second processing on a modern CPU to access a key - * *light* needs 4MB memory and 100 millisecond processing on a modern CPU to access a key +Similarly to the reusable [Go libraries](/docs/dapp/native), the mobile wrappers focus on +three main usage areas: -As such, *light* is more suitable for mobile applications, but you should be aware of the -trade-offs nonetheless. +- Simplified client side account management +- Remote node interfacing via different transports +- Contract interactions through auto-generated bindings -*For those interested in the cryptographic and/or implementation details, the key-store -uses the `secp256k1` elliptic curve as defined in the [Standards for Efficient -Cryptography](sec2), implemented by the [`libsecp256k`][secp256k1] library and wrapped by -[`github.com/ethereum/go-ethereum/accounts`][accounts-go]. Accounts are stored on disk in -the [Web3 Secret Storage][secstore] format.* +The Geth mobile API is broadly equivalent to the [Go API](/docs/dapp/native). +The source code can be found in the `mobile` section of Geth's +[Github](https://github.com/ethereum/go-ethereum/tree/master/mobile). + +## Mobile Account Management + +Best practise for account management is to do it client-side, with all sensitive information +self-contained inside the local application. This ensures the developer/user retains +fine-grained control over the access permissions for user-data instead of outsourcing security +to a third party. + +To support this, Geth provides an accounts library that includes the tools required +for secure account management via encrypted keystores and passphrase protected accounts, +similarly to running a full Geth node. + +### Encrypted keystores + +Access keys to Ethereum accounts should never be stored in plain-text. Instead, they should +be stored encrypted so that even if the mobile device is accessed by a malicious third party +the keys are still hidden under an additional layer of security. Geth provides a keystore +that enables developers to store keys securely using the [`secp256k1` elliptic curve](sec2), +implemented using [`libsecp256k`][secp256k1] and wrapped by [Geth accounts][accounts-go]. +Accounts are stored on disk in the [Web3 Secret Storage][secstore] format. Developers should be +aware of these implementation details but are not required to deeply understand the cryptographic +primitives in order to use the keystore. + +One thing that should be understood, though, is that the cryptographic primitives underpinning +the keystore can operate in *light* or *standard* mode. Light mode is computationally cheaper, while +standard mode has extra security. Light mode is appropriate for mobile devices, but developers +should be aware that there is a security trade-off. + +* *standard* needs 256MB memory and 1 second processing on a modern CPU to access a key +* *light* needs 4MB memory and 100 millisecond processing on a modern CPU to access a key ### Keystores on Android (Java) The encrypted keystore on Android is implemented by the `KeyStore` class from the -`org.ethereum.geth` package. The configuration constants (for the *standard* or *light* -security modes described above) are located in the `Geth` abstract class, similarly from -the `org.ethereum.geth` package. Hence to do client side account management on Android, -you'll need to import two classes into your Java code: +`org.ethereum.geth` package. The configuration constants are located in the `Geth` +abstract class, similarly from the `org.ethereum.geth` package. +Hence to do client side account management on Android, two classes should be +imported into the Java code: ```java import org.ethereum.geth.Geth; import org.ethereum.geth.KeyStore; ``` -Afterwards you can create a new encrypted keystore via: +Then new encrypted keystore can be created via: ```java KeyStore ks = new KeyStore("/path/to/keystore", Geth.LightScryptN, Geth.LightScryptP); ``` -The path to the keystore folder needs to be a location that is writable by the local -mobile application but non-readable for other installed applications (for security reasons -obviously), so we'd recommend placing it inside your app's data directory. If you are -creating the `KeyStore` from within a class extending an Android object, you will most -probably have access to the `Context.getFilesDir()` method via `this.getFilesDir()`, so -you could set the keystore path to `this.getFilesDir() + "/keystore"`. +The keystore should be in a location writable by the local mobile application but +on-readable for other installed applications such as inside the app's data directory. +If the `KeyStore` is created from within a class extending an Android object, access +to the `Context.getFilesDir()` method is probably provided via `this.getFilesDir()`, +so the keystore path could be set to `this.getFilesDir() + "/keystore"`. The last two arguments of the `KeyStore` constructor are the crypto parameters defining -how resource-intensive the keystore encryption should be. You can choose between +how resource-intensive the keystore encryption should be. The choices are `Geth.StandardScryptN, Geth.StandardScryptP`, `Geth.LightScryptN, Geth.LightScryptP` or -specify your own numbers (please make sure you understand the underlying cryptography for -this). We recommend using the *light* version. +custom numbers. The *light* version is recommended. + ### Keystores on iOS (Swift 3) The encrypted keystore on iOS is implemented by the `GethKeyStore` class from the `Geth` -framework. The configuration constants (for the *standard* or *light* security modes -described above) are located in the same namespace as global variables. Hence to do client -side account management on iOS, you'll need to import the framework into your Swift code: +framework. The configuration constants are located in the same namespace as global +variables. Hence to do client side account management on iOS, `Geth` framework should be +imported into the Swift code: ```swift import Geth ``` -Afterwards you can create a new encrypted account manager via: +Then a new encrypted account manager can be created using: ```swift let ks = GethNewKeyStore("/path/to/keystore", GethLightScryptN, GethLightScryptP); ``` -The path to the keystore folder needs to be a location that is writable by the local -mobile application but non-readable for other installed applications (for security reasons -obviously), so we'd recommend placing it inside your app's document directory. You should -be able to retrieve the document directory via `let datadir = -NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0]`, so you -could set the keystore path to `datadir + "/keystore"`. +The keystore folder needs to be in a location writable by the local mobile application +but non-readable for other installed applications such as inside the app's document +directory. The document directory shopuld be retrievable using +`let datadir = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0]`, +so the keystore path could be `datadir + "/keystore"`. The last two arguments of the `GethNewKeyStore` factory method are the crypto parameters -defining how resource-intensive the keystore encryption should be. You can choose between +defining how resource-intensive the keystore encryption should be. The choices are `GethStandardScryptN, GethStandardScryptP`, `GethLightScryptN, GethLightScryptP` or -specify your own numbers (please make sure you understand the underlying cryptography for -this). We recommend using the *light* version. +custom numbers. The *light* version is recommended. -## Account lifecycle +### Account lifecycle -Having created an encrypted keystore for your Ethereum accounts, you can use this for the -entire account lifecycle requirements of your mobile application. This includes the basic -functionality of creating new accounts and deleting existing ones; as well as the more -advanced functionality of updating access credentials, exporting existing accounts, and -importing them on another device. +The encyrpted keystore can be used for the entire account lifecycle requirements of a mobile +application. This includes the basic functionality of creating new accounts and deleting +existing ones as well as more advanced functions like updating access credentials and account +import/export. -Although the keystore defines the encryption strength it uses to store your accounts, +Although the keystore defines the encryption strength it uses to store accounts, there is no global master password that can grant access to all of them. Rather each account is maintained individually, and stored on disk in its [encrypted format][secstore] individually, ensuring a much cleaner and stricter separation of credentials. -This individuality however means that any operation requiring access to an account will +This individuality means that any operation requiring access to an account will need to provide the necessary authentication credentials for that particular account in the form of a passphrase: * When creating a new account, the caller must supply a passphrase to encrypt the account - with. This passphrase will be required for any subsequent access, the lack of which - will forever forfeit using the newly created account. + with. This passphrase will be required for any subsequent access. * When deleting an existing account, the caller must supply a passphrase to verify ownership of the account. This isn't cryptographically necessary, rather a protective measure against accidental loss of accounts. * When updating an existing account, the caller must supply both current and new passphrases. After completing the operation, the account will not be accessible via the - old passphrase any more. + old passphrase. * When exporting an existing account, the caller must supply both the current passphrase to decrypt the account, as well as an export passphrase to re-encrypt it with before returning the key-file to the user. This is required to allow moving accounts between devices without sharing original credentials. * When importing a new account, the caller must supply both the encryption passphrase of - the key-file being imported, as well as a new passhprase with which to store the + the key-file being imported, as well as a new passphrase with which to store the account. This is required to allow storing account with different credentials than used for moving them around. @@ -153,9 +202,9 @@ guarantee that account credentials cannot be brute forced in any meaningful time ### Accounts on Android (Java) An Ethereum account on Android is implemented by the `Account` class from the -`org.ethereum.geth` package. Assuming we already have an instance of a `KeyStore` called -`ks` from the previous section, we can easily execute all of the described lifecycle -operations with a handful of function calls. +`org.ethereum.geth` package. Assuming an instance of a `KeyStore` called +`ks` exists, all of the described lifecycle operations can be executed with +a handful of function calls: ```java // Create a new account with the specified encryption passphrase. @@ -176,16 +225,15 @@ ks.deleteAccount(newAcc, "Update password"); Account impAcc = ks.importKey(jsonAcc, "Export password", "Import password"); ``` -*Although instances of `Account` can be used to access various information about specific +Although instances of `Account` can be used to access various information about specific Ethereum accounts, they do not contain any sensitive data (such as passphrases or private -keys), rather act solely as identifiers for client code and the keystore.* +keys), rather they act solely as identifiers for client code and the keystore. ### Accounts on iOS (Swift 3) An Ethereum account on iOS is implemented by the `GethAccount` class from the `Geth` -framework. Assuming we already have an instance of a `GethKeyStore` called `ks` from the -previous section, we can easily execute all of the described lifecycle operations with a -handful of function calls. +framework. Assuming an instance of a `GethKeyStore` called `ks` exists, all of the described +lifecycle operations can be executed with a handful of function calls: ```swift // Create a new account with the specified encryption passphrase. @@ -206,20 +254,20 @@ try! ks?.delete(newAcc, passphrase: "Update password") let impAcc = try! ks?.importKey(jsonKey, passphrase: "Export password", newPassphrase: "Import password") ``` -*Although instances of `GethAccount` can be used to access various information about +Although instances of `GethAccount` can be used to access various information about specific Ethereum accounts, they do not contain any sensitive data (such as passphrases or -private keys), rather act solely as identifiers for client code and the keystore.* +private keys), rather they act solely as identifiers for client code and the keystore. ## Signing authorization As mentioned above, account objects do not hold the sensitive private keys of the -associated Ethereum accounts, but are merely placeholders to identify the cryptographic +associated Ethereum accounts - they are merely placeholders to identify the cryptographic keys with. All operations that require authorization (e.g. transaction signing) are performed by the account manager after granting it access to the private keys. There are a few different ways one can authorize the account manager to execute signing -operations, each having its advantages and drawbacks. Since the different methods have -wildly different security guarantees, it is essential to be clear on how each works: +operations. Since the different methods have very different security guarantees, +it is essential to be clear on how each works: * **Single authorization**: The simplest way to sign a transaction via the keystore is to provide the passphrase of the account every time something needs to be signed, which @@ -240,17 +288,12 @@ wildly different security guarantees, it is essential to be clear on how each wo manager can sign transactions while a particular account is unlocked** (e.g. device left unattended; application running untrusted code). -*Note, creating transactions is out of scope here, so the remainder of this section will -assume we already have a transaction to sign, and will focus only on creating an -authorized version of it. Creating an actually meaningful transaction will be covered -later.* ### Signing on Android (Java) -Assuming we already have an instance of a `KeyStore` called `ks` from the previous -sections, we can create a new account to sign transactions with via it's already -demonstrated `newAccount` method; and to avoid going into transaction creation for now, we -can hard-code a random transaction to sign instead. +Assuming an instance of a `KeyStore` called `ks` exists, a new account to sign transactions +can be created using its `newAccount` method. For this demonstation a hard-coded +example transaction is created to sign: ```java // Create a new account to sign transactions with @@ -261,8 +304,7 @@ Transaction tx = new Transaction( BigInt chain = new BigInt(1); // Chain identifier of the main net ``` -With the boilerplate out of the way, we can now sign transaction using the authorization -mechanisms described above: +The transaction `tx` can be signed using the authorization mechanisms described above: ```java // Sign a transaction with a single authorization @@ -280,10 +322,9 @@ signed = ks.signTx(signer, tx, chain); ### Signing on iOS (Swift 3) -Assuming we already have an instance of a `GethKeyStore` called `ks` from the previous -sections, we can create a new account to sign transactions with via it's already -demonstrated `newAccount` method; and to avoid going into transaction creation for now, we -can hard-code a random transaction to sign instead. +Assuming an instance of a `GethKeyStore` called `ks` exists, a new account +can be created to sign transactions with its `newAccount` method. For +this demonstation a hard-coded example transaction is created to sign: ```swift // Create a new account to sign transactions with @@ -295,13 +336,12 @@ let tx = GethNewTransaction(1, to, GethNewBigInt(0), GethNewBigInt(0), GethNe let chain = GethNewBigInt(1) // Chain identifier of the main net ``` -*Note, although Swift usually rewrites `NSError` returns to throws, this particular +*Although Swift usually rewrites `NSError` returns to throws, this particular instance seems to have been missed for some reason (possibly due to it being a constructor). It will be fixed in a later version of the iOS bindings when the appropriate -fixed are implemented upstream in the `gomobile` project.* +fixes are implemented upstream in the `gomobile` project.* -With the boilerplate out of the way, we can now sign transaction using the authorization -methods described above: +The transaction `tx` can now be signed using the authorization methods described above: ```swift // Sign a transaction with a single authorization @@ -317,6 +357,11 @@ try! ks?.timedUnlock(signer, passphrase: "Signer password", timeout: 1000000000) signed = try! ks?.signTx(signer, tx: tx, chainID: chain) ``` +## Summary + +This page introduced Geth for mobile. In addition to download and installation instructions, basic +account management was demonstrated for mobile applications on iOS and Android. + [sec2]: https://www.secg.org/sec2-v2.pdf [accounts-go]: https://godoc.org/github.com/ethereum/go-ethereum/accounts [secp256k1]: https://github.com/bitcoin-core/secp256k1 diff --git a/content/docs/developers/dapp-developer/mobile/mobile.md b/content/docs/developers/dapp-developer/mobile/mobile.md deleted file mode 100644 index efcc107451..0000000000 --- a/content/docs/developers/dapp-developer/mobile/mobile.md +++ /dev/null @@ -1,180 +0,0 @@ ---- -title: Mobile API -sort_key: F ---- - -The Ethereum blockchain along with its two extension protocols Whisper and Swarm was -originally conceptualized to become the supporting pillar of web3, providing the -consensus, messaging and storage backbone for a new generation of distributed (actually, -decentralized) applications called DApps. - -The first incarnation towards this dream of web3 was a command line client providing an -RPC interface into the peer-to-peer protocols. The client was soon enough extended with a -web-browser-like graphical user interface, permitting developers to write DApps based on -the tried and proven HTML/CSS/JS technologies. - -As many DApps have more complex requirements than what a browser environment can handle, -it became apparent that providing programmatic access to the web3 pillars would open the -door towards a new class of applications. As such, the second incarnation of the web -dream is to open up all our technologies for other projects as reusable components. - -Starting with the 1.5 release family of `go-ethereum`, we transitioned away from providing -only a full blown Ethereum client and started shipping official Go packages that could be -embedded into third party desktop and server applications. It took only a small leap from -here to begin porting our code to mobile platforms. - -## Quick overview - -Similarly to our reusable Go libraries, the mobile wrappers also focus on four main usage -areas: - -- Simplified client side account management -- Remote node interfacing via different transports -- Contract interactions through auto-generated bindings -- In-process Ethereum, Whisper and Swarm peer-to-peer node - -You can watch a quick overview about these in Peter's (@karalabe) talk titled "Import -Geth: Ethereum from Go and beyond", presented at the Ethereum Devcon2 developer conference -in September, 2016 (Shanghai). Slides are [available -here](https://ethereum.karalabe.com/talks/2016-devcon.html). - -[![Peter's Devcon2 talk](https://img.youtube.com/vi/R0Ia1U9Gxjg/0.jpg)](https://www.youtube.com/watch?v=R0Ia1U9Gxjg) - -## Library bundles - -The `go-ethereum` mobile library is distributed either as an Android `.aar` archive -(containing binaries for `arm-7`, `arm64`, `x86` and `x64`); or as an iOS XCode framework -(containing binaries for `arm-7`, `arm64` and `x86`). We do not provide library bundles -for Windows phone the moment. - -### Android archive - -The simplest way to use `go-ethereum` in your Android project is through a Maven -dependency. We provide bundles of all our stable releases (starting from v1.5.0) through -Maven Central, and also provide the latest develop bundle through the Sonatype OSS -repository. - -#### Stable dependency (Maven Central) - -To add an Android dependency to the **stable** library release of `go-ethereum`, you'll -need to ensure that the Maven Central repository is enabled in your Android project, and -that the `go-ethereum` code is listed as a required dependency of your application. You -can do both of these by editing the `build.gradle` script in your Android app's folder: - -```gradle -repositories { - mavenCentral() -} - -dependencies { - // All your previous dependencies - compile 'org.ethereum:geth:1.5.2' // Change the version to the latest release -} -``` - -#### Develop dependency (Sonatype) - -To add an Android dependency to the current version of `go-ethereum`, you'll need to -ensure that the Sonatype snapshot repository is enabled in your Android project, and that -the `go-ethereum` code is listed as a required `SNAPSHOT` dependency of your application. -You can do both of these by editing the `build.gradle` script in your Android app's -folder: - -```gradle -repositories { - maven { - url "https://oss.sonatype.org/content/groups/public" - } -} - -dependencies { - // All your previous dependencies - compile 'org.ethereum:geth:1.5.3-SNAPSHOT' // Change the version to the latest release -} -``` - -#### Custom dependency - -If you prefer not to depend on Maven Central or Sonatype; or would like to access an older -develop build not available any more as an online dependency, you can download any bundle -directly from [our website](https://geth.ethereum.org/downloads/) and insert it into your -project in Android Studio via `File -> New -> New module... -> Import .JAR/.AAR Package`. - -You will also need to configure `gradle` to link the mobile library bundle to your -application. This can be done by adding a new entry to the `dependencies` section of your -`build.gradle` script, pointing it to the module you just added (named `geth` by default). - -```gradle -dependencies { - // All your previous dependencies - compile project(':geth') -} -``` - -#### Manual builds - -Lastly, if you would like to make modifications to the `go-ethereum` mobile code and/or -build it yourself locally instead of downloading a pre-built bundle, you can do so using a -`make` command. This will create an Android archive called `geth.aar` in the `build/bin` -folder that you can import into your Android Studio as described above. - -```bash -$ make android -[...] -Done building. -Import "build/bin/geth.aar" to use the library. -``` - -### iOS framework - -The simplest way to use `go-ethereum` in your iOS project is through a -[CocoaPods](https://cocoapods.org/) dependency. We provide bundles of all our stable -releases (starting from v1.5.3) and also latest develop versions. - -#### Automatic dependency - -To add an iOS dependency to the current stable or latest develop version of `go-ethereum`, -you'll need to ensure that your iOS XCode project is configured to use CocoaPods. -Detailing that is out of scope in this document, but you can find a guide in the upstream -[Using CocoaPods](https://guides.cocoapods.org/using/using-cocoapods.html) page. -Afterwards you can edit your `Podfile` to list `go-ethereum` as a dependency: - -```ruby -target 'MyApp' do - # All your previous dependencies - pod 'Geth', '1.5.4' # Change the version to the latest release -end -``` - -Alternatively, if you'd like to use the latest develop version, replace the package -version `1.5.4` with `~> 1.5.5-unstable` to switch to pre-releases and to always pull in -the latest bundle from a particular release family. - -#### Custom dependency - -If you prefer not to depend on CocoaPods; or would like to access an older develop build -not available any more as an online dependency, you can download any bundle directly from -[our website](https://geth.ethereum.org/downloads/) and insert it into your project in -XCode via `Project Settings -> Build Phases -> Link Binary With Libraries`. - -Do not forget to extract the framework from the compressed `.tar.gz` archive. You can do -that either using a GUI tool or from the command line via (replace the archive with your -downloaded file): - -``` -tar -zxvf geth-ios-all-1.5.3-unstable-e05d35e6.tar.gz -``` - -#### Manual builds - -Lastly, if you would like to make modifications to the `go-ethereum` mobile code and/or -build it yourself locally instead of downloading a pre-built bundle, you can do so using a -`make` command. This will create an iOS XCode framework called `Geth.framework` in the -`build/bin` folder that you can import into XCode as described above. - -```bash -$ make ios -[...] -Done building. -Import "build/bin/Geth.framework" to use the library. -``` diff --git a/content/docs/fundamentals/Managing-your-accounts.md b/content/docs/fundamentals/Managing-your-accounts.md index cdd6305a22..1df734593d 100644 --- a/content/docs/fundamentals/Managing-your-accounts.md +++ b/content/docs/fundamentals/Managing-your-accounts.md @@ -1,48 +1,22 @@ --- -title: Managing Your Accounts -sort_key: B +title: Account Management +sort_key: C --- -**WARNING** -Remember your password. +It is recommended to use the external key manager Clef for interacting with Geth because it can be run from secure external devices and has additional security benefits such as the ability to sign transactions according to custom rules. Instructions for setting up and using Clef can be found on the [Clef page](../clef/tutorial). However, Geth also has its own built-in account management tools that are more convenient and secure enough for many use-cases. This page will describe how to manage accounts using Geth's built in tools. The command line is considered first and then managing accounts from the Javascript console is considered in a [separate section](#accounts-in-the-javascript-console). -If you lose the password you use to encrypt your account, you will not be able to access that account. -Repeat: It is NOT possible to access your account without a password and there is no _forgot my password_ option here. Do not forget it. -The ethereum CLI `geth` provides account management via the `account` command: +## Account command + +Interacting with accounts is achieved using Geth's `account` command: ``` -$ geth account [options...] [arguments...] +geth account [options...] [arguments...] ``` -Manage accounts lets you create new accounts, list all existing accounts, import a private -key into a new account, migrate to newest key format and change your password. - -It supports interactive mode, when you are prompted for password as well as -non-interactive mode where passwords are supplied via a given password file. -Non-interactive mode is only meant for scripted use on test networks or known safe -environments. - -Make sure you remember the password you gave when creating a new account (with new, update -or import). Without it you are not able to unlock your account. - -Note that exporting your key in unencrypted format is NOT supported. - -Keys are stored under `/keystore`. Make sure you backup your keys regularly! See -[DATADIR backup & restore](../install-and-build/backup-restore) -for more information. If a custom datadir and keystore option are given the keystore -option takes preference over the datadir option. - -The newest format of the keyfiles is: `UTC----
`. The -order of accounts when listing, is lexicographic, but as a consequence of the timestamp -format, it is actually order of creation +The account command enables the user to create new accounts, list existing accounts, import private keys into a new account, update key formats and update the passwords that lock each account. In interactive mode, the user is prompted for passwords in the console when the `account` functions are invoked, whereas in non-interactive mode passwords to unlock accounts are saved to text files whose path is passed to Geth at startup. Non-interactive mode is only intended for use on private networks or known safe environments. -It is safe to transfer the entire directory or the individual keys therein between -ethereum nodes. Note that in case you are adding keys to your node from a different node, -the order of accounts may change. So make sure you do not rely or change the index in your -scripts or code snippets. - -And again. **DO NOT FORGET YOUR PASSWORD** +The `account` subcommands are: ``` COMMANDS: @@ -52,7 +26,8 @@ COMMANDS: import Import a private key into a new account ``` -You can get info about subcommands by `geth account --help`. +Information about the subcommands can be displayed in the terminal using `geth account --help`. For example, for the `list` subcommand: + ``` $ geth account list --help list [command options] [arguments...] @@ -60,293 +35,224 @@ list [command options] [arguments...] Print a short summary of all accounts OPTIONS: - --datadir "/home/bas/.ethereum" Data directory for the databases and keystore - --keystore Directory for the keystore (default = inside the datadir) + --datadir "/home/.ethereum" Data directory for the databases and keystore + --keystore Directory for the keystore (default = inside the datadir) ``` -Accounts can also be managed via the [Javascript Console](../interface/javascript-console) -## Examples -### Interactive use +## Creating new accounts -#### creating an account +New accounts can be created using `account new`. This generates a new key pair and adds them to the `keystore` directory in the `datadir`. To create a new account in the default data directory: -``` + +```shell $ geth account new +``` + +This returns the following to the terminal: + +```terminal Your new account is locked with a password. Please give a password. Do not forget this password. Passphrase: Repeat Passphrase: Address: {168bc315a2ee09042d83d7c5811b533620531f67} ``` -#### Listing accounts in a custom keystore directory - -``` -$ geth account list --keystore /tmp/mykeystore/ -Account #0: {5afdd78bdacb56ab1dad28741ea2a0e47fe41331} keystore:///tmp/mykeystore/UTC--2017-04-28T08-46-27.437847599Z--5afdd78bdacb56ab1dad28741ea2a0e47fe41331 -Account #1: {9acb9ff906641a434803efb474c96a837756287f} keystore:///tmp/mykeystore/UTC--2017-04-28T08-46-52.180688336Z--9acb9ff906641a434803efb474c96a837756287f - -``` +It is critical to backup the account password safely and securely as it cannot be retrieved or reset. -#### Import private key into a node with a custom datadir - -``` -$ geth account import --datadir /someOtherEthDataDir ./key.prv -The new account will be encrypted with a passphrase. -Please enter a passphrase now. -Passphrase: -Repeat Passphrase: -Address: {7f444580bfef4b9bc7e14eb7fb2a029336b07c9d} -``` +{% include note.html content=" If the password provided on account creation is lost or forgotten, there is no way to retrive it and the account will simply stay locked forever. The password MUST be backed up safely and securely! **IT IS CRITICAL TO BACKUP THE KEYSTORE AND REMEMBER PASSWORDS**" %} -#### Account update +The newly generated key files can be viewed in `/keystore/`. The file naming format is `UTC----
` where `date` is the date and time of key creation formatted according to [UTC 8601](https://www.iso.org/iso-8601-date-and-time-format.html) with zero time offset and seconds precise to eight decimal places. `address` is the 40 hexadecimal characters that make up the account address without a leading `0x`, for example: -``` -$ geth account update a94f5374fce5edbc8e2a8697c15331677e6ebf0b -Unlocking account a94f5374fce5edbc8e2a8697c15331677e6ebf0b | Attempt 1/3 -Passphrase: -0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b -Account 'a94f5374fce5edbc8e2a8697c15331677e6ebf0b' unlocked. -Please give a new password. Do not forget this password. -Passphrase: -Repeat Passphrase: -0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b -``` +`UTC--2022-05-19T12-34-36.47413510Z--0b85e5a13e118466159b1e1b6a4234e5f9f784bb` -### Non-interactive use -You supply a plaintext password file as argument to the `--password` flag. The data in the -file consists of the raw characters of the password, followed by a single newline. +## Listing Accounts -**Note**: Supplying the password directly as part of the command line is not recommended, -but you can always use shell trickery to get round this restriction. +Listing all existing accounts is achieved using the `account list` command. If the keystore is located anywhere other than the default location its path should be included with the `keystore` flag. For example, if the datadir is `some-dir`: +```shell +geth account list --keystore some-dir/keystore ``` -$ geth account new --password /path/to/password -$ geth account import --datadir /someOtherEthDataDir --password /path/to/anotherpassword ./key.prv -``` +This command returns the following to the terminal for a keystore with two files: -# Creating accounts +```terminal +Account 0: {5afdd78bdacb56ab1dad28741ea2a0e47fe41331} keystore:///tmp/mykeystore/UTC--2017-04-28T08-46-27.437847599Z--5afdd78bdacb56ab1dad28741ea2a0e47fe41331 +Account 1: {9acb9ff906641a434803efb474c96a837756287f} keystore:///tmp/mykeystore/UTC--2017-04-28T08-46-52.180688336Z--9acb9ff906641a434803efb474c96a837756287f +``` -## Creating a new account +The ordering of accounts when they are listed is lexicographic, but is effectively chronological based on time of creation due to the timestamp in the file name. It is safe to transfer the entire `keystore` directory or individual key files between Ethereum nodes. This is important because when accounts are added from other nodes the order of accounts in the keystore may change. It is therefore important not to rely on account indexes in scripts or code snippets. -``` -$ geth account new -$ geth account new --password /path/to/passwdfile -$ geth account new --password <(echo $mypassword) -``` +## Importing accounts -Creates a new account and prints the address. +### Import a keyfile -On the console, use: +It is also possible to create a new account by importing a private key. For example, a user might already have some ether at an address they created using a browser wallet and now wish to use a new Geth node to interact with their funds. In this case, the private key can be exported from the browser wallet and imported into Geth. Geth requires the private key to be stored as a file which contains the private key as unencrypted canonical elliptic curve bytes encoded into hex (i.e. plain text key without leading 0x). The new account is then saved in encrypted format, protected by a passphrase the user provides on request. As always, this passphrase must be securely and safely backed up - there is no way to retrieve or reset it if it is forgotten! +```shell +$ geth account import --datadir /some-dir ./keyfile ``` -> personal.newAccount() -... you will be prompted for a password ... -or - -> personal.newAccount("passphrase") +The following information will be displayed in the terminal, indicating a successful import: +```terminal +Please enter a passphrase now. +Passphrase: +Repeat Passphrase: +Address: {7f444580bfef4b9bc7e14eb7fb2a029336b07c9d} ``` -The account is saved in encrypted format. You **must** remember this passphrase to unlock -your account in the future. +This import/export process is not necessary for transferring accounts between Geth instances because the key files can simply be copied directly from one keystore to another. -For non-interactive use the passphrase can be specified with the `--password` flag: +It is also possible to import an account in non-interactive mode by saving the account password as plaintext in a `.txt` file and passing its path with the `--password` flag on startup. +```shell +geth account import --password path/password.txt path/keyfile ``` -geth account new --password +In this case, it is important to ensure the password file is not readable by anyone but the intended user. This can be achieved by changing the file permissions. On Linux, the following commands update the file permissions so only the current user has access: + +```shell +chmod 700 /path/to/password +cat > /path/to/password + ``` -Note, this is meant to be used for testing only, it is a bad idea to save your -password to file or expose in any other way. +### Import a presale wallet -## Creating an account by importing a private key +Assuming the password is known, importing a presale wallet is very easy. The `wallet import` commands are used, passing the path to the wallet. -``` - geth account import +```shell +geth wallet import /path/presale.wallet ``` -Imports an unencrypted private key from `` and creates a new account and prints -the address. -The keyfile is assumed to contain an unencrypted private key as canonical EC raw bytes -encoded into hex. +## Updating accounts -The account is saved in encrypted format, you are prompted for a passphrase. +The `account update` subcommand is used to unlock an account and migrate it to the newest format. This is useful for accounts that may have been created in a format that has since been deprecated. The same command can be used to update the account password. The current password and account address are needed in order to update the account, as follows: -You must remember this passphrase to unlock your account in the future. +```shell +geth account update a94f5374fce5edbc8e2a8697c15331677e6ebf0b +``` -For non-interactive use the passphrase can be specified with the `--password` flag: +The following will be returned to the terminal: -``` -geth account import --password +```terminal +Unlocking account a94f5374fce5edbc8e2a8697c15331677e6ebf0b | Attempt 1/3 +Passphrase: +0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b +Account 'a94f5374fce5edbc8e2a8697c15331677e6ebf0b' unlocked. +Please give a new password. Do not forget this password. +Passphrase: +Repeat Passphrase: +0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b ``` -**Note**: Since you can directly copy your encrypted accounts to another ethereum -instance, this import/export mechanism is not needed when you transfer an account between -nodes. +Alternatively, in non-interactive mode the path to a password file containing the account password in unencrypted plaintext can be passed with the `--password` flag: -**Warning:** when you copy keys into an existing node's keystore, the order of accounts -you are used to may change. Therefore you make sure you either do not rely on the account -order or doublecheck and update the indexes used in your scripts. +```shell +geth account update a94f5374fce5edbc8e2a8697c15331677e6ebf0b --password path/password.txt +``` -**Warning:** If you use the password flag with a password file, best to make sure the file -is not readable or even listable for anyone but you. You achieve this with: +Updating the account replaces the original file with a new one - this means the original file is no longer available after it has been updated. -``` -touch /path/to/password -chmod 700 /path/to/password -cat > /path/to/password ->I type my pass here^D -``` -## Updating an existing account +## Unlocking accounts -You can update an existing account on the command line with the `update` subcommand with -the account address or index as parameter. You can specify multiple accounts at once. +In Geth, accounts are locked unless they are explicitly unlocked. If an account is intended to be used by apps connecting to Geth via RPC then it can be unlocked in non-interactive mode by passing the `--unlock` flag with a comma-separated list of account addresses (or keystore indexes) to unlock. This unlocks the accounts for one session only. Including the `--unlock` flag without any account addresses defaults to unlocking the first account in the keystore. -``` -geth account update 5afdd78bdacb56ab1dad28741ea2a0e47fe41331 9acb9ff906641a434803efb474c96a837756287f -geth account update 0 1 2 +```shell +geth --unlock 0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b ``` -The account is saved in the newest version in encrypted format, you are prompted -for a passphrase to unlock the account and another to save the updated file. +Geth will start and prompt the user to input the account password in the terminal. Alternatively, the user can provide a password as a text file and pass its path to `--password`: -This same command can therefore be used to migrate an account of a deprecated -format to the newest format or change the password for an account. +```shell +geth --unlock 0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b --password path/password.txt +``` -After a successful update, all previous formats/versions of that same key are removed! +{% include note.html content=" By default, account **unlocking is forbidden when HTTP or Websocket access is enabled** (i.e. by passing `--http` or `ws` flag). This is because an attacker that manages to access the node via the externally-exposed HTTP/WS port can then control the unlocked account. It is possible to force account unlock by including the `--allow-insecure-unlock` flag but this is unsafe and **not recommended** except for expert users that completely understand how it can be used safely. This is not a hypothetical risk: **there are bots that continually scan for http-enabled Ethereum nodes to attack**" %} -# Importing your presale wallet -Importing your presale wallet is very easy. If you remember your password that is: +## Accounts in the Javascript console -``` -geth wallet import /path/to/my/presale.wallet +Account management can also be achieved in the Javascript console attached to a running Geth instance. Assuming Geth is already running, in a new terminal attach a Javascript console using the `geth.ipc` file. This file can be found in the data directory. Assuming the data directory is named `data` the console can be started using: + +```shell +geth attach data/geth.ipc ``` -will prompt for your password and imports your ether presale account. It can be used -non-interactively with the --password option taking a passwordfile as argument containing -the wallet password in cleartext. +### New accounts -# Listing accounts and checking balances +New accounts can be generated using the Javascript console using `personal.newAccount()`. A new password is requested in the console and successful account creation is confirmed by the new account address being displayed. -### Listing your current accounts +```shell +personal.newAccount() +``` -From the command line, call the CLI with: +Accounts can also be created by importing private keys directly in the Javascript console. The private key is passed as an unencrypted hex-encoded string to `personal.importRawKey()` along with a passphrase that will be used to encrypt the key. A new key file will be generated from the private key and saved to the keystore. -``` -$ geth account list -Account #0: {5afdd78bdacb56ab1dad28741ea2a0e47fe41331} keystore:///tmp/mykeystore/UTC--2017-04-28T08-46-27.437847599Z--5afdd78bdacb56ab1dad28741ea2a0e47fe41331 -Account #1: {9acb9ff906641a434803efb474c96a837756287f} keystore:///tmp/mykeystore/UTC--2017-04-28T08-46-52.180688336Z--9acb9ff906641a434803efb474c96a837756287f +```shell +personal.importRawKey("hexstringkey", "password") ``` -to list your accounts in order of creation. +### Listing accounts -**Note**: -This order can change if you copy keyfiles from other nodes, so make sure you either do not rely on indexes or make sure if you copy keys you check and update your account indexes in your scripts. +The `accounts` function in the `eth` namespace can be used to list the accounts that currently exist in the keystore.: -When using the console: ``` -> eth.accounts -["0x5afdd78bdacb56ab1dad28741ea2a0e47fe41331", "0x9acb9ff906641a434803efb474c96a837756287f"] +eth.accounts ``` -or via RPC: -``` -# Request -$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_accounts","params":[],"id":1}' -H 'Content-type: application/json' http://127.0.0.1:8545 +or alternatively the same is achieved using: -# Result -{ - "id":1, - "jsonrpc": "2.0", - "result": ["0x5afdd78bdacb56ab1dad28741ea2a0e47fe41331", "0x9acb9ff906641a434803efb474c96a837756287f"] -} +``` +personal.listAccounts ``` -If you want to use an account non-interactively, you need to unlock it. You can do this on -the command line with the `--unlock` option which takes a comma separated list of accounts -(in hex or index) as argument so you can unlock the accounts programmatically for one -session. This is useful if you want to use your account from Dapps via RPC. `--unlock ` -will unlock the first account. This is useful when you created your account -programmatically, you do not need to know the actual account to unlock it. +This returns an array of account addresses to the terminal. -Create account and start node with account unlocked: -``` -geth account new --password <(echo this is not secret!) -geth --password <(echo this is not secret!) --unlock primary --rpccorsdomain localhost --verbosity 6 2>> geth.log -``` -Instead of the account address, you can use integer indexes which refers to the address -position in the account listing (and corresponds to order of creation) +### Unlocking accounts -The command line allows you to unlock multiple accounts. In this case the argument to -unlock is a comma delimited list of accounts addresses or indexes. +To unlock an account, the `personal.unlockAccount` function can be used: ``` -geth --unlock "0x407d73d8a49eeb85d32cf465507dd71d507100c1,0,5,e470b1a7d2c9c5c6f03bbaa8fa20db6d404a0c32" +personal.unlockAccount(eth.accounts[1]) ``` -If this construction is used non-interactively, your password file will need to contain -the respective passwords for the accounts in question, one per line. - -On the console you can also unlock accounts (one at a time) for a duration (in seconds). +The account passphrase is requested: +```terminal +Unlock account 0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b +Passphrase: +true ``` -personal.unlockAccount(address, "password", 300) + +This unlocked account can now be used to sign and send transactions. it is also possible to pass the passphrase as an argument to `personal.unlockAccount()` along with a duration after which the accout will automatically re-lock (in seconds), as follows: + +```shell +personal.unlockAccount(eth.accounts[1], "passphrase", 60) ``` -Note that we do NOT recommend using the password argument here, since the console history -is logged, so you may compromise your account. You have been warned. +This unlocks the account for 60 seconds. However, this is not recommended because the command history is logged by the Javascript console which could compromise the security of the account. An unlocked account can be manually re-locked using `personal.lockAccount()`, passing the address as the sole argument. -### Checking account balances -To check your the etherbase account balance: -``` -> web3.fromWei(eth.getBalance(eth.coinbase), "ether") -6.5 -``` +### Unlocking for transactions -Print all balances with a JavaScript function: -``` -function checkAllBalances() { - var totalBal = 0; - for (var acctNum in eth.accounts) { - var acct = eth.accounts[acctNum]; - var acctBal = web3.fromWei(eth.getBalance(acct), "ether"); - totalBal += parseFloat(acctBal); - console.log(" eth.accounts[" + acctNum + "]: \t" + acct + " \tbalance: " + acctBal + " ether"); - } - console.log(" Total balance: " + totalBal + " ether"); -}; -``` -That can then be executed with: -``` -> checkAllBalances(); - eth.accounts[0]: 0xd1ade25ccd3d550a7eb532ac759cac7be09c2719 balance: 63.11848 ether - eth.accounts[1]: 0xda65665fc30803cb1fb7e6d86691e20b1826dee0 balance: 0 ether - eth.accounts[2]: 0xe470b1a7d2c9c5c6f03bbaa8fa20db6d404a0c32 balance: 1 ether - eth.accounts[3]: 0xf4dd5c3794f1fd0cdc0327a83aa472609c806e99 balance: 6 ether -``` +Sending transactions from the Javascript console also requires the sender account to be unlocked. There are two ways to send transactions: `eth.sendTransaction` and `personal.sendTransaction`. The difference between these two functions is that `eth.sendTransaction` requires the account to be unlocked globally, at the node level (i.e., by unlocking it on the command line at the start of the Geth session). On the other hand, `personal.sendTransaction` takes a passphrase argument that unlocks the account temporarily in order to sign the transaction, then locks it again immediately afterwards. For example, to send 5 ether between two accounts in the keystore: -Since this function will disappear after restarting geth, it can be helpful to store -commonly used functions to be recalled later. The -[loadScript](../interface/javascript-console) -function makes this very easy. +```shell +var tx = {from: eth.accounts[1], to: eth.accounts[2], value: web3.toWei(5, "ether")} -First, save the `checkAllBalances()` function definition to a file on your computer. For -example, `/Users/username/gethload.js`. Then load the file from the interactive console: +# this requires global account unlock for eth.accounts[1] +eth.sendTransaction(tx) +# this unlocks eth.accounts[1] temporarily just to send the transaction +personal.sendTransaction(tx, "password") ``` -> loadScript("/Users/username/gethload.js") -true -``` -The file will modify your JavaScript environment as if you has typed the commands -manually. Feel free to experiment! +## Summary + +This page has demonstrated how to use Geth's built-in account management tools, both on the command line and in the Javascript console. Accounts are stored encrypted by a password. It is critical that the account passwords and the keystore directory are safely and securely backed up. diff --git a/content/docs/fundamentals/mining.md b/content/docs/fundamentals/mining.md new file mode 100644 index 0000000000..d2d91f9cf7 --- /dev/null +++ b/content/docs/fundamentals/mining.md @@ -0,0 +1,208 @@ +--- +title: Mining +sort_key: F +--- + + +The Ethereum blockchain grows when nodes add blocks and distribute them to their peers. Nodes +that add blocks are rewarded with ether payouts. This creates competition for the right to add +blocks to the blockchain. This means some mechanism must exist to select a single block for each +position in the blockchain that all nodes can agree upon. The mechanism that currently achieves +this for Ethereum is "proof-of-work" (PoW). This involved computing a certain value that can +only be calculated using repeated random guesses. Only if a node can demonstrate that they have +calculated this value, and therefore expended energy, will their block be accepted by other nodes. +This secures the network. This process of creating blocks and securing them using proof-of-work +is known as "mining". + +Much more information about mining, including details about the specific algorithm ("Ethash") used by +Ethereum nodes is available on +[ethereum.org](https://ethereum.org/en/developers/docs/consensus-mechanisms/pow/mining). + + +## Mining and the Merge + +[The Merge](https://ethereum.org/en/upgrades/merge) is an upcoming upgrade to Ethereum that +will swap the existing PoW for a [proof-of-stake (PoS)](https://ethereum.org/en/developers/docs/consensus-mechanisms/pos) consensus mechanism. This marks the end of mining on Ethereum. Instead, nodes can [stake ether](https://ethereum.org/en/staking/solo/#get-started-on-the-staking-launchpad) directly and earn ether rewards by running [validators](https://ethereum.org/en/developers/docs/consensus-mechanisms/pos/#validators). The merge is expected to happen in the second half of 2022. Until then, Ethereum will continue to be secured by PoW miners. It is no longer recommended to purchase new hardware to participate in Ethereum mining because the chances of returning a profit before The Merge are low. + + +## CPU vs GPU + +Participating in Ethereum's PoW mining requires running an algorithm called +["Ethash"](https://ethereum.org/en/developers/docs/consensus-mechanisms/pow/mining-algorithms/ethash). Geth includes +a CPU miner which runs Ethash within the Geth process. This might be useful for mining on some testnets. However, this is CPU mining is not viable on Ethereum Mainnet because CPU miners are easily out-competed by more efficient GPU miners. GPU mining is the recommended method for mining real ether on Ethereum Mainnet, but it is not part of the standard Geth installation. To mine using GPUs an additional piece of third-paty software is required. The recommended GPU mining software is [Ethminer](https://github.com/ethereum-mining/ethminer). + +Regardless of the mining method, the blockchain must be fully synced before mining is started, otherwise the miner will build on an incorrect chain, invalidating the block rewards. + + +## GPU Mining + +### Installing Ethminer + +The Ethminer software can be installed from a downloaded binary or built from source. The relevant downloads +and installation instructions are available from the [Ethminer Github](https://github.com/ethereum-mining/ethminer/#build). Standalone executables are available for Linux, macOS and Windows. + +### Using Ethminer with Geth + + +An account to receive block rewards must first be defined. The address of the account is all that is required to start mining - the mining rewards will be credited to that address. This can be an existing address or one that is newly created by Geth. More detailed instructions on creating and importing accounts are available on the [Account Management](/docs/interface/managing-your-accounts) page. + +The account address can be provided to `--mining.etherbase` when Geth is started. This instructs Geth to direct any block rewards to this address. Once started, Geth will sync the blockchain. If Geth has not connected to this network before, or if the data directory has been deleted, this can take several days. Also, enable HTTP traffic with the `--http` command. + +```shell +geth --http --miner.etherbase 0xC95767AC46EA2A9162F0734651d6cF17e5BfcF10 +``` + +The progress of the blockchain syncing can be monitored by attaching a JavaScript console in another terminal. More detailed information about the console can be found on the [Javascript Console](/docs/interface/javascript-console) page. To attach and open a console: + +```shell +geth attach http://127.0.0.1:8545 +``` + +Then in the console, to check the sync progress: + +```shell +eth.syncing +``` + +If the sync is progressing correctly the output will look similar to the following: + +```terminal +{ + currentBlock: 13891665, + healedBytecodeBytes: 0, + healedBytecodes: 0, + healedTrienodeBytes: 0, + healedTrienodes: 0, + healingBytecode: 0, + healingTrienodes: 0, + highestBlock: 14640000, + startingBlock: 13891665, + syncedAccountBytes: 0, + syncedAccounts: 0, + syncedBytecodeBytes: 0, + syncedBytecodes: 0, + syncedStorage: 0, + syncedStorageBytes: 0 +} +``` + +Once the blockchain is sync'd, mining can begin. In order to begin mining, Ethminer must be run and connected to Geth in a new terminal. OpenCL can be used for a wide range of GPUs, CUDA can be used specifically for Nvidia GPUs: + +```shell +#OpenCL +ethminer -v 9 -G -P http://127.0.0.1:8545 +``` + +```shell +#CUDA +ethminer -v -U -P http://127.0.0.1:8545 +``` + +Ethminer communicates with Geth on port 8545 (Geth's default RPC port) but this can be changed by providing a custom +port to the `http.port` command. The corresponding port must also be configured in Ethminer by providing +`-P http://127.0.0.1:`. This is necessary when multiple instances of Geth/Ethminer will coexist on the same machine. + +If using OpenCL and the default for `ethminer` does not work, specifying the device using the `--opencl--device X` command is a common fix. `X` is an integer `1`, `2`, `3` etc. The Ethminer `-M` (benchmark) command should display something that looks like: + +```terminal +Benchmarking on platform: { "platform": "NVIDIA CUDA", "device": "GeForce GTX 750 Ti", "version": "OpenCL 1.1 CUDA" } + +Benchmarking on platform: { "platform": "Apple", "device": "Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz", "version": "OpenCL 1.2 " } +``` + +Note that the Geth command `miner.hashrate` only works for CPU mining - it always reports zero for GPU mining. To check the GPU mining hashrate, check the logs `ethminer` displays to its terminal. More verbose logs can be configured using `-v` and a value between 0-9. + +The Ethash algorithm is [memory-hard](https://crypto.stackexchange.com/questions/84002/memory-hard-vs-memory-bound-functions) and requires a large dataset to be loaded into memory. Each GPU requires 4-5 GB of RAM. The error message `Error GPU mining. GPU memory fragmentation?` indicates that there is insufficient memory available. + + +## CPU Mining with Geth + +When Geth is started is is not mining by default. Unless it is specifically instructed to mine, it acts only as a node, not a miner. Geth starts as a (CPU) miner if the `--mine` flag is provided. The `--miner.threads` parameter can +be used to set the number parallel mining threads (defaulting to the total number of processor cores). + +```shell +geth --mine --miner.threads=4 +``` + +CPU mining can also be started and stopped at runtime using the [console](/docs/interface/javascript-console). The command `miner.start` takes an optional parameter for the number of miner threads. + +```js +miner.start(8) + true +miner.stop() + true +``` + +Note that mining for real ether only makes sense if you are in sync with the network (since you mine on top of the consensus block). Therefore the Ethereum blockchain downloader/synchroniser will delay mining until syncing is complete, and after that mining automatically starts unless you cancel your intention with `miner.stop()`. + +Like with GPU mining, an etherbase account must be set. This defaults to the primary account in the keystore but can be set to an alternative address using the `--miner.etherbase` command: + +```shell +geth --miner.etherbase '0xC95767AC46EA2A9162F0734651d6cF17e5BfcF10' --mine +``` +If there is no account available the miner will not start. The Javascript console can also be used to reset the etherbase account at runtime: + +```shell +miner.setEtherbase(eth.accounts[2]) +``` + +Note that your etherbase does not need to be an address of a local account, it just has to be set to an existing one. + +There is an option to add extra data (32 bytes only) to the mined blocks. By convention this is interpreted as a unicode string, so it can be used to add a short vanity tag using `miner.setExtra` in the Javascript console. + +```shell +miner.setExtra("ΞTHΞЯSPHΞЯΞ") +``` + +The console can also be used to check the current hashrate in units H/s (Hash operations per second): + +```shell +eth.hashrate + 712000 +``` + +After some blocks have been mined, the etherbase account balance with be >0. Assuming the etherbase is a local account: + +```shell +eth.getBalance(eth.coinbase).toNumber(); + '34698870000000' +``` + +It is also possible to check which blocks were mined by a particular miner (address) using the following code snippet in the Javascript console: + +```js +function minedBlocks(lastn, addr) { + addrs = []; + if (!addr) { + addr = eth.coinbase + } + limit = eth.blockNumber - lastn + for (i = eth.blockNumber; i >= limit; i--) { + if (eth.getBlock(i).miner == addr) { + addrs.push(i) + } + } + return addrs +} + +// scans the last 1000 blocks and returns the blocknumbers of blocks mined by your coinbase +// (more precisely blocks the mining reward for which is sent to your coinbase). +minedBlocks(1000, eth.coinbase) +[352708, 352655, 352559] + +``` + +The etherbase balance will fluctuate because quite often a mined block may be re-org'd out +of the canonical chain. This means that when the local Geth node includes the mined block +in its own local blockchain the account balance appears higher because the block rewards are +applied. When the node switches to another version of the chain due to information received +from peers, that block may not be included and the block rewards are not applied. + +The logs show locally mined blocks confirmed after 5 blocks. + + +## Summary + +The page describes how to start Geth as a mining node. Mining can be done on CPUs - in which case Geth's built-in +miner can be used - or on GPUs which requires third party software. GPUs are required to mine real ether on Ethereum +Mainnet. It is important to note that Ethereum will swap its consensus mechanism from PoW to PoS in the second half of 2022. This swap, known as "The Merge" will end mining on Ethereum. diff --git a/content/docs/getting_started/getting-started-with-clef.md b/content/docs/getting_started/getting-started-with-clef.md new file mode 100644 index 0000000000..684996f7ed --- /dev/null +++ b/content/docs/getting_started/getting-started-with-clef.md @@ -0,0 +1,597 @@ +--- +title: Geth and Clef +permalink: docs/getting-started/geth-and-clef +sort_key: B +--- + +This page explains how to set up Geth and execute some basic tasks using the command line tools. +In order to use Geth, the software must first be installed. There are several ways Geth can be +installed depending on the operating system and the user's choice of installation method, for example +using a package manager, container or building from source. Instructions for installing Geth can be +found on the ["Install and Build"](install-and-build/installing-geth) pages. The tutorial on this +page assumes Geth and the associated developer tools have been installed successfully. + +This page provides step-by-step instructions covering the fundamentals of using Geth. This +includes generating accounts, joining an Ethereum network, syncing the blockchain and sending ether +between accounts. This tutorial also uses [Clef](/docs/clef/tutorial). Clef is an account management tool +external to Geth itself that allows users to sign transactions. It is developed and maintained by +the Geth team and is intended to eventually replace the account management tool built in to Geth. + +## Prerequisites + +In order to get the most value from the tutorials on this page, the following skills are necessary: + +- Experience using the command line +- Basic knowledge about Ethereum and testnets +- Basic knowledge about HTTP and JavaScript + +Users that need to revisit these fundamentals can find helpful resources relating to the command line +[here][cli], +Ethereum and its testnets [here](https://ethereum.org/en/developers/tutorials/), http +[here](https://developer.mozilla.org/en-US/docs/Web/HTTP) and Javascript [here](https://www.javascript.com/learn). + +{% include note.html content="If Geth was installed from source on Linux, `make` saves the binaries for +Geth and the associated tools in `/build/bin`. To run these programs it is convenient to move them to +the top level project directory (e.g. running `mv ./build/bin/* ./`) from `/go-ethereum`. Then `./` must +be prepended to the commands in the code snippets in order to execute a particular program, e.g. `./geth` +instead of simply `geth`. If the executables are not moved then either navigate to the `bin` directory to +run them (e.g. `cd ./build/bin` and `./geth`) or provide their path (e.g. `./build/bin/geth`). These +instructions can be ignored for other installations." %} + +## Background + +Geth is an Ethereum client written in Go. This means running Geth turns a computer into an Ethereum node. +Ethereum is a peer-to-peer network where information is shared directly between nodes rather than being +managed by a central server. Nodes compete to generate new blocks of transactions to send to its peers +because they are rewarded for doing so in Ethereum's native token, ether (ETH). On receiving a new block, +each node checks that it is valid and adds it to their database. The sequence of discrete blocks is called +a "blockchain". The information provided in each block is used by Geth to update its "state" - the ether +balance of each account on Ethereum. There are two types of account: externally-owned accounts (EOAs) and +contract accounts. Contract accounts execute contract code when they receive transactions. EOAs are accounts +that users manage locally in order to sign and submit transactions. Each EOA is a public-private key pair, +where the public key is used to derive a unique address for the user and the private key is used to protect +the account and securely sign messages. Therefore, in order to use Ethereum, it is first necessary to generate +an EOA (hereafter, "account"). This tutorial will guide the user through creating an account, funding it with +ether and sending some to another address. + +Read more about Ethereum accounts [here](https://ethereum.org/en/developers/docs/accounts/). + +## Step 1: Generating accounts + +There are several methods for generating accounts in Geth. This tutorial demonstrates how to generate accounts +using Clef, as this is considered best practice, largely because it decouples the users' key management from Geth, +making it more modular and flexible. It can also be run from secure USB sticks or virtual machines, offering +security benefits. For convenience, this tutorial will execute Clef on the same computer that will also run Geth, +although more secure options are available (see [here](https://github.com/ethereum/go-ethereum/blob/master/cmd/clef/docs/setup.md)). + +An account is a pair of keys (public and private). Clef needs to know where to save these keys to so that they +can be retrieved later. This information is passed to Clef as an argument. This is achieved using the following command: + +```shell +clef newaccount --keystore geth-tutorial/keystore +``` + +The specific function from Clef that generates new accounts is `newaccount` and it accepts a +parameter, `--keystore`, that tells it where to store the newly generated keys. In this +example the keystore location is a new directory that will be created automatically: `geth-tutorial/keystore`. +Clef will return the following result in the terminal: + +```terminal +WARNING! + +Clef is an account management tool. It may, like any software, contain bugs. + +Please take care to +- backup your keystore files, +- verify that the keystore(s) can be opened with your password. + +Clef is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY +without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR +PURPOSE. See the GNU General Public License for more details. + +Enter 'ok' to proceed: +> +``` + +This is important information. The `geth-tutorial/keystore` directory will soon contain a secret +key that can be used to access any funds held in the new account. If it is compromised, the funds +can be stolen. If it is lost, there is no way to retrieve the funds. This tutorial will only use +dummy funds with no real world value, but when these steps are repeated on Ethereum mainnet is +critical that the keystore is kept secure and backed up. + + +Typing `ok` into the terminal and pressing `enter` causes Clef to prompt for a password. +Clef requires a password that is at least 10 characters long, and best practice would be +to use a combination of numbers, characters and special characters. Entering a suitable +password and pressing `enter` returns the following result to the terminal: + +```terminal +----------------------- +DEBUG[02-10|13:46:46.436] FS scan times list="92.081µs" set="12.629µs" diff="2.129µs" +INFO [02-10|13:46:46.592] Your new key was generated address=0xCe8dBA5e4157c2B284d8853afEEea259344C1653 +WARN [02-10|13:46:46.595] Please backup your key file! path=keystore:///.../geth-tutorial/keystore/UTC--2022-02-07T17-19-56.517538000Z--ca57f3b40b42fcce3c37b8d18adbca5260ca72ec +WARN [02-10|13:46:46.595] Please remember your password! +Generated account 0xCe8dBA5e4157c2B284d8853afEEea259344C1653 +``` + +It is important to save the account address and the password somewhere secure. +They will be used again later in this tutorial. Please note that the account address shown +in the code snippets above and later in this tutorials are examples - those generated by followers +of this tutorial will be different. The account generated above can be used as the main account +throughout the remainder of this tutorial. However in order to demonstrate transactions between +accounts it is also necessary to have a second account. A second account can be added to the +same keystore by precisely repeating the previous steps, providing the same password. + +## Step 2: Start Clef + +The previous commands used Clef's `newaccount` function to add new key pairs to the keystore. +Clef uses the private key(s) saved in the keystore is used to sign transactions. In order to do +this, Clef needs to be started and left running while Geth is running simultaneously, so that the +two programs can communicate between one another. + +To start Clef, run the Clef executable passing as arguments the keystore file location, +config directory location and a chain ID. The config directory was automatically created +inside the `geth-tutorial` directory during the previous step. The [chain ID](https://chainlist.org/) +is an integer that defines which Ethereum network to connect to. Ethereum mainnet has chain ID 1. +In this tutorial Chain ID 5 is used which is that of the Goerli testnet. It is very important that +this chain ID parameter is set to 5. The following command starts Clef on Goerli: + +```shell + +clef --keystore geth-tutorial/keystore --configdir geth-tutorial/clef --chainid 5 + +``` + +After running the command above, Clef requests the user to type “ok” to proceed. On typing "ok" and pressing enter, Clef returns the following to the terminal: + +```terminal +INFO [02-10|13:55:30.812] Using CLI as UI-channel +INFO [02-10|13:55:30.946] Loaded 4byte database embeds=146,841 locals=0 local=./4byte-custom.json +WARN [02-10|13:55:30.947] Failed to open master, rules disabled err="failed stat on geth-tutorial/clef/masterseed.json: stat geth-tutorial/clef/masterseed.json: no such file or directory" +INFO [02-10|13:55:30.947] Starting signer chainid=5 keystore=geth-tutorial/keystore light-kdf=false advanced=false +DEBUG[02-10|13:55:30.948] FS scan times list="133.35µs" set="5.692µs" diff="3.262µs" +DEBUG[02-10|13:55:30.970] Ledger support enabled +DEBUG[02-10|13:55:30.973] Trezor support enabled via HID +DEBUG[02-10|13:55:30.976] Trezor support enabled via WebUSB +INFO [02-10|13:55:30.978] Audit logs configured file=audit.log +DEBUG[02-10|13:55:30.981] IPCs registered namespaces=account +INFO [02-10|13:55:30.984] IPC endpoint opened url=geth-tutorial/clef/clef.ipc +------- Signer info ------- +* intapi_version : 7.0.1 +* extapi_version : 6.1.0 +* extapi_http : n/a +* extapi_ipc : geth-tutorial/clef/clef.ipc +``` + +This result indicates that Clef is running. This terminal should be left running for the duration of this +tutorial. If the tutorial is stopped and restarted later Clef must also be restarted by running the previous command. + +## Step 3: Start Geth + +Geth is the Ethereum client that will connect the computer to the Ethereum network. In this tutorial the +network is Goerli, an Ethereum testnet. Testnets are used to test Ethereum client software and smart +contracts in an environment where no real-world value is at risk. To start Geth, run the Geth executable +file passing argument that define the data directory (where Geth should save blockchain data), +signer (points Geth to Clef), the network ID and the sync mode. For this tutorial, snap sync is recommended +(see [here](https://blog.ethereum.org/2021/03/03/geth-v1-10-0/) for reasons why). The final argument +passed to Geth is the `--http` flag. This enables the http-rpc server that allows external programs to +interact with Geth by sending it http requests. By default the http server is only exposed locally using +port 8545: `localhost:8545`. + +The following command should be run in a new terminal, separate to the one running Clef: + +```shell +geth --datadir geth-tutorial --signer=geth-tutorial/clef/clef.ipc --goerli --syncmode snap --http +``` + +Running the above command starts Geth. The terminal should rapidly fill with status updates, starting with: + +```terminal +INFO [02-10|13:59:06.649] Starting Geth on goerli testnet... +INFO [02-10|13:59:06.649] Dropping default light client cache provided=1024 updated=128 +INFO [02-10|13:59:06.652] Maximum peer count ETH=50 LES=0 total=50 +INFO [02-10|13:59:06.655] Using external signer url=geth-tutorial/clef/clef.ipc +INFO [02-10|13:59:06.660] Set global gas cap cap=50,000,000 +INFO [02-10|13:59:06.661] Allocated cache and file handles database=/.../geth-tutorial/geth/chaindata cache=64.00MiB handles=5120 +INFO [02-10|13:59:06.855] Persisted trie from memory database nodes=361 size=51.17KiB time="643.54µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B +INFO [02-10|13:59:06.855] Initialised chain configuration config="{ChainID: 5 Homestead: 0 DAO: nil DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 1561651, Muir Glacier: nil, Berlin: 4460644, London: 5062605, Arrow Glacier: nil, MergeFork: nil, Engine: clique}" +INFO [02-10|13:59:06.862] Added trusted checkpoint block=5,799,935 hash=2de018..c32427 +INFO [02-10|13:59:06.863] Loaded most recent local header number=6,340,934 hash=483cf5..858315 td=9,321,576 age=2d9h29m +INFO [02-10|13:59:06.867] Configured checkpoint oracle address=0x18CA0E045F0D772a851BC7e48357Bcaab0a0795D signers=5 threshold=2 +INFO [02-10|13:59:06.867] Gasprice oracle is ignoring threshold set threshold=2 +WARN [02-10|13:59:06.869] Unclean shutdown detected booted=2022-02-08T04:25:08+0100 age=2d9h33m +INFO [02-10|13:59:06.870] Starting peer-to-peer node instance=Geth/v1.10.15-stable/darwin-amd64/go1.17.5 +INFO [02-10|13:59:06.995] New local node record seq=1,644,272,735,880 id=d4ffcd252d322a89 ip=127.0.0.1 udp=30303 tcp=30303 +INFO [02-10|13:59:06.996] Started P2P networking self=enode://4b80ebd341b5308f7a6b61d91aa0ea31bd5fc9e0a6a5483e59fd4ea84e0646b13ecd289e31e00821ccedece0bf4b9189c474371af7393093138f546ac23ef93e@127.0.0.1:30303 +INFO [02-10|13:59:06.997] IPC endpoint opened url=/.../geth-tutorial/geth.ipc +INFO [02-10|13:59:06.998] HTTP server started endpoint=127.0.0.1:8545 prefix= cors= vhosts=localhost +WARN [02-10|13:59:06.998] Light client mode is an experimental feature +WARN [02-10|13:59:06.999] Failed to open wallet url=extapi://geth-tutorial/clef/cle.. err="operation not supported on external signers" +INFO [02-10|13:59:08.793] Block synchronisation started +``` + +This indicates that Geth has started up and is searching for peers to connect to. Once it finds peers it +can request block headers from them, starting at the genesis block for the Goerli blockchain. Geth continues +to download blocks sequentially, saving the data in files in `/go-ethereum/geth-tutorial/geth/chaindata/`. +This is confirmed by the logs printed to the terminal. There should be a rapidly-growing sequence of logs in +the terminal with the following syntax: + +```terminal +INFO [04-29][15:54:09.238] Looking for peers peercount=2 tried=0 static=0 +INFO [04-29][15:54:19.393] Imported new block headers count=2 elapsed=1.127ms number=996288 hash=09f1e3..718c47 age=13h9m5s +INFO [04-29][15:54:19:656] Imported new block receipts count=698 elapsed=4.464ms number=994566 hash=56dc44..007c93 age=13h9m9s +``` + +These logs indicate that Geth is running as expected. Sending an empty Curl request to the http server +provides a quick way to confirm that this too has been started without any issues. In a third terminal, +the following command can be run: + +```shell +curl http://localhost:8545 +``` + +If there is no error message reported to the terminal, everything is OK. Geth must be running in order for +a user to interact with the Ethereum network. If this terminal is closed down then Geth must be restarted +in a new terminal. Geth can be started and stopped easily, but it must be running for any interaction with +Ethereum to take place. To shut down Geth, simply press `CTRL+C` in the Geth terminal. To start it again, +run the previous command `geth --datadir ... ..`. + +{% include note.html content="Snap syncing Goerli will take some time and until the sync is finished you +can't use the node to transfer funds. You can also try doing a [light sync](interface/les) which will be +much quicker but depends on light servers being available to serve your node the data it needs." %} + +## Step 4: Get Testnet Ether + +In order to make some transactions, the user must fund their account with ether. On Ethereum mainnet, +ether can only be obtained in three ways: 1) by receiving it as a reward for mining/validating; +2) receiving it in a transfer from another Ethereum user or contract; 3) receiving it from an exchange, +3) having paid for it with fiat money. On Ethereum testnets, the ether has no real world value so it +4) can be made freely available via faucets. Faucets allow users to request a transfer of testnet ether to their account. + +The address generated by Clef in Step 1 can be pasted into the Paradigm Multifaucet faucet +[here](https://fauceth.komputing.org/?chain=1115511). This requires a Twitter login as proof of +personhood. The faucets adds ether to the given address on multiple testnets simultaneously, +including Goerli. In the next steps Geth will be used to check that the ether has been sent +to the given address and send some of it to the second address created earlier. + + +## Step 5: Interact with Geth + +For interacting with the blockchain, Geth provides JSON-RPC APIs. +[JSON-RPC](https://ethereum.org/en/developers/docs/apis/json-rpc/) is a way to execute specific +tasks by sending instructions to Geth in the form of [JSON](https://www.json.org/json-en.html) objects. +RPC stands for "Remote Procedure Call" and it refers to the ability to send these JSON-encoded +instructions from locations outside of those managed by Geth. It is possible to interact with Geth +by sending these JSON encoded instructions directly over Geth's exposed http port using tools like Curl. +However, this is somewhat user-unfriendly and error-prone, especially for more complex instructions. +For this reason, there are a set of libraries built on top of JSON-RPC that provide a more user-friendly +interface for interacting with Geth. One of the most widely used is Web3.js. + +Geth provides a Javascript console that exposes the Web3.js API. This means that with Geth running in one terminal, a Javascript environment can be opened in another allowing the user to interact with Geth using Web3.js. There are three transport protocols that can be used to connect the Javascript environment to Geth: + +- IPC (Inter-Process Communication): Provides unrestricted access to all APIs, but only works when the console is run on the same host as the geth node. + +- HTTP: By default provides access to the `eth`, `web3` and `net` method namespaces. + +- Websocket: By default provides access to the `eth`, `web3` and `net` method namespaces. + +This tutorial will use the HTTP option. Note that the terminals running Geth and Clef should both still be active. +In a new (third) terminal, the following command can be run to start the console and connect it to Geth +using the exposed http port: + +```shell +geth attach http://127.0.0.1:8545 +``` + +This command causes the terminal to hang because it is waiting for approval from Clef. Approving the request +in the terminal running Clef will lead to the following welcome message being displayed in the Javascript console: + +```terminal +Welcome to the Geth JavaScript console! + +instance: Geth/v1.10.15-stable/darwin-amd64/go1.17.5 +at block: 6354736 (Thu Feb 10 2022 14:01:46 GMT+0100 (WAT)) + modules: eth:1.0 net:1.0 rpc:1.0 web3:1.0 + +To exit, press ctrl-d or type exit +``` + +The console is now active and connected to Geth. It can now be used to interact with the Ethereum (Goerli) network. + + +### List of accounts + +In this tutorial, the accounts are managed using Clef. This means that requesting information about +the accounts requires explicit approval in Clef, which should still be running in its own terminal. +Earlier in this tutorial, two accounts were created using Clef. The following command will display the +addresses of those two accounts and any others that might have been added to the keystore before or since. + +```javascript +eth.accounts +``` + +The console will hang, because Clef is waiting for approval. The following message will be +displayed in the Clef terminal: + +```terminal +-------- List Account request-------------- +A request has been made to list all accounts. +You can select which accounts the caller can see + [x] 0xca57F3b40B42FCce3c37B8D18aDBca5260ca72EC + URL: keystore:///.../geth-tutorial/keystore/UTC--2022-02-07T17-19-56.517538000Z--ca57f3b40b42fcce3c37b8d18adbca5260ca72ec + [x] 0xCe8dBA5e4157c2B284d8853afEEea259344C1653 + URL: keystore:///.../geth-tutorial/keystore/UTC--2022-02-10T12-46-45.265592000Z--ce8dba5e4157c2b284d8853afeeea259344c1653 +------------------------------------------- +Request context: + NA - ipc - NA + +Additional HTTP header data, provided by the external caller: + User-Agent: "" + Origin: "" +Approve? [y/N]: + +``` + +Entering `y` approves the request from the console. In the terminal running the Javascript console, +the account addresses are now displayed: + +```terminal +["0xca57f3b40b42fcce3c37b8d18adbca5260ca72ec", "0xce8dba5e4157c2b284d8853afeeea259344c1653"] +``` + +It is also possible for this request to time out if the Clef approval took too long - +in this case simply repeat the request and approval. + + +### Checking account balance. + +Having confirmed that the two addresses created earlier are indeed in the keystore and accessible +through the Javascript console, it is possible to retrieve information about how much ether they own. +The Goerli faucet should have sent 1 ETH to the address provided, meaning that the balance of one of +the accounts should be 1 ether and the other should be 0. The following command displays the account +balance in the console: + +```javascript +web3.fromWei(eth.getBalance("0xca57F3b40B42FCce3c37B8D18aDBca5260ca72EC"), "ether") +``` + +There are actually two instructions sent in the above command. The inner one is the `getBalance` +function from the `eth` namespace. This takes the account address as its only argument. By default, +this returns the account balance in units of Wei. There are 1018 Wei to one ether. +To present the result in units of ether, `getBalance` is wrapped in the `fromWei` function from +the `web3` namespace. Running this command should provide the following result +(for the account that received faucet funds): + +```terminal +1 +``` + +Repeating the command for the other account should yield: + +```terminal +0 +``` + +### Send ether to another account + +The command `eth.sendTransaction` can be used to send some ether from one address to another. +This command takes three arguments: `from`, `to` and `value`. These define the sender and +recipient addresses (as strings) and the amount of Wei to transfer. It is far less error +prone to enter the transaction value in units of ether rather than Wei, so the value field can +take the return value from the `toWei` function. The following command, run in the Javascript +console, sends 0.1 ether from one of the accounts in the Clef keystore to the other. Note that +the addresses here are examples - the user must replace the address in the `from` field with +the address currently owning 1 ether, and the address in the `to` field with the address +currently holding 0 ether. + +```javascript +eth.sendTransaction({ + from: "0xca57f3b40b42fcce3c37b8d18adbca5260ca72ec", + to: "0xce8dba5e4157c2b284d8853afeeea259344c1653", + value: web3.toWei(0.1, "ether") +}) +``` + +Note that submitting this transaction requires approval in Clef. In the Clef terminal, +Clef will prompt for approval and request the account password. If the password is correctly +entered, Geth proceeds with the transaction. The transaction request summary is presented by +Clef in the Clef terminal. This is an opportunity for the sender to review the details and +ensure they are correct. + +```terminal +--------- Transaction request------------- +to: 0xCe8dBA5e4157c2B284d8853afEEea259344C1653 +from: 0xca57F3b40B42FCce3c37B8D18aDBca5260ca72EC [chksum ok] +value: 10000000000000000 wei +gas: 0x5208 (21000) +maxFeePerGas: 2425000057 wei +maxPriorityFeePerGas: 2424999967 wei +nonce: 0x3 (3) +chainid: 0x5 +Accesslist + +Request context: + NA - ipc - NA + +Additional HTTP header data, provided by the external caller: + User-Agent: "" + Origin: "" +------------------------------------------- +Approve? [y/N]: + +Please enter the password for account 0xca57F3b40B42FCce3c37B8D18aDBca5260ca72EC + +``` + +After approving the transaction, the following confirmation screen in displayed in +the Clef terminal: + +```terminal +----------------------- +Transaction signed: + { + "type": "0x2", + "nonce": "0x3", + "gasPrice": null, + "maxPriorityFeePerGas": "0x908a901f", + "maxFeePerGas": "0x908a9079", + "gas": "0x5208", + "value": "0x2386f26fc10000", + "input": "0x", + "v": "0x0", + "r": "0x66e5d23ad156e04363e68b986d3a09e879f7fe6c84993cef800bc3b7ba8af072", + "s": "0x647ff82be943ea4738600c831c4a19879f212eb77e32896c05055174045da1bc", + "to": "0xce8dba5e4157c2b284d8853afeeea259344c1653", + "chainId": "0x5", + "accessList": [], + "hash": "0x99d489d0bd984915fd370b307c2d39320860950666aac3f261921113ae4f95bb" + } +``` + +In the Javascript console, the transaction hash is displayed. This will be used in the +next section to retrieve the transaction details. + +```terminal +"0x99d489d0bd984915fd370b307c2d39320860950666aac3f261921113ae4f95bb" +``` + +It is also advised to check the account balances using Geth by repeating the instructions from +earlier. At this point in the tutorial, the two accounts in the Clef keystore should have balances +just below 0.9 ether (because 0.1 ether has been transferred out and some small amount paid in +transaction gas) and 0.1 ether. + + +### Checking the transaction hash + +The transaction hash is a unique identifier for this specific transaction that can be used +later to retrieve the transaction details. For example, the transaction details can be +viewed by pasting this hash into the [Goerli block explorer](https://goerli.etherscan.io/). +The same information can also be retrieved directly from the Geth node. The hash returned in +the previous step can be provided as an argument to `eth.getTransaction` to return the +transaction information: + +```javascript +eth.getTransaction("0x99d489d0bd984915fd370b307c2d39320860950666aac3f261921113ae4f95bb") +``` + +This returns the following response (although the actual values for each field will vary +because they are specific to each transaction): + +```terminal +{ + accessList: [], + blockHash: "0x1c5d3f8dd997b302935391b57dc3e4fffd1fa2088ef2836d51f844f993eb39c4", + blockNumber: 6355150, + chainId: "0x5", + from: "0xca57f3b40b42fcce3c37b8d18adbca5260ca72ec", + gas: 21000, + gasPrice: 2425000023, + hash: "0x99d489d0bd984915fd370b307c2d39320860950666aac3f261921113ae4f95bb", + input: "0x", + maxFeePerGas: 2425000057, + maxPriorityFeePerGas: 2424999967, + nonce: 3, + r: "0x66e5d23ad156e04363e68b986d3a09e879f7fe6c84993cef800bc3b7ba8af072", + s: "0x647ff82be943ea4738600c831c4a19879f212eb77e32896c05055174045da1bc", + to: "0xce8dba5e4157c2b284d8853afeeea259344c1653", + transactionIndex: 630, + type: "0x2", + v: "0x0", + value: 10000000000000000 +} +``` + +## Using Curl + +Up to this point this tutorial has interacted with Geth using the convenience library Web3.js. +This library enables the user to send instructions to Geth using a more user-friendly interface +compared to sending raw JSON objects. However, it is also possible for the user to send these +JSON objects directly to Geth's exposed HTTP port. Curl is a command line tool that sends HTTP +requests. This part of the tutorial demonstrates how to check account balances and send a +transaction using Curl. + +### Checking account balance + +The command below returns the balance of the given account. This is a HTTP POST request to the +local port 8545. The `-H` flag is for header information. It is used here to define the format +of the incoming payload, which is JSON. The `--data` flag defines the content of the payload, +which is a JSON object. That JSON object contains four fields: `jsonrpc` defines the spec version +for the JSON-RPC API, `method` is the specific function being invoked, `params` are the function +arguments, and `id` is used for ordering transactions. The two arguments passed to `eth_getBalance` +are the account address whose balance to check and the block to query (here `latest` is used to +check the balance in the most recently mined block). + +```shell +curl -X POST http://127.0.0.1:8545 \ + -H "Content-Type: application/json" \ + --data '{"jsonrpc":"2.0", "method":"eth_getBalance", "params":["0xca57f3b40b42fcce3c37b8d18adbca5260ca72ec","latest"], "id":1}' +``` + +A successful call will return a response like the one below: + +```terminal +{"jsonrpc":"2.0","id":1,"result":"0xc7d54951f87f7c0"} +``` + +The balance is in the `result` field in the returned JSON object. However, it is denominated in +Wei and presented as a hexadecimal string. There are many options for converting this value to a +decimal in units of ether, for example by opening a Python console and running: + +```python +0xc7d54951f87f7c0 / 1e18 +``` +This returns the balance in ether: + +```terminal +0.8999684999998321 +``` + +### Checking the account list + +The curl command below returns the list of all accounts. + +```shell +curl -X POST http://127.0.0.1:8545 \ + -H "Content-Type: application/json" \ + --data '{"jsonrpc":"2.0", "method":"eth_accounts","params":[], "id":1}' +``` + +This requires approval in Clef. Once approved, the following information is returned to the terminal: + +```terminal +{"jsonrpc":"2.0","id":1,"result":["0xca57f3b40b42fcce3c37b8d18adbca5260ca72ec"]} +``` + +### Sending Transactions + +Sending a transaction between accounts can also be achieved using Curl. Notice that the value of the +transaction is a hexadecimal string in units of Wei. To transfer 0.1 ether, it is first necessary to +convert this to Wei by multiplying by 1018 then converting to hex. 0.1 ether is +`"0x16345785d8a0000"` in hex. As before, update the `to` and `from` fields with the addresses in +the Clef keystore. + + +```shell +curl -X POST http://127.0.0.1:8545 \ + -H "Content-Type: application/json" \ + --data '{"jsonrpc":"2.0", "method":"eth_sendTransaction", "params":[{"from": "0xca57f3b40b42fcce3c37b8d18adbca5260ca72ec","to": "0xce8dba5e4157c2b284d8853afeeea259344c1653","value": "0x16345785d8a0000"}], "id":1}' +``` + +This requires approval in Clef. Once the password for the sender account has been provided, +Clef will return a summary of the transaction details and the terminal that made the Curl +request will display a response containing the transaction hash. + +```terminal +{"jsonrpc":"2.0","id":5,"result":"0xac8b347d70a82805edb85fc136fc2c4e77d31677c2f9e4e7950e0342f0dc7e7c"} +``` + +## Summary + +This tutorial has demonstrated how to generate accounts using Clef, fund them with testnet ether and use +those accounts to interact with Ethereum (Goerli) through a Geth node. Checking account balances, sending +transactions and retrieving transaction details were explained using the web3.js library via the +Geth console and using the JSON-RPC directly using Curl. For more detailed information about Clef, please see +[the Clef docs](/docs/clef/tutorial). + + +[cli]: https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Understanding_client-side_tools/Command_line diff --git a/content/docs/getting_started/index.md b/content/docs/getting_started/index.md index 86b3f3a2ca..448e8eb5aa 100644 --- a/content/docs/getting_started/index.md +++ b/content/docs/getting_started/index.md @@ -4,134 +4,138 @@ permalink: docs/getting-started sort_key: A --- -This page explains how to set up Geth and execute some basic tasks using the command line tools. In order to use Geth, the software must first be installed. There are several ways Geth can be installed depending on the operating system and the user's choice of installation method, for example using a package manager, container or building from source. Instructions for installing Geth can be found on the ["Install and Build"](install-and-build/installing-geth) pages. The tutorial on this page assumes Geth and the associated developer tools have been installed successfully. - -This page provides step-by-step instructions covering the fundamentals of using Geth. This includes generating accounts, joining an Ethereum network, syncing the blockchain and sending ether between accounts. This tutorial also uses [Clef](clef/tutorial). Clef is an account management tool external to Geth itself that allows users to sign transactions. It is developed and maintained by the Geth team and is intended to eventually replace the account management tool built in to Geth. +This page explains how to set up Geth and execute some basic tasks using the command line tools. +In order to use Geth, the software must first be installed. There are several ways Geth can be +installed depending on the operating system and the user's choice of installation method, for +example using a package manager, container or building from source. Instructions for installing +Geth can be found on the ["Install and Build"](install-and-build/installing-geth) pages. +The tutorial on this page assumes Geth and the associated developer tools have been installed successfully. + +This page provides step-by-step instructions covering the fundamentals of using Geth. This includes +generating accounts, joining an Ethereum network, syncing the blockchain and sending ether between accounts. +It is considered best-practice to use [Clef](/docs/clef/Introduction) for account management - this +is explained in the [Geth with Clef](/docs/getting-started/geth_with_clef) tutorial. In this +introductory tutorial, Geth's built-in account management tools are used instead. + +{:toc} +- this will be removed by the toc ## Prerequisites -In order to get the most value from the tutorials on this page, the following skills are necessary: +In order to get the most value from the tutorials on this page, the following skills are +necessary: - Experience using the command line - Basic knowledge about Ethereum and testnets - Basic knowledge about HTTP and JavaScript -Users that need to revisit these fundamentals can find helpful resources relating to the command line [here](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Understanding_client-side_tools/Command_line), Ethereum and its testnets [here](https://ethereum.org/en/developers/tutorials/), http [here](https://developer.mozilla.org/en-US/docs/Web/HTTP) and Javascript [here](https://www.javascript.com/learn). +Users that need to revisit these fundamentals can find helpful resources relating to the command +line [here][cli], Ethereum and its testnets [here](https://ethereum.org/en/developers/tutorials/), +http [here](https://developer.mozilla.org/en-US/docs/Web/HTTP) and +Javascript [here](https://www.javascript.com/learn). -{% include note.html content="If Geth was installed from source on Linux, `make` saves the binaries for Geth and the associated tools in `/build/bin`. To run these programs it is convenient to move them to the top level project directory (e.g. running `mv ./build/bin/* ./`) from `/go-ethereum`. Then `./` must be prepended to the commands in the code snippets in order to execute a particular program, e.g. `./geth` instead of simply `geth`. If the executables are not moved then either navigate to the `bin` directory to run them (e.g. `cd ./build/bin` and `./geth`) or provide their path (e.g. `./build/bin/geth`). These instructions can be ignored for other installations." %} +{% include note.html content="If Geth was installed from source on Linux, `make` saves the +binaries for Geth and the associated tools in `/build/bin`. To run these programs it is +convenient to move them to the top level project directory (e.g. running `mv ./build/bin/* ./`) +from `/go-ethereum`. Then `./` must be prepended to the commands in the code snippets in order to +execute a particular program, e.g. `./geth` instead of simply `geth`. If the executables are not +moved then either navigate to the `bin` directory to run them (e.g. `cd ./build/bin` and `./geth`) +or provide their path (e.g. `./build/bin/geth`). These instructions can be ignored for other installations." %} ## Background -Geth is an Ethereum client written in Go. This means running Geth turns a computer into an Ethereum node. Ethereum is a peer-to-peer network where information is shared directly between nodes rather than being managed by a central server. Nodes compete to generate new blocks of transactions to send to its peers because they are rewarded for doing so in Ethereum's native token, ether (ETH). On receiving a new block, each node checks that it is valid and adds it to their database. The sequence of discrete blocks is called a "blockchain". The information provided in each block is used by Geth to update its "state" - the ether balance of each account on Ethereum. There are two types of account: externally-owned accounts (EOAs) and contract accounts. Contract accounts execute contract code when they receive transactions. EOAs are accounts that users manage locally in order to sign and submit transactions. Each EOA is a public-private key pair, where the public key is used to derive a unique address for the user and the private key is used to protect the account and securely sign messages. Therefore, in order to use Ethereum, it is first necessary to generate an EOA (hereafter, "account"). This tutorial will guide the user through creating an account, funding it with ether and sending some to another address. +Geth is an Ethereum client written in Go. This means running Geth turns a computer into an Ethereum node. +Ethereum is a peer-to-peer network where information is shared directly between nodes rather than being +managed by a central server. Nodes compete to generate new blocks of transactions to send to its peers +because they are rewarded for doing so in Ethereum's native token, ether (ETH). On receiving a new block, +each node checks that it is valid and adds it to their database. The sequence of discrete blocks is called +a "blockchain". The information provided in each block is used by Geth to update its "state" - the ether +balance of each account on Ethereum. There are two types of account: externally-owned accounts (EOAs) and +contract accounts. Contract accounts execute contract code when they receive transactions. EOAs are accounts +that users manage locally in order to sign and submit transactions. Each EOA is a public-private key pair, +where the public key is used to derive a unique address for the user and the private key is used to protect +the account and securely sign messages. Therefore, in order to use Ethereum, it is first necessary to generate +an EOA (hereafter, "account"). This tutorial will guide the user through creating an account, funding it +with ether and sending some to another address. Read more about Ethereum accounts [here](https://ethereum.org/en/developers/docs/accounts/). ## Step 1: Generating accounts -There are several methods for generating accounts in Geth. This tutorial demonstrates how to generate accounts using Clef, as this is considered best practice, largely because it decouples the users' key management from Geth, making it more modular and flexible. It can also be run from secure USB sticks or virtual machines, offering security benefits. For convenience, this tutorial will execute Clef on the same computer that will also run Geth, although more secure options are available (see [here](https://github.com/ethereum/go-ethereum/blob/master/cmd/clef/docs/setup.md)). - -An account is a pair of keys (public and private). Clef needs to know where to save these keys to so that they can be retrieved later. This information is passed to Clef as an argument. This is achieved using the following command: - -```shell - -clef newaccount --keystore geth-tutorial/keystore +To generate a new account in Geth: +```sh +geth account new ``` -The specific function from Clef that generates new accounts is `newaccount` and it accepts a parameter, `--keystore`, that tells it where to store the newly generated keys. In this example the keystore location is a new directory that will be created automatically: `geth-tutorial/keystore`. Clef will return the following result in the terminal: - -```terminal -WARNING! +This returns a prompt for a password. Once provided, a new account will be created and added to the +default keystore (`/datadir/keystore`). A custom keystore can also be provided by passing `--keystore `. +In this tutorial the keys will be stored in a new data directory `geth-tutorial`. Create that diredctory, then run: -Clef is an account management tool. It may, like any software, contain bugs. - -Please take care to -- backup your keystore files, -- verify that the keystore(s) can be opened with your password. - -Clef is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; -without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR -PURPOSE. See the GNU General Public License for more details. - -Enter 'ok' to proceed: -> +```sh +geth account new --keystore geth-tutorial/keystore ``` - -This is important information. The `geth-tutorial/keystore` directory will soon contain a secret key that can be used to access any funds held in the new account. If it is compromised, the funds can be stolen. If it is lost, there is no way to retrieve the funds. This tutorial will only use dummy funds with no real world value, but when these steps are repeated on Ethereum mainnet is critical that the keystore is kept secure and backed up. - - -Typing `ok` into the terminal and pressing `enter` causes Clef to prompt for a password. Clef requires a password that is at least 10 characters long, and best practice would be to use a combination of numbers, characters and special characters. Entering a suitable password and pressing `enter` returns the following result to the terminal: +The following will be returned to the console, confirming the new account has been created and +added to the keystore. ```terminal ------------------------ -DEBUG[02-10|13:46:46.436] FS scan times list="92.081µs" set="12.629µs" diff="2.129µs" -INFO [02-10|13:46:46.592] Your new key was generated address=0xCe8dBA5e4157c2B284d8853afEEea259344C1653 -WARN [02-10|13:46:46.595] Please backup your key file! path=keystore:///.../geth-tutorial/keystore/UTC--2022-02-07T17-19-56.517538000Z--ca57f3b40b42fcce3c37b8d18adbca5260ca72ec -WARN [02-10|13:46:46.595] Please remember your password! -Generated account 0xCe8dBA5e4157c2B284d8853afEEea259344C1653 -``` - -It is important to save the account address and the password somewhere secure. They will be used again later in this tutorial. Please note that the account address shown in the code snippets above and later in this tutorials are examples - those generated by followers of this tutorial will be different. The account generated above can be used as the main account throughout the remainder of this tutorial. However in order to demonstrate transactions between accounts it is also necessary to have a second account. A second account can be added to the same keystore by precisely repeating the previous steps, providing the same password. +Your new account is locked with a password. Please give a password. Do not forget this password. +Password: +Repeat password: -## Step 2: Start Clef +Your new key was generated -The previous commands used Clef's `newaccount` function to add new key pairs to the keystore. Clef uses the private key(s) saved in the keystore is used to sign transactions. In order to do this, Clef needs to be started and left running while Geth is running simultaneously, so that the two programs can communicate between one another. - -To start Clef, run the Clef executable passing as arguments the keystore file location, config directory location and a chain ID. The config directory was automatically created inside the `geth-tutorial` directory during the previous step. The [chain ID](https://chainlist.org/) is an integer that defines which Ethereum network to connect to. Ethereum mainnet has chain ID 1. In this tutorial Chain ID 5 is used which is that of the Goerli testnet. It is very important that this chain ID parameter is set to 5. The following command starts Clef on Goerli: - -```shell - -clef --keystore geth-tutorial/keystore --configdir geth-tutorial/clef --chainid 5 +Public address of the key: 0xca57F3b40B42FCce3c37B8D18aDBca5260ca72EC +Path of the secret key file: /home/go-ethereum/geth-tutorial/keystore/UTC--2022-07-25T08-27-59.433905560Z--ca57F3b40B42FCce3c37B8D18aDBca5260ca72EC +- You can share your public address with anyone. Others need it to interact with you. +- You must NEVER share the secret key with anyone! The key controls access to your funds! +- You must BACKUP your key file! Without the key, it's impossible to access account funds! +- You must REMEMBER your password! Without the password, it's impossible to decrypt the key! ``` -After running the command above, Clef requests the user to type “ok” to proceed. On typing "ok" and pressing enter, Clef returns the following to the terminal: - -```terminal -INFO [02-10|13:55:30.812] Using CLI as UI-channel -INFO [02-10|13:55:30.946] Loaded 4byte database embeds=146,841 locals=0 local=./4byte-custom.json -WARN [02-10|13:55:30.947] Failed to open master, rules disabled err="failed stat on geth-tutorial/clef/masterseed.json: stat geth-tutorial/clef/masterseed.json: no such file or directory" -INFO [02-10|13:55:30.947] Starting signer chainid=5 keystore=geth-tutorial/keystore light-kdf=false advanced=false -DEBUG[02-10|13:55:30.948] FS scan times list="133.35µs" set="5.692µs" diff="3.262µs" -DEBUG[02-10|13:55:30.970] Ledger support enabled -DEBUG[02-10|13:55:30.973] Trezor support enabled via HID -DEBUG[02-10|13:55:30.976] Trezor support enabled via WebUSB -INFO [02-10|13:55:30.978] Audit logs configured file=audit.log -DEBUG[02-10|13:55:30.981] IPCs registered namespaces=account -INFO [02-10|13:55:30.984] IPC endpoint opened url=geth-tutorial/clef/clef.ipc -------- Signer info ------- -* intapi_version : 7.0.1 -* extapi_version : 6.1.0 -* extapi_http : n/a -* extapi_ipc : geth-tutorial/clef/clef.ipc -``` - -This result indicates that Clef is running. This terminal should be left running for the duration of this tutorial. If the tutorial is stopped and restarted later Clef must also be restarted by running the previous command. - -## Step 3: Start Geth - -Geth is the Ethereum client that will connect the computer to the Ethereum network. In this tutorial the network is Goerli, an Ethereum testnet. Testnets are used to test Ethereum client software and smart contracts in an environment where no real-world value is at risk. To start Geth, run the Geth executable file passing argument that define the data directory (where Geth should save blockchain data), signer (points Geth to Clef), the network ID and the sync mode. For this tutorial, snap sync is recommended (see [here](https://blog.ethereum.org/2021/03/03/geth-v1-10-0/) for reasons why). The final argument passed to Geth is the `--http` flag. This enables the http-rpc server that allows external programs to interact with Geth by sending it http requests. By default the http server is only exposed locally using port 8545: `localhost:8545`. - -The following command should be run in a new terminal, separate to the one running Clef: +It is important to save the account address and the password somewhere secure. They will be used +again later in this tutorial. Please note that the account address shown in the code snippets +above and later in this tutorials are examples - those generated by followers of this tutorial +will be different. The account generated above can be used as the main account throughout the +remainder of this tutorial. However in order to demonstrate transactions between accounts it is +also necessary to have a second account. A second account can be added to the same keystore by +precisely repeating the previous steps, providing the same password. + +Notice that the path to the secret key includes a long filename that starts `UTC--`. This is the +name of the file that contains the keys for the new account. It is **extremely important** that +this file stays secure because it contains the secret key used to control access to any funds +associated with the account. The file should be backed up securely along with the password +used to encrypt it. If the file or the password is lost, then so is access to the funds in +the account. If someone else gains access to the keyfile and password, they have access to any +assets in the account. + +## Step 2: Start Geth + +Geth is the Ethereum client that will connect the computer to the Ethereum network. +In this tutorial the network is Goerli, an Ethereum testnet. Testnets are used to test +Ethereum client software and smart contracts in an environment where no real-world value +is at risk. To start Geth, run the Geth executable file passing argument that define the +data directory (where Geth should save blockchain data), the network ID and the sync mode. +For this tutorial, snap sync is recommended +(see [here](https://blog.ethereum.org/2021/03/03/geth-v1-10-0/) for reasons why). + +The following command should be run in the terminal: ```shell - -geth --datadir geth-tutorial --signer=geth-tutorial/clef/clef.ipc --goerli --syncmode snap --http - +geth --datadir geth-tutorial --goerli --syncmode snap ``` - -Running the above command starts Geth. The terminal should rapidly fill with status updates, starting with: +Running the above command starts Geth. The terminal should rapidly fill with status updates that look like the following: ```terminal INFO [02-10|13:59:06.649] Starting Geth on goerli testnet... INFO [02-10|13:59:06.649] Dropping default light client cache provided=1024 updated=128 INFO [02-10|13:59:06.652] Maximum peer count ETH=50 LES=0 total=50 -INFO [02-10|13:59:06.655] Using external signer url=geth-tutorial/clef/clef.ipc INFO [02-10|13:59:06.660] Set global gas cap cap=50,000,000 INFO [02-10|13:59:06.661] Allocated cache and file handles database=/.../geth-tutorial/geth/chaindata cache=64.00MiB handles=5120 INFO [02-10|13:59:06.855] Persisted trie from memory database nodes=361 size=51.17KiB time="643.54µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B -INFO [02-10|13:59:06.855] Initialised chain configuration config="{ChainID: 5 Homestead: 0 DAO: DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 1561651, Muir Glacier: , Berlin: 4460644, London: 5062605, Arrow Glacier: , MergeFork: , Engine: clique}" +INFO [02-10|13:59:06.855] Initialised chain configuration config="{ChainID: 5 Homestead: 0 DAO: nil DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 1561651, Muir Glacier: nil, Berlin: 4460644, London: 5062605, Arrow Glacier: nil, MergeFork: nil, Engine: clique}" INFO [02-10|13:59:06.862] Added trusted checkpoint block=5,799,935 hash=2de018..c32427 INFO [02-10|13:59:06.863] Loaded most recent local header number=6,340,934 hash=483cf5..858315 td=9,321,576 age=2d9h29m INFO [02-10|13:59:06.867] Configured checkpoint oracle address=0x18CA0E045F0D772a851BC7e48357Bcaab0a0795D signers=5 threshold=2 @@ -141,13 +145,15 @@ INFO [02-10|13:59:06.870] Starting peer-to-peer node instance=Geth INFO [02-10|13:59:06.995] New local node record seq=1,644,272,735,880 id=d4ffcd252d322a89 ip=127.0.0.1 udp=30303 tcp=30303 INFO [02-10|13:59:06.996] Started P2P networking self=enode://4b80ebd341b5308f7a6b61d91aa0ea31bd5fc9e0a6a5483e59fd4ea84e0646b13ecd289e31e00821ccedece0bf4b9189c474371af7393093138f546ac23ef93e@127.0.0.1:30303 INFO [02-10|13:59:06.997] IPC endpoint opened url=/.../geth-tutorial/geth.ipc -INFO [02-10|13:59:06.998] HTTP server started endpoint=127.0.0.1:8545 prefix= cors= vhosts=localhost WARN [02-10|13:59:06.998] Light client mode is an experimental feature -WARN [02-10|13:59:06.999] Failed to open wallet url=extapi://geth-tutorial/clef/cle.. err="operation not supported on external signers" INFO [02-10|13:59:08.793] Block synchronisation started ``` -This indicates that Geth has started up and is searching for peers to connect to. Once it finds peers it can request block headers from them, starting at the genesis block for the Goerli blockchain. Geth continues to download blocks sequentially, saving the data in files in `/go-ethereum/geth-tutorial/geth/chaindata/`. This is confirmed by the logs printed to the terminal. There should be a rapidly-growing sequence of logs in the terminal with the following syntax: +This indicates that Geth has started up and is searching for peers to connect to. Once it finds peers +it can request block headers from them, starting at the genesis block for the Goerli blockchain. +Geth continues to download blocks sequentially, saving the data in files in `/go-ethereum/geth-tutorial/geth/chaindata/`. +This is confirmed by the logs printed to the terminal. There should be a rapidly-growing sequence of logs in the +terminal with the following syntax: ```terminal @@ -157,51 +163,76 @@ INFO [04-29][15:54:19:656] Imported new block receipts count=698 elapsed=4.46 ``` -These logs indicate that Geth is running as expected. Sending an empty Curl request to the http server provides a quick way to confirm that this too has been started without any issues. In a third terminal, the following command can be run: - -```shell - -curl http://localhost:8545 - -``` - -If there is no error message reported to the terminal, everything is OK. Geth must be running in order for a user to interact with the Ethereum network. If this terminal is closed down then Geth must be restarted in a new terminal. Geth can be started and stopped easily, but it must be running for any interaction with Ethereum to take place. To shut down Geth, simply press `CTRL+C` in the Geth terminal. To start it again, run the previous command `geth --datadir ... ..`. - -{% include note.html content="Snap syncing Goerli will take some time and until the sync is finished you can't use the node to transfer funds. You can also try doing a [light sync](interface/les) which will be much quicker but depends on light servers being available to serve your node the data it needs." %} - -## Step 4: Get Testnet Ether - -In order to make some transactions, the user must fund their account with ether. On Ethereum mainnet, ether can only be obtained in three ways: 1) by receiving it as a reward for mining/validating; 2) receiving it in a transfer from another Ethereum user or contract; 3) receiving it from an exchange, having paid for it with fiat money. On Ethereum testnets, the ether has no real world value so it can be made freely available via faucets. Faucets allow users to request a transfer of testnet ether to their account. - -The address generated by Clef in Step 1 can be pasted into the Paradigm Multifaucet faucet [here](https://fauceth.komputing.org/?chain=1115511). This requires a Twitter login as proof of personhood. The faucets adds ether to the given address on multiple testnets simultaneously, including Goerli. In the next steps Geth will be used to check that the ether has been sent to the given address and send some of it to the second address created earlier. - - -## Step 5: Interact with Geth via IPC or RPC - -For interacting with the blockchain, Geth provides JSON-RPC APIs. [JSON-RPC](https://ethereum.org/en/developers/docs/apis/json-rpc/) is a way to execute specific tasks by sending instructions to Geth in the form of [JSON](https://www.json.org/json-en.html) objects. RPC stands for "Remote Procedure Call" and it refers to the ability to send these JSON-encoded instructions from locations outside of those managed by Geth. It is possible to interact with Geth by sending these JSON encoded instructions directly over Geth's exposed http port using tools like Curl. However, this is somewhat user-unfriendly and error-prone, especially for more complex instructions. For this reason, there are a set of libraries built on top of JSON-RPC that provide a more user-friendly interface for interacting with Geth. One of the most widely used is Web3.js. - -Geth provides a Javascript console that exposes the Web3.js API. This means that with Geth running in one terminal, a Javascript environment can be opened in another allowing the user to interact with Geth using Web3.js. There are two transport protocols that can be used to connect the Javascript environment to Geth: - -- IPC (Inter-Process Communication): This provides unrestricted access to all APIs, but only works when the console is run on the same host as the geth node. +These logs indicate that Geth is running as expected. + +If there is no error message reported to the terminal, everything is OK. Geth must be running in +order for a user to interact with the Ethereum network. If this terminal is closed down then Geth +must be restarted again. Geth can be started and stopped easily, but it must be running for any +interaction with Ethereum to take place. To shut down Geth, simply press `CTRL+C` in the Geth terminal. +To start it again, run the previous command `geth --datadir ... ..`. + +{% include note.html content="Snap syncing Goerli will take some time and until the sync is finished +you can't use the node to transfer funds. You can also try doing a [light sync](interface/les) +which will be much quicker but depends on light servers being available to serve your node the data it needs." %} + +## Step 3: Get Testnet Ether + +In order to make some transactions, the user must fund their account with ether. On Ethereum mainnet, +ether can only be obtained in three ways: 1) by receiving it as a reward for mining/validating; 2) +receiving it in a transfer from another Ethereum user or contract; 3) receiving it from an exchange, +having paid for it with fiat money. On Ethereum testnets, the ether has no real world value so it +can be made freely available via faucets. Faucets allow users to request a transfer of testnet ether +to their account. + +The address generated by `geth account new` can be pasted into the Paradigm Multifaucet faucet +[here](https://fauceth.komputing.org/?chain=1115511). This requires a Twitter login as proof of +personhood. The faucets adds ether to the given address on multiple testnets simultaneously, +including Goerli. In the next steps Geth will be used to check that the ether has been sent to +the given address and send some of it to the second address created earlier. + + +## Step 4: Interact with Geth + +For interacting with the blockchain, Geth provides JSON-RPC APIs. +[JSON-RPC](https://ethereum.org/en/developers/docs/apis/json-rpc/) is a way to execute specific tasks +by sending instructions to Geth in the form of [JSON](https://www.json.org/json-en.html) objects. +RPC stands for "Remote Procedure Call" and it refers to the ability to send these JSON-encoded +instructions from locations outside of those managed by Geth. It is possible to interact with Geth +by sending these JSON encoded instructions directly to Geth using tools such as Curl. However, +this is somewhat user-unfriendly and error-prone, especially for more complex instructions. For this +reason, there are a set of libraries built on top of JSON-RPC that provide a more user-friendly +interface for interacting with Geth. One of the most widely used is Web3.js. + +Geth provides a Javascript console that exposes the Web3.js API. This means that with Geth running in +one terminal, a Javascript environment can be opened in another allowing the user to interact with +Geth using Web3.js. There are three transport protocols that can be used to connect the Javascript +environment to Geth: + +- IPC (Inter-Process Communication): Provides unrestricted access to all APIs, but only works when the +- console is run on the same host as the Geth node. -- HTTP: This connection method by default provides access to the `eth`, `web3` and `net` method namespaces. - -This tutorial will use the HTTP option. Note that the terminals running Geth and Clef should both still be active. In a new (third) terminal, the following command can be run to start the console and connect it to Geth using the exposed http port: +- HTTP: By default provides access to the `eth`, `web3` and `net` method namespaces. -```shell +- Websocket: By default provides access to the `eth`, `web3` and `net` method namespaces. -geth attach http://127.0.0.1:8545 +This tutorial will use the IPC option. To do this, the path to Geth's `ipc` file must be known. +By default, this is the `datadir`, in this case `geth-tutorial`. In a new terminal, the following +command can be run to start the Javascript console and connect it to Geth using the `geth.ipc` +file from the datadir: +```shell +geth attach geth-tutorial/geth.ipc ``` -This command causes the terminal to hang because it is waiting for approval from Clef. Approving the request in the terminal running Clef will lead to the following welcome message being displayed in the Javascript console: +The following welcome message will be displayed in the Javascript console: ```terminal Welcome to the Geth JavaScript console! instance: Geth/v1.10.15-stable/darwin-amd64/go1.17.5 at block: 6354736 (Thu Feb 10 2022 14:01:46 GMT+0100 (WAT)) - modules: eth:1.0 net:1.0 rpc:1.0 web3:1.0 + datadir: /home/go-ethereum/geth-tutorial + modules: admin:1.0 clique:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0 To exit, press ctrl-d or type exit ``` @@ -211,78 +242,58 @@ The console is now active and connected to Geth. It can now be used to interact ### List of accounts -In this tutorial, the accounts are managed using Clef. This means that requesting information about the accounts requires explicit approval in Clef, which should still be running in its own terminal. Earlier in this tutorial, two accounts were created using Clef. The following command will display the addresses of those two accounts and any others that might have been added to the keystore before or since. +Earlier in this tutorial, at least one account was created using `geth account new`. The following +command will display the addresses of those two accounts and any others that might have been added +to the keystore before or since. ```javascript - eth.accounts - ``` -The console will hang, because Clef is waiting for approval. The following message will be displayed in the Clef terminal: - ```terminal - --------- List Account request-------------- -A request has been made to list all accounts. -You can select which accounts the caller can see - [x] 0xca57F3b40B42FCce3c37B8D18aDBca5260ca72EC - URL: keystore:///.../geth-tutorial/keystore/UTC--2022-02-07T17-19-56.517538000Z--ca57f3b40b42fcce3c37b8d18adbca5260ca72ec - [x] 0xCe8dBA5e4157c2B284d8853afEEea259344C1653 - URL: keystore:///.../geth-tutorial/keystore/UTC--2022-02-10T12-46-45.265592000Z--ce8dba5e4157c2b284d8853afeeea259344c1653 -------------------------------------------- -Request context: - NA -> ipc -> NA - -Additional HTTP header data, provided by the external caller: - User-Agent: "" - Origin: "" -Approve? [y/N]: -> y - -``` - -Entering `y` approves the request from the console. In the terminal running the Javascript console, the account addresses are now displayed: - -```terminal - ["0xca57f3b40b42fcce3c37b8d18adbca5260ca72ec", "0xce8dba5e4157c2b284d8853afeeea259344c1653"] - ``` -It is also possible for this request to time out if the Clef approval took too long - in this case simply repeat the request and approval. - ### Checking account balance. -Having confirmed that the two addresses created earlier are indeed in the keystore and accessible through the Javascript console, it is possible to retrieve information about how much ether they own. The Goerli faucet should have sent 1 ETH to the address provided, meaning that the balance of one of the accounts should be 1 ether and the other should be 0. The following command displays the account balance in the console: +Having confirmed that the two addresses created earlier are indeed in the keystore and accessible +through the Javascript console, it is possible to retrieve information about how much ether they +own. The Goerli faucet should have sent 1 ETH to the address provided, meaning that the balance +of one of the accounts should be 1 ether and the other should be 0. The following command displays +the account balance in the console: ```javascript - web3.fromWei(eth.getBalance("0xca57F3b40B42FCce3c37B8D18aDBca5260ca72EC"), "ether") - ``` -There are actually two instructions sent in the above command. The inner one is the `getBalance` function from the `eth` namespace. This takes the account address as its only argument. By default, this returns the account balance in units of Wei. There are 1018 Wei to one ether. To present the result in units of ether, `getBalance` is wrapped in the `fromWei` function from the `web3` namespace. Running this command should provide the following result (for the account that received faucet funds): +There are actually two instructions sent in the above command. The inner one is the `getBalance` +function from the `eth` namespace. This takes the account address as its only argument. By default, +this returns the account balance in units of Wei. There are 1018 Wei to one ether. To +present the result in units of ether, `getBalance` is wrapped in the `fromWei` function from the +`web3` namespace. Running this command should provide the following result (for the account that +received faucet funds): ```terminal - 1 - ``` -Repeating the command for the other account should yield: +Repeating the command for the other new account that was not funded from the faucet should yield: ```terminal - 0 - ``` - ### Send ether to another account -The command `eth.sendTransaction` can be used to send some ether from one address to another. This command takes three arguments: `from`, `to` and `value`. These define the sender and recipient addresses (as strings) and the amount of Wei to transfer. It is far less error prone to enter the transaction value in units of ether rather than Wei, so the value field can take the return value from the `toWei` function. The following command, run in the Javascript console, sends 0.1 ether from one of the accounts in the Clef keystore to the other. Note that the addresses here are examples - the user must replace the address in the `from` field with the address currently owning 1 ether, and the address in the `to` field with the address currently holding 0 ether. +The command `eth.sendTransaction` can be used to send some ether from one address to another. +This command takes three arguments: `from`, `to` and `value`. These define the sender and +recipient addresses (as strings) and the amount of Wei to transfer. It is far less error prone +to enter the transaction value in units of ether rather than Wei, so the value field can take the +return value from the `toWei` function. The following command, run in the Javascript console, +sends 0.1 ether from one of the accounts in the keystore to the other. Note that the addresses +here are examples - the user must replace the address in the `from` field with the address +currently owning 1 ether, and the address in the `to` field with the address currently holding 0 ether. ```javascript eth.sendTransaction({ @@ -292,82 +303,48 @@ eth.sendTransaction({ }) ``` -Note that submitting this transaction requires approval in Clef. In the Clef terminal, Clef will prompt for approval and request the account password. If the password is correctly entered, Geth proceeds with the transaction. The transaction request summary is presented by Clef in the Clef terminal. This is an opportunity for the sender to review the details and ensure they are correct. - -```terminal ---------- Transaction request------------- -to: 0xCe8dBA5e4157c2B284d8853afEEea259344C1653 -from: 0xca57F3b40B42FCce3c37B8D18aDBca5260ca72EC [chksum ok] -value: 10000000000000000 wei -gas: 0x5208 (21000) -maxFeePerGas: 2425000057 wei -maxPriorityFeePerGas: 2424999967 wei -nonce: 0x3 (3) -chainid: 0x5 -Accesslist - -Request context: - NA -> ipc -> NA - -Additional HTTP header data, provided by the external caller: - User-Agent: "" - Origin: "" -------------------------------------------- -Approve? [y/N]: -> y - - -Please enter the password for account 0xca57F3b40B42FCce3c37B8D18aDBca5260ca72EC -> -``` - -After approving the transaction, the following confirmation screen in displayed in the Clef terminal: - -```terminal ------------------------ -Transaction signed: - { - "type": "0x2", - "nonce": "0x3", - "gasPrice": null, - "maxPriorityFeePerGas": "0x908a901f", - "maxFeePerGas": "0x908a9079", - "gas": "0x5208", - "value": "0x2386f26fc10000", - "input": "0x", - "v": "0x0", - "r": "0x66e5d23ad156e04363e68b986d3a09e879f7fe6c84993cef800bc3b7ba8af072", - "s": "0x647ff82be943ea4738600c831c4a19879f212eb77e32896c05055174045da1bc", - "to": "0xce8dba5e4157c2b284d8853afeeea259344c1653", - "chainId": "0x5", - "accessList": [], - "hash": "0x99d489d0bd984915fd370b307c2d39320860950666aac3f261921113ae4f95bb" - } - +This command will return an error message indicating that `authentication is needed: password or unlock`. +This is a security feature that prevents unauthorized access to sensitive account operations. +There are two ways to unlock the account. The first is to start Geth with the account permanently +unlocked (by passing `--unlock
` at startup). This is not recommended because the account +remains unlocked all the time Geth is running, creating a security weakness. Instead, it is better +to temporarily unlock the account for the specific transaction. This requires using the `sendTransaction` +method from the `personal` namespace instead of the `eth` namespace. The password can be provided as a +string in the method call as follows: + +```sh +personal.sendTransaction({ + from: "0xca57f3b40b42fcce3c37b8d18adbca5260ca72ec", + to: "0xce8dba5e4157c2b284d8853afeeea259344c1653", + value: web3.toWei(0.1, "ether") +}, "password") ``` -In the Javascript console, the transaction hash is displayed. This will be used in the next section to retrieve the transaction details. +In the Javascript console, the transaction hash is displayed. This will be used in the next section +to retrieve the transaction details. ```terminal - "0x99d489d0bd984915fd370b307c2d39320860950666aac3f261921113ae4f95bb" - ``` -It is also advised to check the account balances using Geth by repeating the instructions from earlier. At this point in the tutorial, the two accounts in the Clef keystore should have balances just below 0.9 ether (because 0.1 ether has been transferred out and some small amount paid in transaction gas) and 0.1 ether. - +It is also advised to check the account balances using Geth by repeating the instructions from earlier. +At this point in the tutorial, the two accounts in the keystore should have balances just below 0.9 +ether (because 0.1 ether has been transferred out and some small amount paid in transaction gas) and 0.1 ether. ### Checking the transaction hash -The transaction hash is a unique identifier for this specific transaction that can be used later to retrieve the transaction details. For example, the transaction details can be viewed by pasting this hash into the [Goerli block explorer](https://goerli.etherscan.io/). The same information can also be retrieved directly from the Geth node. The hash returned in the previous step can be provided as an argument to `eth.getTransaction` to return the transaction information: +The transaction hash is a unique identifier for this specific transaction that can be used later to +retrieve the transaction details. For example, the transaction details can be viewed by pasting this +hash into the [Goerli block explorer](https://goerli.etherscan.io/). The same information can also +be retrieved directly from the Geth node. The hash returned in the previous step can be provided as +an argument to `eth.getTransaction` to return the transaction information: ```javascript - eth.getTransaction("0x99d489d0bd984915fd370b307c2d39320860950666aac3f261921113ae4f95bb") - ``` -This returns the following response (although the actual values for each field will vary because they are specific to each transaction): +This returns the following response (although the actual values for each field will vary because they +are specific to each transaction): ```terminal { @@ -393,21 +370,31 @@ This returns the following response (although the actual values for each field w } ``` - ## Using Curl -Up to this point this tutorial has interacted with Geth using the convenience library Web3.js. This library enables the user to send instructions to Geth using a more user-friendly interface compared to sending raw JSON objects. However, it is also possible for the user to send these JSON objects directly to Geth's exposed HTTP port. Curl is a command line tool that sends HTTP requests. This part of the tutorial demonstrates how to check account balances and send a transaction using Curl. +Up to this point this tutorial has interacted with Geth using the convenience library Web3.js. +This library enables the user to send instructions to Geth using a more user-friendly interface +compared to sending raw JSON objects. However, it is also possible for the user to send these JSON +objects directly to Geth's exposed HTTP port. Curl is a command line tool that sends HTTP requests. +This part of the tutorial demonstrates how to check account balances and send a transaction using Curl. +This requires Geth to expose an HTTP port to listen for requests. This can be configured at startup +by passing the `--http` flag. If no other commands are passed with it, `--http` will expose the +default `localhost:8545` port. ### Checking account balance -The command below returns the balance of the given account. This is a HTTP POST request to the local port 8545. The `-H` flag is for header information. It is used here to define the format of the incoming payload, which is JSON. The `--data` flag defines the content of the payload, which is a JSON object. That JSON object contains four fields: `jsonrpc` defines the spec version for the JSON-RPC API, `method` is the specific function being invoked, `params` are the function arguments, and `id` is used for ordering transactions. The two arguments passed to `eth_getBalance` are the account address whose balance to check and the block to query (here `latest` is used to check the balance in the most recently mined block). +The command below returns the balance of the given account. This is a HTTP POST request to the local +port 8545. The `-H` flag is for header information. It is used here to define the format of the incoming +payload, which is JSON. The `--data` flag defines the content of the payload, which is a JSON object. +That JSON object contains four fields: `jsonrpc` defines the spec version for the JSON-RPC API, `method` +is the specific function being invoked, `params` are the function arguments, and `id` is used for ordering +transactions. The two arguments passed to `eth_getBalance` are the account address whose balance to check +and the block to query (here `latest` is used to check the balance in the most recently mined block). ```shell - curl -X POST http://127.0.0.1:8545 \ -H "Content-Type: application/json" \ --data '{"jsonrpc":"2.0", "method":"eth_getBalance", "params":["0xca57f3b40b42fcce3c37b8d18adbca5260ca72ec","latest"], "id":1}' - ``` A successful call will return a response like the one below: @@ -416,22 +403,19 @@ A successful call will return a response like the one below: {"jsonrpc":"2.0","id":1,"result":"0xc7d54951f87f7c0"} ``` -The balance is in the `result` field in the returned JSON object. However, it is denominated in Wei and presented as a hexadecimal string. There are many options for converting this value to a decimal in units of ether, for example by opening a Python console and running: +The balance is in the `result` field in the returned JSON object. However, it is denominated in Wei and +presented as a hexadecimal string. There are many options for converting this value to a decimal in units +of ether, for example by opening a Python console and running: ```python - 0xc7d54951f87f7c0 / 1e18 - ``` This returns the balance in ether: ```terminal - 0.8999684999998321 - ``` - ### Checking the account list The curl command below returns the list of all accounts. @@ -442,33 +426,27 @@ curl -X POST http://127.0.0.1:8545 \ --data '{"jsonrpc":"2.0", "method":"eth_accounts","params":[], "id":1}' ``` -This requires approval in Clef. Once approved, the following information is returned to the terminal: +The following information is returned to the terminal: ```terminal - {"jsonrpc":"2.0","id":1,"result":["0xca57f3b40b42fcce3c37b8d18adbca5260ca72ec"]} - ``` ### Sending Transactions -Sending a transaction between accounts can also be achieved using Curl. Notice that the value of the transaction is a hexadecimal string in units of Wei. To transfer 0.1 ether, it is first necessary to convert this to Wei by multiplying by 1018 then converting to hex. 0.1 ether is `"0x16345785d8a0000"` in hex. As before, update the `to` and `from` fields with the addresses in the Clef keystore. +It is possible to send transactions using raw curl requests too, but this requires unlocking the sender +account. It is recommended to do this using Clef to manage access to accounts or to use `ipc` instead. The +combination of HTTP and unlocked accounts pose a security risk. +## Summary -```shell -curl -X POST http://127.0.0.1:8545 \ - -H "Content-Type: application/json" \ - --data '{"jsonrpc":"2.0", "method":"eth_sendTransaction", "params":[{"from": "0xca57f3b40b42fcce3c37b8d18adbca5260ca72ec","to": "0xce8dba5e4157c2b284d8853afeeea259344c1653","value": "0x16345785d8a0000"}], "id":1}' -``` - -This requires approval in Clef. Once the password for the sender account has been provided, Clef will return a summary of the transaction details and the terminal that made the Curl request will display a response containing the transaction hash. - -```terminal - -{"jsonrpc":"2.0","id":5,"result":"0xac8b347d70a82805edb85fc136fc2c4e77d31677c2f9e4e7950e0342f0dc7e7c"} - -``` +This tutorial has demonstrated how to generate accounts using Geth's built-in account management tool, +fund them with testnet ether and use those accounts to interact with Ethereum (Goerli) through a Geth +node. Checking account balances, sending transactions and retrieving transaction details were explained using +the web3.js library via the Geth console and using the JSON-RPC directly using Curl. Note that this is an +entry-level tutorial designed to help users get familiar with basic Geth processes, we strongly recommend +following this with the [Geth with Clef](/docs/getting-started/geth_with_clef) tutorial which will help to +adopt more secure account management practices than those outlined here. -## Summary -This tutorial has demonstrated how to generate accounts using Clef, fund them with testnet ether and use those accounts to interact with Ethereum (Goerli) through a Geth node. Checking account balances, sending transactions and retrieving transaction details were explained using the web3.js library via the Geth console and using the JSON-RPC directly using Curl. +[cli]: https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Understanding_client-side_tools/Command_line diff --git a/content/docs/tools/Clef/Rules.md b/content/docs/tools/Clef/Rules.md index 9441181716..b7a7416cce 100644 --- a/content/docs/tools/Clef/Rules.md +++ b/content/docs/tools/Clef/Rules.md @@ -1,26 +1,58 @@ --- title: Rules -sort_key: B +sort_key: C --- -The `signer` binary contains a ruleset engine, implemented with [OttoVM](https://github.com/robertkrimen/otto) +This page provides a fairly low-level explanation for how rules are implemented in +Clef. It is a good idea to read the [Introduction to Clef](/docs/clef/introduction) +and the [Clef tutorial](/docs/clef/tutorial) before diving in to this page. -It enables usecases like the following: +{:toc} +- this will be removed by the toc + +## Introduction -* I want to auto-approve transactions with contract `CasinoDapp`, with up to `0.05 ether` in value to maximum `1 ether` per 24h period -* I want to auto-approve transaction to contract `EthAlarmClock` with `data`=`0xdeadbeef`, if `value=0`, `gas < 44k` and `gasPrice < 40Gwei` +Rules in Clef are sets of conditions that determine whether a given action can be +approved automatically without requiring manual intervention from the user. This can be +useful for automatically approving transactions between a user's own accounts, or +approving patterns that are commonly used by applications. Automatic signing also +requires Clef to have access to account passwords which is configured independently +of the ruleset. -The two main features that are required for this to work well are; +Rules can define arbitrary conditions such as: -1. Rule Implementation: how to create, manage and interpret rules in a flexible but secure manner -2. Credential managements and credentials; how to provide auto-unlock without exposing keys unnecessarily. +* Auto-approve 10 transactions with contract `CasinoDapp`, with value between `0.05 ether` and + `1 ether` per 24h period. -The section below deals with both of them +* Auto-approve transactions to contract `Uniswapv2` with `value` up to 1 ether, if + `gas < 44k` and `gasPrice < 40Gwei`. + +* Auto-approve signing if the data to be signed contains the string `"approve_me"`. + +* Auto-approve any requests to list accounts in keystore if the request arrives over IPC + +Because the rules are Javascript files they can be customized to implement any arbitrary logic on +the available request data. + +This page will explain how rules are implemented in Clef and how best to manage credentials +when automatic rulesets are enabled. ## Rule Implementation -A ruleset file is implemented as a `js` file. Under the hood, the ruleset-engine is a `SignerUI`, implementing the same methods as the `json-rpc` methods -defined in the UI protocol. Example: + +The ruleset engine acts as a gatekeeper to the command line interface - it auto-approves +any requests that meet the conditions defined in a set of authenticated rule files. This +prevents the user from having to manually approve or reject every request - instead they +can define common patterns in a rule file and abstract that task away to the ruleset engine. +The general architecture is as follows: + +![Clef ruleset logic](/static/images/clef_ruleset.png) + +When Clef receives a request, the ruleset engine evaluates a Javascript file for +each method defined in the internal [UI API docs](/docs/clef/apis). For example the code +snippet below is an example ruleset that calls the function `ApproveTx`. The call to `ApproveTx` +is invoking the `ui_approveTx` [JSON_RPC API endpoint](/docs/clef/apis/#ui-api). Every time an RPC +method is invoked the Javascript code is executed in a freshly instantiated virtual machine. ```js function asBig(str) { @@ -39,7 +71,8 @@ function ApproveTx(req) { return "Approve" } // If we return "Reject", it will be rejected. - // By not returning anything, it will be passed to the next UI, for manual processing + // By not returning anything, the decision to approve/reject + // will be passed to the next UI, for manual processing } // Approve listings if request made from IPC @@ -48,98 +81,165 @@ function ApproveListing(req){ } ``` -Whenever the external API is called (and the ruleset is enabled), the `signer` calls the UI, which is an instance of a ruleset-engine. The ruleset-engine -invokes the corresponding method. In doing so, there are three possible outcomes: - -1. JS returns "Approve" - * Auto-approve request -2. JS returns "Reject" - * Auto-reject request -3. Error occurs, or something else is returned - * Pass on to `next` ui: the regular UI channel. - -A more advanced example can be found below, "Example 1: ruleset for a rate-limited window", using `storage` to `Put` and `Get` `string`s by key. - -* At the time of writing, storage only exists as an ephemeral unencrypted implementation, to be used during testing. - -### Things to note - -The Otto vm has a few [caveats](https://github.com/robertkrimen/otto): +When a request is made via the external API, the logic flow is as follows: + +* Request is made to the `signer` binary using external API + +* `signer` calls the UI - in this case the ruleset engine + +* UI evaluates whether the call conforms to rules in an attested rulefile + +* Assuming the call returns "Approve", request is signed. + + +There are five possible outcomes from the ruleset engine that are +handled in different ways: + +| Return value | Action | +| ------------------| ----------------------------------------- | +| "Approve" | Auto-approve request | +| "Reject" | Auto-reject request | +| Anything else | Pass decision to UI for manual approval | + + +There are some additional noteworthy implementation details that are important +for defining rules correctly in `ruleset.js`: + +* The code in `ruleset.js` **cannot** load external Javascript files. + +* The Javascript engine can access `storage` and `console` + +* The only preloaded library in the Javascript environment is + `bignumber.js` version `2.0.3`. + +* Each invocation is made in a fresh virtual machine meaning data cannot be + stored in global variables between invocations. + +* Since no global variable storage is available, disk backed `storage` must be + used - rules should not rely on ephemeral data. + +* Javascript API parameters are always objects. This ensures parameters are + accessed by _key_ to avoid misordering errors. + +* Otto VM uses ES5. ES6-specific features (such as Typed Arrays) are not supported. + +* The regular expression engine (re2/regexp) in Otto VM is not fully compatible + with the [ECMA5 specification](https://tc39.es/ecma262/#sec-intro). + +* [Strict mode](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Strict_mode) + is not supported. "Use strict" will parse but it does nothing. -* "use strict" will parse, but does nothing. -* The regular expression engine (re2/regexp) is not fully compatible with the ECMA5 specification. -* Otto targets ES5. ES6 features (eg: Typed Arrays) are not supported. -Additionally, a few more have been added - -* The rule execution cannot load external javascript files. -* The only preloaded library is [`bignumber.js`](https://github.com/MikeMcl/bignumber.js) version `2.0.3`. This one is fairly old, and is not aligned with the documentation at the github repository. -* Each invocation is made in a fresh virtual machine. This means that you cannot store data in global variables between invocations. This is a deliberate choice -- if you want to store data, use the disk-backed `storage`, since rules should not rely on ephemeral data. -* Javascript API parameters are _always_ an object. This is also a design choice, to ensure that parameters are accessed by _key_ and not by order. This is to prevent mistakes due to missing parameters or parameter changes. -* The JS engine has access to `storage` and `console`. +## Credential management -#### Security considerations +The ability to auto-approve transaction requires that the signer has +the necessary credentials, i.e. account passwords, to decrypt keyfiles. +These are stored encrypted as follows: -##### Security of ruleset +When the `signer` is started it generates a seed that is locked with a +user specified password. The seed is saved to a location that defaults to +`$HOME/.clef/masterseed.json`. The `seed` itself is a blob of bytes. -Some security precautions can be made, such as: +The `signer` uses the `seed` to: -* Never load `ruleset.js` unless the file is `readonly` (`r-??-??-?`). If the user wishes to modify the ruleset, he must make it writeable and then set back to readonly. - * This is to prevent attacks where files are dropped on the users disk. -* Since we're going to have to have some form of secure storage (not defined in this section), we could also store the `sha3` of the `ruleset.js` file in there. - * If the user wishes to modify the ruleset, he'd then have to perform e.g. `signer --attest /path/to/ruleset --credential ` +* Generate the `path` where the configuration and credentials data are stored. + * `$HOME/.clef/790046d38025/config.json` + * `$HOME/.clef/790046d38025/credentials.json` +* Generate the encryption password for the config and credentials files. -##### Security of implementation +`config.json` stores the hashes of any attested rulesets. `credentials.json` +stores encrypted account passwords. The masterseed is required to decrypt +these files. The decrypted account passwords can then be used to decrypt keyfiles. -The drawbacks of this very flexible solution is that the `signer` needs to contain a javascript engine. This is pretty simple to implement, since it's already -implemented for `geth`. There are no known security vulnerabilities in, nor have we had any security-problems with it so far. +## Security -The javascript engine would be an added attack surface; but if the validation of `rulesets` is made good (with hash-based attestation), the actual javascript cannot be considered -an attack surface -- if an attacker can control the ruleset, a much simpler attack would be to implement an "always-approve" rule instead of exploiting the js vm. The only benefit -to be gained from attacking the actual `signer` process from the `js` side would be if it could somehow extract cryptographic keys from memory. +### The Javascript VM +The downside of the very flexible rule implementation included in Clef is +that the `signer` binary needs to contain a Javascript engine. This is an +additional attack surface. The only viable attack is for an adversary to +somehow extract cryptographic keys from memory during the Javascript VM execution. +The hash-based rule attestation condition means the actual Javascript code +executed by the Javascript engine is not a viable attack surface -- since if the attacker can control the ruleset, a much simpler +attack would be to surreptitiously insert an attested "always-approve" rule +instead of attempting to exploit the Javascript virtual machine. The Javascript +engine is quite simple to implement and there are currently no known security +vulnerabilities, not have there been any security problems identified for the +similar Javascript VM implemented in Geth. -##### Security in usability +### Writing rules -Javascript is flexible, but also easy to get wrong, especially when users assume that `js` can handle large integers natively. Typical errors -include trying to multiply `gasCost` with `gas` without using `bigint`:s. +Since the user has complete freedom to write custom rules, it is plausible that those rules +could create unintended security vulnerabilities. This can only really be protected by +coding very carefully and trying to test rulesets (e.g. on a private testnet) before +implementing them on a public network. -It's unclear whether any other DSL could be more secure; since there's always the possibility of erroneously implementing a rule. +Javascript is very flexible but also easy to write incorrectly. For example, users +might assume that javascript can handle large integers natively rather than explicitly +using `bigInt`. This is an error commonly encountered in the Ethereum context when +users attempt to multiply `gas` by `gasCost`. +It’s unclear whether any other language would be more secure - there is alwas the possibility +of implementing an insecure rule. -## Credential management +### File security -The ability to auto-approve transaction means that the signer needs to have necessary credentials to decrypt keyfiles. These passwords are hereafter called `ksp` (keystore pass). -### Example implementation +### Credential security -Upon startup of the signer, the signer is given a switch: `--seed ` -The `seed` contains a blob of bytes, which is the master seed for the `signer`. +Clef implements a secure, encrypted vault for storing sensitive data. This vault is +encrypted using a `masterseed` which the user is responsible for storing and backing +up safely and securely. Since this `masterseed` is used to decrypt the secure vault, +and its security is not handled by Clef, it could represent a security vulnerability +if the user does not implement best practise in keeping it safe. -The `signer` uses the `seed` to: +The same is also true for keys. Keys are not stored by Clef, they are only accessed +using account passwords that Clef does store in its vault. The keys themselves are stored +in an external `keystore` whose security is the responsibility of the user. If the +keys are compromised, the account is not safe irrespective of the security benefits +derived from Clef. -* Generate the `path` where the settings are stored. - * `./settings/1df094eb-c2b1-4689-90dd-790046d38025/vault.dat` - * `./settings/1df094eb-c2b1-4689-90dd-790046d38025/rules.js` -* Generate the encryption password for `vault.dat`. +## Ruleset examples -The `vault.dat` would be an encrypted container storing the following information: +Below are some examples of `ruleset.js` files. -* `ksp` entries -* `sha256` hash of `rules.js` -* Information about pair:ed callers (not yet specified) +### Example 1: Allow destination -### Security considerations +```js +function ApproveTx(r) { + if (r.transaction.to.toLowerCase() == "0x0000000000000000000000000000000000001337") { + return "Approve" + } + if (r.transaction.to.toLowerCase() == "0x000000000000000000000000000000000000dead") { + return "Reject" + } + // Otherwise goes to manual processing +} +``` -This would leave it up to the user to ensure that the `path/to/masterseed` is handled in a secure way. It's difficult to get around this, although one could -imagine leveraging OS-level keychains where supported. The setup is however in general similar to how ssh-keys are stored in `.ssh/`. +### Example 2: Allow listing +```js +function ApproveListing() { + return "Approve" +} +``` -# Implementation status +### Example 3: Approve signing data -This is now implemented (with ephemeral non-encrypted storage for now, so not yet enabled). +```js +function ApproveSignData(req) { + if (req.address.toLowerCase() == "0xd9c9cd5f6779558b6e0ed4e6acf6b1947e7fa1f3") { + if (req.messages[0].value.indexOf("bazonk") >= 0) { + return "Approve" + } + return "Reject" + } + // Otherwise goes to manual processing +} +``` -## Example 1: ruleset for a rate-limited window +### Example 4: Rate-limited window ```js function big(str) { @@ -214,24 +314,11 @@ function OnApprovedTx(resp) { } ``` -## Example 2: allow destination +## Summary -```js -function ApproveTx(r) { - if (r.transaction.from.toLowerCase() == "0x0000000000000000000000000000000000001337") { - return "Approve" - } - if (r.transaction.from.toLowerCase() == "0x000000000000000000000000000000000000dead") { - return "Reject" - } - // Otherwise goes to manual processing -} -``` +Rules are sets of conditions encoded in Javascript files that enable certain actions to +be auto-approved by Clef. This page outlined the implementation details and security +considerations that will help to build suitrable ruleset files. See the +[Clef Github](https://github.com/ethereum/go-ethereum/tree/master/cmd/clef) for further reading. -## Example 3: Allow listing -```js -function ApproveListing() { - return "Approve" -} -```