Merge pull request #2410 from ethereum/e2ereadme

update readme

README.md

@@ -141,69 +141,129 @@ To run the Selenium tests via Nightwatch:
     for Firefox: `npm run nightwatch_local_firefox`, or 
 
     for Google Chrome: `npm run nightwatch_local_chrome`
- - Run a specific test case instead, use one of following commands: 
+ - Run a specific test case instead, use a command like this: 
  
 		- npm run nightwatch_local_ballot
-
-        - npm run nightwatch_local_usingWorker
-		
-		- npm run nightwatch_local_libraryDeployment
-		
-		- npm run nightwatch_local_solidityImport
-		
-		- npm run nightwatch_local_recorder
 		
-		- npm run nightwatch_local_transactionExecution
-		
-		- npm run nightwatch_local_staticAnalysis
-		
-		- npm run nightwatch_local_signingMessage
+	The package.json file contains a list of all the tests you can run.
+        
+**NOTE:**
+
+- **The `ballot` tests suite** requires to run `ganache-cli` locally.
 
-        - npm run nightwatch_local_specialFunctions
+- **The `remixd` tests suite** requires to run `remixd` locally.
+
+- **The `gist` tests suite** requires specifying a github access token in **.env file**. 
+```
+    gist_token = <token> // token should have permission to create a gist
+```
 
-        - npm run nightwatch_local_solidityUnitTests
+### Using 'select_test' for locally running specific tests
 
-        - npm run nightwatch_local_remixd # remixd needs to be run
+There is a script to allow selecting the browser and a specific test to run:
 
-		- npm run nightwatch_local_terminal
+```
+npm run select_test
+```
 
-        - npm run nightwatch_local_gist
+You need to have 
 
-        - npm run nightwatch_local_workspace
+- selenium running 
 
-        - npm run nightwatch_local_defaultLayout
+- the IDE running
 
-        - npm run nightwatch_local_pluginManager
+- optionally have remixd or ganache running
 
-        - npm run nightwatch_local_publishContract
+### Splitting tests with groups
 
-        - npm run nightwatch_local_generalSettings
+Groups can be used to group tests in a test file together. The advantage is you can avoid running long test files when you want to focus on a specific set of tests within a test file.x
 
-        - npm run nightwatch_local_fileExplorer
+These groups only apply to the test file, not across all test files. So for example group1 in the ballot is not related to group1 in another test file.
 
-        - npm run nightwatch_local_debugger
+Running a group only runs the tests marked as belonging to the group + all the tests in the test file that do not have a group tag. This way you can have tests that run for all groups, for example to peform common actions.
 
-        - npm run nightwatch_local_editor
+There is no need to number the groups in a certain order. The number of the group is arbitrary.
 
-        - npm run nightwatch_local_compiler
+A test can have multiple group tags, this means that this test will run in different groups.
 
-        - npm run nightwatch_local_txListener
+You should write your tests so they can be executed in groups and not depend on other groups.
 
-        - npm run nightwatch_local_fileManager
+To do this you need to:
 
-        - npm run nightwatch_local_runAndDeploy
-		
-        
-**NOTE:**
+- Add a group to tag to a test, they are formatted as #group followed by a number: so it becomes #group1, #group220, #group4. Any number will do. You don't have to do it in specific order. 
 
-- **The `ballot` tests suite** requires to run `ganache-cli` locally.
+```
+  'Should generate test file #group1': function (browser: NightwatchBrowser) {
+    browser.waitForElementPresent('*[data-id="verticalIconsKindfilePanel"]')
+```
 
-- **The `remixd` tests suite** requires to run `remixd` locally.
+- add '@disable': true to the test file you want to split:
 
-- **The `gist` tests suite** requires specifying a github access token in **.env file**. 
 ```
-    gist_token = <token> // token should have permission to create a gist
+module.exports = {
+  '@disabled': true,
+  before: function (browser: NightwatchBrowser, done: VoidFunction) {
+    init(browser, done) // , 'http://localhost:8080', false)
+  },
+```
+- change package json to locally run all group tests:
+
+```
+    "nightwatch_local_debugger": "npm run build:e2e && nightwatch --config dist/apps/remix-ide-e2e/nightwatch.js dist/apps/remix-ide-e2e/src/tests/debugger_*.spec.js --env=chrome",
+```
+
+- run the build script to build the test files if you want to run the locally
+
+```
+npm run build:e2e
+```
+
+### Locally testing group tests
+
+You can tag any test with a groupname, for example, #group10 and easily run the test locally.
+
+- make sure you have nx installed globally
+- group tests are run like any other test, just specify the correct group number
+
+#### method 1
+
+This script will give you an option menu, just select the test you want
+```
+npm run select_test
+```
+#### method 2
+
+```
+npm run group_test --test=debugger --group=10 --env=chromeDesktop
+```
+- specify chromeDesktop to see the browser action, use 'chrome' to run it headless
+
+### Run the same (flaky) test across all instances in CircleCI
+
+In CircleCI all tests are divided across instances to run in paralel. 
+You can also run 1 or more tests simultaneously across all instances.
+This way the pipeline can easily be restarted to check if a test is flaky.
+
+For example:
+
 ```
+  'Static Analysis run with remixd #group3 #flaky': function (browser) {
+```
+
+Now group3 of this test will be executed in firefox and chrome 80 times.
+If you mark more groups in other tests they will also be executed. 
+
+**CONFIGURATION**
+
+It's important to set a parameter in the .circleci/config.yml, set it to false then the normal tests will run.
+Set it to true to run only tests marked with flaky.
+```
+parameters:
+  run_flaky_tests:
+    type: boolean
+    default: true
+```
+
 
 ## Important Links