diff --git a/README.md b/README.md index b8f700f796..a96b99c249 100644 --- a/README.md +++ b/README.md @@ -141,69 +141,129 @@ To run the Selenium tests via Nightwatch: for Firefox: `npm run nightwatch_local_firefox`, or for Google Chrome: `npm run nightwatch_local_chrome` - - Run a specific test case instead, use one of following commands: + - Run a specific test case instead, use a command like this: - npm run nightwatch_local_ballot - - - npm run nightwatch_local_usingWorker - - - npm run nightwatch_local_libraryDeployment - - - npm run nightwatch_local_solidityImport - - - npm run nightwatch_local_recorder - - npm run nightwatch_local_transactionExecution - - - npm run nightwatch_local_staticAnalysis - - - npm run nightwatch_local_signingMessage + The package.json file contains a list of all the tests you can run. + +**NOTE:** + +- **The `ballot` tests suite** requires to run `ganache-cli` locally. - - npm run nightwatch_local_specialFunctions +- **The `remixd` tests suite** requires to run `remixd` locally. + +- **The `gist` tests suite** requires specifying a github access token in **.env file**. +``` + gist_token = // token should have permission to create a gist +``` - - npm run nightwatch_local_solidityUnitTests +### Using 'select_test' for locally running specific tests - - npm run nightwatch_local_remixd # remixd needs to be run +There is a script to allow selecting the browser and a specific test to run: - - npm run nightwatch_local_terminal +``` +npm run select_test +``` - - npm run nightwatch_local_gist +You need to have - - npm run nightwatch_local_workspace +- selenium running - - npm run nightwatch_local_defaultLayout +- the IDE running - - npm run nightwatch_local_pluginManager +- optionally have remixd or ganache running - - npm run nightwatch_local_publishContract +### Splitting tests with groups - - npm run nightwatch_local_generalSettings +Groups can be used to group tests in a test file together. The advantage is you can avoid running long test files when you want to focus on a specific set of tests within a test file.x - - npm run nightwatch_local_fileExplorer +These groups only apply to the test file, not across all test files. So for example group1 in the ballot is not related to group1 in another test file. - - npm run nightwatch_local_debugger +Running a group only runs the tests marked as belonging to the group + all the tests in the test file that do not have a group tag. This way you can have tests that run for all groups, for example to peform common actions. - - npm run nightwatch_local_editor +There is no need to number the groups in a certain order. The number of the group is arbitrary. - - npm run nightwatch_local_compiler +A test can have multiple group tags, this means that this test will run in different groups. - - npm run nightwatch_local_txListener +You should write your tests so they can be executed in groups and not depend on other groups. - - npm run nightwatch_local_fileManager +To do this you need to: - - npm run nightwatch_local_runAndDeploy - - -**NOTE:** +- Add a group to tag to a test, they are formatted as #group followed by a number: so it becomes #group1, #group220, #group4. Any number will do. You don't have to do it in specific order. -- **The `ballot` tests suite** requires to run `ganache-cli` locally. +``` + 'Should generate test file #group1': function (browser: NightwatchBrowser) { + browser.waitForElementPresent('*[data-id="verticalIconsKindfilePanel"]') +``` -- **The `remixd` tests suite** requires to run `remixd` locally. +- add '@disable': true to the test file you want to split: -- **The `gist` tests suite** requires specifying a github access token in **.env file**. ``` - gist_token = // token should have permission to create a gist +module.exports = { + '@disabled': true, + before: function (browser: NightwatchBrowser, done: VoidFunction) { + init(browser, done) // , 'http://localhost:8080', false) + }, +``` +- change package json to locally run all group tests: + +``` + "nightwatch_local_debugger": "npm run build:e2e && nightwatch --config dist/apps/remix-ide-e2e/nightwatch.js dist/apps/remix-ide-e2e/src/tests/debugger_*.spec.js --env=chrome", +``` + +- run the build script to build the test files if you want to run the locally + +``` +npm run build:e2e +``` + +### Locally testing group tests + +You can tag any test with a groupname, for example, #group10 and easily run the test locally. + +- make sure you have nx installed globally +- group tests are run like any other test, just specify the correct group number + +#### method 1 + +This script will give you an option menu, just select the test you want +``` +npm run select_test +``` +#### method 2 + +``` +npm run group_test --test=debugger --group=10 --env=chromeDesktop +``` +- specify chromeDesktop to see the browser action, use 'chrome' to run it headless + +### Run the same (flaky) test across all instances in CircleCI + +In CircleCI all tests are divided across instances to run in paralel. +You can also run 1 or more tests simultaneously across all instances. +This way the pipeline can easily be restarted to check if a test is flaky. + +For example: + ``` + 'Static Analysis run with remixd #group3 #flaky': function (browser) { +``` + +Now group3 of this test will be executed in firefox and chrome 80 times. +If you mark more groups in other tests they will also be executed. + +**CONFIGURATION** + +It's important to set a parameter in the .circleci/config.yml, set it to false then the normal tests will run. +Set it to true to run only tests marked with flaky. +``` +parameters: + run_flaky_tests: + type: boolean + default: true +``` + ## Important Links