In the parameter window, you can set the URL of the software under test, the timeouts, variable, execution concurrency, and the environments:
One project consists of one URL. For a new project don’t forget to set it.
Navigation timeout is a time when you navigate (e.g. click a link) and the new page appears after some time. Timeout is for other cases, such as when a value is wrong. It is usually less than the navigation timeout.
As a default, if you execute the test cases for a feature or the project, the test cases are executed sequentially one by one. You can execute them in parallel, by checking ‘Concurrent running’.
Different variables can be set for different environments. Just add a key-value pair:
and add the same key into a low-level step (use ${}):
Environment
You can execute your tests on several environments. First, select the ‘Parameters’, then press ‘New environment’ and add your environment name (each name should be different):
The result is a new environment (here production), where you can set a different URL, timeouts, etc. necessary for this environment:
By pressing the three dots next to the environment name, you can modify or delete it:
Flaky test elimination
Test flakiness is considered one of the main challenges of automated testing. Fortunately, there is a significant difference between failed and flaky tests, namely, a failed test will always fail at the same execution point, but a flaky test may pass. If a flaky test passes, then the (part of) requirement covered by the test is correct. If so it’s enough to re-execute them to pass.
In Harmony, you can set the number of retries. If it’s 2, then failed test cases will be re-executed twice:
According to our experiences, 2 retries are usually enough. In the test report, you can check if a test case only passed during the retrial period as the the error is displayed but the test passed (green):
