Test automation stability is the primary challenge for any QA automation engineer. Why is it so important? Validating what tests failed because of bugs or changes to the code and what tests are the result of an unstable environment is the daily practice of a QA automation engineer. If automated tests are unstable, the number of false positives will continue to rise as the project grows. As a result, an enormous amount of time is wasted on filtering false positives from truly failed tests.
Before we started any work on the project, the Union54 team had already written a couple of tests with the Mocha framework, so we decided to give it a try. As new test scenarios became more elaborate and the test suites increased in size, we noticed significant stability issues and communicated our concerns to our client. To resolve this issue, we switched to a different framework, rewrote all the existing tests in Cypress, and placed them into a separate repository.
We agreed on Cypress because of the numerous benefits it offers. Cypress allows you to automatically retry failed tests and set a custom number of retry attempts, either for all tests or just for a specific one. This feature is super handy for reducing test flakiness and CI build failures caused by the temporary unavailability of third-party services or random network errors. Other Cypress advantages are logical and straightforward patterns, high readability of test scenarios, and scalable code; the already written patterns can be reused in future tests. Cypress is also a good match for running complex, multi-step test scenarios.
CI for the automated tests is implemented with the help of Bitbucket Pipeline. Every night at 2 a.m., the latest commits from the develop branch are pulled into the CI pipeline. Therefore, our tests cover all changes to the codebase even if developers work late into the night. Thanks to the Bitbucket integration with Slack, we are instantly notified of all commits, pull requests, and pipeline statuses, both successful and failed.
A complete test run including 1500 test scenarios in the CI pipeline takes about 1.5 hours, which is quite decent, considering the test scenario complexity and number. Having achieved the stability of automated tests, we see a clear picture every morning, spending minimum time on troubleshooting.
As new capabilities are being added to the Union54 website, we also cover the front-end part with automated functional tests.