Evolv

Introduction

Evolv empowers businesses through its AI-driven SaaS platform designed for launching multiple UX experiments which go far beyond traditional A/B testing. Evolv helps their clients significantly increase key metrics such as conversion rates by introducing the right UX improvements to their websites or apps. Evolv started in 2019 and is the result of a technology spin-off from Sentient Technologies who spent the better part of the decade as an AI research and development firm. That team incubated some innovative algorithms, which are now included as a building block in Evolv’s AI-driven optimization platform. Initially, we provided our service to Sentient Technologies; after Sentient sold the majority of its business to Cognizant, Evolv was born from the intellectual property of the offering previously called Sentient Ascend.

A platform with such an extensive business logic requires a complex and unique architecture with multiple modules. To keep the platform robust and healthy, Evolv continues to invest in ongoing software testing and quality assurance, and entrusts QAwerk with helping with this crucial mission.

Evolv’s Digital Growth Optimization solution enables brands to capitalize on changing trends and customer preferences using proven UX improvement research. They help clients analyze their current digital customer experience strategy, generate new ideas to be explored, and leverage their proprietary AI software to quickly determine winning experiences and route traffic to them in real-time.

Suppose a business owner wants more people to use a lead magnet on a service page. However, searching for options to optimize the user journey on that page with conventional A/B testing consumes too many resources, both time and money. Evolv allows testing multiple UX candidates simultaneously, automatically excluding the least performing options. On top of that, it has a second phase of UX experiments – EVO – when variants from different variables get combined based on the platform’s machine learning algorithms. As a result, having all the statistical reports at hand, the business owner will know exactly what UX changes to implement to achieve their optimization target.

Challenge

At the very beginning, it was challenging for us to perform comprehensive testing without a complete understanding of the platform’s business logic and its scope. Back then, we had only a few test cases to start with, and we worked with an older version of Evolv’s web app. Later on, we started testing their brand new desktop app – Web Editor – designed for developing UX improvements for clients’ websites, and a web app – Manager – designed for launching and managing AI-driven testing as well as viewing all the relevant statistics on the experiments. One of our biggest challenges was to validate that the A/B test is designed correctly, that the UX experiment goes as intended, and that the Manager app shows accurate reports.

All the QA services we provide for Evolv can be grouped into the following categories:

• Testing UX improvements for the experiments run by Evolv’s clients – offering custom-tailored UX solutions for each visitor and customer is an essential component of Evolv’s digital growth strategy. Our task in this part is to make sure all the suggested UX changes for digital growth optimization are implemented properly.
• Testing product releases + regression testing – Evolv releases software updates every month, so we regularly test new features, bug fixes, and perform regression testing of the core functionality to keep the platform running 24/7.
• Automated testing – our major challenge in this domain is to automate regression and smoke tests so that they could be run in the CI/CD pipeline. We also strive to create easily maintainable and reusable tests with a focus on codeless test automation.

Solution

The obstacles we faced at the initial testing stage were quickly resolved thanks to close communication with Evolv’s team and thorough documentation. While we were writing test cases in CucumberStudio, Evolv’s team prepared debugging documents for us to better understand the platform’s logic. Once Evolv transitioned to a new platform, it was easier for us to test it because we were part of every step of its development, so we knew the architecture and all the logic behind it very well.

So far, we have finished a set of regression test cases that cover the platform’s core functionality. We perform regression testing for every release, and we managed to speed up this process from 3-4 days to 2.

We are also excited about extending our services to the domain of automated testing, which significantly reduces the time our QAwerk team spends on checking the main functionality in every new build. When implementing automated tests, QA engineers often have to deal with continuous code editing or complete code rewriting to get automated tests running. The challenge centers on the need for the platform to continue being developed and further upgraded and how the user journey may be changed multiple times in the process. Therefore, each change in the user journey will lead to respective edits in the test automation code, which consumes time and effort. We found a more productive approach to automating regression and smoke tests, which allows us to write maintainable and reusable code and easily create new test scenarios without writing any code: we only form new sets of scenarios in CucumberStudio.

Test Automation

Let’s dive deeper into how we came to almost no-code test automation. Initially, Evolv relied on CucumberStudio as their BDD collaboration tool. When we started working with Evolv, they initially used it to write and run some manual test cases and scenarios. On our end, we relied on CodeceptJS, namely Puppeteer, for the automation project. Yet, we wanted to leverage the benefits of both tools and optimize the automation process and make it time-efficient and transparent even for employees with weak or no programming backgrounds.

Combining CucumberStudio’s automation capabilities, such as tests export for BDD frameworks like CodeceptJS and .feature files import, with some Javascript magic (AST parsing/code generation), we’ve built a tool to automatically generate glue code for CucumberStudio and CodeceptJS-based automation project back-and-forth integration. This setup allows easy integration into virtually any CI/CD pipeline.

With this approach, it is possible to significantly decrease the amount of code a QA engineer has to write to automate common test scenarios. On top of that, this setup allows manual QA engineers to contribute to test automation with minimum time spent on onboarding. Here is what they can do:

• Build up new test scenarios in CucumberStudio project (into which .feature file generated by our custom tool had been imported before) – using about 100-plus Action Words and Test Scenario Context (variables)
• Add newly built test scenarios to a test runner in CucumberStudio (with CI/CD pipeline already configured)
• Verify newly built test scenarios – passes in CI/CD
• Refactor low-level (group) Action Words in newly built test scenarios to business-logic-level ones – aka true Gherkin Phrases in CucumberStudio

It’s no breaking news that test automation can easily defeat its purpose if done incorrectly. Fixing test inconsistencies due to changes in GUI (which happen quite often) is one of the major pain points in test automation. If a button or a text box are moved, the element locator won’t find it, resulting in a failed test run. To make tests green again, the automation tester will have to modify code and push/PR/merge the changes – quite a time-consuming process.

Our approach to test automation allows reducing the time needed for such (and many other kinds of) fixes, be it changes in locators’ names or steps order. Manual QA engineers can efficiently perform this kind of edits by correcting test steps in CucumberStudio and synchronizing these changes with test runs.