Evolv

#1 AI-driven digital growth solution

tested and perfected together with QAwerk
Girl Image Boy Image Clip Icon Blue Clip Icon Red
×
Where do you want us to send our Evolv case study?
Please enter your business email

The Evolv Digital Growth Optimization solution allows brands to keep pace with changing market dynamics. Using AI, Evolv is able to work across all digital touchpoints in the customer journey, including web and mobile, and empowers companies to meet the ever-increasing customer expectations.

All Customers

Web & Desktop Testing

We thoroughly test Evolv’s web and desktop apps to make sure they work as intended. With our continuous QA support, Evolv can deliver new capabilities quickly and continue to drive value for their clients while ensuring non-stop availability of their platform.

Learn more

Automated Testing

Combining substantial experience in automated testing and proactive thinking, our QA engineers help Evolv maintain a cost-effective strategy for automating regression and smoke tests. With our continuous automated testing, Evolv can release software updates frequently and safely.

Learn more

Introduction

Evolv empowers businesses through its AI-driven SaaS platform designed for launching multiple UX experiments which go far beyond traditional A/B testing. Evolv helps their clients significantly increase key metrics such as conversion rates by introducing the right UX improvements to their websites or apps. Evolv started in 2019 and is the result of a technology spin-off from Sentient Technologies who spent the better part of the decade as an AI research and development firm. That team incubated some innovative algorithms, which are now included as a building block in Evolv’s AI-driven optimization platform. Initially, we provided our service to Sentient Technologies; after Sentient sold the majority of its business to Cognizant, Evolv was born from the intellectual property of the offering previously called Sentient Ascend.

A platform with such an extensive business logic requires a complex and unique architecture with multiple modules. To keep the platform robust and healthy, Evolv continues to invest in ongoing software testing and quality assurance, and entrusts QAwerk with helping with this crucial mission.

Evolv’s Digital Growth Optimization solution enables brands to capitalize on changing trends and customer preferences using proven UX improvement research. They help clients analyze their current digital customer experience strategy, generate new ideas to be explored, and leverage their proprietary AI software to quickly determine winning experiences and route traffic to them in real-time.

Suppose a business owner wants more people to use a lead magnet on a service page. However, searching for options to optimize the user journey on that page with conventional A/B testing consumes too much resources, both time and money. Evolv allows testing multiple UX candidates simultaneously, automatically excluding the least performing options. On top of that, it has a second phase of UX experiments – EVO – when variants from different variables get combined based on the platform’s machine learning algorithms. As a result, having all the statistical reports at hand, the business owner will know exactly what UX changes to implement to achieve their optimization target.

Challenge

At the very beginning, it was challenging for us to perform comprehensive testing without a complete understanding of the platform’s business logic and its scope. Back then, we had only a few test cases to start with, and we worked with an older version of Evolv’s web app. Later on, we started testing their brand new desktop app – Web Editor – designed for developing UX improvements for clients’ websites, and a web app – Manager – designed for launching and managing AI-driven testing as well as viewing all the relevant statistics on the experiments. One of our biggest challenges was to validate that the A/B test is designed correctly, that the UX experiment goes as intended, and that the Manager app shows accurate reports.

All the QA services we provide for Evolv can be grouped into the following categories:

  • Testing UX improvements for the experiments run by Evolv’s clients – offering custom-tailored UX solutions for each visitor and customer is an essential component of Evolv’s digital growth strategy. Our task in this part is to make sure all the suggested UX changes for digital growth optimization are implemented properly.
  • Testing product releases + regression testing – Evolv releases software updates every month, so we regularly test new features, bug fixes, and perform regression testing of the core functionality to keep the platform running 24/7.
  • Automated testing – our major challenge in this domain is to automate regression and smoke tests so that they could be run in the CI/CD pipeline. We also strive to create easily maintainable and reusable tests with a focus on codeless test automation.

Solution

The obstacles we faced at the initial testing stage were quickly resolved thanks to close communication with Evolv’s team and thorough documentation. While we were writing test cases in CucumberStudio, Evolv’s team prepared debugging documents for us to better understand the platform’s logic. Once Evolv transitioned to a new platform, it was easier for us to test it because we were part of every step of its development, so we knew the architecture and all the logic behind it very well.

So far, we have finished a set of regression test cases that cover the platform’s core functionality. We perform regression testing for every release, and we managed to speed up this process from 3-4 days to 2.

We are also excited about extending our services to the domain of automated testing, which significantly reduces the time our QAwerk team spends on checking the main functionality in every new build. When implementing automated tests, QA engineers often have to deal with continuous code editing or complete code rewriting to get automated tests running. The challenge centers on the need for the platform to continue being developed and further upgraded and how the user journey may be changed multiple times in the process. Therefore, each change in the user journey will lead to respective edits in the test automation code, which consumes time and effort. We found a more productive approach to automating regression and smoke tests, which allows us to write maintainable and reusable code and easily create new test scenarios without writing any code: we only form new sets of scenarios in CucumberStudio.

Test Automation

Let’s dive deeper into how we came to almost no-code test automation. Initially, Evolv relied on CucumberStudio as their BDD collaboration tool. When we started working with Evolv, they initially used it to write and run some manual test cases and scenarios. On our end, we relied on CodeceptJS, namely Puppeteer, for the automation project. Yet, we wanted to leverage the benefits of both tools and optimize the automation process and make it time-efficient and transparent even for employees with weak or no programming backgrounds.

Combining CucumberStudio’s automation capabilities, such as tests export for BDD frameworks like CodeceptJS and .feature files import, with some Javascript magic (AST parsing/code generation), we’ve built a tool to automatically generate glue code for CucumberStudio and CodeceptJS-based automation project back-and-forth integration. This setup allows easy integration into virtually any CI/CD pipeline.

With this approach, it is possible to significantly decrease the amount of code a QA engineer has to write to automate common test scenarios. On top of that, this setup allows manual QA engineers to contribute to test automation with minimum time spent on onboarding. Here is what they can do:

  • Build up new test scenarios in CucumberStudio project (into which .feature file generated by our custom tool had been imported before) – using about 100-plus Action Words and Test Scenario Context (variables)
  • Add newly built test scenarios to a test runner in CucumberStudio (with CI/CD pipeline already configured)
  • Verify newly built test scenarios – passes in CI/CD
  • Refactor low-level (group) Action Words in newly built test scenarios to business-logic-level ones – aka true Gherkin Phrases in CucumberStudio

It’s no breaking news that test automation can easily defeat its purpose if done incorrectly. Fixing test inconsistencies due to changes in GUI (which happen quite often) is one of the major pain points in test automation. If a button or a text box are moved, the element locator won’t find it, resulting in a failed test run. To make tests green again, the automation tester will have to modify code and push/PR/merge the changes – quite a time-consuming process.

Our approach to test automation allows reducing the time needed for such (and many other kinds of) fixes, be it changes in locators’ names or steps order. Manual QA engineers can efficiently perform this kind of edits by correcting test steps in CucumberStudio and synchronizing these changes with test runs.

Bugs Found

The biggest portion of our time goes on testing Evolv’s SaaS platform consisting of Web Editor and Manager; therefore, the bugs displayed refer to these apps’ functionality.

Bug

Actual result: Combination report in Manager shows an incorrect visitor count. Total Unique Visitors for the combination is a negative number.

Expected result: Combination report in Manager shows a correct visitor count.

Actual result: QA mode in Web Editor: selected UX changes cannot be applied and are visible only in the Edit Mode.

Expected result: QA mode in Web Editor: selected UX changes are successfully applied and are visible in QA mode.

Bug

Actual result: Unable to create New Organization in Manager, receiving: message: {statusCode: 403, error: “Forbidden”, message: “Forbidden resource”}

Expected result: It’s possible to create New Organization in Manager; no error messages are received.

Result

Having a reliable technology partner is crucial to a company’s continuous growth. With the joint effort of both teams, Evolv managed to release its new product – Evolv 1.0., which allows their clients to conduct even more elaborate UX experiments. We are proud to be part of such an innovative project and continue working tirelessly to assure Evolv’s long lasting success.

Exit: Acquisition

Evolv acquired Sentient Ascend™, a world-famous conversion optimization platform in March 2019 Evolv acquired Sentient Ascend™, a world-famous conversion optimization platform in March 2019 Read More

Awarded

Frost & Sullivan, Best Practices Award 2020 Frost & Sullivan, Best Practices Award 2020 Read More
I feel like our QA was so bad, but we definitely improved it with QAwerk, and looking forward to some new improvement with you.
Sinéad Ryan, Head of Global Services at Evolv Technologies
There's a real commitment to get the task done in a timeframe that is expected. The quality of the work is very high. I would certainly recommend working with QAwerk's team.
Robert Severn, VP of Engineering at Evolv Technologies

Need comprehensive QA automation?

Let’s talk

Tools

JiraJira
CucumberStudioCucumberStudio
CodeceptJSCodeceptJS
PuppeteerPuppeteer
GithubGithub
Github ActionsGithub Actions
29specialists engaged in project
10QA engineers
200+test cases written
100+UX experiments tested

QAwerk Team Comment

Alexander

Alexander
QA engineer

What makes the project truly exciting for me is the tech stack I get to work with: React, TypeScript, Material UI, MobX, GraphQL, Apollo, RxJS, VX, D3, Formik, Auth0, Electron, Jest. While testing, I had to dive deeper into some of these technologies, which gave me a great technical boost. I also really appreciate the opportunity to perform both manual and automated testing, which allows me to comprehend the platform better, and as a result, achieve the set goals faster. As for test automation, the CucumberStudio +CodeceptJS +Puppeteer combination turned out to work very well, allowing us to create tests as BDD scenarios and achieve simple implementation of the complex logic using action words. In terms of communication, I have slightly shifted my work hours so that I can communicate with Evolv’s team in real-time, which makes our cooperation much more efficient.

Media
Media
Media
Media
Media

Impressed?

Hire us

Other Case Studies

Unfold

Unfold

United States

Transformed a crashy storymaker into a billion-user content creation platform acquired by Squarespace

Station

Station

France

Performed regression testing of desktop app with 600+ integrations on WidowsmacOSand Ubuntu

Unpakt

Unpakt

United States

Helped popular US moving service detect adaptation issues and smoothly transition to new stack