Evolv

AI-Driven Digital Growth Optimization Solution

tested and perfected together with QAwerk

Evolv
Evolv TechnologiesBerkeley, CA, United States

The Evolv Digital Growth Optimization solution allows brands to keep pace with changing market dynamics. Using AI, Evolv is able to work across all digital touchpoints in the customer journey, including web and mobile, and empowers companies to meet the ever-increasing customer expectations.

All Customers
Web & Desktop Testing

We thoroughly test Evolv’s web and desktop apps to make sure they work as intended. With our continuous QA support, Evolv can deliver new capabilities quickly and continue to drive value for their clients while ensuring non-stop availability of their platform.

Learn more
Automated Testing

Combining substantial experience in automated testing and proactive thinking, our QA engineers help Evolv maintain a cost-effective strategy for automating regression and smoke tests. With our continuous automated testing, Evolv can release software updates frequently and safely.

Learn more

Introduction

Evolv empowers businesses through its AI-driven SaaS platform designed for launching multiple UX experiments which go far beyond traditional A/B testing. Evolv helps their clients significantly increase key metrics such as conversion rates by introducing the right UX improvements to their websites or apps. Evolv started in 2019 and is the result of a technology spin-off from Sentient Technologies who spent the better part of the decade as an AI research and development firm. That team incubated some innovative algorithms, which are now included as a building block in Evolv’s AI-driven optimization platform. Initially, we provided our service to Sentient Technologies; after Sentient sold the majority of its business to Cognizant, Evolv was born from the intellectual property of the offering previously called Sentient Ascend.

A platform with such an extensive business logic requires a complex and unique architecture with multiple modules. To keep the platform robust and healthy, Evolv continues to invest in ongoing software testing and quality assurance, and entrusts QAwerk with helping with this crucial mission.

Evolv’s Digital Growth Optimization solution enables brands to capitalize on changing trends and customer preferences using proven UX improvement research. They help clients analyze their current digital customer experience strategy, generate new ideas to be explored, and leverage their proprietary AI software to quickly determine winning experiences and route traffic to them in real-time.

Suppose a business owner wants more people to use a lead magnet on a service page. However, searching for options to optimize the user journey on that page with conventional A/B testing consumes too much resources, both time and money. Evolv allows testing multiple UX candidates simultaneously, automatically excluding the least performing options. On top of that, it has a second phase of UX experiments – EVO – when variants from different variables get combined based on the platform’s machine learning algorithms. As a result, having all the statistical reports at hand, the business owner will know exactly what UX changes to implement to achieve their optimization target.

Challenge

At the very beginning, it was challenging for us to perform comprehensive testing without a complete understanding of the platform’s business logic and its scope. Back then, we had only a few test cases to start with, and we worked with an older version of Evolv’s web app. Later on, we started testing their brand new desktop app – Web Editor – designed for developing UX improvements for clients’ websites, and a web app – Manager – designed for launching and managing AI-driven testing as well as viewing all the relevant statistics on the experiments. One of our biggest challenges was to validate that the A/B test is designed correctly, that the UX experiment goes as intended, and that the Manager app shows accurate reports.

All the QA services we provide for Evolv can be grouped into the following categories:

  • Testing UX improvements for the experiments run by Evolv’s clients – offering custom-tailored UX solutions for each visitor and customer is an essential component of Evolv’s digital growth strategy. Our task in this part is to make sure all the suggested UX changes for digital growth optimization are implemented properly.
  • Testing product releases + regression testing – Evolv releases software updates every month, so we regularly test new features, bug fixes, and perform regression testing of the core functionality to keep the platform running 24/7.
  • Automated testing – our major challenge in this domain is to automate regression and smoke tests so that they could be run in the CI/CD pipeline. We also strive to create easily maintainable and reusable tests with a focus on codeless test automation.

Solution

The obstacles we faced at the initial testing stage were quickly resolved thanks to close communication with Evolv’s team and thorough documentation. While we were writing test cases in CucumberStudio, Evolv’s team prepared debugging documents for us to better understand the platform’s logic. Once Evolv transitioned to a new platform, it was easier for us to test it because we were part of every step of its development, so we knew the architecture and all the logic behind it very well.

So far, we have finished a set of regression test cases that cover the platform’s core functionality. We perform regression testing for every release, and we managed to speed up this process from 3-4 days to 2.

We are also excited about extending our services to the domain of automated testing, which significantly reduces the time our QAwerk team spends on checking the main functionality in every new build. When implementing automated tests, QA engineers often have to deal with continuous code editing or complete code rewriting to get automated tests running. The challenge centers on the need for the platform to continue being developed and further upgraded and how the user journey may be changed multiple times in the process. Therefore, each change in the user journey will lead to respective edits in the test automation code, which consumes time and effort. We found a more productive approach to automating regression and smoke tests, which allows us to write maintainable and reusable code and easily create new test scenarios without writing any code: we only form new sets of scenarios in CucumberStudio.

Test Automation

Let’s dive deeper into how we came to almost no-code test automation. Initially, Evolv relied on CucumberStudio as their BDD collaboration tool. When we started working with Evolv, they initially used it to write and run some manual test cases and scenarios. On our end, we relied on CodeceptJS, namely Puppeteer, for the automation project. Yet, we wanted to leverage the benefits of both tools and optimize the automation process and make it time-efficient and transparent even for employees with weak or no programming backgrounds.

Combining CucumberStudio’s automation capabilities, such as tests export for BDD frameworks like CodeceptJS and .feature files import, with some Javascript magic (AST parsing/code generation), we’ve built a tool to automatically generate glue code for CucumberStudio and CodeceptJS-based automation project back-and-forth integration. This setup allows easy integration into virtually any CI/CD pipeline.

With this approach, it is possible to significantly decrease the amount of code a QA engineer has to write to automate common test scenarios. On top of that, this setup allows manual QA engineers to contribute to test automation with minimum time spent on onboarding. Here is what they can do:

  • Build up new test scenarios in CucumberStudio project (into which .feature file generated by our custom tool had been imported before) – using about 100-plus Action Words and Test Scenario Context (variables)
  • Add newly built test scenarios to a test runner in CucumberStudio (with CI/CD pipeline already configured)
  • Verify newly built test scenarios – passes in CI/CD
  • Refactor low-level (group) Action Words in newly built test scenarios to business-logic-level ones – aka true Gherkin Phrases in CucumberStudio

It’s no breaking news that test automation can easily defeat its purpose if done incorrectly. Fixing test inconsistencies due to changes in GUI (which happen quite often) is one of the major pain points in test automation. If a button or a text box are moved, the element locator won’t find it, resulting in a failed test run. To make tests green again, the automation tester will have to modify code and push/PR/merge the changes – quite a time-consuming process.

Our approach to test automation allows reducing the time needed for such (and many other kinds of) fixes, be it changes in locators’ names or steps order. Manual QA engineers can efficiently perform this kind of edits by correcting test steps in CucumberStudio and synchronizing these changes with test runs.

Bugs Found

The biggest portion of our time goes on testing Evolv’s SaaS platform consisting of Web Editor and Manager; therefore, the bugs displayed refer to these apps’ functionality.

Actual result: Combination report in Manager shows an incorrect visitor count.
Total Unique Visitors for the combination is a negative number.

Expected result: Combination report in Manager shows a correct visitor count.

Actual result: QA mode in Web Editor: selected UX changes cannot be applied and are visible only in the Edit Mode.

Expected result: QA mode in Web Editor: selected UX changes are successfully applied and are visible in QA mode.

Actual result: Unable to create New Organization in Manager, receiving: message: {statusCode: 403, error: “Forbidden”, message: “Forbidden resource”}

Expected result: It’s possible to create New Organization in Manager; no error messages are received.

Result

Having a reliable technology partner is crucial to a company’s continuous growth. With the joint effort of both teams, Evolv managed to release its new product – Evolv 1.0., which allows their clients to conduct even more elaborate UX experiments. We are proud to be part of such an innovative project and continue working tirelessly to assure Evolv’s long lasting success.

Acquisition

Initially, we provided QA services for Sentient Technologies, a leading AI-based technology company specializing in evolutionary intelligence. In 2019, Evolv acquired Sentient Ascend™, a world-famous conversion optimization platform, the result of a spin-off from Sentient Technologies Holdings Ltd. After the acquisition, we preserved our contract and now provide software testing services to Evolv.

Awarded

Best Practices Award, Frost & Sullivan

Frost & Sullivan,
Best Practices Award 2020

There's a real commitment to get the task done in a timeframe that is expected. The quality of the work is very high. I would certainly recommend working with QAwerk's team

Robert Severn, VP of Engineering at Evolv Technologies

Need experienced QA engineers to speed up your test automation?

Let’s talk

29

people engaged in the project

10

QA engineers

200+

test cases written

100+

UX experiments tested

QAwerk Team Comment
Denys

Denys
QA engineer

What makes the project truly exciting for me is the tech stack I get to work with: React, TypeScript, Material UI, MobX, GraphQL, Apollo, RxJS, VX, D3, Formik, Auth0, Electron, Jest. While testing, I had to dive deeper into some of these technologies, which gave me a great technical boost. I also really appreciate the opportunity to perform both manual and automated testing, which allows me to comprehend the platform better, and as a result, achieve the set goals faster. As for test automation, the CucumberStudio +CodeceptJS +Puppeteer combination turned out to work very well, allowing us to create tests as BDD scenarios and achieve simple implementation of the complex logic using action words. In terms of communication, I have slightly shifted my work hours so that I can communicate with Evolv’s team in real-time, which makes our cooperation much more efficient.

Evolv
Evolv
Evolv
Evolv
Evolv

Impressed?

Hire us

Other Case Studies

United States
Mobile toolkit for storytellers
France
Desktop app unifying work tools
United States
Moving services marketplace