Penpot

Challenge

Penpot turned to QAwerk to make sure they could launch their product to the general public within the set timeframe. They wanted their in-house team to be laser-focused on developing value-adding features and polishing the product functionality rather than worrying about bugs and ways to capture them. 

Here is a brief overview of the things we were expected to deliver:

• Devise test strategy & plan. Before partnering with QAwerk, the Penpot team reported bugs with the help of internal team members and their large community of beta version users. With Penpot’s growth in popularity and intention to remove the beta label from their app, this approach was no longer sustainable. That’s why we were asked to plan the entire testing effort and establish proper QA workflows to lay the foundation for ongoing professional testing in the future.
• Go from beta to official release. Eliminating bugs and improving performance before releasing the product to the public as commercially ready was paramount for Penpot. Our task here was to methodically go through all the existing functionality and find as many issues as possible to ensure no critical bugs were still in the prod and the app runs without lags or data-saving problems. 
• Polish new functionality. The production release included some long-awaited killer features, such as flexible interfaces, accessing code right from the workspace area, and accessibility improvements, among other things. Our task was to make sure these upgrades were integrated seamlessly, caused no disruptions to the existing functionality, and made users genuinely excited about using Penpot.

Besides that, we wanted to help Penpot achieve better coverage and release updates with peace of mind. So we offered them to enhance manual testing with test automation.

Solution

When we just started testing, we encountered a great deal of internal server errors that would appear quite randomly, and it wasn’t clear what caused this behavior initially. These bugs required immediate fixing since users experiencing them could lose their work.

So we would record videos to capture the exact steps to reproduce this type of bug, trying to find a common pattern between such occurrences and figure out what user actions could have triggered the errors and what app areas are most prone to them. We would share our findings with Penpot’s developers, and eventually, they eradicated this problem entirely. 

Our testing effort included several testing types, such as:

• Functional Testing. We checked if the user authorization went smoothly. We thoroughly tested the user profile, dashboard, and workspace modules. We made sure all changes were applied to the design without any lags, especially during collaborative editing. 
• Regression Testing. The Penpot team continuously ships new features and upgrades of the existing features, which necessitates regular regression testing. We combine both manual and automated regression testing to achieve the best coverage and consistent results.
• UI Testing. The sleek visual appearance of the user interface is paramount for any modern app, especially for a design tool like Penpot used by UI/UX and product professionals at large corporations. We checked element positioning and alignment on every interface and ensured UI patterns were consistent throughout the app. 
• Compatibility Testing. Our QA engineers tested Penpot in three major browsers – Firefox, Chrome, and Safari. We tested Penpot in several Safari versions, as some bugs appeared only in older versions. We also used three different operating systems, Windows, macOS, and Linux, to ensure all users can benefit from Penpot regardless of their devices.

As for test automation, we primarily focused on regression test cases.

Test Automation

Test automation was performed using the Playwright framework. Playwright is designed to simplify cross-browser and cross-platform automation, so it suits our needs well. 

When selecting test cases for automation, we followed two rules: test cases should be of high impact, and their preparation shouldn’t consume too much time. This way, you can see a positive ROI.

For example, we didn’t automate cases like sign-up or forgot password because here you have to deal with third-party APIs such as Gmail, and automating this flow is not worth the effort. Or cases with drag-and-drop or copy-paste actions because these are not always handled correctly by Playwright and tests can be unstable. 

We automated test cases like login, creating and deleting files, opening and renaming files, creating and deleting boards, adding blur to boards, changing the background color, among many others. They are available in Penpot’s public repository. 

We also leveraged Playwright’s visual comparisons for snapshot testing. To minimize the number of failed tests because of UI changes, we used screenshots that captured only a single element or section, such as a created shape. This way, if new sections were added to the design panel, they didn’t affect the test stability.

About 10% of our test cases require capturing the entire screen, for example, when we need to check not only the layers but also the toolbar. Here we used masking to ignore elements with dynamic content, such as timestamps or usernames. Even though it’s impossible to eliminate false positives completely, updating such test cases is easy and quick.