Couple Up! Love Show Story

Challenge

Native Games Studio turned to QAwerk to address the challenges of a growing user base. They wanted to ensure the Couple Up! game wouldn’t be sluggish or crashy under a substantially increased influx of players.

Here is what we were expected to deliver: 

• Load Testing. We needed to test several API endpoints, gradually increasing server requests and tracking the server response time. Load testing helps identify the app’s peak operating capacity and breaking point and make necessary adjustments before users experience any negative issues.
•  Server Configuration Audit. Our task was to scrutinize their Hetzner server configurations, such as checking network bottlenecks, nginx configuration, auditing docker containers, kernel and service settings. This part was performed by our DevOps engineer.
• Code Review. We were also asked to review the server code and point out areas for improvement. We checked it against criteria like code quality, error handling, caching, and testability. Since the code was a REST API built with Flask, it was reviewed by our senior Python developer.

Before proceeding with the project, we provided the Couple Up! team with an estimation of our efforts to bring clarity and transparency to our cooperation. We also asked them to fill the gaps in the API documentation so that we had an accurate understanding of the underlying infrastructure. 

Solution

Load Testing

We tested one GET and three POST API calls to measure their performance under varied load conditions. Load testing is usually done with the help of a tool that simulates different numbers of users interacting with your app simultaneously or sequential increase in the app’s demand. Our QA engineers chose Apache JMeter because it’s a reliable open-source tool, specifically designed to make load testing rapid yet effective.

One problem we faced was an improper description of API headers. The latter are vital in API testing because they contain metadata about the API request and response. They also help the API understand your request and retrieve the exact information in the needed format.

We solved this issue by writing a script that allowed us to describe the requests in JMeter correctly. Having analyzed the system, expected and peak load, we set respective test data parameters.

Here are the test cases we used:

• Add 1 thread (user) in 1 sec
• Add 100 threads (users) in 1 sec
• Add 1000 threads (users) in 10 sec
• Add 10000 threads(users) in 1000 sec

We measured the server’s performance in terms of response times over time, active threads over time, and internal server error rates.

Server Configuration Audit

Our server configuration audit aimed at identifying opportunities for improving the server’s performance. Since mongod processes cause the main CPU & RAM load on the server, their work should be optimized.

We provided the Couple Up! Team with a list of exact configurations that needed to be set in place. For example, we broke down how to tweak Linux kernel settings, such as max map count, swappiness, dirty_ratio, max number of queued connections, local port range, and several others, in relation to the anticipated load of 50K connections.

To reduce the CPU & RAM load, we advised our client to check how queries are formed and optimize those running slow and parallelize queries of the same type. We also suggested indexing and re-indexing routinely-queried fields and doing this on time. With the right indexing strategy, you can significantly increase your database performance.

Additionally, we mentioned what default configurations should be changed and how.

Code Review

The QAwerk team performed a comprehensive code review and provided Couple Up! with quick and cost-effective action steps as well as significant architectural changes for better performance and scalability long term.

We highlighted the areas in need of refactoring to help the Couple Up! developers maintain, add new business logic, and scale the code without pain in the future. We noted the need for unit and integration tests because they will make refactoring and expanding the code base easier.

Also, we recommended improving error handling so that the error messages were more informative and always showed correct status codes. This would simplify debugging and testing and help standardize the app’s communication with the API.

Proper caching can significantly boost performance for a fraction of the cost compared to migrating to Amazon. That’s why we offered our client to implement caching.

Last but certainly not least is the CI/CD pipeline. In case frequent releases are planned, CI/CD is essential for catching bugs before deploying updates and speeding up the delivery of new capabilities to users. So CI/CD setup was also on the list of our recommendations.