The Automation Journey of Kilo Health

By Justinas Židonis, Head of QA at Kilo Health

We created the QA Department at Kilo Health nearly 3 years ago. Since the beginning, we have put a high emphasis on automation. Test automation provides confidence in your product and enables you to not only move faster in your release cycle but also spend the time you saved in a more productive way. 

With this blog post, I would like to share our ongoing journey, providing you with insights and reasoning behind our decisions, and, hopefully, give you a headstart with your journey to web automation.

The beginning

I was the first QA engineer at Kilo Health. When I joined, there were no automation tests or solid infrastructure to support it. 

From the beginning, I was eager to start automation testing, but there were a few challenges ahead:

1. Products were not testable. All the attributes of HTML elements were auto-generated, making it difficult to get the elements automation would interact with. However, we did have testing environments (staging, dev) to test on.
2. We had no infrastructure to execute tests on. Executing tests on your local environment is less than ideal, and your machine would never win the “resources war” against the server. 
3. Developers were not familiar with this testing technique in practice. Most of the people in IT do know about things like TDD or automated testing; however, not all have work experience with them. 

Obviously, during the initial implementation of automation, many more challenges arose, but these three were the main ones to tackle first.

#1 – Tackling the not testable products

Sounds cliche, but to test a product, it has to be testable. 

This goes both for manual and automated testing. It encapsulates a whole bunch of things, but when focusing on automation, there are only a few more important than selectors. 

Automatically generated attributes or selecting an element by text are the best ways to make your tests even flakier. To avoid that, we want a particular element to have its own unique selector. 

To address this, I made a simple document with screenshots of all elements which need selectors in our product. A task was made out of the document, and a few weeks later, I had all the selectors I needed to fully automate our product End-to-End.  

Another thing I ran into while trying out multiple threads while running automation was insufficient resources for testing environment servers. Make sure that your testing environment will be able to keep up with automation and be able to process multiple browsers hitting that system at the same time. 

This often gets overlooked but is a key in avoiding test flakiness as well. 

#2 – Setting up the infrastructure

There are numerous reasons why you should always execute your automation on dedicated servers, but, in my opinion, the most important are:

• Having a whole lot more resources at your disposal
• Better time management while tests are running
• Stable and unchanging execution environments

With automation, you want to save time. That’s a fact. Executing on dedicated servers enables you to use multithreading and run multiple browsers at the same time while keeping your own computing resources available to be spent elsewhere. 

Also, it does help with debugging since the execution environment is always the same, and there are fewer “I cannot reproduce it locally” types of arguments. 

Addressing this challenge, there are quite a few paths to choose from. You can either go with cloud-stored solutions to execute your tests or save the initial setup. But the trade-off can mean that you are sacrificing potential customization capabilities. Price may be another factor since, in the beginning, you are not sure if all the fancy features are really needed.

I think that you’ve already figured it out, but we did not go with a cloud-stored solution – we set up our execution server ourselves on Amazon AWS. 

We did spend a reasonable amount of time on the setup and configuration but had the ability to configure everything to fit your needs. The technology we chose for this one is TeamCity. Not only is it free (if 3 run agents are enough), but the setup was not complicated, and the feature-rich administration panel helped configure everything. 

#3 – Onboarding the team

The IT field is constantly changing, and new technologies and methodologies are coming up constantly. So it is impossible to have a working knowledge of everything new. This is true for automation as well. 

By default, it is not necessary to “sell” automation to developers (project managers are a different beast in this regard) because they do understand the benefits. However, depending on your situation, proving that automation really brings value may be essential as well. The easiest way to prove that is for your tests to catch some bugs. 

The bigger the impact of a caught bug, the bigger the “visible” value. In my situation, I got lucky to be placed in a professional and eager-to-improve team.

After the benefits of automation are clear, there are a few things from the QA side of things you can do to onboard developers to automation:

Choose a programming language your team is familiar with

Most of the developers were working or had knowledge of PHP language; therefore, the Codeception framework was chosen. 

As a tester, I think that if you know how to automate things in one language, it is quite easy to jump to another one. But when it’s completely new to you, you get a headstart when it’s written in the form you understand. Also, the stack trace of a failed test becomes a whole lot more understandable. 

Make it easy to run and understand the outcome

Make sure that your test framework is easily run and provides you with an output (screenshots, videos, etc.). Shocker, but people tend to use things when they are easy to use. 

To address this point, I made a Slack bot that, after typing a command in a channel, would send a GET request to TeamCity and automatically start tests. After the execution, the outcome would also be posted to Slack (out-of-the-box TeamCity feature). 

This means that tests can be executed and the result seen inside a Slack channel. 

Avoid flakiness at all costs

There are multiple techniques and approaches to avoid flakiness, and you, as a QA, should strive to make your tests as stable as possible. 

If we do have flaky tests, there is a possibility of developers getting “used to the red color” of a falling test and start ignoring the outcome. Tests should provide confidence in a product and give feedback to the developer that his changes did not break anything. 

We should never lose that confidence. 

Show that automation is a tool, not a bottleneck

Constantly updating your framework, adding new features and test types that were requested by the team, and reacting fast to changes help show the value and potential of automation.

What will be your results?

To sum up, when introducing an automation technique to a team where it did not exist before, make sure that:

• Products can be tested – Elements you will interact with have unique selectors, and testing environments are set up and able to function under a heavy load.
• There is an infrastructure to support it – We need a dedicated server and a system to run tests on. Ideally, that system should be customizable and provide different ways to start execution and get test results afterward.
• Devs are onboarded – Make sure developers understand the benefits of having automated tests, they feel comfortable with the code and stack traces, they know how to execute tests and how to understand the output. 
• Everything is easy to use and can be tailored to your needs – From the starting execution to getting the results back – everything should not be a hassle to use. Any custom solutions (e.g., start a test from Slack, use custom reporting, etc.) are more than welcome here. 
• The tests are not flaky – We do not want a failure to become the new normal. A test should fail only when the condition is met. Focusing on reducing flakiness is one of the most important things when introducing automated testing to a team.

The forward-thinking mentality

To recap, we’ve already had working automation tests, infrastructure, reporting, and the right kind of attitude to make everything work in our processes. 

However, as the team grew and the rate of development increased, the demand to be even more flexible, reduce test running time, and start adopting CI/CD pushed us to improve our automation setup.

To reduce running time and have more testing and reporting capabilities, it was decided to move from Codeception to Playwright. NodeJs was chosen as a programming language because it is an asynchronous, open-source, and cross-platform JavaScript runtime environment. 

This was a great choice for multiple reasons:

1. All the developers in our team knew JavaScript.
2. Asynchronous programming in the area of automation testing does help a lot since if we start an action, our tests are waiting for its resolution.
3. Along with NodeJs, npm came, which enabled us to be even more dynamic regarding the test types we are using. For example, if you wanted to make automated lighthouse reporting, you were only one package away from implementing it.
4. Cross-platform was very important as well because our Agents in TeamCity are Linux, I was developing with macOS, and a few of the team members had Windows. Tests were working no matter the environment.
5. Playwright itself. This framework is being updated regularly, and new features are coming up constantly. It does support multiple browsers and multiple devices to test on, has great reporting capabilities, has mocking of services, is easily configurable and integrable into CI/CD, and lots more. I may be biased, but I really recommend any Tester or Developer to give it a try.

Since we were not prepared to adopt CI/CD just yet, I spent lots of time improving the current framework and introducing new test types:

• Visual testing. With Playwright, it came out of the box – no additional services or dependencies were needed. 
• Automated Lighthouse reporting. I am sure most of you know that you can run lighthouse reports straight from your browser. However, Playwright’s flexibility allowed us to produce such reports in an automated way with the help of the playwright-lighthouse package.
• API testing. Previously, we tested our APIs using Postman and even had tests there (yes, you can have automated tests with Postman – read more here). But now that we had npm, we were one package away from enabling our framework to test API. With the help of Axios, we did just that.

During this time, I learned a lot. I saw the direct impact of having a very flexible tool and the freedom to make decisions and not be constrained by the processes. This, again, proved to me that automation is a tool and should be treated as such. 

I was in a situation where having an automation framework and processes around it made everyday tasks take too much time, and making custom and reactive decisions to ever-changing products became nearly impossible. 

The tool should help you achieve your goals easily. If a tool starts interfering with your goals and starts slowing you down, you cannot quickly implement new features or tests – it’s time for a change. In other aspects of life, people are skeptical of changes; however, in the IT field, it should be embraced and welcomed because it enables us to move forward and improve.

The CI/CD Saga

In my opinion, the natural progression of development is when a team adopts CI/CD, which, in a few words, is a modern software development practice in which incremental code changes are made frequently and reliably. 

In CI/CD pipelines, you do have some static code checks and Unit test jobs, but in our case, we wanted to expand it to include E2E automation suites as well. Since our framework was written in NodeJS and had a package.json file, we were `npm install` away from installing all the dependencies needed to execute tests anywhere. 

To integrate your automation tests into the pipeline, there are only a few steps from the development side of things:

• Prepare Docker image. If your tests are in a different repository than the project you are testing, you need to add two files to your automation framework:
  – gitlab-ci file to tell the gitlab to build your docker image (Playwright has a docker image)
  – Dockerfile to include all the commands needed to assemble an image
• Prepare script files in your current automation framework, which would change variables before launching tests. For example, you may want to change the URL of an app you are testing or use some kind of proxy, or execute not all of the tests.
• Create custom testing jobs in the pipeline. Pretty straightforward; just do not forget to add your job to the correct stages, add dependencies (for example, you may want to environment build before launching tests), and set up artifacts that would include all the tests’ output. 
• Create custom reporting jobs. Ideally, we want to receive some kind of notification after the tests and pipeline itself are run. For that purpose, we created a Slack bot that would post an outcome of tests to a channel with a hyperlink to access it. This goes back to my point that providing a team with easy-to-use/one-click solutions drastically increases the adoption of a solution.  

There are also workflow-related aspects you should consider. For example, when running a CI/CD pipeline, you want to get feedback as soon as possible, so running the entire suite of your tests as a default step may not be an ideal solution because the run time would increase dramatically. We, by default, run only regression tests and leave the ability to manually launch the entire suite. 

Furthermore, what if we needed to push a hotfix quickly? Waiting for an entire pipeline to succeed and all the tests to pass may not be an option. In such cases, with the magic of GitLab’s CI/CD job variables, we allowed developers to skip these steps if they used special tags in the commit messages. 

What if we wanted to enable different types of tests for different projects (e.g., front-end vs. back-end)? Again, GitLab’s job variables came to the rescue. 

Using CI/CD pipelines not only made us move a lot faster in the development process, but it did not take away any flexibility we had before that. Our code is constantly tested, and if we need any custom solutions, they are easily implemented. Having workflows is good, but you have to remember to always improve those workflows because, again, change in IT is a good thing.

Lastly, I wanted to point out that in this paragraph, I talked about QA automation only. 

We expanded current pipelines with a focus on quality and not increasing run times by a lot. We still have a TeamCity to run tests on-demand, but a move to CI/CD-focused testing, in my opinion, will enable us to move even faster and in a confident, quality-focused way. 

To sum up

It’s been a long journey, and there is a long way to go. During the past 3 years, our approach to automation has changed a lot. We learned from our mistakes, improved on them, and planned for the future. 

We tried to set ourselves up to be able to adapt to the ever-changing development sphere, introduce changes quickly, improve current solutions, and not be prisoners of workflows and processes. You have to have a plan and be able to react quickly.

In this blog post, I focused more on a theory and the experiences that worked for us. 

Your journey may be different, but I still hope you’ll find something useful and it will help you hit the ground running.

If you are interested in technological solutions, I’d be happy to talk about them via LinkedIn PM.