Improve your test execution time

Writing tests is just the first step.

We need to run tests with every change we add in the code. The smaller the changes, the faster we can identify and fix issues. The problem with large test suites is that they usually take a longer time to execute. We will analyse a few methods on how to speedup our test execution.

There are different kind of tests you write to increase the chances to catch bugs before reaching production.

Unit tests

Usually you are using a mocking library like Mockito, EasyMock or similar. They help you create a specific environment where the code you added acts in a specific way. All the tests should be independent, but these should be also fast and easy to run.

In this very simple example we verify our service delegates the save call to the repository. We let Mockito handle the test setup — we don’t create any object, we just tell the library we are going to test our personService which may depend on some other objects. By default if there aren’t mocks defined we should expect null values in our target object.

If you are new to Mockito, checkout the Mockito javadoc. It provides useful details and examples about many use cases.

Integration tests

We usually group our logic in layers: the persistance layer to store data, service layer with business logic, controllers that map specific resources to some endpoints etc. In an integration test we execute code across multiple layers. The more layers we add, the better the coverage we get. Unfortunately the time execution degrades and it becomes messy to have a simple setup. One can also choose to connect to a dedicated (in-memory?) database, event bus etc. The price we pay for these involves time and resources so having something very simple and easy to run should be wise.

Spring Boot has a solution with the @ SpringBootTest annotation. It starts a Test application context similar with the real one. You can fine tune it to include a web environment or not, have some specific properties or profiles activated or include only some specific beans. These variations can help us find a simple setup to test the code we add. Other options are @ DataJpaTest, @ WebMvcTest or your own.

In this sample project we use an embedded cassandra to store the data we get from a kafka instance. I generated many tests to mimic a real word example application. This helps us understand which are the methods we can use to speed up our test execution (in the pipeline)

In the example below, we spin up a local kafka instance, we send a message with a person payload and we verify our service saved the person. We could also use an embedded cassandra instance to verify all the fields were properly mapped to the Person table. However we can leave this test for the PersonRepository test.

One important aspect in this test is the asynchronous nature of the send/receive methods. With the @ MockBean annotation we can inject our test logic into the service.save method to wait for it to be called. This is possible because our KafkaConsumer is stateless and it delegates the processing logic to the service layer that can be tested separately.

We can see the test took around ~500ms in our IDE, but if we read the logs we can see the entire time was ~17 seconds (Test started at 15:24:32.224 and finished at 15:24:49.361) This happens because Spring initialises a Test Application context which takes time — remember we create the entire ApplicationContext (including a local Kafka instance, a web server, an embedded cassandra instance — not being used and any other thing our application uses)

If we add (webEnvironment = SpringBootTest.WebEnvironment.NONE) we can see a small improvement.

A bigger improvement is to include in the test application context only the things we actually need. With the change below we see it took around ~500ms to run the code but the entire test duration dropped to ~3 seconds (start: 15:42:38.036, finish 15:42:41.393)

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.NONE,
classes = {KafkaAutoConfiguration.class, KafkaConsumer.class, KafkaProducer.class})

We can do the same to test our PersonRepository. We run an embedded cassandra instance to execute some cql code.

We can notice a significant time difference between the two test executions. This happens because Spring Test caches the application context from the first test and reuses it for the second run. You may want to check this spring.test.context.cache.maxSize property to see if it helps in your application.

To validate this, I created 50 tests for each example (cassandra/kafka) and we can still see a very fast execution for the new tests.

Running our tests in the pipeline could be done by using a maven plugin. In this example we will use maven-surefire-plugin

It provides some useful configuration properties to achieve a faster test execution.

• Run multiple JVM instances depending on number of CPU cores available.

<forkCount>4C</forkCount>
<reuseForks>true</reuseForks>

• Run multiple threads

<parallel>all</parallel>

While this approach sounds like a good idea, it is important to understand some consequences :

• if we start additional services it may be a good idea to share the same instance and separate the test data on the topic level (for kafka) or schemas for databases (if we can’t automatically rollback our test changes)
• we need to use different ports for each new service. Each JVM starts it’s own embedded local instances. We can use Spel to generate some random values.

@Value("${random.int[9000,9999]}")
int cqlPort;

• reusing the same test application context can be faster than creating a new jvm instance that creates the same context again and again.
• if you are using wiremock to mock some external APIs with the spring-cloud-contract-wiremock library you can easily inject the random port into your http client using a magic Spel

${wiremock.server.https-port}

• troubleshooting errors when running multiple tests in the same time can lead to some (long) troubleshooting sessions.

Don’t forget to check the *.dump files

If your application starts very slow, you may want to also increase your visibility on which are the steps to create the application context. By default, spring uses a no-op ApplicationStartup implementation that you can change.

For example, adding a FlightRecorderApplicationStartup instance to your application can generate custom events with the time it took for each related step.

Another alternative is to use the BufferingApplicationStartup and see the steps from the /actuator/startup endpoint.

There isn’t a general solution for all the problems. It is important to understand which are the tools you can use and gradually improve your testing code that hopefully saves you some time.