Let’s continue with our blog series “Treat your processes like code – test them!” Today we will take a closer look at a frequently discussed topic: Test coverage. But what role does coverage play in testing BPMN models and how can coverage be measured?

In this post we will address the following questions:

  • Why do we need coverage?
  • How can we measure coverage in BPMN processes?
  • How can we monitor the coverage in a transparent way?
  • This post will also give a short preview of how Camunda BPM Process Test Coverage is evolving.

For this purpose, we are going to extend the example from our previous posts. You can find it in our GitHub repository. If you haven’t read them already, check out the other posts in this series:

Why is test coverage an important metric for developing automated processes? The short answer is: From a technical perspective, BPMN is a programming language, so it should be treated as one. High test coverage brings many benefits, for example:

  • Greater confidence in the model: Automated processes include workflow logic, dependencies on data, and can grow over time. If there is no certainty that all paths will still work when a change is made, this has a negative impact on agility. This is because a lack of trust in a change to the model often means that new versions are deployed less frequently and acceptance tests become more costly.
  • Errors are detected at an early stage: When testing models in detail, technical and functional errors in the model can be detected at an early stage. After all, not only errors in the process logic are relevant. Testing at the technical level often reveals states in the model that are not desired from a business perspective.
  • Tests are definitely executed: If test coverage is included as a key figure in the development and integrated into the pipeline, then there is no way around executing the tests. This creates a higher level of confidence in the deployment. In addition, seamless and automated integration means that measuring this metric is not an additional effort.

But how can we monitor test coverage when developing automated processes?

There is a community extension for this: Camunda BPM Process Test Coverage. It visualizes and checks the coverage of the process model. Running JUnit tests creates HTML files in the build folder. These look like this:

The extension offers the following features:

  • Visualization of the results of test methods and test classes in the BPMN model
  • Visualization of expressions on sequence flows and service tasks
  • Visualization of transaction boundaries
  • Calculation and visualization of the degree of coverage

Let’s extend our example. For this, we first check out the branch 02_process_dependencies in our GitHub repository. There are only three steps required for the initial setup:

Adding the maven dependencies

1. First, we have to add the following dependency to our project’s pom.xml:

1.	<dependency>
2.	  <groupId>org.camunda.bpm.extension</groupId>
3.	  <artifactId>camunda-bpm-process-test-coverage</artifactId>
4.	  <version>0.3.2</version>
5.	  <scope>test</scope>
</dependency>

2. Updating the camunda-cfg.xml file:

1.	<bean id="processEngineConfiguration"
2.	   class="org.camunda.bpm.extension.process_test_coverage.junit.rules.ProcessCoverageInMemProcessEngineConfiguration">
3.	   ...
</bean>

3. Using TestCoverageProcessEngineRule in the test:

For this, let’s customize our WorkflowTest and change the @Rule as follows:

1.	    @Rule
2.	    @ClassRule
3.	    public static TestCoverageProcessEngineRule rule = TestCoverageProcessEngineRuleBuilder.create()
4.	            .excludeProcessDefinitionKeys(DELIVERY_PROCESS_KEY)
            .build();

It is important that we exclude the Delivery Process from the coverage. This is because it is only provided as a mock in the test and is therefore not available.

Now we can run the test. The test results will end up in ./target/process-test-coverage.

As we can see, we reached 95% coverage even though everything was tested successfully. However, this is due to a bug in the coverage library that does not track compensation events properly.

Defining the minimum coverage

Let’s now have a look at how we can define the minimum coverage that will be automatically monitored. We have two options for this:

  • Coverage on the class level
  • Coverage on the method level

These can also be combined with each other. At the class level we can define the coverage as follows:

    @Rule
    @ClassRule
    public static TestCoverageProcessEngineRule rule = TestCoverageProcessEngineRuleBuilder.create()
            .excludeProcessDefinitionKeys(DELIVERY_PROCESS_KEY)
            .assertClassCoverageAtLeast(0.9)
            .build();

Requiring test coverage of 1.0, meaning 100%, is not recommended either because of the previously mentioned bugs. At the method level, we can add a coverage requirement as follows:

    @Test
    public void shouldExecuteHappyPath() {
        Scenario.run(testOrderProcess)
                .startByKey(PROCESS_KEY, withVariables(VAR_CUSTOMER, "john"))
                .execute();

        verify(testOrderProcess)
                .hasFinished(END_EVENT_ORDER_FULLFILLED);

        rule.addTestMethodCoverageAssertionMatcher("shouldExecuteHappyPath", greaterThanOrEqualTo(0.5));
    }

There are several org.hamcrest.Matchers available, for this use case the following are helpful:

  • greaterThan
  • greaterThanOrEqualTo
  • lessThan
  • lessThanOrEqualTo

The Camunda BPM Process Test Coverage offers a few more features, like switching off the coverage on class or method level. It is worth to have a look at the GitHub repository.

Traceable test coverage

The graphical representation of the test coverage in the form of HTML files is sufficient for local development. However, there are specifically two areas where the library reaches its limits:

Traceability

Test coverage is a great metric if we measure it. But measuring it also means keeping track of historical data. It’s about making improvements or increasing transparency to provide motivation and traceability. Therefore, we need a solution to collect and graphically present the reports we create.

CI/CD Integration

On the one hand, we have local development. After the process has been adapted, the corresponding test is implemented and executed, the test result is checked and then the whole process starts again. Once the model and test are completed, the corresponding code is pushed to Git.

The test is then run again in the build pipeline. But how are the results of a pull request, for example, visualized? How can a change in coverage compared to the previous state be tracked? With Camunda BPM Process Test Coverage this is not possible out-of-the-box.

These are just some of the reasons why we developed FlowCov – a platform that covers precisely these requirements. It enables you to upload the generated reports as part of your own build pipeline in order to process them graphically and make them accessible to all stakeholders. If you want to learn more about it, we recommend our Docs.

Further development

At the moment the Camunda BPM Process Test Coverage is being completely re-developed. Thus various enhancements and features will be implemented:

  • Compatibility with JUnit5
  • Splitting up coverage measurement and report generation
  • Redesign of the local HTML report
  • Seamless integration in FlowCov (without custom extension)
  • Possible usage for integration tests

If you want to participate and bring in your own ideas, we would be happy to receive contributions and discussions! The development takes place in this fork: https://github.com/holunda-io/camunda-bpm-process-test-coverage/tree/develop/extension

If you’d like to dive deeper into testing topics, check out Testing Cheesecake – Integrate Your Test Reports Easily with FlowCov, presented at CamundaCon LIVE 2020.2 and available free on-demand.

Dominik Horn is Co-Founder of FlowSquad — specialists in process automation and individual software development. Stay in touch with Flowsquad on TwitterGitHub or email — info@flowsquad.io.

  • Welcome to the Camunda Community Summit

    Camunda has always been focused on creating an excellent experience for developers everywhere who are trying to automate processes. From our days as a small process automation consulting firm to the powerful products used by hundreds around the globe, our engineering mindset and commitment to our community has always been at the heart of what we do.  Out of this commitment to our community comes our first Camunda Community Summit — coming live to your laptop on April 27-28. This inaugural summit builds upon our first Community Unconference last October. The feedback we received was tremendous and people have been asking for another event like it ever since. This virtual conference, created by developers for developers, offers peer-to-peer engagement and...

    Read more
  • chor-js – an Editor for BPMN Choreography...

    Let’s say you are craving your favorite pizza from that pizza place a ten-minute walk away. You call them up, place your order, and are then asked whether you would like to have the pizza delivered or pick it up yourself. In your mind, you think about how convenient it would be to have a delivery driver knock on your door and hand you the pizza within half an hour or so, but then again it is only a short walk, and some fresh air would really do you good… Putting somewhat existential questions aside, what we just described is a prime example for a so-called choreography in Business Process Management. Different participants (you, the pizza place and the delivery...

    Read more
  • Announcing Camunda for Common Good

    Helping NGOs and Non-Profits around the world reach even more individuals Mission-driven organizations are often short-staffed, insufficiently funded, and underestimated. Our new program, Camunda for Common Good, offers an opportunity to give back to organizations that are making a tangible difference in the lives of people around the world. There’s a need for process automation and making operations more efficient no matter what industry you’re in and no matter who you’re serving, but the potential for NGOs and non-profit organizations, in particular, to vastly increase their impact on the world through automation, is compelling. With the launch of Camunda for Common Good, we’re bringing our powerful process automation platform, combined with our expert knowledge, to support qualified NGOs and non-profit...

    Read more