Should data be embedded in business processes? It depends!

In October, I had the opportunity to lead a session at Camunda Unconference.

This was a rather unique event, with an “unconference” format designed specifically for the community; it was most conducive to great peer-to-peer discussions, and exhange of ideas, experiences and learnings.

What was also interesting was that the topics for the sessions were sourced from within the community, and each topic was voted in to a final shortlist. The sessions themselves were discussion-led, to encourage collaboration and creativity.

Data in Process

I had voted for “Data in Process” as one of the topics; it has always intrigued me, and needless to say, I was thrilled at the opportunity to lead the session!

I have been consulting for customers building Cloud Native applications using BPM engines such as Camunda for their business processes, for the past several years. “Control flow” and “data flow” are key to any business process solution that the customers build.

Based on my experience across several projects, I believe designing for “how to handle data in business process” is a particularly complex design problem, and is key to the success of your implementation.

You can create a business process instance that has all the data required. Or you can just pass references to external data, and the business process retrieves the data during run-time. Or you can use a mixed strategy. There really is no ‘one size fits all’ solution to this question.

This personal observation was validated during the session, when one participant mentioned how they discussed “Data in Process” when they began their BPM journey a few years ago.

There are a lot of nuances to the topic irrespective of the BPM engine or tool you use. And it is primarily a design decision.

I structured my discussions around the The Why, The What and The How framework.

The Why

In a business process, data is necessary not just to work with from a business perspective, but to also drive the business process to the correct, next steps. Data can be:

  • input by the users
  • loaded from external systems
  • used for decision making
  • denerated or transformed during the execution of business process

One participant also pointed out how Data for KPI and Reporting is of interest. As you operate business process workflows in production, measuring various KPIs such as average time to complete the workflow, time required for a particular step are all very important. This type of data is inferred based on workflow execution.

The What

The most complex decision while designing a BPM solution, in my experience, has been the type of data, or “what” you want included in the business process.

Do you “store the entire payload,” or “store only references to data in external systems,” or do you use a mixed strategy?

I’ve come to realize the answer is, always, “it depends,” and that practical experiences and learnings across projects is what helps identify and refine that strategy.

As expected, this is the area we spent most of our discussion time. I took back some interesting insights myself, as I learnt from the experiences of the participants.

The discussions were lively, with several of us agreening and disagreeing on the thoughts shared. But this only allowed us to reflect with a keener observation, on our own experiences, and that brought about additional points for us to consider. Here’s how we began:

  • “Store only references to data.” Most participants agreed this is the typical default everyone should turn to.
  • “Store the entire payload.” Choose this approach provided you don’t have any other system of record.
  • In many cases a “mixed strategy” would be ideal.

One point that stood out for me was how availability of  a “source system” was a challenge for the “Store only references to data” option. One participant explained that their source system was a mainframe, and how the workflow process stalled when the source system was unavailable.

Another problem highlighted was the performance of process execution, if we were to access data from a source system each time we needed it.

Ease of Testing / Troubleshooting was a grey area. Some strongly disagreeing that it is easy to do in one approach vs the other. This is primarily a result of number of tools and techniques available to help test and debug activities.

We used a Miro board for our discussion, which I believe is a great tool for collaboration! Here is our final board. The discussions were rich, and we ran out of time before we could arrive at a consensus!

The How

We discussed options such as “process variables,” solutions that allow us to model complex data objects, concurrent access issues and so on. I only wish we had had more time to discuss this aspect a bit more!

At the end of the day, it was an amazing event, made thoroughly engaging and interactive through the “unconference” format!

Thank you, Camunda, the rest of the event organizing team, and participants for the opportunity!


  1. Best Practice document from Camunda. This article provides great information about data in process and how to handle.
  2. Great tool to handle data in process – something that I learnt of during the conference.

This guest blog was originally published on

Rajesh Rajagopalan is Managing Partner at PeerIslands, heading the Cayman operations. A problem solver and a coder at heart. A technology evangelist who brings in a valuable blend of technical and business competency and a passion for innovation. He brings in 20+ years of industry experience architecting solutions, product development; helping customers navigate digital transformation and technology adoption in wide range of areas including Cloud-native microservices, IoT, Big Data and Machine Learning. Built high performing team of developers from diverse cultures working towards common company goals.

  • Welcome to the Camunda Community Summit

    Camunda has always been focused on creating an excellent experience for developers everywhere who are trying to automate processes. From our days as a small process automation consulting firm to the powerful products used by hundreds around the globe, our engineering mindset and commitment to our community has always been at the heart of what we do.  Out of this commitment to our community comes our first Camunda Community Summit — coming live to your laptop on April 27-28. This inaugural summit builds upon our first Community Unconference last October. The feedback we received was tremendous and people have been asking for another event like it ever since. This virtual conference, created by developers for developers, offers peer-to-peer engagement and...

    Read more
  • chor-js – an Editor for BPMN Choreography...

    Let’s say you are craving your favorite pizza from that pizza place a ten-minute walk away. You call them up, place your order, and are then asked whether you would like to have the pizza delivered or pick it up yourself. In your mind, you think about how convenient it would be to have a delivery driver knock on your door and hand you the pizza within half an hour or so, but then again it is only a short walk, and some fresh air would really do you good… Putting somewhat existential questions aside, what we just described is a prime example for a so-called choreography in Business Process Management. Different participants (you, the pizza place and the delivery...

    Read more
  • Announcing Camunda for Common Good

    Helping NGOs and Non-Profits around the world reach even more individuals Mission-driven organizations are often short-staffed, insufficiently funded, and underestimated. Our new program, Camunda for Common Good, offers an opportunity to give back to organizations that are making a tangible difference in the lives of people around the world. There’s a need for process automation and making operations more efficient no matter what industry you’re in and no matter who you’re serving, but the potential for NGOs and non-profit organizations, in particular, to vastly increase their impact on the world through automation, is compelling. With the launch of Camunda for Common Good, we’re bringing our powerful process automation platform, combined with our expert knowledge, to support qualified NGOs and non-profit...

    Read more