In this post, we’ll share at a high level why we think workflow automation is and will continue to be important to organizations of all shapes and sizes and why we created Zeebe.

Workflows Are Everywhere

At Camunda, after more than ten years in the business process automation space, we’ve learned at least one thing: workflows are everywhere.

We define a “workflow” here simply as a sequence of tasks that result in some business outcome. Nearly all companies, regardless of size or industry, can be modeled as a collection of workflows.

In e-commerce, receiving and fulfilling a customer order is a workflow.

In retail banking, detecting a potentially-fraudulent transaction, freezing the affected account, and notifying the customer is a workflow.

In insurance, processing a new application and pricing a policy for the applicant is a workflow.

In subscription-based streaming services, a new user’s signup and on-boarding experience is a workflow.

The list goes on.

Considering this central role of workflows in a business, it makes sense that the workflow engine–a piece of software that makes it possible for users to define and automate the workflows critical to day-to-day operations–was created.

And there are a number of well-defined benefits that result from modeling a business in terms of workflows then automating them with a workflow engine, especially as the engines themselves have become more powerful and more comprehensive.

The workflow-first approach and the associated benefits apply regardless of how the workflows are carried out.

It doesn’t really matter whether workflows are mostly handled by humans, by machines, or by some combination of both. Modeling a business in terms of workflows and orchestrating work using a workflow engine is broadly applicable and can be implemented across different industries and across different types of teams.

We’re grateful to have been a part of this evolution of workflow engines and the role they play, because we’ve seen firsthand the impact that widespread adoption of workflow automation can have on an organization. Let’s talk through this in more detail.

What’s Supposed To Happen (And What Could Go Wrong) Is Modeled Explicitly

To model a business (or a single use case) in terms of workflows, it’s first important to have a clear understanding what’s supposed to happen–the so-called Happy Path–and to think about how to handle problems and exceptions.

The act of defining a workflow is itself valuable. Different stakeholders must come together and get aligned on business requirements. Workflow definition is documenting the undocumented, making concrete what would otherwise be implicit.

Once a team has come to an understanding about what’s supposed to happen in a given workflow, it’s straightforward to create workflow models in BPMN 2.0, a well-established ISO standard for graphical modelling that’s easy for technical and non-technical users to understand.

BPMN is mature as a result of input from 15 years of real-world use cases, and it supports a wide range of complex business logic. A BPMN model also serves as source code that can be directly executed by a workflow engine.

For these reasons (and more), we believe it’s the best standard to use to define workflows of all types.

A simple BPMN model for an e-commerce order process

A Workflow Engine Gives You Visibility into the Live State of the Business And a Means of Fixing Problems

If all running workflow instances are being orchestrated by a workflow engine, you should have, at any given point in time, a snapshot of the live state of your entire business.

(A quick aside: when we say “workflow instance”, we mean a single execution of a workflow model–in an e-commerce company, one workflow instance might be one customer order that needs to be fulfilled.)

You’ll be able to answer questions such as:

  • How many instances are currently running?
  • Are there errors or outages that are affecting a large number of instances?
  • Which instances are stuck and why? Do we need to do something about it?

And just as important as visibility into running instances is the power to actually fix problems when they occur. A workflow engine should then also provide users a means of updating running instances–adding a piece of missing data such as a customer ID or an item price, or example–and then either restarting or canceling the instance.

These capabilities are critical for the technical operations teams that are responsible for monitoring running workflows and ensuring they finish according to their definition.

Operate is a tool for monitoring and managing workflows running in Zeebe.

A Workflow Engine Provides a Historic Record of Your Business

A workflow engine gives you a log of historic workflow data for auditing and analysis. Every execution of every workflow instance can be stored and analyzed–including metadata associated with each instance, the paths taken at decision points, the time spent on each step, and more.

In some industries, this ability to show why and when certain business decisions were made is not a nice-to-have–it’s a legal requirement.

But regardless of whether a company is legally required to maintain a detailed audit trail, there’s lots to be gained from the ongoing analysis of workflows; it’s an opportunity to uncover recurring problems and make improvements. If you can dig into data generated by a workflow engine that tells you exactly how your business is running, you can then determine how your business might be improved.

Analyze historic workflow performance

But the Scale of These Workflows–and How They’re Executed–Is Changing

A couple of years ago, we started to hear from users who wanted to apply workflow automation to large-scale, emerging use cases but couldn’t find a workflow engine that was the right fit.

Digital-first businesses–along with well-established businesses in the midst of digital transformation efforts–are dealing with an order of magnitude more data than they were a decade ago, and data volume is expected to grow significantly in the years to come [1].

Many modern workflow automation use cases require an engine that can process not just tens or hundreds of workflow instances per second, but hundreds or thousands or even more workflow instances per second. This is scale that traditional, transactional workflow engines often aren’t able to deliver, mostly because of how they’re built–these engines rely on a relational database to store the state of running workflow instances, and a relational database inevitably becomes a scaling bottleneck at a certain point.

But the scalability requirement is just one part of a larger theme. Software architectures are evolving to keep pace with fast-changing business requirements. Organizations are moving from monoliths to microservices architectures, where decoupled services coordinate to deliver a business outcome. Developers increasingly prefer to deploy Docker containers on Kubernetes, use Apache Kafka for communication between services, and store data in Elasticsearch and other NoSQL systems.

Many existing workflow engines were built with a different sort of architecture in mind and often don’t support these technologies natively.

Enter Zeebe

We set out to build Zeebe so that organizations can automate workflows of any scale, in any architecture.

We often refer to Zeebe a “workflow engine for microservices orchestration”, and we have found that Zeebe is particularly well-suited for orchestrating workflows that span multiple microservices.

Microservices orchestration is itself a big and meaningful topic. When we asked users about the biggest challenges they were facing with their microservices architectures, the two most frequently cited were:

  1. Lack of visibility into processes that span multiple microservices
  2. Ambiguous error handling, leading to unaddressed errors at the boundaries between services

A scalable workflow engine is an effective way to address these challenges.

But like we said earlier, modeling a business in terms of workflows is an approach that can be applied in many domains and software architectures. We believe that Zeebe can have an impact on use cases beyond microservices orchestration, too. For instance, we’ve already seen an example of how to use Zeebe to orchestrate serverless functions.

We built Zeebe from the ground up because we thought it was time for a new approach to the workflow engine–one designed according to cloud-native principles and with horizontal scalability top of mind.

With the help of our community of early users, we’ve made a lot of progress in the past two years. We still have a lot left to do. And we’re still firm believers in the value of the workflow-first mindset as organizations become even more software driven in the years to come.

[1] “The Digitization of the World From Edge to Core” by IDC

  • Camunda Platform 8.1 Released – What’s New

    We’re extremely excited to announce the release of Camunda Platform 8.1.  In this post, we will go into the highlights of this release including Web Modeler, Connectors, Zeebe, Operate, Optimize, and more. Here’s the highlights of what’s included in this release: Easier Installation and maintenance for Camunda Platform Self-managed Hot backups for Self-Managed Token-Based authentication Modeler updates Process Templates FEEL Expression Editor Inline Annotations Changes to Camunda forms Connectors updates Workflow engine updates Operate updates Optimize updates Let’s get started! Easier installation and maintenance for Camunda Platform Self-Managed Improved Guides and Support for OpenShift and Amazon EKS (AWS) With this release, we are introducing official support for deploying and running Camunda Platform 8 Self-Managed on OpenShift and Amazon EKS (AWS)....

    Read more
  • Camunda Platform 7.18 Released – What’s New

    We’re extremely excited to announce the release of Camunda Platform 7.18 Here are some of the highlights: Cockpit: Improved instance selection to avoid unintended batch operations Cockpit: Easier navigation of high batch operation volumes MDC logging features Camunda Forms in Tasklist: Populate select options via process data Exception error codes Filter and order tasks by last updated WebSphere Liberty New and retired supported environments You can download Camunda Platform or run it with Docker. For a complete list of the changes, please check out the release notes. For patched security vulnerabilities, see our security notices. If you want to dig deeper, check out the source code on GitHub. Cockpit: Improved instance selection to avoid unintended batch operations Previously, when performing...

    Read more
  • Increase your resilience with new regions in...

    We’re excited to announce new regions for Camunda Platform 8 SaaS that further strengthen the resilience, performance, and data requirements for our SaaS customers. Enterprise customers can now choose to create their clusters in a number of regions including Europe West, US Central, US East, and the most recent addition, Australia South East. This provides multiple benefits, such as increasing availability, allowing to set up fail-safes, adhering to regulatory requirements for regional data storage, and reduced latency due to closer physical proximity. Resilience and redundancy in technology are important for all modern organizations and are top of mind for many. Across industries, organizations need a solution that scales effectively and is resilient to provide a good base for process orchestration....

    Read more

Ready to get started?

Still have questions?