In this post, we’ll share at a high level why we think workflow automation is and will continue to be important to organizations of all shapes and sizes and why we created Zeebe.
- Workflows Are Everywhere
- But the Scale of These Workflows–and How They’re Executed–Is Changing
- Enter Zeebe
Workflows Are Everywhere
At Camunda, after more than ten years in the business process automation space, we’ve learned at least one thing: workflows are everywhere.
We define a “workflow” here simply as a sequence of tasks that result in some business outcome. Nearly all companies, regardless of size or industry, can be modeled as a collection of workflows.
In e-commerce, receiving and fulfilling a customer order is a workflow.
In retail banking, detecting a potentially-fraudulent transaction, freezing the affected account, and notifying the customer is a workflow.
In insurance, processing a new application and pricing a policy for the applicant is a workflow.
In subscription-based streaming services, a new user’s signup and on-boarding experience is a workflow.
The list goes on.
Considering this central role of workflows in a business, it makes sense that the workflow engine–a piece of software that makes it possible for users to define and automate the workflows critical to day-to-day operations–was created.
And there are a number of well-defined benefits that result from modeling a business in terms of workflows then automating them with a workflow engine, especially as the engines themselves have become more powerful and more comprehensive.
The workflow-first approach and the associated benefits apply regardless of how the workflows are carried out.
It doesn’t really matter whether workflows are mostly handled by humans, by machines, or by some combination of both. Modeling a business in terms of workflows and orchestrating work using a workflow engine is broadly applicable and can be implemented across different industries and across different types of teams.
We’re grateful to have been a part of this evolution of workflow engines and the role they play, because we’ve seen firsthand the impact that widespread adoption of workflow automation can have on an organization. Let’s talk through this in more detail.
What’s Supposed To Happen (And What Could Go Wrong) Is Modeled Explicitly
To model a business (or a single use case) in terms of workflows, it’s first important to have a clear understanding what’s supposed to happen–the so-called Happy Path–and to think about how to handle problems and exceptions.
The act of defining a workflow is itself valuable. Different stakeholders must come together and get aligned on business requirements. Workflow definition is documenting the undocumented, making concrete what would otherwise be implicit.
Once a team has come to an understanding about what’s supposed to happen in a given workflow, it’s straightforward to create workflow models in BPMN 2.0, a well-established ISO standard for graphical modelling that’s easy for technical and non-technical users to understand.
BPMN is mature as a result of input from 15 years of real-world use cases, and it supports a wide range of complex business logic. A BPMN model also serves as source code that can be directly executed by a workflow engine.
For these reasons (and more), we believe it’s the best standard to use to define workflows of all types.
A Workflow Engine Gives You Visibility into the Live State of the Business And a Means of Fixing Problems
If all running workflow instances are being orchestrated by a workflow engine, you should have, at any given point in time, a snapshot of the live state of your entire business.
(A quick aside: when we say “workflow instance”, we mean a single execution of a workflow model–in an e-commerce company, one workflow instance might be one customer order that needs to be fulfilled.)
You’ll be able to answer questions such as:
- How many instances are currently running?
- Are there errors or outages that are affecting a large number of instances?
- Which instances are stuck and why? Do we need to do something about it?
And just as important as visibility into running instances is the power to actually fix problems when they occur. A workflow engine should then also provide users a means of updating running instances–adding a piece of missing data such as a customer ID or an item price, or example–and then either restarting or canceling the instance.
These capabilities are critical for the technical operations teams that are responsible for monitoring running workflows and ensuring they finish according to their definition.
A Workflow Engine Provides a Historic Record of Your Business
A workflow engine gives you a log of historic workflow data for auditing and analysis. Every execution of every workflow instance can be stored and analyzed–including metadata associated with each instance, the paths taken at decision points, the time spent on each step, and more.
In some industries, this ability to show why and when certain business decisions were made is not a nice-to-have–it’s a legal requirement.
But regardless of whether a company is legally required to maintain a detailed audit trail, there’s lots to be gained from the ongoing analysis of workflows; it’s an opportunity to uncover recurring problems and make improvements. If you can dig into data generated by a workflow engine that tells you exactly how your business is running, you can then determine how your business might be improved.
But the Scale of These Workflows–and How They’re Executed–Is Changing
A couple of years ago, we started to hear from users who wanted to apply workflow automation to large-scale, emerging use cases but couldn’t find a workflow engine that was the right fit.
Digital-first businesses–along with well-established businesses in the midst of digital transformation efforts–are dealing with an order of magnitude more data than they were a decade ago, and data volume is expected to grow significantly in the years to come [1].
Many modern workflow automation use cases require an engine that can process not just tens or hundreds of workflow instances per second, but hundreds or thousands or even more workflow instances per second. This is scale that traditional, transactional workflow engines often aren’t able to deliver, mostly because of how they’re built–these engines rely on a relational database to store the state of running workflow instances, and a relational database inevitably becomes a scaling bottleneck at a certain point.
But the scalability requirement is just one part of a larger theme. Software architectures are evolving to keep pace with fast-changing business requirements. Organizations are moving from monoliths to microservices architectures, where decoupled services coordinate to deliver a business outcome. Developers increasingly prefer to deploy Docker containers on Kubernetes, use Apache Kafka for communication between services, and store data in Elasticsearch and other NoSQL systems.
Many existing workflow engines were built with a different sort of architecture in mind and often don’t support these technologies natively.
Enter Zeebe
We set out to build Zeebe so that organizations can automate workflows of any scale, in any architecture.
We often refer to Zeebe a “workflow engine for microservices orchestration”, and we have found that Zeebe is particularly well-suited for orchestrating workflows that span multiple microservices.
Microservices orchestration is itself a big and meaningful topic. When we asked users about the biggest challenges they were facing with their microservices architectures, the two most frequently cited were:
- Lack of visibility into processes that span multiple microservices
- Ambiguous error handling, leading to unaddressed errors at the boundaries between services
A scalable workflow engine is an effective way to address these challenges.
But like we said earlier, modeling a business in terms of workflows is an approach that can be applied in many domains and software architectures. We believe that Zeebe can have an impact on use cases beyond microservices orchestration, too. For instance, we’ve already seen an example of how to use Zeebe to orchestrate serverless functions.
We built Zeebe from the ground up because we thought it was time for a new approach to the workflow engine–one designed according to cloud-native principles and with horizontal scalability top of mind.
With the help of our community of early users, we’ve made a lot of progress in the past two years. We still have a lot left to do. And we’re still firm believers in the value of the workflow-first mindset as organizations become even more software driven in the years to come.
[1] “The Digitization of the World From Edge to Core” by IDC
