Don’t Compromise on Your Orchestration Engine

It seems like orchestration is the word on everyone’s mind right now. In February, Forrester published a report called AI Is Reshaping Automation Markets that predicts the formation of a process orchestration market in 2026.

In June, they published Leverage The Power Of Process Orchestration To Drive Innovation, which describes how process orchestration enables organizations to innovate and increase efficiency. And in May, Gartner introduced the concept of business orchestration and automation technologies (BOAT), which they’ve since covered in more detail in Quick Answer: Beyond RPA, BPA and Low Code — The Future Is BOAT (click that link to download a free copy).

It’s exciting to see industry analysts recognize the importance of orchestration because at Camunda, we’ve focused on delivering enterprise-grade process orchestration to our customers for years. We’ve also learned a valuable lesson along with our customers: You’re going to hurt your business if you compromise when choosing an orchestration engine.

In a way, it’s simple. Processes are the algorithms that define how your business runs. An orchestration engine executes those processes—no matter how automated or manual they are, and no matter how many different tools and technologies they involve. Therefore, if the orchestration engine can’t deliver the capabilities and performance that your business needs, it’s the business that’s going to suffer.

But not all orchestration engines are created equally, and there are many technical factors that determine whether a particular engine is the right choice for your organization. Let’s look at these factors in three categories:

• What matters at design time
• What matters at runtime
• What matters in operations

What matters at design time

Design time features allow users to build, test, and deploy process and decision logic. You should consider these design time capabilities:

• A standards-based approach to process and decision modeling that provides a visual way for technical users and business stakeholders to collaborate on process design. Standards facilitate communication and alignment between the subject matter experts in the business and the software developers who implement automation projects.
• Support for advanced workflow patterns that involve branching logic, dynamic parallel branches, message correlation, task cancellation, time-based escalation, fallback flows, and more.
• Integrated yet loosely coupled decision automation so that automated business decisions and rules can be delivered as part of processes or executed independently from processes.
• Support for process and decision versioning so users can visualize and understand how process and decision models have changed over time, and can easily roll back to previous versions if needed.
• A process simulator that shows how a process will function under different conditions to ensure that process logic works the way it should without requiring IT to provision a special environment.
• The ability to integrate with automated testing pipelines so teams can efficiently perform regression testing on process and decision models. This assurance enables teams to change and improve processes much faster.

What matters at runtime

At runtime, you need an orchestration engine that delivers massive scale, resilience, security, and durability for your most critical business processes. You should consider these runtime capabilities:

• The ability to manage long-running business processes—that is, processes that run for hours, days, or weeks—with a high level of data consistency.
• Scalable, resilient, performant state persistence that maintains the state of long-running process instances while avoiding the performance bottleneck of a central state management database.
• Support for correlating external events to process instances that are running in the orchestration engine, even when a process instance runs for a long time or when there are thousands of process instances running simultaneously.
• Timers and timeouts that enable a process to wait for a period of time and then take action, ensuring that SLAs are met and long-running processes don’t lead to a poor customer experience.
• The ability to carry data payloads through process instances. Business data is often stored in many different places, and while an orchestration engine is normally not the single source of truth for business data, it needs to make the right data available to process instances at the right time.
• Integration that works both synchronously and asynchronously, so the engine can trigger task automation tools to do work; wait for a response; trigger other tasks in parallel; and fall back to a different flow if the task automation tool fails.

What matters in operations

When operating an orchestration engine, you need features that make it easy to install, run, and upgrade the tooling. Look for:

• An open architecture that’s not tied to a specific tech stack or monolithic application architecture, but that instead can be integrated into your existing automation landscape.
• The ability to allow users to modify running process instances, for example by changing the values of process variables, so people can solve problems and keep processes flowing.
• The ability to allow users to migrate running process instances to a new model with minimal or no interruption, depending on how extensively the process model has changed. This ensures you can update long-running processes without interrupting business.
• Support for consistent high availability, ideally through a distributed architecture that doesn’t rely on a single workflow node or central database.
• Support for high uptime and easy in-flight upgrades, so you can take care of routine maintenance and software upgrades without affecting running process instances.
• Reliable software backups that ensure the data in the backup is consistent;  that minimizes the impact on engine performance; and that can run without interrupting process instances.
• A comprehensive audit log that keeps track of every automated and manual action that’s taken, with a configurable level of detail that enables IT teams to pinpoint issues and that’s available in a human-readable format for business compliance purposes.

Learn more

To learn more about what matters when choosing an orchestration engine, check out our deep dive into advanced workflow engines. And to get a checklist for evaluating your software options, download our Buyer’s Guide for End-to-End Process Orchestration.