What to look for in an advanced workflow engine
A workflow engine is an essential part of any process automation tool, and you need an engine with advanced capabilities to ensure your most complex, business-critical processes run reliably, every time.
Consider workflow engine options holistically
When considering an advanced workflow engine, you should think about three functional areas:
- Design: Features that allow users to build, test, and deploy process and decision logic.
- Runtime: Features that deliver massive scale, resilience, security, and durability for your most critical business processes.
- Operations: Features that make it easy to install, run, and upgrade the workflow engine.
Design
Process logic
All workflow engines can execute process models that users have built. An advanced workflow engine uses a standards-based process modeling approach that provides a visual way for technical users and business stakeholders to collaborate on process design. The engine then interprets and runs the process model.
Why it matters
An open process modeling standard such as BPMN facilitates communication and alignment between the subject matter experts in the business and the software developers who implement automation projects. Because BPMN is executable, it ensures that what users build is exactly what the workflow engine executes at runtime, so business logic isn’t buried in code.
Advanced workflow patterns
The processes that build and deliver a company’s products and services are normally more complex than a basic sequence of tasks. They often require workflow patterns that involve branching logic, dynamic parallel branches, message correlation, task cancellation, time-based escalation, fallback flows, and more.
Why it matters
An advanced workflow engine that leverages the BPMN process modeling standard enables users to express complex workflows elegantly, in a way that’s easy to visualize and understand. Visualization is critical for the initial design of a process and for its maintenance, as new business requirements arise and teams uncover opportunities for improvement.
Integrated decision logic
Businesses run on decisions, and decision automation can bring a level of efficiency, consistency, and time and cost savings that just isn’t possible with manual work. AI-led decisions offer some promise, but are difficult to implement, involve a high risk of unconscious bias, and lack transparency about why a particular decision was made. An advanced workflow engine offers decisioning capability that’s integrated yet loosely coupled, so automated decisions can be delivered as part of business processes or executed independently. An advanced engine also uses a standards-based decision modeling approach for decision automation to facilitate collaboration.
Why it matters
Similar to BPMN, an open decision modeling standard such as DMN enables communication and alignment between business stakeholders and IT users. DMN is particularly well-suited for enabling subject matter experts to independently build, test, and deploy decision tables without impacting process models or interrupting in-flight process instances.
Process and decision versioning
An advanced workflow engine keeps a version history as teams modify process and decision models. It allows users to roll back to previous versions if needed and enables them to deploy new versions to test and production environments.
Why it matters
Business processes change over time as manual tasks are automated, new requirements are introduced, and teams find ways to optimize process performance. Versioning helps teams understand how processes and decisions have changed and allows them to undo changes by rolling back. Versioning is also important for changing the processes that are running in production. It enables teams to improve processes without interruption and experiment with new process flows (for example, through A/B testing).
Process simulation
A workflow engine that supports process simulation allows users to understand how a process will function under different conditions; for example, when a variable’s value is changed or an error state is triggered. Simulation complements automated testing and facilitates understanding of processes.
Why it matters
Process simulation helps business teams ensure that process logic works the way it should without requiring IT to provision a special environment. It enables rapid prototyping of business solutions, reduces the time required to build a working process model, and increases the quality of the model.
Process testing
Testing is a given any time you’re developing software, but some process automation tools and low-code application development platforms don’t make it easy to test what you’ve built in the same way you test other code. An advanced workflow engine can be integrated into a CI/CD pipeline so you can automate unit tests, integration tests, performance tests, and so on.
Why it matters
Test automation is essential for quality because it ensures that you can efficiently perform regression testing on your process and decision models. This assurance enables teams to change and improve processes much faster. A suite of automated tests that you can trust reduces the overhead for small changes and provides more confidence for large changes.
It’s important to understand if the workflow engine and other tools in your hyperautomation stack are holding you back from making changes, gaining visibility into how well your business processes are working, and scaling your end-to-end automation efforts. The Buyer’s Guide to End-to-End Process Orchestration covers key questions to ask when evaluating technologies and vendors.
Runtime
Long-running process management
All workflow engines must be able to manage long-running processes; this is a critical differentiator between a workflow engine and a tool that can execute one automated task at a time. An advanced workflow engine can maintain the state of process instances that run for hours, days, weeks, or months with a high level of data consistency.
Why it matters
Long-running processes are common in business contexts, and maintaining the consistency of business data is critical for those processes to be successful. Imagine a mortgage application or insurance claim; these are processes that may take days or weeks and require employees to collect large amounts of sensitive customer data. This data must continue to be available to the process instance until it is finished.
Scalable, resilient, performant state persistence
It’s challenging to maintain the state of running process instances in a way that scales up with high transaction volumes, that’s resilient to failures, and that performs well at scale. An advanced workflow engine uses appropriate technology such as event-streaming to persist the state of process instances outside of a central database, thus avoiding the performance bottleneck of database I/O.
Why it matters
State persistence is a capability you’ll find in every workflow engine, but event-streaming technology is an innovation that delivers unparalleled scalability and performance. It ensures that the engine doesn’t encounter a performance bottleneck when reading from and writing to a central database. A distributed architecture ensures resilience, even when load is high.
Event/message correlation
As a business process runs, many events happen in systems that are outside the workflow engine. The engine must be able to receive messages about those events and correlate them to the correct persisted process instances. Message correlation must continue to work even when a process instance runs for a long time. It must also work at scale, when there are thousands or millions of process instances and events in the system.
Why it matters
Message correlation is critical for even the simplest scenarios, such as ensuring that a payment is applied to the correct order from an ecommerce website. Without message correlation, a workflow engine cannot effectively work with other systems to satisfy business use cases. In addition, growing adoption of event-driven architecture in the automation market means that smooth interaction with events is increasingly important for workflow engines to provide.
Timers and timeouts
Keeping track of time is a key requirement for any workflow engine. It’s common for business processes to wait for a certain period of time and then take action. For example, imagine a flight reservation process that holds your ticket for 24 hours and then releases it if you haven’t completed the purchase. Or imagine a process that needs to escalate a customer care case to a supervisor if an SLA is about to expire.
Why it matters
An advanced workflow engine can handle scenarios that involve timekeeping without requiring users to write any code. When combined with state persistence, timers and timeouts are a powerful way to ensure that long-running processes don’t lead to a poor customer experience.
Payload/data
A workflow engine is normally not the single source of truth for business data. Rather, it should only require the minimum data it needs to execute a process. An advanced workflow engine can carry the data payload through a process and make it available to process instances as needed, all with a high level of consistency.
Why it matters
Business data is critical to process execution, but it’s often stored in many different places: CRM and ERP systems, data lakes, data warehouses, and so on. While a workflow engine might not store business data for long periods of time, it’s important that the engine is capable of retrieving data from various systems, maintaining the integrity of that data, and ensuring that each task in each process—whether it’s automated or assigned to a person—has enough contextual data to execute correctly.
Synchronous and asynchronous endpoint integration
Integration between a workflow engine and endpoints such as APIs, microservices, and RPA bots is nothing new, but an advanced workflow engine supports both synchronous and asynchronous communication with endpoints.
Why it matters
Endpoint integration is particularly important when working with APIs or RPA bots that are prone to timeouts. You want a workflow engine that can wait for a response, execute tasks in parallel while it’s waiting, and fall back to a different flow if the endpoint fails to respond after a certain period of time.
Check out this on-demand webinar to learn how Camunda’s advanced workflow engine, Zeebe, delivers unparalleled performance and resilience for the most demanding, business-critical processes.
Operations
Open architecture
The tech stack around a workflow engine can greatly affect its performance. An advanced workflow engine is not tied to a specific tech stack or monolithic application architecture; instead, it’s designed to easily fit into diverse and complex enterprise environments. For example, it leverages widely used technology such as Java or Docker, and is verified to work with common technical architectures and frameworks such as Spring Boot or Kubernetes.
Why it matters
When you can easily integrate a workflow engine into your existing tech stack, you can start delivering process automation projects and get to value quickly. An open architecture reduces the learning curve for IT teams and helps future-proof automation efforts.
Process instance modification
Sometimes, automated business processes require human intervention. For example, a process instance might be stuck because an external system returned invalid data. An advanced workflow engine shows these problems to users in the context of the process model and allows users to modify active process instances; for example, by changing the values of process variables.
Why it matters
Process instance modification isn’t just convenient; it can be a powerful customer support tool. Imagine you’re in the middle of a complicated auto insurance claim. It would be a poor customer experience if you had to start over just because an automated identity check or rental car booking failed. Process instance modification empowers customer support staff to identify and fix problems so that processes keep flowing.
Process instance migration
Business doesn’t stop when a process needs to change, but deploying a new process model without interrupting customer experiences is challenging. An advanced workflow engine supports migration of process instances to a new version of the process model with minimal or no interruption, depending on the changes in the model.
Why it matters
Process instance migration allows teams to deliver bug fixes and new functionality without stopping process instances that are in progress. An advanced workflow engine will show users exactly what will happen when process instances are modified and will offer flexible options for mapping elements in the running process to elements in the new process model. This reduces the operational complexity of running multiple versions of a process in parallel.
High availability
Organizations undergoing digital transformation execute an ever-increasing number of digital transactions, which requires a workflow engine that’s highly available. An advanced workflow engine with a distributed architecture is ideal for delivering high availability because the software doesn’t rely on a single workflow node or a central database and can resiliently keep working in case of hardware failures, even if whole datacenters go down.
Why it matters
Today, users expect fast, reliable performance from every application they interact with. If those expectations aren’t met, it’s likely that users will look to your competition—making high availability a business requirement. Plus, some organizations have to contend with SLAs defined by regulatory requirements, with the potential for large fines if they fail to meet uptime requirements. It’s no longer enough to lift-and-shift your existing workflow engine technology into the cloud; to deliver consistent high availability, you need the benefits that distributed architecture offers.
Uptime and upgrades
Critical business processes often have a high uptime requirement. Downtime can be caused by technical problems, routine hardware or software maintenance, or software upgrades. An advanced workflow engine with a distributed architecture can deliver exceptional uptime and can be upgraded without interrupting process instances that are in progress. It can also buffer requests during short periods of downtime.
Why it matters
Downtime of critical business processes directly affects both customers and employees. If a customer wants to purchase a product or service, open a support case, or otherwise engage with your business, they expect the processes that support that engagement to be up and running. An advanced workflow engine enables teams to keep your business operational while fixing issues or updating the engine.
Backups
In enterprise situations, reliable and consistent software backups are a must-have. An advanced workflow engine isn’t just designed to be backed up; it’s also designed with software restoration in mind. IT teams should be able to rely on the engine to work properly after being restored from a backup, with minimal to no data loss.
Why it matters
Software backups don’t mean much if you can’t restore in emergency situations. An advanced workflow engine supports hot backups that ensure the data in the backup is consistent; that minimize the impact of backup on the engine’s performance; and that can run without interrupting running process instances.
Auditability
A workflow engine will typically log automated tasks for troubleshooting purposes. An advanced workflow engine keeps track of every automated and manual action that’s taken, with a configurable level of detail that enables IT teams to pinpoint issues and the option to export logs to other systems for analysis. It also provides audit information in a human-readable format for business compliance purposes.
Why it matters
Audit logs are critical for both technical troubleshooting and regulatory requirements. You don’t want to be surprised by a situation where you don’t know what happened in a process because every task wasn’t logged. An advanced workflow engine helps IT teams, business users, and internal and external auditors find the data they need without a lot of manual data parsing. It also helps users visualize what happened by combining audit logs with a visualization of the process model.
