Today, we’re releasing Camunda Platform 8, the next major version of Camunda Platform. In this blog post, we want to specifically look at what this means for our large Camunda Platform 7 community.
The Story So Far
Camunda Platform 7.0 started as a fork of Activiti 5 in 2013. Since then, Camunda has released 17 minor versions and more than 100 patch releases. Now is the time to increment the big number in the front and move from major version 7 to 8. Check out the press release and the release blog post to learn more about what this release means in general. In this blog post, we’re going to specifically focus on how Camunda Platform 8 features relate to the existing Camunda community.
I specifically want to discuss how some of the core features of Camunda Platform 8 improve upon Camunda Platform 7—thanks to the lessons learned from years of development. I’ll briefly touch on migrating existing Camunda Platform 7 solutions, as Camunda Platform 8 won’t be a drop-in replacement. But there’s already an important thing to understand about this major release, we will not abandon Camunda Platform 7. For at least the next five years, we’ll continue to actively develop Camunda Platform 7 and fully support it. This means we’ll provide for both Camunda Platform 7 and Camunda Platform 8 in parallel, as detailed below. This also means that existing users can choose whether they prefer to stay with Camunda Platform 7 or to migrate to Camunda Platform 8. There will be more on this later.
Comparing Camunda Platform 8 and Camunda Platform 7
There are a few methods to utilize Camunda Platform 7 based on your preferred production environment and configuration. This flexibility makes a comparison with Camunda Platform 8 complicated, as it depends on your architecture. For example, Camunda Platform 7 can be embedded as a JAR file in your Java application or be operated as a Docker container orchestrating your polyglot microservices via REST. Those are very different architectures. For this post, let’s look at the default distribution of Camunda Platform 7, which is Camunda Run, a standalone Java application containing all web applications that persists to a relational database. The applications using the workflow engine connect via REST.
In Camunda Platform 8, the default environment is our SaaS Edition, where Camunda hosts a cluster of Zeebe workflow engines in a Kubernetes environment. The engine exports data to Elasticsearch, the datastore used by the web applications. Applications using the workflow engine typically embed the provided SDK for the programming language of choice.
The applications will seem quite familiar to Camunda Platform 7 users. Operate is equivalent to Cockpit. There’s a new Tasklist which is also helpfully called Tasklist, and for those using Optimize, you’ll be delighted to know that it is compatible with both Camunda Platform 8 and Camunda Platform 7, so no change there. Desktop Modeler now also supports modeling for both versions of Camunda, you just need to download the latest version.
The use case for both platforms is exactly the same: orchestrating systems, services, and people by executing a predefined BPMN 2.0 process model. The differences between Camunda Platform 7 and Camunda Platform 8 aren’t about the use cases, but are in the technical implementation of the underlying engines. If you’re interested in more in-depth information on this, you might find this blog post helpful.
5 Key Improvements of Camunda Platform 8
Let’s quickly look at the most prominent improvements of Camunda Platform 8 over Camunda Platform 7. In general, the changes can be grouped into the following categories:
- Architecture: Which now allows for true clustering, resilience, horizontal scalability, and geographic redundancy.
- Installation: As Camunda Platform 8 runs on a Kubernetes cluster and automatically has all the tools you need.
- Web Modeler: Where modeling for execution and deployment can finally happen all in web-based tooling.
- Connectors: Making it easier to manage integration with external systems.
- Feature improvements: The existing features you love in Camunda Platform 7 have experienced overhauls and improvements.
Let’s go over this in detail now.
In Camunda Platform 7, the clustering model was to run multiple workflow engine instances and point them to the same relational database. Then, you could add a load balancer upfront to distribute incoming traffic between the nodes. The database was the single point of failure and could become the bottleneck in this architecture.
The Zeebe engine has clustering built-in, and the default recommendation is to always run at least three brokers (a broker is similar to a Camunda Platform 7 workflow engine node) for resilience, as this eliminates any single point of failure.
The provided gateway deals with incoming requests and routes them to the right broker, while also caring about load balancing. Brokers store their data as an event stream, locally on disk (as “local” as a disk can be in Kubernetes), and this state is replicated across brokers. This is why there is no need for a relational database as was needed with Camunda Platform 7.
The way Camunda Platform 7 scales has one very obvious limitation, the central database. All engines in a cluster have to connect to that same database instance. This limits the throughput of an engine to what the database can handle, so at some point, you’ll reach a hard limit of what a cluster can do.
Camunda Platform 8 does not have this kind of limitation, and the performance continues to scale with each node, which is called horizontal scalability. If you have more brokers running, you can scale for a higher load. It’s pretty easy.
As the Camunda engine matured and stabilized over the years, it has become trusted in more and more mission-critical processes. Hence, we were regularly asked how we would deal with the problem of an entire data center going down. The general solution to that is geographic redundancy, where you run your application in two different physical locations.
With Camunda Platform 7, doing this was possible, but there was a massive hit on performance as we needed to synchronize to relational database instances over a low latency connection. As this needed to be done as part of the technical database transactions, it led to transactions being open for quite long, leading to cascading problems. Long story short—while it was possible, it was painful and never easy.
When designing Camunda Platform 8, we had those use cases at the top of mind. In general, Zeebe can now replicate its data to a remote broker in a different physical data center without harming overall throughput or stability. Of course, the latency of replication is still there, but it’s just a constant latency being added and is not a problem at all. In one customer case, we replicated from Europe to the U.S. smoothly.
True parallel execution
There will be a detailed blog post all about this single topic coming very soon, so I’ll just summarize it a bit here. In order to maintain a consistent state, Camunda Platform 7 leverages locking mechanisms extensively. In the experiment you’ll read all about soon, you’ll clearly see that this hits you when you execute a large number of tasks through parallel branches or parallel multi-instance tasks. The Camunda Platform 7 engine will either produce a lot of OptimisticLockExceptions, or you’ll need to configure it to sequentially do things in one process instance, leading to bad performance for this use case.
Camunda Platform 8 doesn’t rely on the same system for ensuring a consistent state, instead, it uses append-only logs, buffers, and a clever layout of single writers. This design removes the contention on the data storage layer (like the database in Camunda Platform 7), and you don’t need to make things run sequentially, which speeds up the above example massively.
Streamlined deployment and architecture options
As someone who likes to help out by answering questions on our forum or conducting workshops, I can say with confidence that the biggest pain for people getting started with Camunda Platform 7 is:
- Understanding the different possible architectures and how this should affect things like where to write your code.
- How to deploy your models.
- How to update the engine.
For example, deploying a model as part of a Spring Boot project is done by putting your model in the resources folder, while with Camunda Run, it’s best to deploy via the REST API. Mixing up these options can result in the code not running, or deployments being lost or overwritten.
With Camunda Platform 8, we simplified things massively. We’re offering a single method for running and deploying processes, and a single method for running your code—that is to deploy your model remotely via gRPC or REST, and then run your code externally the same way as you would with the external task pattern.
Installation of all components
With Camunda Platform 7, you had to define the exact configuration of your components and take care of the installation. While this is relatively simple, there’s still a certain amount of effort involved in setting completely decoupled components like Camunda Optimize.
The whole installation procedure of Camunda Platform 8 now relies on Kubernetes, and we provide Helm charts to set up and wire together the whole environment, including Camunda Optimize. You won’t even have to think about Kubernetes or Helm if you’re using Camunda Platform 8 SaaS—you’ll have access to all components with the click of a button.
Set up internal SaaS offerings
Camunda is most often introduced step by step. A typical example is when an organization starts a lighthouse project, and once it’s successful, it inspires other teams to use the engine for their project. This is a great way to get started, low risk and high reward. However, after a few years, a typical issue that arises from this is that a single organization with about 20 different projects will end up having different databases and engine versions. Since these projects were often completely independently done, there’s no easy way to get an overview of them all. To solve this, some users have developed their own solutions, following the pattern sketched out with the Engine Harvester.
With Camunda Platform 8, you can set up your own organization-internal SaaS service. All you need is a Kubernetes environment to install Camunda Platform 8 clusters (we plan to provide the web console to provision new engines for self-managed users soon). While this SaaS can be operated centrally, it can be available to all projects in self-service (as you would expect from any other cloud service). It greatly simplifies maintenance by having a single point to provision new engines, monitor them, or upgrade to newer versions.
Camunda Platform 7 has always had pretty good integration with its web apps, but we’ve always found it challenging to properly integrate Modeler in a way that is a good technical and UX fit. With Camunda Platform 8 being hosted in the cloud, it gives us a lot more flexibility with how we can set up a modeler to better integrate with the process engine because we no longer require the end user to have to tinker around with the settings. From the end user’s perspective, building and deploying a process model is as seamless as all the required components that are hosted in the cloud. This doesn’t mean that you’ll need to change the way you deploy. You can also keep and maintain models locally and deploy them from Desktop Modeler in the same way you can with Camunda Platform 7.
Easy integration with connectors
An orchestrator’s most common job is communicating with external systems. In projects using Camunda, those connections are typically programmed from the ground up by the people building the process solution. But some of this logic ends up being quite generic. Do you need to call a REST endpoint? Send an AMQP message? Or connect to that bespoke inhouse system in any other project?
For those situations, we want to offer more convenience and make reusability easier.
So with Camunda Platform 8, we’ll be adding the ability to add pre-defined connectors. Camunda will provide some connectors, but the ecosystem shall allow users to build their own connectors to be reused within an organization. We also looked at the whole user experience around connectors and decided that it should be added as part of Web Modeler. I expect these connectors to save developers a lot of time and effort when building their processes, while still not restricting what they can do. We don’t intend for people to start seeing Camunda as a low-code platform.
Feature improvements like BPMN message buffering
Let’s use BPMN messages as one example of how we can improve core features of the platform with the switch to Camunda Platform 8. The BPMN specification defines messages in a way that a token needs to be ready to receive that message when it is sent.
In Camunda Platform 7, if you send a message and there isn’t a token waiting on a message receive event, that message will fail. While this follows the standard perfectly, we’ve learned how impractical that is in real life. Especially in complex distributed scenarios, we often saw messages overtaking each other, making message receiving with BPMN unnecessarily complex.
With Camunda Platform 8, we’re introducing message buffering. Messages now have a time to live—so you can send your message whenever you want, and it will wait in a queue until the message can be delivered.
To Migrate or Not to Migrate?
Camunda Platform 7 users are a hard group to address as a single entity, there are numerous combinations that can give a variety of possible architecture options. The two most common groups of users are: the ones that have either used Camunda Platform 7 as an embedded engine to orchestrate their Java code (using Spring Boot or Tomcat), or the ones using Camunda as an independent orchestrator, implementing external tasks that fetch and complete work from the engine. Let’s look at both use cases.
If you’re using an embedded engine, you might want to stick with Camunda Platform 7, unless you have good reason to migrate. This is a perfectly reasonable choice as Camunda Platform 7 will remain fully supported for at least the next five years. As a pure migration rarely results in immediate benefits, those projects are often hard to justify.
However, if you modify your application anyway or you are experiencing problems with scaling your system, you should definitely consider migration. Camunda Platform 8 should be the natural choice for new projects.
If you’re using Camunda Platform 7 as an external task orchestrator, you’re better off using Camuna Platform 8, as it greatly improves the support for this use case.
If you’re looking for a deep dive into how your exact setup will be affected by Camunda Platform 8 or how to approach migration, I highly suggest you check out this document on migrating from Camunda Platform 7 to Camunda Platform 8.
A Dot-zero Release?
We’re big fans of agile software development and believe in “release early and often.” In this spirit, we also released Camunda Platform 8.0 with it not yet having all the features that we eventually want to have in it. This means there are some features you might miss in the Camunda Platform 8.0 release that you know from Camunda Platform 7.
For example, token move and process instance migration, but they are on the way to be released soon. In addition, some BPMN elements aren’t present yet, but they’ll be added soon. You can also expect more work around Web Modeler, connectors, and other parts of the platform within the upcoming months. So if there’s a feature you’re really missing, just let us know. This is exactly the feedback we want to improve our priorities for the broader community.
At the same time, Camunda Platform 8.0 is the incremental improvement of Camunda Cloud 1.3 (so Camunda Platform 8.0 = Camunda Cloud 1.4), which is already in production for a lot of users. While we might have spared some features, we’ve already hardened the core engine, so you can safely trust your valuable processes with it.
Shape Camunda Platform 8’s Future!
The Camunda Platform 8.0 release is really just the start. With our great developer relations team, we are ready to shape the future based on our community’s needs, now more than ever before. How you use our software and envision future use cases can make a big difference, so make sure you reach out to us with your feedback. We’re all very excited to put this release out into the wild, and we hope you’ll enjoy having it as part of your system, and trust it to be the fundamental component of your processes.