*Camunda Platform 8, our cloud-native solution for process orchestration, launched in April 2022. Images and supporting documentation in this post may reflect an earlier version of our cloud and software solutions.
Zeebe is the workflow automation engine that powers Camunda Platform 8, Camunda’s hosted service offering. Zeebe is source-available and free to use as self-hosted.
A common question asked about Zeebe is:
“How is communication secured between components, particularly between client applications and a Zeebe cluster?”
In this article, we are going to take a look at Zeebe’s communication channels and how to secure them in a production deployment.
Understanding how to secure Zeebe will help you to secure your production instance if you are self-hosting; and will help you to quantify the value of the security aspect of using the Camunda-hosted service. The current maximum possible security is configured in production on the hosted Camunda Platform 8.
Securing Concerns
Securing client-server communications has several aspects:
- Encryption – Communication between components that travels over the wire is encrypted so that an intermediary who intercepts the communication cannot read the content of the communication.
- Authentication – The server will accept communication only from an authenticated client.
- Authorization – The authentication layer provides the server with an identity for the client, and specific operations on the server can be allowed or disallowed based on the identity of the client.
- Auditing – Operations performed by clients are logged, to allow future forensic analysis of “who did what.”
Zeebe System Architecture
Let’s take a look at the architecture of a Zeebe system before we examine how each component in the system handles security.
A basic Zeebe system consists of:
- A Zeebe Gateway
- One or more Zeebe brokers
- An ElasticSearch cluster (which may have a single node)
- Client applications, using a Zeebe client library
Note that we are covering only the core broker and client applications in this article. We will not be covering the Operate and Tasklist components. Operate provides similar features to Camunda Cockpit – inspecting and managing process instances, both running and historical. And Tasklist allows you to complete user tasks in processes.
Zeebe’s Communication Channels
The Zeebe workflow engine, which powers Camunda Platform 8, uses gRPC for communication with external applications. gRPC is a modern open source high-performance Remote Procedure Call (RPC) framework that can run in any environment.
It was originally developed by Google for inter-application communication. Like Kubernetes, which originated with Google as their container scheduling system, gRPC has escaped the Google data centers and has become a popular technology in the broader community.
There are three categories of gRPC communication in a Zeebe system:
- Gateway to broker communication – The Zeebe Gateway sends commands to brokers, and receives responses from the Process ID of a new process instance or jobs for workers.
- Broker to broker communication – The Zeebe brokers replicate the state and send cluster metadata messages to each other.
- Gateway to client communication – Applications written with a Zeebe client library start process instances, resolve incidents, and request jobs to complete from the Zeebe Gateway.
Support for Security Concerns in Zeebe
- Encryption – Zeebe provides built-in support for TLS communication between clients and the gateway, and has support from release 1.3 between the gateways and the brokers. Encryption between the gateway and clients is not enabled by default. You can enable it, and this involves provisioning certificates to the brokers — and configuring certificate stores on the broker and the clients if you use self-signed certificates. You may find it makes more sense to put the cluster inside a private network and use a proxy to put the configuration concern in a single component in the system.
- Authentication – Zeebe doesn’t directly provide authentication for external clients. To implement this, you will need to use a reverse proxy in front of the gateway. This is how authentication and encryption are implemented in the hosted Camunda Platform 8. If you implement this, you will also implement TLS here. In that case, you may treat the gateway and the brokers as an internal network and leave the communication between the proxy and the gateway unencrypted. This coalesces your security configuration in the proxy, and simplifies the deployment and configuration of your cluster.
- Authorization – You can implement your own granular authorization in the gateway using custom interceptors if you are running Zeebe as self-hosted. At the moment, the hosted Camunda Platform 8 provides no granularity of operation authorization. Any authenticated client can perform any operation. We are analyzing how to implement authorization in the hosted offering. Moreover, you can add a comment to this GitHub issue, if you’d like to share your take on the matter.
- Auditing – There is currently no built-in auditing capability. This is a longer-term plan for Zeebe after granular authorization is implemented. You could implement your own auditing solution using custom interceptors if you are running Zeebe as self-hosted.
Encrypting Communication
[ Gateway | Broker ] to Broker
From version 1.3, Zeebe supports encrypting communication between the gateway and brokers, and between brokers.
This is disabled by default. To configure intra-cluster TLS encryption, you need to set several configuration parameters and supply a TLS certificate for each node.
As an alternative, rather than encrypting communication between nodes (for example: if you’re using a version prior to 1.3 or want to simplify your configuration and deployment), you could place your brokers in a private network and put a publicly-accessible proxy in front of the gateway.
This configuration means the communication inside the cluster is insecure, but it’s on a publicly-inaccessible network.
Gateway to Client
You should use the default configuration. Zeebe’s gRPC communication between external client applications and the gateway is not secured.
Connecting to the cluster using the zbctl command-line client requires you to use the — insecure flag out of the box.
The Zeebe Gateway allows you to configure TLS encryption and supply a certificate to encrypt communication between clients and the gateway. This secures data in transit between client applications and the Zeebe Gateway.
If you configure TLS encryption on the gateway, then you don’t need to use the — insecure flag with zbctl.
The Java and Go client libraries enable TLS by default. This means that if you’re developing a client application using a local Zeebe broker with the default configuration, you need to explicitly tell the client to disable TLS.
The first thing to do in securing communications with client applications is to encrypt communications between the gateway and the clients with TLS. This encrypts communication on the wire, making it impossible for a man-in-the-middle to read the data.
Configuring TLS encryption this way may not be your best option — for two reasons:
- It requires additional configuration and the provision of certificates to the Zeebe Gateway. This makes it a cross-cutting concern with the deployment of Zeebe. If you’re not a Java shop, configuring the certificate store adds additional complexity.
- If you want authentication, you need a proxy — so you might want to put all your security concerns in that layer.
An alternative approach is to put the gateway behind an nginx proxy or similar, and terminate the TLS connection from the client at the proxy. In this setup, your Zeebe cluster is on a private network and deployed without TLS being configured. All TLS will be handled in the proxy.
Camunda Platform 8, the hosted SaaS platform, secures communication between clients and the Zeebe Gateway through an nginx proxy.
However, this does not provide authentication — any client that can access the gateway can still send commands to it.
Authentication and Authorization
Encrypting the communication between client applications and the Zeebe Gateway is a good first step to securing the system. However, any client that can access the gateway, whether directly or via a proxy, can issue commands.
To limit access to the gateway to only authorized clients, you need to implement authentication.
There is no intrinsic support in the Zeebe Gateway for authentication or authorization. The Zeebe client libraries support transparently adding an authorization header to gRPC calls. However, the Zeebe Gateway does not check this. You will need to implement a proxy that examines the gRPC headers, and allows or denies commands based on this header.
The Zeebe client libraries take an OAuth configuration, and make a POST request to the configuration token endpoint, passing a client ID and client secret, to receive the authorization token to use in the headers.
Refer to the Authorization section of the Zeebe documentation and the source code of the Zeebe clients for more details on how to implement this.
Camunda Platform 8, Camunda’s hosted service, has an nginx proxy that examines headers for the authorization token, and a token issuing endpoint where client applications can swap API credentials for an authorization token.
There is a fair amount of infrastructure required to implement an authentication and authorization solution for Zeebe — and it’s possible. But it may not be the best use of your time. Camunda Platform 8 provides authentication/authorization out-of-the-box, allowing you to focus on business logic, rather than infrastructure.
