Using the Zeebe Kafka Connector to Orchestrate, Integrate, and More

The Zeebe team just implemented a Kafka Connect Zeebe connector. This is a feature users have been asking for since the start of the project, so let’s give it a closer look.

What is Kafka Connect?

Kafka Connect is the ecosystem of connectors into or out of Kafka. There are lots of existing connectors, e.g. for databases, key-value stores or file systems. So for example you can read data from a RDMS and push it to Elasticsearch or flat files.

What is Kafka Connect? From Confluent.

A Kafka Connect example from https://www.confluent.io/blog/the-simplest-useful-kafka-connect-data-pipeline-in-the-world-or-thereabouts-part-2

Kafka Connect Zeebe

You can easily develop your own Kafka Connect connectors, and that’s what we did for Zeebe. Based on a POC, which I showed in a talk at Kafka Summit San Francisco (recording available), the Zeebe team cleaned up the code (which is pretty important if I wrote parts of it ;-)) and just released a version that is suitable for real-life usage. You can find it on https://github.com/zeebe-io/kafka-connect-zeebe.

What can the Kafka connector for Zeebe do?

The connector can do two things:

Send messages to a Kafka topic when a workflow instance reaches a specific activity. When I say message, I really refer to records in Kafka, where a lot of people also simply speak of events. This is a source in Kafka Connect speak.

Consume messages from a Kafka topic and correlate them to a workflow. This is a Kafka Connect sink.

A Zeebe sink via Kafka Connect

Use cases and examples

A simple example to test if the connector is working for you is the ping-pong example on GitHub. It simply sends a message to Kafka which is directly correlated back.

There are multiple use cases for the connector, and let’s quickly highlight two of them.

A common use case is around microservices orchestration. We have a simple example around the domain of order fulfillment in the microservices-orchestration example on GitHub.

An order process and payment microservice communicating via Kafka, orchestrated by Zeebe

It leverages the connector’s source to push a message to Kafka whenever a payment is required, expecting some payment service to process it and emit an event (or send a response message) most probably asynchronously. This response is correlated to the waiting order fulfillment process using the connector’s sink.

Another great example is around messaging integration patterns. For example we might want to wait for a couple of events to arrive before we want to trigger some activity. If events are missing within a certain time frame we need to escalate.

An order process and payment microservice communicating via Kafka, orchestrated by Zeebe

Quickstart and screencast

You can easily follow our quickstart guide to play around with the ping-pong example. Or you could also sit back, relax, and watch this screencast:

Status and roadmap

The connector is basically ready to be used. We do have some open issues, but we want to wait for more real-life user feedback before we decide how to move forward and what to prioritize.

So if the connector could be useful for you, try it out today and make sure to let us know how it’s going (via the forum or Slack). Also, don’t hesitate to open issues or vote for existing ones to make sure your voice is heard!

As soon as it stabilizes, we plan to publish the connector on the Confluent Hub.

  • Camunda Platform 8.1 Released – What’s New

    We’re extremely excited to announce the release of Camunda Platform 8.1.  In this post, we will go into the highlights of this release including Web Modeler, Connectors, Zeebe, Operate, Optimize, and more. Here’s the highlights of what’s included in this release: Easier Installation and maintenance for Camunda Platform Self-managed Hot backups for Self-Managed Token-Based authentication Modeler updates Process Templates FEEL Expression Editor Inline Annotations Changes to Camunda forms Connectors updates Workflow engine updates Operate updates Optimize updates Let’s get started! Easier installation and maintenance for Camunda Platform Self-Managed Improved Guides and Support for OpenShift and Amazon EKS (AWS) With this release, we are introducing official support for deploying and running Camunda Platform 8 Self-Managed on OpenShift and Amazon EKS (AWS)....

    Read more
  • Camunda Platform 7.18 Released – What’s New

    We’re extremely excited to announce the release of Camunda Platform 7.18 Here are some of the highlights: Cockpit: Improved instance selection to avoid unintended batch operations Cockpit: Easier navigation of high batch operation volumes MDC logging features Camunda Forms in Tasklist: Populate select options via process data Exception error codes Filter and order tasks by last updated WebSphere Liberty New and retired supported environments You can download Camunda Platform or run it with Docker. For a complete list of the changes, please check out the release notes. For patched security vulnerabilities, see our security notices. If you want to dig deeper, check out the source code on GitHub. Cockpit: Improved instance selection to avoid unintended batch operations Previously, when performing...

    Read more
  • Increase your resilience with new regions in...

    We’re excited to announce new regions for Camunda Platform 8 SaaS that further strengthen the resilience, performance, and data requirements for our SaaS customers. Enterprise customers can now choose to create their clusters in a number of regions including Europe West, US Central, US East, and the most recent addition, Australia South East. This provides multiple benefits, such as increasing availability, allowing to set up fail-safes, adhering to regulatory requirements for regional data storage, and reduced latency due to closer physical proximity. Resilience and redundancy in technology are important for all modern organizations and are top of mind for many. Across industries, organizations need a solution that scales effectively and is resilient to provide a good base for process orchestration....

    Read more

Ready to get started?

Still have questions?