The Zeebe team just implemented a Kafka Connect Zeebe connector. This is a feature users have been asking for since the start of the project, so let’s give it a closer look.
What is Kafka Connect?
Kafka Connect is the ecosystem of connectors into or out of Kafka. There are lots of existing connectors, e.g. for databases, key-value stores or file systems. So for example you can read data from a RDMS and push it to Elasticsearch or flat files.
A Kafka Connect example from https://www.confluent.io/blog/the-simplest-useful-kafka-connect-data-pipeline-in-the-world-or-thereabouts-part-2
Kafka Connect Zeebe
You can easily develop your own Kafka Connect connectors, and that’s what we did for Zeebe. Based on a POC, which I showed in a talk at Kafka Summit San Francisco (recording available), the Zeebe team cleaned up the code (which is pretty important if I wrote parts of it ;-)) and just released a version that is suitable for real-life usage. You can find it on https://github.com/zeebe-io/kafka-connect-zeebe.
What can the Kafka connector for Zeebe do?
The connector can do two things:
Send messages to a Kafka topic when a workflow instance reaches a specific activity. When I say message, I really refer to records in Kafka, where a lot of people also simply speak of events. This is a source in Kafka Connect speak.
Consume messages from a Kafka topic and correlate them to a workflow. This is a Kafka Connect sink.
Use cases and examples
A simple example to test if the connector is working for you is the ping-pong example on GitHub. It simply sends a message to Kafka which is directly correlated back.
There are multiple use cases for the connector, and let’s quickly highlight two of them.
A common use case is around microservices orchestration. We have a simple example around the domain of order fulfillment in the microservices-orchestration example on GitHub.
It leverages the connector’s source to push a message to Kafka whenever a payment is required, expecting some payment service to process it and emit an event (or send a response message) most probably asynchronously. This response is correlated to the waiting order fulfillment process using the connector’s sink.
Another great example is around messaging integration patterns. For example we might want to wait for a couple of events to arrive before we want to trigger some activity. If events are missing within a certain time frame we need to escalate.
Quickstart and screencast
You can easily follow our quickstart guide to play around with the ping-pong example. Or you could also sit back, relax, and watch this screencast:
Status and roadmap
The connector is basically ready to be used. We do have some open issues, but we want to wait for more real-life user feedback before we decide how to move forward and what to prioritize.
So if the connector could be useful for you, try it out today and make sure to let us know how it’s going (via the forum or Slack). Also, don’t hesitate to open issues or vote for existing ones to make sure your voice is heard!
As soon as it stabilizes, we plan to publish the connector on the Confluent Hub.
