Stack and Technology Questions: Monitoring & Orchestrating Your Microservices Landscape using Workflow Automation

Back in March, I conducted the webinar: “Monitoring & Orchestrating Your Microservices Landscape using Workflow Automation”. Not only was I overwhelmed by the number of attendees, but we also got a huge list of interesting questions before and, especially, during the webinar. I was able to answer some of these, but ran out of time to answer them all. So I want to answer all open questions in this series of seven blog posts – you can click on the hyperlinks below to navigate to the other entries.

In this blog, we’ll be exploring Stack and Technology questions.

  1. BPMN & modeling-related questions (6 answers)
  2. Architecture related questions (12)
  3. Stack & technology questions (6)
  4. Camunda product-related questions (5)
  5. Camunda Optimize specific questions (3)
  6. Questions about best practices (5)
  7. Questions around project layout, journey and value proposition (3)

Q: How could Camunda be used in conjunction with Apache streaming and queuing platform?

The downside of questions via chat is that you simply can’t ask counter questions to understand the exact meaning. So I am not sure what Apache streaming and queuing platform relates to. But if my consulting past teaches me just one thing … this does not need to stop me from answering 😉

Let’s assume this platform is what I know as Apache Kafka. Then there are two interesting aspects to look at:

First: There is an overlap of the use case of streaming and workflow, especially when streaming is used to build event-chains between microservices, as I described with the choreography in an earlier answer.

But, there are also good examples of how to use streaming and workflow together. My ah-ha moment was when saw one use case around vehicle maintenance, which I describe in detail in Zeebe loves Kafka: Act on Insights of your Streams.

Assume that you have a huge number of sensors that constantly send measurements (oil pressure is 80 psi) via Kafka. Now you have some clever logic generating insights based on these measures (oil pressure is critically high). All these insights are also sent via Kafka.

But now you want to act on this information, e.g. to alert an operations person to organize some maintenance. In order to do so you have to get from the world of stateless event streams into the world of stateful workflows, from a world of a massive amount of information (the measurements might be sent every second) to a world with lower numbers, especially as you want to start a workflow only once per insight.

The second interesting aspect is actually about the integration of both tools, which is exactly the next question!

Q: Integration with Apache Kafka, best practices?

Integrating a workflow engine with Apache Kafka is technically speaking relatively easy. You need to:

  • Ingest records from Kafka into the workflow engine, which is done via message receive events/tasks in BPMN.
  • Publish records onto Kafka from the workflow, which is done via message send events/tasks in BPMN (or service tasks if you prefer).

You can find some pieces of code doing this with Java and Camunda BPM e.g. in the flowing-retail example using Spring Boot and Spring Cloud Streams. Or if you use Camunda Cloud (Zeebe) you could also leverage the Kafka connector for Zeebe, as you can see in this example on GitHub.

Q: Integrating with Spring Cloud?

Yes, this is possible. Camunda BPM offers a Spring Boot starter, so it is easy to hook it into the Spring universe including Spring Cloud. There are some examples using e.g. Spring Cloud Streams or also Spring Cloud to run Camunda on Cloud Foundry/PCF.

Q: Do you have any hands-on experience with a heterogeneous Camunda architecture? If yes, do you have any suggestions or best practices?

This question would need to be revised with more details. Almost every customer uses Camunda in a heterogeneous context. I could imagine this question is related to the next two: What if I don’t develop in Java?

Q: You mentioned the benefits of embedded Camunda. In this scenario, what languages or tech stack is Camunda compatible with? We like our microservices written in Go and Node.JS.

The Camunda engine is written in Java. So it is not possible to run it embedded as a library in any other language. Sorry 🙁

BUT: You can use Camunda from other languages. I wrote about that in 2017: Use Camunda as an easy-to-use REST-based orchestration and workflow engine (without touching Java). The main idea is: Camunda provides a REST API which allows you to code in whatever language you like and just talk REST with Camunda:

This works very well for a lot of customers in different worlds, like for example Go, Node.JS, C# and Ruby.

With the upcoming 7.13 release we further improved the experience by introducing two features:
Camunda Run: A distribution of Camunda BPM that is highly configurable and doesn’t require you to have any Java know-how.
OpenAPI Support (aka Swagger): This allows you to use the REST API from the programming language of your choice, for example by generating a client for it.

Q: Is it OK to use Camunda with a full .NET solution?

Yes, of course. See my answer to the last question, which also applies to .NET. We still do not provide a ready-to-be-used client library as a NuGet package, but we are not that far away anymore. And we plan to improve support for other languages over time too.

Ready for more?

Next week we’ll be diving into Camunda product-related questions. But if you can’t wait until the next blog, you can check out the original here on my Medium site.

Ultimate Guide to Process Automation and Microservices

The Ultimate Guide to Process Automation and Microservices is the top-to-bottom guide you need to achieve success with your microservices-based architecture.

  • Monitoring Camunda Platform 7 with Prometheus

    Monitoring is an essential facet of running applications in a production system. Through this process, organizations collect and analyze data, and determine if a program is performing as expected within set boundaries. When combined with alerting, monitoring allows for detecting unexpected system behavior to mitigate exceptional situations as fast as possible. Furthermore, tracking the performance of a system enables organizations to improve those aspects that have the biggest impact with higher priority. One essential aspect of monitoring is the list of key metrics you want to observe. There are different categories of statistics that can be of interest here. To observe the defined metrics, there are plenty of application monitoring tools on the market today. They differ in many aspects...

    Read more
  • Securing Camunda 8 self-managed cluster and applications...

    Directory services are an effective way to manage an organization’s users, groups, printers, devices, and more. Most organizations accomplish this using Active Directory, Apache Directory, Oracle Internet Directory, or other similar tools. Recently I worked with a customer who wanted to see how he could secure the Camunda 8 Platform and process applications with such a directory. Their requirements consisted of: Allowing Directory users to access Camunda applications (Tasklist, Operate, Optimize) Accessing secured Tasklist & Operate APIs from our custom project Securing the custom project In this article, I’ll briefly explain the 3 easy steps taken to fulfill their requirements which include: Federate users from the Directory service into Keycloak Declare an application in Identity to access Camunda APIs Configure...

    Read more
  • Accelerate Connectivity with Camunda Platform 8.1

    We’re thrilled to announce Camunda Platform 8.1, the latest release of our process orchestration solution. This new version introduces features that accelerate connectivity to the many different systems and technologies that are required for true digital transformation, including: Create custom Connectors with our Integration Framework to accelerate connectivity New out-of-the-box Connectors for popular services Enhancements to Camunda Modeler that improve productivity Hot backups and official support for Amazon EKS and Red Hat OpenShift Plus, several upgrades requested by Camunda Platform 7 customers Organizations across all industries rely on complex technology stacks to adapt and enhance their operations in response to market dynamics, new disruptive companies, and increasing consumer expectations. Your technology stack likely includes everything from cutting-edge technologies to legacy...

    Read more

Ready to get started?

Still have questions?