From Project to Program: Managing, Monitoring and Leveraging the Cloud

How can you move beyond your first project and automate hundreds of processes successfully using an agile step-by-step approach? For the last 10 years we’ve been helping businesses to automate workflow processes, migrating from monolithic systems into agile, scalable ways of working. And we’ve discovered that you don’t have to start with a big bang approach – in fact, starting small is the fastest and most effective route to digital transformation.

You can catch up on the previous blogs in the series, if you’d like more background before diving straight into this blog, on managing decentralized workflow engines.

Managing Decentralized Workflow Engines

Instead of central platforms I advocate for an approach that every team runs its own engine, especially in a microservices context. The main advantage is to allow for scale by isolating teams.

Managing Decentralized Workflow Engines

This means that with microservices you deliberately accept a wild mix of Camunda installations! Typically it is not a problem for your teams to set up Camunda, as they will simply leverage the Camunda documentation, as well as your own best practices or samples.

But how can you get an overview of what is actually running? How can you make sure the Camunda installations have all the important patches installed? Are all engines doing well? And, in case you run the enterprise edition, you want to collect metrics from various engines to check that you are in your license limits? Typically these questions are asked by the center of excellence, your Camunda champion or an enterprise architect with responsibility for Camunda.

In a recent POC, we validated a very simple idea. We automatically harvested the relevant data from different engines within the company. In order to do so, you can leverage the out-of-the-box REST API and retrieve the data via the metrics and version endpoints. You can find a screenshot and the source code on GitHub.

REST API

Of course, you need to register the endpoints of your engines centrally. But this is actually a chance for the center of excellence to get in touch with the Camunda users. As an alternative, you could also push this data to the harvester, e.g. by writing a simple process engine plugin.

 Process Engine Plugin

Leverage Cloud to Ease Provisioning

Provisioning and governance is much easier with managed services. So running multiple engines becomes super easy with Camunda Cloud, as it already has a control plane built-in. This shows exactly the above mentioned information at a glance. It goes even further, as it allows you to update or patch engines automatically or with the click of a button.

Monitor Business Processes End-To-End

As you scale Camunda usage, an organization’s entire end-to-end process typically exceeds the boundary of one workflow engine. Maybe the process is spread across different microservices using different Camunda engines or third-party workflow engines. Or some steps are executed by legacy software. Either way, you will still need visibility into the end-to-end process.

Trying to force everybody into the same Camunda engine has not proven to be a good approach, as this would limit the independence of different teams.

Most companies rely on business intelligence or data warehousing solutions to gain that overview. While this is a valid approach, our customers report that this is not easy to set up and typically misses the business process perspective. Other tools around observability or distributed tracing are typically too technical. This is why we introduced “process events monitoring” into our process monitoring and reporting tool Optimize.

Do a step-by-step approach and avoid falling into paralysis by analysis, for example because you want to discuss the end-to-end monitoring upfront with all stakeholders involved.

Ready for more?

In our next blog, we’ll take a closer look at establishing a Center of Excellence, managing architecture decisions and process architecture, as well as landscapes.

If you’re interested in more insights, have a look at the Camunda Consulting team’s Customer Success Path, which steps through the most effective way of introducing Camunda as a new BPM platform inside any company.

  • Publishing “Practical Process Automation”

    A Book about Orchestration and Integration in Microservices and Cloud-Native Architectures I am happy to share that my new book called “Practical Process Automation” is officially published by O’Reilly today. In this book, I distilled my practical experience implementing process automation solutions from the last two decades. What You Will Find In The Book A general introduction to process automation and the different forms of automation An explanation of how developer-friendly process automation can be applied in modern system architectures and software development practices A hands-on guide to lightweight workflow engines and BPMN as the core elements to make this happen Architecture guidelines and best practices to implement your own process solutions A discussion of typical misconceptions around process automation...

    Read more
  • Camunda Closes $100 Million Series B Funding...

    I am excited to share that we just completed a €82 million (approximately $100 million) funding round led by Insight Partners. The round also included our existing investor Highland Europe from the Series A. You can find the official press release here. In this blog post I want to comment on this funding round in my own words and briefly walk you through the story of Camunda. But first things first: Who is Camunda and what do we do? Who is Camunda and what do we do? Camunda provides process automation software that is used by our customers, such as Allianz, ING, Vodafone, or Atlassian, to automate processes they need tailor-made for their business. In essence, Camunda provides software to build software. For example...

    Read more
  • Communication Between Loosely Coupled Microservices

    In our recent webinar titled “Communication Between Loosely Coupled Microservices” we got a lot of great questions and because of the limited time some were left unanswered. As community questions are really important to me I want to follow my tradition to answer remaining questions in a blog post (as I have for example also done roughly a year ago in “Webinar FAQ for Monitoring & Orchestrating Your Microservices Landscape using Workflow Automation”). What Was The Webinar About? You can find the slides (here) and recording (here) online. The webinar covered different styles of communication. This was explained by looking at how ordering food works: Synchronous blocking: A call uses a synchronous protocol, like HTTP, and blocks for the result. This is you, calling a pizza...

    Read more