*Camunda Platform 8, our cloud-native solution for process orchestration, launched in April 2022. Images and supporting documentation in this post may reflect an earlier version of our cloud and software solutions.
A beginners journey to the heart of process automation as a service
I joined the Zeebe engineering team at Camunda in February 2020. Zeebe is the workflow engine behind Camunda Platform 8. I was intrigued by the concept of this tool and the underlying technology.
I had dabbled with BPMN models in the past, but not in any serious or professional manner. I was very curious to learn how automation worked and what it could be used for. The question “should we use a workflow engine for this?” had come up in the past during my professional career, but was always answered with “no, that is too much overhead.” So I was the skeptic in the land of believers.
Today, I want to share my story so other beginners know what to expect. I want to paint a picture of how workflow engines work in general before I highlight some of the benefits of Camunda Platform 8. And ultimately, I did convert, and you’ll learn the reasons why.
Phase I – Disenchantment
Starting at Camunda, I was surrounded by smart people who all told me how great BPMN was and the benefits a workflow engine can deliver. But I just didn’t “get it.”
Looking back, I think I had the wrong image in my head. One image that is often used to illustrate the concept of workflow automation is the image of a conductor. For me this image didn’t work. A conductor, in my mind, is an artist. The conductor hires and inspires musicians. They decide which music to perform to delight the audience. Conducting is just a sliver of the vast spectrum of responsibilities of a conductor.
For me, a conductor is a charismatic, active, passionate person – a force of nature. And a workflow engine seemed more like…a bureaucrat. It waits for instructions, and then it waits some more for documents to come in and then it follows those instructions. Not so exciting. But then again, the CPU in your computer could also be described as a bureaucrat. It, too, waits for instructions and carries them out dutifully. Not sexy, but powerful.
So maybe I need to say goodbye to my unrealistic expectations and experience it for what it is.
Phase II – Getting to Know Each Other
I want to offer a different image for what a workflow engine does, it’s a general manager.
A general manager on their own doesn’t do much without a team. But, if you hire a general manager, provide business processes for guidance and give them employees to execute tasks, suddenly you have a profitable business.
In the same way, you need the trinity of workflow engines, business process descriptions and workers to unlock the full power of process automation.
This is what Camunda Platform 8 does, and more. Camunda Platform 8 offers on-demand process automation as a service. It is designed for cloud-native environments maximizing performance, scale and availability. It is built with a powerful execution engine called Zeebe for running BPMN workflows at scale, and paired with essential tools for businesses to collaborate, monitor and improve workflows.
With Camunda Platform 8, you have a general manager (Zeebe), an accountant (Operate) giving you real time insights into the operations of your business, a foreman (Tasklist) to organize manual tasks and a facility manager (cloud console and backend) that makes sure everything is running smoothly. Pair that with experts to optimize and model your business processes and you have a powerful team at your disposal.
Phase III – Let’s See How it Goes
An interesting challenge appeared after working with the team for a couple of months – we wanted to automate complicated end-to-end tests for Zeebe. We also wanted to run tests in the cloud and ideally do so for different resource configurations – called cluster plans. So I pitched the idea that we could use Zeebe to test Zeebe.
I was always fascinated by that pivotal point in the evolution of a new programming language – when the compiler is written in the language that the compiler should compile. Plus, internally we had been looking for opportunities to use Zeebe more widely.
I started drafting some processes using Camunda Modeler. Then adding some workers and soon some more workers. But the process needed refinement, and suddenly I started appreciating the real power of Camunda Cloud. Can testing be done in different cluster plans? Sure, just use a multi-instance.
Can tests be run with a daily schedule? Sure, just use timers and event subprocesses.
Today, we have a modest testbench. Tests run every day, in all cluster plans, and results are published to a Google Sheet. If there is an error, developers will be notified through Slack. It’s been running in production for almost a year now. There were a couple of kinks in the beginning, but overall it’s been smooth sailing.
Phase IV – Falling in Love
What was most surprising is that it was so much fun. Here are the top three things I discovered and now love about Camunda Platform 8:
- It’s developer-friendly
- Has future proofing
- Saves time by leveraging code I don’t need to write
Developer-friendly
Like I said, it was so much fun working iteratively with Zeebe. Just starting with a draft process, and a draft worker, then refine from there. And of course the first iteration is buggy. We all know how tedious it can be when an exception occurs, is logged away and now you are tasked to figure out what went wrong and how to fix it.
But with Camunda Platform 8, rather than looking through log files, those exceptions were elevated to incidents. All I had to do was open Operate and look at them.
Let me explain the difference it made – it’s the debugger of my dreams:
- It runs in production
- When an error occurs, it pauses the invocation in which the error occurred. But all other work continues undisturbed.
- It shows the variables at the time the error occurred
- It shows the path that led to the error
- You can change the variables, and retry (again and again)
- You can also keep the process paused, drink tea, commiserate about what went wrong, have an epiphany, implement a fix, sleep, wake up, implement a test for your fix, redeploy the worker and then retry the job
That is what Camunda Platform 8 provides.
Future proofing
The other thing that you get is a good deal of future proofing. I deliberately started out small and simple. For example, the job to publish the test results just writes to the end of a Google Sheet. That probably won’t scale. But I am not worried about that. Maybe we want to have sheets per month, or sheets per test, or we want to move away from Google Sheets altogether. All I know is that if and when the time comes, I only need to reimplement that particular worker.
Same thing with deployment. As of now, all workers run in a single container. But if one of the workers gets busier, I could move it into its own container and have it scale up and down using Kubernetes.
And finally it’s so easy to just change the processes. I have already made adjustments and improvements over the life of this project:
And the process to prepare a cluster became a lot more elaborate:
All this was easy and very enjoyable to implement.
The process models and workers don’t even have to live in a single repository. Another team can utilize the workers provided so far, and simply add their own custom processes and workers. With independent deployments and everything.
Best of all, I was able to stick to my bread and butter skills – in this case, Java. But I could have implemented the worker in any of the languages for which a client exists. To that end, Camunda Platform 8 is flexible, open and extensible.
Code lines and time saved
The whole solution comes in at about 2,000 lines of code (LoC) at the time of writing. And this fact continues to amaze me. It feels like that number should be much higher.
And the reason is that the calculation does not account for the LoC that I didn’t need to write. A more accurate assessment should include the 2,000 LoC that I did write, plus the ~ 50,000 LoC from Zeebe which I didn’t write, the X LoC of Slack client that I didn’t need to write, the Y LoC of Google Sheets client I didn’t need to write, and the Z LoC of internal APIs that I also didn’t have to write. Add these up and you get closer to the 50,000 LoC total of Zeebe.
And this is exactly the beauty of it. There is so much code that I didn’t need to write, test, babysit into production or maintain. And all that translates into significant time savings.
Phase V – Inseparable
I am happy that I overcame my initial struggles and doubts with this technology. I am grateful that I had the opportunity to experience Camunda Platform 8 from the perspective of a customer. And the experience further increased my motivation to work on the product.
Camunda Platform 8 does a great job lowering the barrier to get started. And it’s getting better and better with each release. But there is still a learning curve related to BPMN that might put some people off. To all people struggling, I can only say, keep going!
Yes, it takes time to get familiar with all the moving parts, but mastery only comes with experience. And it is definitely worth it.
