If you attend a software development conference and ask the attendees whether the application(s) they work on use microservices, the answer is an overwhelming “yes.” Microservices provide a scalable infrastructure that not only improves performance by scaling up when there is a heavy load, but also helps save money by scaling them down when the extra processing power isn’t needed.
Follow that question with “How do you feel about microservices?” and you will get varying responses. Visit any online programming forum and you’ll find a wide range of opinions on microservices. Why? A single paradigm is rarely the answer to anything. There are places where it makes sense; there are places where it doesn’t; and most things fall somewhere in the middle, with people trying to find a balance that suits their company’s needs.
In the end, modern software architectures like microservices are great at optimizing resource usage and cost … but have you ever had to explain to a non-technical stakeholder what happens when a certain action is taken in your application? How does data get from A to B and what happens in between? Try diagramming it …
… it looks like you threw spaghetti at the wall! It’s quite hard to diagram data and logic flows using an architecture diagram. You ultimately end up with a list of dependencies and relationships, instead of a meaningful picture of what happens when a user takes an action.
Not just that, but many modern software architectures have problems that are difficult to solve. For instance, what if a particular data flow requires manual approval? Or what if you need to add timers (alerts, stuck or idle processes, notifications) or interrupting messages? What happens when the stakeholders slide into your DMs with a non-trivial change to the existing business logic? These are, of course, not impossible issues to solve, but if you didn’t account for these needs when designing the system, they can often be quite difficult to add later.
What if there was a better way?
Forget services—think processes
As developers, we tend to think in terms of systems: this code gets deployed here; this data gets pushed to this database; this message goes into this queue, which is picked up by those systems. This is how I was taught to think about developing applications and it is still my default mental model: Start with the architecture.
For just a moment, forget that mental model. Close your eyes and take a deep breath. I want to introduce you to another way of thinking—not in terms of services or systems, but in terms of processes.
The dictionary defines a process as “…a collection of related, structured activities or tasks performed by people or equipment.” Put another way, your application is made up of a series of processes—some independent, some not—that have a defined start and end.
Take, for example, this loan application process. Nearly every financial institution in the world follows a similar process. It starts when the applicant submits the loan application. A series of steps is then taken to either approve or deny the loan. This isn’t a linear process: not all steps are taken for every application. There are decisions involved that change what happens next (in this example, whether the loan was approved or not).
As a thought experiment, try imagining what the software design/architecture may look like to support this process. I would guess that everyone who reads this article will imagine a different system. That’s because this process doesn’t care about whether you’re using microservices or a monolith, whether it’s running in the cloud or on prem, or what language it’s written in. Thinking instead about the process, not the services, allows the other implementation details to fit in later.
Let’s continue this thought experiment with a real world scenario.
Designing a real-world process
When you’re driving through town (in the United States at least) and you see a very large truck and trailer driving by, very often that truck had to apply and pay for a permit to lawfully drive on those roads. What permit they apply for and how much they have to pay depends on a lot of factors: the size of the truck and trailer, the route they are taking, how long it will take, whether it is a one way trip. But the process the trucking company follows is roughly the same every time: apply for a permit, get matched with a permit and a fee, pay the fee, and the permit is issued.
Continuing that thought experiment, take a moment and imagine what the software underneath looks like. In my head, I see a UI that serves a form to fill out, a service that takes the form data and matches a permit, another service to collect the payment, then a final service that issues the permit. Perhaps those are independent microservices or perhaps those are just different classes inside a larger application, but it feels like a very straightforward implementation to me.
But then…
NEW REQUIREMENT!
You’ve started building the software and have written a lot of the code, when the stakeholders show up with a new requirement: “We need the permit turned into a PDF, and email notifications sent to the user and to us.” Because you are trying to problem solve by thinking in terms of processes, you update your process diagram first:
Easy! This new requirement doesn’t change any of the existing plan, you just need to build two new services: one to generate a PDF, and one to send out emails. Then, when the permit is issued, the system can generate the PDF and send the notifications out. That’s not so bad! You write up the new user stories and get back to writing code…
NEW REQUIREMENT!
In the middle of working on one of the services, the stakeholders arrive with a new requirement: “A user tried to fill in the form, and it doesn’t match a permit, it needs a manual review!” You open your process diagram and update it…
This is more complicated than the previous requirement, but at least the “Match Permit” service you already wrote doesn’t need to change much. Somehow you need to build a way for a manual validation to happen, and for that to route to the payment step…
NEW REQUIREMENT!
Another new requirement: “If they don’t pay in X days, the permit needs to be canceled!” This one sounds a little harder than the last requirement, and affects a service you’ve already started development on! You open the process diagram and stare at it, wondering how you should implement this new requirement… and how do you draw it in this process diagram?
Introducing BPMN
Want to know a secret? That process model you’ve been working on? It’s awfully close to a BPMN model! BPMN, or Business Process Model and Notation, is an open standard maintained by the OMG Group (who also maintain other open standards like UML). BPMN has a lot of advantages over drawing a process diagram with a tool like Visio or LucidCharts—primarily that it is XML under the hood, which means it can be easily read by software and also easily checked into your source control and versioned with your releases. Because BPMN is designed to model processes, it is a natural fit for this method of problem solving.
To better illustrate what BPMN is, let’s take a moment to update our previous process diagram to use BPMN:
It doesn’t look significantly different. There is an added start and end event to clearly define the start and end; and there are two additional gateways (the diamond shapes) to make sure we bring all the paths together cleanly. The only other difference are those little icons on each box. In BPMN, each of those rectangles is a “Task” and each task has a specific type. BPMN supports many different task types. In this example we have two: a Service Task (the gear icon) and a User Task (the person icon).
However, this post isn’t intended to teach you BPMN (if you want a BPMN tutorial, head here). It’s meant to introduce a different way of thinking about solving problems. For now, it’s enough to know that BPMN is an open standard that allows you define processes in a visual manner, and to give some additional context to each task (for instance, with the task types).
NEW REQUIREMENT!
Remember the last requirement? “If they don’t pay in X days, the permit needs to be canceled!” Let’s not worry about how to implement it yet, but let’s start with adding it to our BPMN model:
In my opinion, this diagram is still very understandable, even if you know almost nothing about BPMN or code. Some sort of timer event happens, and the permit is then canceled, and the process ends.
I can hear what you’re thinking: “That’s great, dude, but it’s still just a picture…”
… or is it?
Introducing BPMN Engines
This is why BPMN matters: Because it is XML, and because it is an open standard, you can execute a BPMN model with a BPMN engine! There are many BPMN engines out there, but not all support the full BPMN specification. (You can view a list of engines and their support for BPMN here.) Camunda, of course, is a BPMN engine!
I’m sure you have a lot of questions right now—primarily, “That’s a neat animation, but how does an executable workflow diagram help me?” In order to answer that, let’s expand our diagram a little bit:
Remember how each task has a type associated with it? We can “connect” each of those tasks to a service in our application. When the process encounters the “Match permit and get fee” task, it will call the “PermitMatch” service, get the response, then move to the next step.
So how does the engine know how to call the PermitMatch service? That’s the best part: it’s entirely up to you! Exactly how you implement it may vary depending on the engine you choose, but if you choose Camunda you have three primary options:
- A job worker: This is a piece of code that you write that runs in your environment and performs a single task.
- A custom Connector: A Connector is like a reusable job worker, with predefined inputs and outputs.
- A RESTful API call: A REST Connector is provided out of the box with Camunda, allowing you to make RESTful calls without writing any additional code.
However, these details don’t matter too greatly yet. Here’s the key point:
The state of a process is managed by the workflow engine, not the application. The data is managed by the application, not the engine.
NEW REQUIREMENT!
Just as you start learning about BPMN and BPMN engines, the stakeholders sneak in another new requirement! “For some permits, they need additional approval before collecting payment.” Great… the payment collection service, which you’ve already completed, now needs more changes!
I know we just spent time talking about BPMN, but take a moment and think about how you might accomplish this in code. The form is submitted by the user and it goes to a service to match a permit. If no permit is matched, it needs to go to a manual review. Otherwise, the process checks to see if additional approvals are needed before proceeding to payment. There is another path, too, when a manual review is needed. There are two different, unique places that the decision to require another approval may be coming from. It’s not impossible, but there isn’t a particularly elegant solution either.
Now let’s consider the BPMN model:
Another decision gateway is added if the permit needs an additional approval, and that’s it! You don’t need to make any changes to the core logic inside your application, because the process engine is handling the state. As long as the result is either an automatic permit match or a flag for manual review, the process will work exactly as needed to fulfill the new requirement!
NEW REQUIREMENT!
“Oh, and we should probably send a payment reminder before we cancel the permit.” Of course the stakeholders come in with another requirement, but this time you’re ready with BPMN and your process engine:
Now the “Collect payment” task has two timer events. The one with the solid outline is an interrupting event, which will end the process execution when it triggers. The timer with the dashed outline is a non-interrupting event, so it can trigger multiple times without canceling the larger process.
The best part? Because you already have an email service to send notifications at the end of the process, you can simply call that existing service to send the payment reminder, and you have added the new requirement without writing any additional services or logic. You added the timer event to the process model, the new “Send payment reminder” task consumes your existing email service, and done!
(You may have noticed some other changes to the model. While you were at it, you cleaned up the diagram by adding an error event to the “Match permit” task. Now that task just needs to return an error that it couldn’t match a permit, rather than returning a flag saying it couldn’t. While it’s functional and valid BPMN either way, I personally find this model to be a bit nicer. What do you think?)
Let’s Review
My goal in writing this is to give you another tool in your developer toolbox. Certainly this approach doesn’t work for all problems; process automation with BPMN works best with processes that are run frequently or are long running. That one clean up job that executes every quarter? That still might work best as a cronjob.
When I was learning to be a software developer, I was never taught to think about solving a problem in this way. Every microservice orchestration tutorial I’ve read solved these problems with message queues (or a similar concept). After learning BPMN, I can think of several projects (including one related to truck permitting!) that could have benefited greatly from this approach.
If you are new to BPMN and business process management, you likely have a lot of questions. That’s OK. Camunda has some great resources to help you get started. I’ve linked some of them below. Or you can join us on our forums and see how others are using BPMN to solve their problems. I hope to see you there!
- Camunda Academy—free courses on BPMN and Camunda
- BPMN 2.0 documentation
- Camunda Code Studio on YouTube
- Camunda Forum
- Camunda free account (signing up lets you create unlimited BPMN models, plus more)
Start the discussion at forum.camunda.io