We recently shared 3 Common Pitfalls in Microservice Integration – and how to avoid them and lots of you wanted more. So let’s take one step back to the things you’ll be considering before migrating to a microservices architecture and applying workflow automation. In this second of our four-blog series, we’ll look at all things architecture – starting with three basic architecture alternatives to set up your microservices landscape.
You can find the source code on GitHub for the simple order fulfillment application, introduced in the first blog in this series, available as the flowing-retail sample application which we use as an example in this blog.
The role of the workflow engine — three architecture alternatives
How can you set up an architecture using a workflow engine to create a balance between spreading your business process across multiple microservices and real visibility into the end-to-end process? Let’s simplify matters and assume we have a greenfield and only three architecture alternatives we can choose from (so no hybrid architectures or legacy).
- Asynchronous communication by commands and events — normally using a message or event bus
- Point-to-point communication by request/response — often REST
- Work distribution by workflow engine
We’re not yet looking at whether to run the workflow engine centralized or decentralized, which is a separate question we’ll tackle in our next blog post.
1: Asynchronous communication by commands and events
This architecture relies on a central bus for asynchronous communication. Different microservices connect to this bus. Orchestration logic and respective orchestration flows are owned by the microservices. Workflows can send new commands to the bus — like “hey payment, please retrieve some money for me”. Or wait for events to happen — “whoever is interested, I retrieved payment for O42”.
- Typical tools: Kafka, RabbitMQ (AMQP), JMS.
- What the workflow engine does: timeout handling, managing activity chains / the flow, support stateful enterprise integration patterns like aggregator or resequencer, consistency and compensation handling aka Saga pattern, which I spoke about in greater depth at JavaZone Oslo.
- Implementation example: https://github.com/berndruecker/flowing-retail/tree/master/kafka/java
- Pro: Temporal decoupling of microservices.
- Event-driven architecture applied right can reduce coupling; many failure scenarios (like e.g. response messages that are missing) are transparent to the developer, so they think properly about these situations.
- Con: Requires message or event bus as central component which is not easy to operate.
- Lack of operations tooling for these components leads to effort going into homegrown “message hospitals.” Most developers are not so familiar with asynchronous communication.
2: Point-to-point communication by request/response
In this architecture you simply actively call other microservices, most often in a synchronous, blocking way. The most prominent way of doing this is REST. Endpoints are typically retrieved from a registry. The workflow engine can orchestrate the REST calls and also help with challenges of remote communication — a topic I discussed in detail at QCon London.
Typical tools: REST, SOAP, gRPC; Could also be implemented with blocking messaging using request/reply queues e.g. in RabbitMQ.
What the workflow engine does: stateful resilience patterns (like stateful retry), timeout handling, managing activity chains / the flow, consistency and compensation handling aka Saga pattern as discussed in my talk Lost in Transaction.
Implementation example: https://github.com/berndruecker/flowing-retail/tree/master/rest
Pro: Easy to setup and understood by most developers; good tooling available.
Con: Calls look like they were local, so developers often forget about the complexity of distributed systems; requires resilience patterns to be applied (e.g. Circuit Breaker, Stateful Retry, etc.).
3: Work distribution by workflow engine
In this architecture the workflow distributes work among microservices, which means it becomes some kind of bus itself. Microservices can subscribe to certain work of a workflow and get tasks via some kind of queue.
- Typical tools: External Tasks (Camunda BPM) or Workers (Zeebe).
- What the workflow engine does: communication channel, timeout handling, managing activity chains / the flow, consistency and compensation handling aka Saga pattern.
- Implementation example: https://github.com/berndruecker/flowing-retail/tree/master/zeebe
- Pro: Easy to set up; good operations tooling.
- Con: Workflow engine becomes a central piece of the architecture and needs to be operated appropriately; communication between microservices only via workflows — or a second way of communication needs to established (e.g. REST or Messaging).
Thoughts and recommendation
It’s hard to give a clear recommendation without knowing the full background of an individual case. Normally I try to figure out what is already established in the company and base my decision on gut feeling about what can be successful in that environment.
For example, if a customer doesn’t have any experience with Kafka or Messaging, it will be very hard to establish this on the go. So they might be better using a REST-based architecture, especially if, for example, they are deep into Spring Boot, making some of the challenges relatively easy to solve. However, if they strategically want to move towards more asynchronism, while I personally support that, I’d still want to make sure they are really able to handle it.
If a customer already embraces Domain-driven design (DDD) and events, or even leverages frameworks like Akka or Axon, an event-driven approach that includes the workflow engine may be the best option.
So, there is a wide range of possibilities, and I think all options can be totally valid. Ask yourself what fits into your organization, what tools you already have and what goals you are after. Don’t forget about your developers who have to do all the nitty-gritty hard work and need to really understand what they are doing. And don’t forget about operations to make sure you really have fun when going live.
In the next blog, we’ll be exploring more architecture and evaluating whether you should run a workflow engine centralized or decentralized.
This blog was originally published on Bernd’s blog – check it out if you want to dive even deeper into microservices!
Interested in learning even more?
Join Bernd Ruecker, our Camunda Co-Founder and Chief technologist, for his upcoming webinar titled, Monitoring & Orchestrating Your Microservices Landscape using Workflow Automation on Wednesday March 11th, 2020. He‘ll discuss how workflow automation supports the orchestration of microservices, ensuring seamless execution of business processes even in a case of a failure.
If you can’t make it to the webinar, no worries. Sign up anyway and we’ll send you the recording as soon as it’s complete.
Getting Started
Getting started on Camunda is easy thanks to our robust documentation and tutorials
