Thinking Outside the Microservice Box: Decision Making (Part 1)

In this short series I’ll do a deep dive into some of the limitations of modern event-based microservices architectures. This is a very popular way to build software today and normally contains a number or independent services communicating to each other via a message broker or event bus. I’ll then show how looking at that architecture through the lens of process orchestration can open up new ideas and solutions for the architecture and even expand a lot on what you thought is possible for distributed systems.

This post is going to tackle the issue of how services are coordinated. Services do two things:

1. Business logic—the primary result from running the service
2. Flow logic—logic which enables communication between other services to achieve something across contexts

The flow logic (or decision logic) is almost always a part of service, but it doesn’t need to be. I’ll look at the consequences and the surprising benefits of thinking more about who should make decisions, how they should be maintained and what happens when you remove flow logic completely from your services.

The complexity of making decisions

Usually, messages or events are responsible for telling a service it needs to run. They follow Martin Fowler’s famous advice of building “Dumb pipes, smart endpoints”. At the time this was in order to avoid dumping a bunch of unmaintainable flow logic in your ESB. Instead, make your services smart enough to decide on how the flow should continue next. This means any given service is responsible for its core business logic as well as deciding how the result of that will affect the process. For example; an approval service could end successfully with a result that requires further checks. It might also prioritize the request as it’s not only approved but deemed to be of high business value. The service has two big responsibilities here. The business logic, with an internal focus of what the service has been asked to achieve, and the flow logic, which has an external focus and answers the question of what should happen now beyond the scope of the service itself.

The interesting thing about these two responsibilities is how they’re maintained. Unlike the business logic, a decision about what will happen after the current task has completed needs a broad scope and an understanding about what exists outside of the microservice. What message needs to be sent? How to prioritize it as the process continues? This requires a good deal of knowledge beyond the current scope.

A simple change—like adding an optional new service activated for priority customers—involves cooperation between teams of microservice developers, but that’s the easy part. What do you do about “in-flight” processes? Adding, for instance, a new data object, like priority, will create a status where newer instances will have the data while older ones won’t. How do you coordinate the deployment services to deal with that kind of inconsistency? All of this additional complexity is only necessary because decision making is embedded in the service when the fact is, it doesn’t actually need to be embedded in service at all.

The importance of a decision maker

A decision about the right message to send after a service has run its business logic requires knowledge of what other services are out there and the message and payload they expect. The other option is to have the recipient decided if the message is for them. Based on the payload, services can decide if they should run or leave the message for someone else. The potential for lost instances is incredibly high in this case.

So, what are the other options?  Well, we’re in the business of building services in this kind of architecture and we don’t need to look beyond the working concepts already defined by the microservice zeitgeist—build a dedicated service.

Allowing for a dedicated orchestrator service is the most obvious and compliant choice. But adding a decision making component like an orchestrator doesn’t just solve those annoying little problems; you also gain a bunch of really interesting benefits. The first is quite obvious: A single source of truth to view and make changes to your decisions. And not just for software developers—if your orchestrator is a BPMN engine like Camunda it also expands access so stakeholders and analysts can see what logic is being run. Clearly a very big improvement for maintenance, but it also offers the opportunity to version those changes and in fact run multiple versions of the same process at the same time. So now if you want to make sure that the new process version is measurably better than the old one, you can do some A-B testing to make sure.

Another great aspect of this is the increased speed of deploying changes. In the majority of cases an improvement to how a decision is made in the process will never affect any of the services. Having a single place to make updates and improvements means a lower barrier to making those changes and in turn increasing the frequency. Because when it’s hard to change something that’s not working as well as it should—people often won’t take the time to change it. It also continues to follow the principle of separating logical components into defined owners, enabling asynchronous deployments, where in the past these would need coordination and syncretism.  

Conclusion

It’s fast to build a distributed system where all business logic and decision making is embedded in your services. That’s why most people build their services that way. But it is much harder to maintain and improve those systems. If you’re building something that you’re going to want to use as the basis for a growing, improving solution you need to think about where frequent and impactful changes are going to happen. Huge impacts to performance can be made when improving an instance passing through the services in your system. But if the effort to make the change is too great—it’ll always be avoided. An orchestrator gives access to a single independent point to improve things quickly and unobtrusively.

In the next post I’ll be discussing the consequences of dealing with diverse endpoints in a distributed architecture. I’ll explore why the current approach misses the big picture and benefits of embracing a less uniform collection of services and systems.