Editor’s note: this post originally appeared on the Esentri blog. It was cross-posted here with Esentri’s permission. We’d like to give a shout-out to author Artun Subasi for his excellent blog post series on orchestrating serverless function with a workflow engine!
I’d like to demonstrate a Proof of Concept (PoC) which integrates Zeebe with the Fn Project in order to orchestrate serverless functions using Business Process Model Notation (BPMN).
Why integrate Zeebe with Fn Project?
Because both Zeebe and Fn Project are cool and using BPMN in a FaaS platform is exciting! If that explanation sounds a bit weird, feel free to check out my previous articles in which I wrote a lot more about my reasons:
- Is it insane to use BPMN to create serverless workflows?
- Scalable serverless workflows using BPMN, Zeebe and Fn Project
This article will be more practical including code, a step by step tutorial and screenshots.
How can Zeebe be integrated with Fn Project?
In order to integrate Zeebe and Fn Project, I implemented a Fn extension PoC and published the code on GitHub: fn-with-zeebe-extension. To demonstrate how the PoC works, I will be using a sample integration project which I also published on GitHub: fn-zeebe-integration.
The fn-zeebe-integration project contains a docker compose file which runs a Fn Server including the PoC as an extension, the Zeebe broker and the experimental Zeebe Simple Monitor. All needed docker images are publicly available in the Docker Hub.
Test Setup for the Zeebe / Fn Project integration
Let us begin with the BPMN workflow we will be using for the demonstration:
This is a slightly simplified version of the workflow I used in my experiment in my first article. Using this simple workflow, we will be able to see:
- how Fn functions can be bound to Zeebe job types to process BPMN tasks
- how Zeebe workflow variables are passed to Fn functions and vice versa
- how errors in Fn functions are handled in the workflow instances
- as bonus, how parallel execution can be handled
To process the workflow tasks, we will be creating Fn functions. In the demo repository, I prepared two functions as follows:
- get-customer-consent in Python
- send-email in Go
Requirements for the test setup
If you want to follow this article in a hands-on way, you will need the following software:
- Linux or MacOS
- Quoting fnproject.io: If you have a Windows machine the easiest thing to do is install VirtualBox and run a free Linux virtual machine.
- Fn CLI v0.5.65
- The Fn Project is being developed very actively. It may be the case, that newer CLI versions won’t work as described here. Therefore, I recommend installing the exact same version which was used in the demo:
curl –LSs https://raw.githubusercontent.com/fnproject/cli/0.5.65/install | sh
- Docker v18.09.2
- Docker Compose v1.23.2
- Following ports must be free or you will have to adjust them in the docker compose file:
- 8080 (Fn Server)
- 8081 (Zeebe Monitor)
- 26500 (Zeebe Broker)
- 81 and 1521 (H2 database)
Running Zeebe and the Fn Project using the test setup
- Download or clone https://github.com/ArtunSubasi/fn-zeebe-integration to a directory which is close to the partition root (for some reasons, Fn server does not like very long paths for which I created an issue)
- Switch to fn-zeebe-integration directory and run run.sh script. This removes old containers which were started by the script earlier, pulls docker images and starts them.
- If everything went fine, you should see Zeebe Simple Monitor by opening http://localhost:8081/ in your browser.
Deploying the Zeebe workflow
BPMN workflows can be deployed programmatically, using Zeebe CLI or using Zeebe Simple Monitor. For the demo, this last option is the easiest. Open the Zeebe Simple Monitor. Click on the New Deployment button on the top right. and choose the “customer-contact.bpmn” file from the bpmn folder. Click on “Deploy” and refresh the page.
We should now see the deployed workflow in this list. Click on the workflow key (“1” in the above screenshot) to display the workflow.
Now we should see the workflow and an empty list of running instances.
Starting a workflow instance
Similar to deploying workflows, workflow instances can also be started programmatically, using the CLI or using the Zeebe Simple Monitor. Let’s start a new instance by using the “New Instance” button on the top right. A modal dialog will appear and ask for the payload which can be used as the workflow variables.
Zeebe payloads are JSON objects. For now, we will just start a new workflow instance with an empty JSON object {} as the payload, click on “Create” and then refresh.
Note the green label on top of the “Get customer consent” task. This shows that there is one job waiting to be processed within the workflow instances.
Displaying a workflow instance
Click on the workflow instance key (3 in the above screenshot) from the list to jump into the workflow instance view.
This looks very similar, at least when we only have one running workflow instance. In the workflow instance view, we can now view the variables, audit logs, incidents, jobs, etc.
A very important feature is that Zeebe displays the path which was taken up to that point. In the above example, you can see a green path and an arrow pointing to the “Get customer consent” task. That is the path that was executed in this process instance up to now.
Here is another important point: In contrast to most BPM engines, Zeebe tasks do not know what steps to execute or what service to call. Instead the microservices, or functions in our case, ask Zeebe for jobs (pull principle). In the above example, Zeebe created a job with the type “get-customer-consent” and is waiting for the job to be completed. This is where the Fn extension kicks in. Using Fn extension, we can configure Fn functions as “workers” so that they connect to Zeebe and process waiting jobs. Let us dive into Fn project before going further.
Deploying and running Fn functions
First, check if the Fn CLI is working by typing: “fn version”. The result should be something like this except that the client version will probably not be the latest version:
Client version is latest version: 0.5.65</code>
Server version: 0.3.675Create an Fn app named “contact” which will contain our functions:
fn create app contact
The fn-zeebe-integration repository contains some predefined Fn functions in the “fn” directory. Have a look at the “func.yaml” file in the get-customer-consent directory.
schema_version: 20180708
name: get–customer–consent
version: 0.0.15
runtime: python
entrypoint: /python/bin/fdk /function/func.py handler
memory: 256
triggers:
– name: gcc
type: http
source: /customer–consentThe func.yaml file contains some metadata for the function. The Fn server loads this file to find out how to deploy and invoke a function. Deploy this function in the contact app by changing to the “get-customer-consent” directory and executing:
fn deploy —app contact —local
The output should contain:
Successfully created function: get–customer–consent with get–customer–consent:0.0.16
Successfully created trigger: gcc</code>
Trigger Endpoint: http://localhost:8080/t/contact/customer-consentNow try calling the trigger endpoint via curl or your browser and see what happens.
What does the Fn extension do?
Using the Fn extension, you can configure an Fn function as a handler to a Zeebe job type. The Fn extension starts Zeebe job workers in the background to fetch available jobs. Whenever a Zeebe job is available, the Fn extension invokes the configured Fn function and handles the needed data mappings. Depending on the function result, the Fn extension then completes or fails the Zeebe job.
Since I like BPMN, I created workflows to visualize what the Fn extension does.
The Job Worker Registration
The Fn extension hooks itself to the Fn lifecycle events in order to start and stop Zeebe job workers.
The above diagrams shows how the function configuration is processed to create Zeebe job worker. The Fn extension only creates a Zeebe job worker for a Fn function if the function contains at least one HTTP trigger and a configuration parameter called “zeebe_job_type”. A function can be configured to handle Zeebe jobs by adding the following lines in the func.yaml file:
config:
zeebe_job_type: get–customer–consentThat is all you have to configure to bind an Fn function to the Zeebe job type before deploying the function.
There is also a global configuration for the Fn extension so that the extension knows where to find the Zeebe gateway. In the demo project, the global configuration is already covered in the docker compose file. Details can be found in the GitHub repository.
The full func.yaml looks like this:
schema_version: 20180
name: get–customer–consent
version: 0.0.15
runtime: python
entrypoint: /python/bin/fdk /functi
memory: 256
config:
zeebe_job_type: get–customer–consent
triggers:
– name: gcc
type: http
source: /customer–consent
Similarly, the Fn extension also listens to other events such an function deletion, as seen above.
The Zeebe workers are also created when the Fn server is restarted. I will spare the details of this process. The more interesting part is the Fn function invocation.
The Fn function invocation process
Whenever a Zeebe job gets available, the corresponding Zeebe job worker fetches the job and reads its payload containing the workflow variables. The Zeebe job worker then invokes the HTTP trigger of the Fn function and passes the workflow variables as a JSON object in the request body.
If the HTTP request ends successfully with a 2xx return code, the Zeebe job worker completes the Zeebe job. In case of other return codes or a connection failure, the Zeebe job worker fails the Zeebe job. Depending on the BPMN definition of the tasks, the failed jobs are retried automatically by Zeebe.
Testing the Fn extension
Remember the last process instance that we started? It should be still waiting for the “Get customer consent” job. Let us deploy our modified “get-customer-consent” function and see what happens. Make sure you are in the fn/get-customer-consent directory and execute:
fn deploy —app contact —local
In the logs of the Fn Server you should see a new log entry:
Creating a Zeebe job worker of type get–customer–consent for function ID 01D779ZJY5NGAG018ZJ0000002
Refresh the workflow instance view in the browser.
Note that the variables are updated with a new variable “notificationPermitted” which is set to false. The instance is completed since the customer notification was not permitted. This is the case because I implemented the demo get-customer-consent function in a weird way. The notification is only permitted if the customerNumber variable is set to an odd number. Feel free to check the function definition in fn/get-customer-consent/func.py file.
Testing the parallel running tasks
To test the other path which should start parallel tasks to send an e-mail and an S, start a new process with the payload:
{ "customerNumber": 1 }
Your new workflow instance should look like this:
Since the tasks to send the e-mail and the SMS are connected with a parallel gateway, Zeebe started two jobs in parallel which are now both waiting.
Now let’s deploy the send-email function, which is already configured with the corresponding Zeebe job type, to the Fn Server. Change to the fn/send-email directory and execute:
fn deploy —app contact —local
The Fn Server logs should show that the function invocation returned the HTTP response code 400. We should also see that Zeebe retried the function invocation three times which is the default. Let us check the workflow instance:
The task is marked with a reddish lightning symbol. Clicking on the incidents tab shows us the error message which was returned from the send-email function. If you have a look at the function implementation in fn/send-email/func.go, you will see that the function checks for an “email” field in the payload.
To retry the job with an updated e-mail, set the “email” variable using the corresponding button on the top right.
After setting the variable, change to the Incidents tab, resolve the issue by setting the number of retries to one and refresh the view.
The workflow instance should now look like the screenshot above. The e-mail job is completed. The workflow now waits for the SMS job to be completed. You may implement and deploy a similar function to process the send SMS job. I will end the tutorial part here in order not to repeat the same steps.
Data flow restrictions
As demonstrated above, the Fn extension reads the workflow variables and passes them as a JSON object in the HTTP request body to the Fn function. Similarly, the response of the Fn function is expected to be JSON object so that it can be merged with the existing process variables. If other data types (plain text, XML, binary, etc) are to be supported, the Fn extension could be extended. However, since Zeebe variables are fundamentally stored as a JSON object, all data types would have to be mapped to a JSON object somehow.
What next?
There is an aspect that I don’t like about the current architecture of the extension. Initially, my plan was to invoke Fn functions directly within the Fn server without using HTTP triggers. This did not work out because of the possible distribution possibilities of the Fn server. In a production environment, the Fn platform can be composed of an Fn API server, an Fn load balancer and multiple Fn worker nodes. The Fn Zeebe extension currently lives within the Fn API server and calls the Fn functions using the Fn load balancer. Therefore the extension shares the resources of the Fn API server and can only scale together with the Fn API server. I tried to demonstrate the architecture in the following diagram:
However, the current implementation of the function invocation using HTTP triggers is pretty generic. An idea would be extracting the function invocation part as a generic standalone application which can bind Zeebe job types to any HTTP endpoints. Let’s call this Zeebe HTTP Connector. The Zeebe HTTP Connector can still be combined with a smaller Fn extension in the Fn API Server which listens to the Fn events and start Zeebe workers for Fn functions:
Discussion
The separation of such a connector would lead to multiple advantages:
- The Zeebe HTTP Connector can be used to bind any kind of HTTP endpoints to Zeebe workers. E.g. Fn functions, AWS Lambda functions, microservices (preferably also over a load balancer).
- The Zeebe HTTP Connector can be scaled independent of the Fn API server.
- Every time the Fn server is restarted, the current extension implementation goes over all apps, all functions and all relevant trigger definitions to check which Zeebe job workers are to be started. By separating the worker registrations and the function invocations, this can be avoided because the Zeebe HTTP Connector can store its own bindings.
- Last but not least: This would allow the Fn extension to be built using the intended ext.yaml file as described in this post on Medium I tried building the current Fn extension this way. But the built Fn Server fails to run due to conflicting Go dependencies due to the vendoring mechanism. Therefore, the current implementation uses a fork of the Fn repository which directly contains the Fn extension.
Maybe the Zeebe HTTP Connector could be a new project for Awesome Zeebe.
On the one hand, this would be yet another application in the infrastructure that needs to be monitored and scaled. On the other hand, putting the Zeebe workers into the Fn API Server can bring performance and scalability issues. I’d love to hear your ideas about this topic. Any feedback on this article is very welcome. You can also follow me on Twitter and GitHub.
Thanks
- My specials thanks go to Denis Makogon who answered countless questions of mine regarding to the Fn project.
- Thanks to everyone in the Fn Project slack who are answering a lot questions day and night, fast.
- Thanks to David Delabassée for challenging my views on my first article.
- Thanks to the Camunda and Zeebe team for their encouragement.
- Thanks a lot Carsten Wiesbaum for introducing me to Zeebe and Fn Project as well as seeding the idea for the PoC.
- Finally, thanks to esentri for giving its employees the opportunity to do the stuff they like.
References
- Scalable serverless workflows using BPMN, Zeebe and Fn Project (Artun Subasi)
- Fn Project (Oracle)
- Zeebe (Camunda)
- Camunda BPMN Workflow Engine
- How to Extend the Fn Project with new Functionality (Travis Reeder)
- BPMN tutorial (Camunda)
