Unraveling Connector Templates

In previous posts, we’ve shared a technical look at the Connector architecture, how to build a new Connector from scratch, how to extend an existing Connector, and how you can contribute to the Camunda Connector ecosystem. Despite all that fantastic information I found myself with some lingering questions as I built my first Connector. In this post, I’d like to take you on a whirlwind tour of the Connector template schema, to dive deep and explore what makes these templates so powerful and extensible!

Camunda’s Protocol Connectors are designed to be extensible, saving developers time by avoiding boilerplate code. I started my personal Connector journey with a simple goal: to extend the REST Connector template to build an Airtable Connector. For those not familiar with it already, Airtable is a powerful no-code data platform, and they provide a comprehensive API for working with your data.

My first goal was to create new records in an existing table, and I also wanted to create a flexible solution that would allow me to add additional features easily in the future. Inspired by Bernd Ruecker’s demonstration in the Camunda Community Summit 2023 keynote, I decided to start with a fully featured Airtable Connector template that I could then create smaller, more specific Connectors from.

Anatomy of a template

There are two pieces to every Connector: the template and the service. Connector templates are JSON documents that are used to pre-populate the properties of a service task in a BPMN model with the data required by the Connector service itself. The Connector service is a piece of code that works in conjunction with the Connector runtime (more on this in the next section) to execute a job.

Each template contains 10 core elements:

• $schema: this element is a URI to the published JSON Schema for Connector templates. This value is required and should be the same for all templates.
• name: this element is the name of the Connector, and what will be displayed in Modeler when applying a Connector to a task. It should be short and friendly.
• id: this is a unique identifier for the Connector. When creating a new template in Web Modeler, the ID is automatically generated for you.
• description: this optional element allows you to describe the functionality of the Connector, and it is visible in Modeler in the details pane to help users implement the Connector.
• icon: this element contains a base64 data URI of the icon that is displayed in Modeler.
• documentationRef: this optional element is a link to the Connector’s documentation. In Modeler, there is a documentation link in the details pane that will link the user directly to the documentation.
• appliesTo: this element is a list of the BPMN elements the Connector can be applied to. You can view a full list of supported BPMN elements in our documentation.
• elementType: this optional element defines what BPMN element type the Connector should be. When the Connector is applied to a task in Modeler, the task is automatically changed to be this type.
• version: this optional numeric field allows you to release new versions of the template without overriding the previous versions.
• properties: this element is a list of all the properties in the Connector. These properties can be part of the Modeler UI, where users can enter information specific to that process; or they can be hidden or static values.

Most of these elements I found to be very easy to understand. For instance, name, icon, and description are very straightforward and don’t require any special knowledge before assigning a value. Most of the magic of Connectors happens in the properties list. Let’s take a look at how all these elements—especially the list of properties—come together to build a Connector.

Building a base template

I started by following Bastian’s fantastic blog post. (I highly recommend using the Camunda Modeler to work with Connector templates because it offers a live preview of the properties panel on the right-hand side of the Modeler interface.) After updating the description and icon, I reached the properties section and encountered my first question: what does this code snippet actually do?

   {
     "type": "Hidden",
     "value": "io.camunda:http-json:1",
     "binding": {
       "type": "zeebe:taskDefinition:type"
     }
   },

When using with job workers, the task definition type is used to specify which job worker should execute that task. When a job worker registers with the Zeebe engine it announces what task definition types it can execute. Connectors don’t register with the Zeebe engine directly, instead they are registered within the Connector runtime. The Connector runtime acts as an extremely lightweight job worker that routes the job to the appropriate Connector. This binding is what makes that magic happen.

The value of “io.camunda:http-json:1” is simply the namespace, name, and version of the Connector itself. When the Connector registers with the Connector Runtime, this is the type of job it announces it can execute. When the engine tells the Runtime it has a task of type “io.camunda:http-json:1” to execute, the Runtime knows exactly which bit of code to invoke. In this case, it’s the REST Connector (you can see the related Java code here):

@OutboundConnector(
    name = "HTTPJSON",
    inputVariables = { /* */ },
    type = "io.camunda:http-json:1")
public class HttpJsonFunction implements OutboundConnectorFunction {

This allows you to build new Connector templates on top of existing Connectors, giving your process designer a tailored experience. Using my Airtable Connector as an example, from this base template I can now build a more specific template just for Create Records. We could then go another level deep and create a template to insert a specific record type. Next, we will explore what that looks like!

Data handling in templates

The Airtable API has many different endpoints, from managing your organization to permissions to the data itself. For this initial version of the Connector, I am only looking to interact with the data, but the API supports several different data types with multiple operations available for each. Even supporting just a single data type, such as records, would make for a very busy and confusing properties panel to support every single option.

Instead, we can extend our new base template and expose specific fields for specific operations. Here’s a comparison of the detail pane for three different iterations of the Connector template:

The first thing I did was hide fields from the base template that no longer needed to be visible. This is easily done by changing the type to ”type”: “Hidden”. For instance, the “Request Body” text area field isn’t needed when working with the more specific “Create Records” template. Then I added a field specific to this endpoint: the records to be created.

Hiding fields and defaulting their values is what allows us to create the third iteration of our template. For example, the third template hides the authentication section and assumes you have set an AIRTABLE_KEY value in your secrets. As your process orchestration practice matures, making these small changes in your templates helps ease maintenance across your processes (by centralizing key parts of the configuration of each Connector within the Connector itself, rather than within the process definition), and accelerates adding the Connector to new processes (by hiding unnecessary fields).

Any property, hidden or not, that binds to zeebe:input will have its value available as a local variable for that task. When using the REST Connector, the zeebe:input data is sent to the Connector runtime as variables for your services to consume. More importantly, any data you bind to zeebe:input is available in FEEL expressions, and FEEL expressions give us control over the data sent to the API. In other words, your process variables don’t need to be in the same format the API expects; you can build the request body however you need with FEEL. (We will see an example of this in the next section!)

Enough of the conceptual discussion, let’s actually apply this to a Connector template!

Putting it all together

We’ve defined all of the fields needed for the request: the base ID, the table ID, and the records we want to insert. Depending on which template you are in, these fields are hidden and defaulted to a value.

   {
     "label": "Personal Access Token",
     "group": "authentication",
     "type": "Hidden",
     "value": "secrets.AIRTABLE_KEY",
     "binding": {
       "type": "zeebe:input",
       "name": "authentication.token"
     }
   },

Because binding to zeebe:input allows the values to be used in FEEL expressions, we can control the request body to match what the Airtable API is expecting.

   {
     "label": "Request Body",
     "description": "Payload to send with the request",
     "group": "input",
     "type": "Hidden",
     "binding": {
       "type": "zeebe:input",
       "name": "body"
     },
     "value": "={ \"data\": { \"records\": records } }"
   },

Creating the body of this request was quite simple. Sometimes, however, FEEL isn’t enough to get the correct object for the request body. The example below, from Camunda’s OpenAI Connector, illustrates how you can combine multiple properties and FEEL expressions to get the correct object:

  {
     "type": "Hidden",
     "binding": {
       "type": "zeebe:input",
       "name": "internal_messages"
     },
     "condition": {
       "property": "operation",
       "equals": "chat"
     },
     "value": "=append(concatenate(if is defined(internal_systemMessage) then [{\"role\": \"system\", \"content\": internal_systemMessage}] else [], if is defined(internal_chatHistory) then internal_chatHistory else []), {\"role\": \"user\", \"content\": internal_prompt})"
   },
   {
     "type": "Hidden",
     "value": "={\"model\": internal_model, \"messages\": internal_messages, \"n\": number(internal_choices)}",
     "binding": {
       "type": "zeebe:input",
       "name": "body"
     },
     "condition": {
       "property": "operation",
       "equals": "chat"
     }
   },

The variable internal_messages is defined in another property using a FEEL expression, then used in the FEEL expression for the body of the request! Being able to combine the values of different properties allows developers to handle complex data with ease.

What’s next?

You can find my experimental Airtable Connector here. Camunda offers many Connectors out of the box (34 at the time of writing, and more coming with every release), and the community is contributing more Connectors every week. Explore the documentation, submit your custom Connectors to the Camunda Community Hub, and find us on the forums if you have any questions. But most of all, have fun!