Understanding Process Migration

Consider the scenarios where process migration makes sense. Learn how and why you need to migrate (if at all). Understand the Migration Plan, how the Job Executor comes in to play, and reference some advanced techniques.
Understanding Process Migration is also related to
  • Process Migration
  • Migration Plan
Understanding Process Migration

Process Migration Defined

In Camunda, when a new version of an existing process definition is deployed to the process engine, that definition will first be parsed and checked for necessary attributes. If there are no errors, then it will be compared to the existing version of the process definition in the database. If the content of the BPMN has changed in any way then the process engine will consider the incoming definition a new version of the existing definition. The integer representing the version will be incremented by one. By default, any request to instantiate a process will automatically use the latest version of the process.

Over time (and depending on the typical time-to-live of your process instance), you could have process instances of the previous version still “in flight” while process instances of the latest current version are also being executed. Depending on the changes made to both process definitions and to supporting artifacts in the deployment, this may or may not cause undesirable issues. For instance, what if a new version actually fixes some kind of error or bug in a previous version? If this is the case, then you might not want the old process instances running on the old process version. Ideally, you could “migrate” a running instance from an old version to a new version seamlessly. In Camunda, you can start by utilizing the Migration API.

Things To Consider Before Migration

When planning your migration, here are some factors to consider:

  • Do I have a good reason to migrate? Technically, you do not have to migrate process instances when using Camunda. Previous process definition instances will simply continue to run as intended (with some important caveats….see other things to consider below). Here are some examples of good reasons to migrate:

    • Your supporting implementation resources have changed.

    • Your latest process definition represents a substantial change in your business process

    • Your latest process definition fixes a bug.

    • Your latest process definition enforces some time-sensitive legal obligations or rules.

  • How big of a difference is there between process definition versions? Not only the definition itself, but the data required to be present at any given time in your instance.

  • Did supporting implementation resources change from the previous deployment? If a service implementation changes in the new deployment and the reference to the implementation did not change from the previous deployment, then older process instances that are in flight will utilize the newer implementation by default upon deployment of the new resources. If that breaks older instances, then you must migrate.

  • Do I have a proper infrastructure to support “real data” testing of my migration plan? This might be the most important aspect. An ideal way to test your process instance migration would be to have prod-like data in some kind of staging environment that represents not only the type and quality of existing production data, but also volume, scale, and size. You run your migration plan there so that you know what to expect when it comes time to migrate in production. You also need the ability to quickly reset this data via some kind of snapshot, so that you can test over and over again. You can expect many iterations of your migration plan before you move forward with a solid plan.

Steps for Executing Process Migration

This section will give a high overview of what is already documented in the Camunda documentation. docs.camunda.org/manual/latest/user-guide/process-engine/process-instance-migration/

Create Migration Plan

A migration plan is simply your description of which activities in the source definition you would like to map to the activities in the target definition. This is done in the API as an “object” represented in the form of key/value pairs. For the Java API, that means using some nice chain methods off of the runtime service. For the REST API, you would simply send a payload in the form of a JSON object (see the endpoint documentation for more details on the format here docs.camunda.org/manual/latest/reference/rest/migration/).

Execute Plan with the Migration API

You use the RuntimeService to execute your migration plan.

The API also allows you to set/override some other important parameters when you decide to execute your plan. Things like:

  • skipCustomListeners: A boolean value to control whether execution listeners should be invoked during migration.

  • skipIoMappings: A boolean value to control whether input/output mappings should be executed during migration.

As in migration plan creation, execution of your plan can be done with either the Java API or the REST API. You can also create and execute this plan from the Cockpit UI if you are running the Enterprise version of Cockpit (see the BONUS section below).

You must also choose a synchronous or an asynchronous method of migration execution.

Things To Consider During Migration

Volume and the Job Executor

The Migration API allows you to execute your migration plan either synchronously (blocking) or asynchronously (not blocking). Read the descriptions on the pros/cons here docs.camunda.org/manual/latest/user-guide/process-engine/process-instance-migration/#executing-a-migration-plan .

If you choose to use a synchronous migration, then no migrations will be committed until the thread is complete. This is a true “all or nothing” transaction. It will succeed or fail as one. This is an approach that makes sense for a small amount of process instances.

However, if you choose to use asynchronous migration, then you are creating a single Camunda batch. More on batches here: docs.camunda.org/manual/latest/user-guide/process-engine/batch/

Let’s say that you had 10,000 process instances to migrate from version 2 to version 3 of a process definition, and you wanted to use asynchronous migration. If you executed all of these in a single batch, then the batch could add thousands of individual Camunda jobs to the job table all at once. These would then get added to the Job Queue to get in line for the available threads in the thread pool. This technique could dominate resources very quickly, especially when it comes to communication with your supporting database, resulting in a very slow migration. This could also affect the performance of a live Camunda system for process definitions that are not actively being migrated.

Knowing how this works ahead of time will allow you to plan accordingly by controlling the size of your batches so that the migration is more “digestible” to your system. See “Advanced Techniques” below for some examples of how to handle this scenario.

Clustered Architectures

It is important that we take a moment to speak about different kinds of Camunda architectures, and how the migration action might look on each.

Homogeneous Cluster

In a homogeneous clustered (same redundantly bundled Camunda application on each node) architecture using a shared, container-managed process engine, you are in an ideal situation for migration. There is one process engine, and one database. You can trigger migration from any deployed Camunda application, or directly from the REST API. The migration is represented in the end by the change of data in the shared database.

If your homogeneous cluster is using embedded applications that each have their own running Job Executor (an example would be if you were deploying many Spring Boot Camunda implementations), then you just need to be aware that all Job Executors are sharing the same Job table. It is unnecessary to trigger the same migration in all nodes. It is only necessary to do this in one node. If you have multiple processes to migrate, you save nothing by triggering one in each node, because your jobs will simply pile up on the job table and get pulled into the job queue in a linear fashion. Job Executor threads from all available nodes will pick up jobs from the queue and work them as long as they have threads from their own thread pool.

Heterogeneous Cluster

In the case of a heterogeneous clustered architecture (Camunda application resources are bundled differently for different deployed applications), it is critically important that your nodes are utilizing the "deployment aware" flag for the process engine. This will guarantee that your migration will not fail while utilizing a resource that might be in one deployment node, but not the other.

Process Instance Data Requirements

It is possible that a new version of a process definition might require the existence of a newly created process instance variable (or many) in order to function properly. This variable might need to be initialized in order for a gateway or a business rule task to work properly, for example.

Also, the value of these new process instance variables might not be constant. It might need to be calculated dynamically at migration time depending on not only the existing values of the current process instance, but also where the token is in any given instance as well.

Different techniques can be used in combination to update process instance data using the API during migration. Read below for the "advanced techniques" to see some examples of doing just that.

Bonus  : Using Cockpit EE Wizard For Migration

If you are using the Enterprise Edition of Camunda Cockpit, then you can utilize the Migration functionality that is built into the product. This is a graphical wizard that offers the following functionality:

  • Graphical representation of the mappings from source to target that is interactive (i.e. drag and drop mappings directly in the UI)

  • Filter your choice of process instances from the source definition that you want to migrate to the target definition.

  • Generate your Migration Plan object.

  • Have the option at the end of the wizard to immediately execute your plan.

migration cockpit ee

Bonus  : Advanced Techniques

If you would like to have greater control over your migration, including manipulating data during the migration, or controlling how many process instances migrations you execute at one time, then it could be beneficial to simply create your own business processes that represents all actions that need to be implemented and orchestrated in order to successfully migrate your process instances. You could author JavaDelegate implementations in Service Tasks in BPMN that utilize the Migration API provided by the Camunda Java API directly. This gives you a lot of flexibility and power to do what is needed for your own custom migration.

Read the readme for all available use cases in this Github repository for descriptions as well as instructions to run immediately.

No guarantee - The statements made in this publication are recommendations based on the practical experience of the authors. They are not part of Camunda’s official product documentation. Camunda cannot accept any responsibility for the accuracy or timeliness of the statements made. If examples of source code are shown, a total absence of errors in the provided source code cannot be guaranteed. Liability for any damage resulting from the application of the recommendations presented here, is excluded.

Copyright © Camunda Services GmbH - All rights reserved. The disclosure of the information presented here is only permitted with written consent of Camunda Services GmbH.