Reporting About Processes

The Camunda engine automatically collects audit information about historical process, case or decision instances. Leverage this data by generating and displaying business relevant reports - preferably in your own in house reporting solution. Add business relevant phases and milestones to your process models serving as a basis for key performance indicators (KPIs).
Reporting About Processes is also related to
Reporting About Processes

Modeling Key Performance Indicators (KPIs)

When modeling a process, we always add information about important key performance indicators implicitly, like by introducing start and end events.

On top of that one can explicitly add

  1. Meaningful additional business milestones by modeling e.g. intermediate events. This might not have any execution semantics other than to leave an entry in the history tables of a process engine: the milestone is met as soon as the process has passed the event. Its status can therefore be passed or not passed.

  2. Meaningful business phases by modeling things like (embedded) sub processes. In contrast to a milestone, a phase’s state can have a variety of statuses. not entered, currently active or passed.

Consider the following example - a "Tweet Approval Process" showing start and end events as well as milestones:

1 After one business day, the reviewer is reminded to speed up - and such reviews are internally marked by passing the end event 'Review done slowly'.
2 Approved tweets will pass the additional intermediate event. The cycle time up till that point is automatically captured too.
3 Furthermore, when tweets are successfully published, we are interested in the ratio of those tweets…​
4 …​when being compared to tweets that do not get published. Therefore, we model two different end events representing those two business end states of the process. Note that duplicate tweets will be not published even though they have been approved before. The more precisely we describe and name the business semantics of events, the better our KPI’s will reflect the reality we want to measure!

When you do not (only) want to concentrate on milestones, but phases in your process, model the phases as sub processes:

1 The phase Review - modeled with a sub process - will be active while the human reviewer will need to find time to complete the task…​
2 …​ whereas the phase Publication will be completed automatically - hence process instances "remaining" there for longer than a few seconds will probably indicate ongoing problems with the uptime and reachability of the used services.

We can now dig into in more interesting and useful things by accessing the process engine’s historical data!

Accessing Historical Data

In order to generate a report, one must access the appropriate historical data produced by the process engine.

Camunda saves this "historical data" not just when a process instance finishes, but also 'on the go' - while a process instance actually runs along certain paths of the underlying process definition. By doing this Camunda separates runtime data from history data - a growing history will not influence the runtime behavior: you should never need to access runtime data for pure reporting purposes!

Historical data can be leveraged via three possible mechanisms:

  1. Query API: Using the public Camunda HistoryService API via Java or REST.

    This has the advantage of being a very robust approach, because this is public API and has certain backwards compatibility guarantees.

  2. SQL: Accessing the Camunda History Tables directly via SQL (see the Documentation of the Database Schema).

    This has the advantage that the data can be retrieved with an improved performance. Therefore, very complex reporting scenarios become possible, too. However, the approach is less robust, because the database schema can principally change in between Camunda releases.

    A lot of generic reporting tools can do reporting on SQL datasources out-of-the-box, making this a frequently chosen option.
  3. Pushing Events: Pushing Camunda history events exactly at the moment they occur.

    This has the advantage that the history database does not need to be queried for reporting. See the following section about BI and DWH to learn more.

    To achieve "push", you can choose to implement a custom history backend which redirects all events or to log just certain events with a simple process engine plugin / parse listener or Spring Eventing Bridge . Additionally you can find an example of a custom history handler here

Using Camunda Optimize

The usage of Camunda Optimize makes reporting regarding your processes very simple.

Camunda Optimize provides continuous monitoring and insights about your deployed business processes. This Big Data solution helps process owners to make informed decisions in order to optimize their processes. It allows you to build reports and dashboards including setting alerts for thresholds.

Optimize uses ElasticSearch for persistance and automatically pulls the data from the Camunda History and stores it in a data structure that is optimized for reporting purposes. During the import routine it is possible to import business data from external systems that are not stored within the Camunda database.

For more information on how to use Optimize please visit documentation page or request help from Camunda.

Using Business Intelligence (BI) or Data Warehouses (DWH)

You might move data from the Camunda History to a decoupled system like a Business Intelligence (BI) solution, a Data Warehouse (DWH) or a Search Server like SOLR. This has the advantage of optimal performance, because the Camunda database does not need to be queried for reports. By doing so your reports can never break runtime performance.

Leveraging typical BI system’s ETL (extract, transform and load) features allows to optimize data structure for your reporting purposes (to speed up report generation) or to combine generic process engine data with business entities (to allow for more in-depth analysis).

To get the data into the BI system, you can leverage one of the mechanisms described above, hence either

  1. Pull a Camunda history delta on a regular basis (via either API or SQL), or

  2. Push all (or selected) Camunda history events right at the time when they occur.

Selecting a Reporting Solution

Understanding the principal ways to access your historical process execution data as just discussed is the first step to making decision on the matter. The further help you make up your mind about the nature of the reporting solution you are already using or now considering to introduce for process reporting purposes, take a look at the table below.

Camunda
Optimize
(Existing)
BI Solution
(Existing)
Report Generator
Custom
Statistics App

Use Camunda Optimize (available out-of-the-box with Camunda EE) to create reports and dashboards for your processes.

Load Camunda History into BI tools and leverage features for reporting and analytics of this tool.

Use report generator to build business readable reports from Camunda History database.

Program a Cockpit Plugin or custom frontend to visualize business readable reports.

Data Source

Elastic Search

Separate (BI) Database

Camunda Database

Camunda Database

Data Scope

Camunda + Business Data (if accessible)

Camunda + Business Data, powerful pre-processing possible

Camunda + Business Data (if available in same database)

Camunda + Business Data (if accessible)

Report Placement

Camunda Optimize Dashboards

Inside BI solution

Dependant on Report Generator, e.g. Custom Application or Standalone HTML page

Wherever you want (e.g. Camunda Cockpit)

Our Greenfield Choice

Technical Approach

Just use it

ETL data into BI solution.

Pull data (typically SQL) and generate report.

Pull data (Java or SQL) and visualise it e.g. with Javascript.

Report Generation Performance

High. Data Structure optimized for report and analysis around your processes.

High. Data Structure optimized for report and analysis.

Lower. All data needed for report generation is directly pulled.

Bonus  Reporting Examples

Camunda recommends using Camunda Optimize or Camunda Cockpit instead of the following report examples. Of course, if there are any limitations for using Optimize the following recommendations are still valid.

SQL + JavaScript, deployed as Cockpit Plugin

  • Query relevant data via SQL,

  • Provide the result via REST and

  • Visualize it using a Javascript framework.

This can be deployed as a Camunda Cockpit Plugin and might be directly leveraged to show cycle times, ratios and number of started instances over time.

KPIs via Javascript + JSF

Another Example shows how to generically query and display either milestones, modeled as intermediate none events or phases (modeled as embedded sub process):

1 A milestone is met as soon as the process has passed the event. The status can be: not passed or passed.
2 A phase, in contrast, can be not entered, currently active or passed.

Bonus  Sample Java-Queries

Query all passed milestones, modeled as Intermediate None Event as described above:

    List<HistoricActivityInstance> activities = engine.getHistoryService().createHistoricActivityInstanceQuery() //
        .activityType("intermediateNoneThrowEvent")
        .orderByActivityId().asc().list();

Query all process instances currently in a Sub Process as used for phases as described above:

    List<HistoricActivityInstance> activities = engine.getHistoryService().createHistoricActivityInstanceQuery() //
        .activityType("subProcess")
        .unfinished()
        .orderByActivityId().asc().list();

Bonus  Sample SQL-Queries

Here we list SQL, which is tested on H2. The queries might differ on other databases.

Retrieving Cycle Times

SELECT
  durationInHours AS DURATION_,
  count(*) AS COUNT_
FROM (
  SELECT actinst_start.PROC_INST_ID_, actinst_start.START_TIME_, actinst_end.END_TIME_, timestampdiff(HOUR, actinst_start.START_TIME_, actinst_end.END_TIME_) as durationInHours
  FROM
    ACT_HI_ACTINST actinst_start,
    ACT_HI_ACTINST actinst_end
  WHERE
    actinst_start.ACT_ID_ = 'start_event_new_tweet'
    and actinst_start.PROC_DEF_KEY_ = 'TwitterDemoProcess'
    and actinst_start.START_TIME_ IS NOT NULL
    and actinst_end.ACT_ID_ = 'gateway_approved'
    and actinst_end.END_TIME_ IS NOT NULL
    and actinst_end.PROC_INST_ID_ = actinst_start.PROC_INST_ID_
)
GROUP BY durationInHours
ORDER BY durationInHours ASC

Retrieving Number of Started Instances

SELECT
  hour_date, ACT_RE_PROCDEF.KEY_,
  count(*) AS instanceCount
FROM (SELECT sysdate AS hour_date UNION SELECT DATEADD('DAY', -1*rownum, sysdate) AS hour_date FROM SYSTEM_RANGE(1, 13))
  JOIN act_hi_procinst ON (DAY_OF_YEAR(start_time_) = DAY_OF_YEAR(hour_date))
  JOIN ACT_RE_PROCDEF ON act_hi_procinst.PROC_DEF_ID_  = ACT_RE_PROCDEF.ID_
WHERE
    ACT_RE_PROCDEF.KEY_ = 'TwitterDemoProcess'
GROUP BY (hour_date, ACT_RE_PROCDEF.KEY_)
ORDER BY ACT_RE_PROCDEF.KEY_, hour_date

Retrieving Ratios

SELECT
  'Rejected' AS OPTION_,
  count(*) AS COUNT_
FROM PUBLIC.ACT_HI_ACTINST
WHERE ACT_HI_ACTINST.ACT_ID_ = 'service_task_send_rejection_notification'
UNION
SELECT
  'Published' AS OPTION_,
  count(*) AS COUNT_
FROM PUBLIC.ACT_HI_ACTINST
WHERE ACT_HI_ACTINST.ACT_ID_ = 'service_task_publish_on_twitter'

No guarantee - The statements made in this publication are recommendations based on the practical experience of the authors. They are not part of Camunda’s official product documentation. Camunda cannot accept any responsibility for the accuracy or timeliness of the statements made. If examples of source code are shown, a total absence of errors in the provided source code cannot be guaranteed. Liability for any damage resulting from the application of the recommendations presented here, is excluded.

Copyright © Camunda Services GmbH - All rights reserved. The disclosure of the information presented here is only permitted with written consent of Camunda Services GmbH.