On This Page
> resources > docs > process mining data > Data Cleaning and Preparation for Process Mining

Data Cleaning and Preparation for Process Mining

Process Intelligence Process Intelligence Available with a Process Intelligence seat or higher. Only seats on this plan or a higher one can use this feature. Compare plans Available with a Process Intelligence seat or higher

Effective process mining starts with good-quality data, and most of the work comes down to the same handful of problems: events that were never recorded, identifiers that do not line up between systems, formats that disagree, and extracts that are too large or too sensitive to handle casually. This page works through those problems and the steps that fix them. For doing the same work inside ProcessMind, see Configuring Your Dataset and Data Quality.

Why Is Data Cleaning and Preparation Important?

Process mining relies on event logs: datasets that contain the detailed sequence of activities within a business process. If those datasets are incomplete, inconsistent, or contain errors, the insights you gain will be unreliable. Clean and properly structured data ensures the tool can map workflows, detect bottlenecks, and highlight areas for improvement.

Key Steps in Data Cleaning and Preparation

1. Collect Data from Every System Involved

Data usually comes from several systems: an ERP for orders and invoices, a CRM for customer steps, a service desk for tickets. Collect from all of them before you start cleaning, because a step that lives in a system you left out cannot be recovered later.

  • Consolidate data sources: list the systems that record part of the process and extract from each one. An order-to-cash process, for example, spans a sales system (for example, Salesforce) and a finance system (for example, SAP).
  • Break down silos: agree with the process owners which systems count and who can export from them. Missing data usually comes from an unreported system, not from a broken query. Integration tools (for example, Apache NiFi, Talend, Informatica, or Power BI) can merge several sources into one file.
  • Plan for what you cannot extract: if a step happens off-system (a phone approval, a paper sign-off), capture at least its outcome in a digital system, or model the step so the process view stays complete. Where to Get Data covers sourcing system by system.

2. Remove Duplicates

Duplicate records distort the analysis by inflating activity counts and making one event look like several. Identify them by identical case ID, activity, and timestamp, then remove them or merge them into one row.

3. Handle Missing Data

Missing timestamps, activities, or case IDs break the sequence of events and produce incomplete process models.

  • Identify missing values: look for blank timestamps, empty activity names, and null case IDs.
  • Fill in the gaps: where you can, use other sources or domain knowledge. If one timestamp is missing, the surrounding event times can narrow it down.
  • Model rather than invent: for steps that were never recorded at all, add them through process modeling instead of fabricating timestamps, so the mined process and the documented process meet in the middle.
  • Impute or drop: for critical gaps, use an imputation technique (for example, mean substitution or a regression model); if a case cannot be recovered, remove it and note why.

4. Normalize Formats, IDs, and Names

  • Timestamps: use one format everywhere (for example, YYYY-MM-DD HH:MM:SS). If the extract mixes time zones, convert everything to UTC. See the list of supported date formats
  • Case IDs: the same process often carries different identifiers in different systems — an order number in the CRM, an invoice number in finance. Build a mapping between them (in the extract or as a lookup table) so every event lands under one case, and give each process instance a unique ID. In ProcessMind, check that the case ID column is mapped the same way for every dataset you combine; see dataset attributes.
  • Activity names: systems label the same step differently. Decide on one wording per step (“Approve Order” and “Order Approval” are the same activity).
  • Values and types: keep numbers as numbers (costs, durations, amounts) and use one unit per column.

5. Remove Irrelevant Events

Not every recorded event belongs to the process you are analyzing; system logins, technical retries, and administrative tasks clutter the map without adding insight.

  • Filter out what does not belong: use domain knowledge to decide which events describe the process and exclude the rest.
  • Choose the right granularity: events that are too coarse (“order handled”) hide the variations you are looking for, while events that are too fine drown the map in technical detail. Group low-level events into the business activity they belong to, with the process experts deciding where the line is.

6. Handle Outliers and Noise

Outliers give an inaccurate picture of how the process normally runs: a task that took exceptionally long because of a rare event can dominate the averages.

  • Identify them: flag durations that sit far outside the normal range for that step.
  • Decide whether to keep them: a genuine exception — a rare but critical failure — is often insight worth keeping; a data error is not. Remove only what is wrong, not what is inconvenient.

7. Check Event Order

If events are out of order, the tool traces the wrong flow.

  • Validate the sequence: each case’s events should follow the timestamps. An “Order Approved” event before “Order Created” in the same case means the timestamps need checking.
  • Sort by timestamp: sort each case’s events by the timestamp column before uploading.

8. Handle Large Datasets

Big extracts are slow to prepare and slow to upload, so plan for size rather than fighting it.

  • Sample to explore, load to analyze: a representative sample is enough to see whether the extracted events make sense; load the full extract when the analysis needs it.
  • Load incrementally: instead of replacing everything, append new periods with incremental (delta) uploads.
  • Prefer columnar formats: Parquet and ORC files are smaller and upload faster than CSV or Excel for large volumes; see Supported Data Formats.

9. Protect Sensitive Data

Event logs carry names, addresses, customer numbers, and user IDs, so treat the extract as personal data.

  • Anonymize or mask: remove or mask what the analysis does not need, and keep user IDs only where resource analysis matters.
  • Limit access: give the extract to as few people as possible, on a need-to-know basis, and rely on role-based access control for the datasets you upload.
  • Follow the rules that apply to you: retention limits, encryption in transit and at rest, and regulations such as GDPR for customer data.

10. Create the Event Log

Once the data is clean, consistent, and ordered, build the event log — the primary dataset for process mining. Every log needs:

  • Case ID: a unique identifier for each process instance.
  • Activity: the name of the step that occurred.
  • Timestamp: when it occurred, which fixes the order of events.

Optional columns add depth: the resource (person or team), department, cost, or any custom attribute you want to slice the process by. The full set of requirements is listed in Supported Data Formats.

11. Validate the Dataset

Before you rely on the results, check that the dataset represents the process you think it does.

  • Spot-check cases: walk through a few process instances end to end and confirm the event sequences make sense.
  • Run a test analysis: mine the process once and look for obvious nonsense — activities that never happen, a start or end event that looks wrong, cases that should have closed.
  • Confirm with the process owners: business experts can tell you in minutes whether the data reflects how work actually runs.

Cleaning Data Inside ProcessMind

Much of the preparation above can be handled inside ProcessMind after upload:

  • Map and configure columns: set display names, data types, and timestamp formats in the dataset attributes instead of reworking the source file.
  • Check quality: the Data Quality dashboard flags missing values and inconsistencies so you can fix them before analysis, and AI data recommendations suggests mappings and fixes.
  • Hide or filter irrelevant data: hide columns in dataset attributes and use filters to exclude events you do not need.
  • Reload instead of re-upload: replace data in place, or use incremental (delta) uploads for continuously growing datasets.

For cleaning outside ProcessMind, general-purpose tools (Python/Pandas, Excel or Google Sheets, ETL tools, OpenRefine) work before upload. For the wider playbook — naming, formats, incremental loading, and archiving — see ETL for Process Mining.

Related Documentation