Incremental Data Loading

What Is Incremental Data Loading?

Incremental data loading (also called delta loading) allows you to append new rows to an existing datatable without replacing the original data. This is useful when your event log data grows over time and you want to keep your process mining analysis up to date without re-uploading the entire dataset.

Common scenarios include:

• Daily or weekly data exports: Append new transactions, orders, or cases as they become available.
• Continuous process monitoring: Keep your dashboards current by regularly adding fresh data.
• Growing event logs: Add new events from ERP systems, CRM tools, or other sources incrementally.

Requirements

• The datatable must already have an initial upload with data loaded.
• The delta file must be in the same format as the original upload:
  • CSV files require CSV delta files
  • Parquet files require Parquet delta files
  • ORC files require ORC delta files
  • JSONL files require JSONL delta files
• Excel (XLSX/XLS/XLSB) and XES files are not supported for delta uploads: convert them to CSV first.
• Column names in the delta file should match the original upload.
• Delta files can be gzip-compressed (.gz), as long as the inner format matches the original upload format.

Uploading Incremental Data

Via the API

Use the ProcessMind API to upload delta files programmatically. See the API Reference: Data page for full endpoint details.

The workflow is:

1. Get a delta presigned upload URL:

   GET /v1/tenant/{tenantId}/datatables/{dataTableId}/uploads/presignedurl
       ?delta=true&filename=delta.csv&filesize=1024&filelastmodified=1704067200000

2. Upload the delta file using the presigned URL:

   curl -X PUT --upload-file "delta.csv" \
       -H "Content-Type: text/csv" \
       "{PreSignedUploadUrl}"

After the upload completes, ProcessMind automatically processes the delta and combines it with the original data. You can poll GET /datatables/{dataTableId} until hasDataLoaded is true to know when processing is done.

Via the UI

Incremental data loading can also be triggered from the dataset’s General tab in the ProcessMind application:

1. Toggle the Append switch to switch the upload area from Replace Data to Append Data.
2. Drop the new file into the Append Data area (or click to browse and select it).

ProcessMind uploads the file as a delta and processes it together with the existing data. Delta uploads are explicit: they are never detected automatically from a normal (replace) upload.

Viewing Upload Parts

After one or more incremental uploads, the datatable’s upload history includes an uploadParts array that tracks each file:

{
	"uploadHistory": {
		"uploadParts": [
			{
				"key": "datatable/123/2.csv",
				"fileName": "orders-january.csv",
				"fileSize": 524288,
				"uploadedAt": "2024-01-15T10:00:00Z"
			},
			{
				"key": "datatable/123/3.csv",
				"fileName": "orders-february.csv",
				"fileSize": 419430,
				"uploadedAt": "2024-02-15T10:00:00Z"
			}
		]
	}
}

Best Practices

• Use consistent column names across all files to avoid data alignment issues.
• Upload in chronological order when possible, so the upload history reflects the data timeline.
• Keep delta files reasonably sized: upload data in moderate batches; very large numbers of tiny files can slow down processing.
• Monitor row counts after each delta to verify data was loaded correctly.

Schema Mismatch Behavior

ProcessMind does not enforce a strict schema between upload parts. This means:

• Extra columns in delta files are ignored during processing: only columns that exist in the original upload schema are used.
• Missing columns result in null/empty values for those fields in the delta rows.
• Different column names are treated as entirely different columns: the delta rows will have nulls for the expected columns.

For best results, ensure all delta files have the same column structure as the original upload.