Your Software Development Lifecycle Data Template
Your Software Development Lifecycle Data Template
- Recommended attributes to collect
- Key activities to track
- Extraction guidance for ServiceNow DevOps
Software Development Lifecycle Attributes
| Name | Description | ||
|---|---|---|---|
| Activity Name ActivityName | The name of the specific development lifecycle event that occurred, such as 'Development Started' or 'Code Review Performed'. | ||
| Description This attribute records the name of each milestone or task completed within the software development lifecycle. These activities form the sequential steps of the process, from creation to deployment. Analyzing the sequence and frequency of these activities is the primary function of process mining. It enables the construction of the process map, helps identify bottlenecks between steps, and highlights non-compliant or inefficient process variations. The defined set of activities includes key stages like design, development, testing, and deployment. Why it matters It defines the steps in the process map, allowing for the analysis of process flow, identification of bottlenecks, and discovery of deviations from the standard SDLC. Where to get This is typically derived by mapping status changes, event records, or audit trail entries to a standardized list of activity names. For example, a 'state' field changing to 'In Progress' could map to 'Development Started'. Examples Development StartedCode CommittedQA Testing CompletedDeployed to Production | |||
| Development Item DevelopmentItem | The unique identifier for a single unit of work, such as a feature, bug, or task, that progresses through the development lifecycle. | ||
| Description The Development Item serves as the primary case identifier, representing a distinct unit of work being tracked. It links all activities, from initial conception and planning through development, testing, and deployment for that specific item. In process mining analysis, this attribute is fundamental for reconstructing the end-to-end journey of each work item. It allows for the visualization of process flows, calculation of total cycle times, and identification of process variants for individual features or bug fixes. Every event in the log must be associated with a Development Item to build a coherent process map. Why it matters This is the core identifier that connects all related development activities into a single process instance, making it possible to analyze the complete lifecycle of each work item. Where to get This identifier is typically the primary key from tables managing stories, bugs, or tasks, such as the 'rm_story', 'rm_bug', or 'task' tables in ServiceNow. Examples STRY0010015BUG0034092TASK0050118 | |||
| Start Time EventTime | The exact timestamp indicating when a specific activity or event occurred. | ||
| Description This attribute provides the date and time for when each activity in the development lifecycle was recorded. It is essential for chronological ordering of events and for all time-based analysis. In process mining, the start time is used to calculate durations between activities, identify waiting times, and measure the overall cycle time of the process. It is a critical component for dashboards analyzing performance, such as the SDLC End-to-End Cycle Time Analysis, and for calculating key performance indicators like Code Review Lead Time. Why it matters This timestamp is essential for ordering events correctly and calculating all performance metrics, including cycle times, durations, and waiting times. Where to get Typically found in system-generated timestamp fields like 'sys_updated_on' or 'sys_created_on' from the audit trail or task tables. Examples 2023-10-26T10:00:00Z2023-10-27T14:35:10Z2023-11-01T09:15:00Z | |||
| Last Data Update LastDataUpdate | Timestamp indicating the last time the data for this event log was refreshed from the source system. | ||
| Description This attribute records when the dataset was last extracted or updated from ServiceNow DevOps. It applies to the entire dataset rather than individual events. This timestamp is vital for understanding the freshness of the analysis. It informs users how current the process insights are and helps in scheduling data refreshes. Displaying this information on dashboards provides context to all metrics and visualizations, ensuring that decisions are made based on timely data. Why it matters Provides crucial context about the timeliness of the data, ensuring users understand how up-to-date the process analysis is. Where to get This timestamp is generated and added during the data extraction process, recording when the extraction was executed. Examples 2023-11-15T08:00:00Z | |||
| Source System SourceSystem | Identifies the system from which the data was extracted, which is ServiceNow DevOps in this case. | ||
| Description This attribute specifies the originating system for the event data. For this process, it will consistently be 'ServiceNow DevOps'. While it may seem static, explicitly including the source system is crucial for data governance and in environments where data might be merged from multiple systems, such as Jira or Azure DevOps. It ensures clarity on data provenance and helps in diagnosing data quality or extraction issues. Why it matters Ensures data traceability and is essential for maintaining data integrity, especially when integrating data from multiple development tools. Where to get This is a static value that should be added during the data extraction and transformation process. Examples ServiceNow DevOps | |||
| Assigned Developer AssignedDeveloper | The name or ID of the developer or user assigned to the development item at the time of the activity. | ||
| Description This attribute identifies the individual responsible for executing a specific task or activity. It is dynamic and can change as the development item moves between different stages and teams. This attribute is critical for analyzing resource allocation, workload, and handoffs. It directly supports the 'Developer Workload and Handoffs' dashboard and the 'Activity Volume per Developer' KPI. By tracking changes in this field, it's possible to measure handoff times and identify collaboration bottlenecks between developers or between development and QA teams. Why it matters This is essential for resource-based analysis, including workload distribution, handoff efficiency, and identifying team-specific performance patterns. Where to get This information is typically stored in the 'assigned_to' field on task-related tables in ServiceNow. Examples David MillerAnna WilliamsJames Brown | |||
| Assignment Group AssignmentGroup | The team or group responsible for the development item at the time of the activity. | ||
| Description This attribute identifies the team assigned to a work item, such as 'Frontend Developers', 'Backend Services', or 'QA Team'. As a work item progresses, it is often handed off between different assignment groups. Tracking the assignment group is essential for understanding cross-functional collaboration and handoffs. It helps identify systemic delays that occur when work moves from one team to another. This attribute supports analysis of team-level performance, workload, and identifying which teams are bottlenecks in the overall flow. Why it matters Tracks which team is responsible for the work, enabling analysis of team performance, workload balancing, and the efficiency of handoffs between teams. Where to get This information is stored in the 'assignment_group' field, which is a standard field on task-related tables in ServiceNow. Examples Platform EngineeringMobile App TeamQuality AssuranceDevOps | |||
| Development Item Cycle Time DevelopmentItemCycleTime | The total time elapsed from the creation of the development item to its final closure or deployment. | ||
| Description This attribute is a calculated metric representing the end-to-end duration for a single development item. It is computed by finding the difference between the timestamp of the very first activity and the very last activity for each case. This is a primary key performance indicator for the entire SDLC process, directly supporting the 'Average SDLC Cycle Time' KPI. It provides a high-level measure of process velocity and efficiency. Analyzing this metric over time and across different dimensions, like priority or team, helps track the impact of process improvement initiatives. Why it matters Represents the total end-to-end duration for a work item, a key metric for measuring overall process efficiency and velocity. Where to get This is not a field in the source system. It is calculated in the process mining tool by subtracting the minimum StartTime from the maximum StartTime for each CaseId. Examples 15 days 4 hours3 days 12 hours32 days 8 hours | |||
| Development Item State DevelopmentItemState | The status or state of the development item at the time of the event, such as 'Open', 'In Progress', or 'Closed'. | ||
| Description This attribute reflects the official status of the development item within ServiceNow. While activities are derived process steps, the state represents the formal stage in the system's workflow. State is often the source from which activities are derived. It can be used for data validation and for creating simpler, high-level views of the process. For example, analyzing the time spent in each state can provide a different view of bottlenecks compared to analyzing time between activities. It is also useful for identifying items that are stalled or have been resolved. Why it matters Provides the official system status of a work item, which is often the source for deriving activities and can be used for validation and high-level status analysis. Where to get This is a standard field, usually named 'state' or 'stage', on task-related tables in ServiceNow. Examples PendingWork in ProgressReady for TestClosed Complete | |||
| Development Item Type DevelopmentItemType | The classification of the work item, such as 'Feature', 'Bug', 'Technical Debt', or 'Task'. | ||
| Description This attribute distinguishes between different kinds of work flowing through the SDLC process. For example, the process for fixing a critical bug might be different and faster than the process for developing a new feature. Analyzing the process based on work item type allows for a more nuanced understanding of performance. It helps answer questions like: 'Do bugs have a higher rework rate than new features?' or 'Is our cycle time for technical debt reduction acceptable?'. This segmentation provides deeper insights than a one-size-fits-all process view. Why it matters Distinguishes between different types of work, like features and bugs, which may have different process paths, priorities, and expected durations. Where to get This can be determined from the source table of the record (e.g., 'rm_story' vs 'rm_bug') or from a 'type' field on a generic task table. Examples FeatureBugTaskSpike | |||
| Is Rework IsRework | A boolean flag that is true if the activity is part of a rework loop, such as returning to development after testing. | ||
| Description This is a derived attribute that identifies activities occurring after a process has looped back to an earlier stage. For instance, if a 'Development Started' activity occurs after a 'QA Testing Completed' activity for the same item, it is flagged as rework. This flag is essential for quantifying and visualizing rework. It directly supports the 'Rework and Rejection Flow Analysis' dashboard and is used to calculate the 'Rework Rate after Testing' KPI. By flagging these events, analysts can easily filter for and analyze the frequency, causes, and impact of rework on the overall cycle time. Why it matters This flag makes it easy to quantify and analyze rework, helping to measure process quality and identify the root causes of repeated work. Where to get This attribute is calculated within the process mining tool by analyzing the sequence of activities for each case to detect backward movements in the process flow. Examples truefalse | |||
| Module/Component Affected ModuleComponentAffected | The specific software module, application, or component that the development item relates to. | ||
| Description This attribute categorizes the development work by the part of the system it impacts. This could be a specific microservice, a UI component, or a backend application. Segmenting the process by module or component is crucial for identifying localized bottlenecks. The 'Component-Specific Bottleneck Insights' dashboard and the 'Avg Stage Duration by Component' KPI rely on this attribute to pinpoint if certain parts of the codebase are consistently associated with longer development cycles, higher rework rates, or more frequent deployment failures. This helps focus improvement efforts where they are most needed. Why it matters Enables analysis to be segmented by application or component, helping to isolate bottlenecks or quality issues specific to certain parts of the system. Where to get This is often a custom field or a reference to the Configuration Management Database (CMDB), linking the work item to a 'cmdb_ci' record. Consult ServiceNow DevOps documentation. Examples Billing ServiceUser Authentication UIReporting DatabaseAPI Gateway | |||
| Priority DevelopmentItemPriority | The priority level assigned to the development item, such as 'High', 'Medium', or 'Low'. | ||
| Description This attribute categorizes development items based on their business urgency. Priority levels help teams focus on the most critical tasks and are often used to manage SLAs and stakeholder expectations. In process mining, priority is a key dimension for comparative analysis. It allows for filtering the process map to see if high-priority items follow a faster or different path. It is essential for the 'High-Priority Feature Delivery Time' dashboard and KPI, helping to validate whether critical items are truly being expedited. Why it matters Allows for filtering and comparing processes for different priority levels, helping to verify if high-priority items are processed faster and more efficiently. Where to get This is a standard field, often named 'priority', on task-related tables in ServiceNow. Examples 1 - Critical2 - High3 - Moderate4 - Low | |||
| Commit ID CommitId | The unique identifier of the source code commit associated with the development work. | ||
| Description This attribute provides a direct link from a development item to the specific code change in the source code repository, like Git. It is captured when a 'Code Committed' activity occurs. In process mining, the Commit ID enriches the analysis by connecting process data with engineering data. It allows analysts to trace a problematic deployment back to the exact code change or to correlate code complexity metrics with development cycle times. This provides a much deeper, more technical layer of root cause analysis. Why it matters Links the process event to a specific code change, enabling deeper root cause analysis by correlating process metrics with code-level details. Where to get This is captured by ServiceNow DevOps integrations with source code management systems like Git or SVN. The data resides in related tables linked to the development item. Examples a1b2c3d4e5f6f0e9d8c7b6a59a8b7c6d5e4f | |||
| Deployment Status DeploymentStatus | Indicates the outcome of a deployment activity, typically 'Success' or 'Failure'. | ||
| Description This attribute records the result of a deployment to a specific environment. It is a critical piece of information for understanding the reliability and stability of the release process. This attribute is essential for the 'Deployment Success and Failure Trends' dashboard and the 'Deployment Failure Rate' KPI. By analyzing the frequency and trends of deployment failures, organizations can identify underlying issues in their testing, infrastructure, or release coordination. It helps focus efforts on improving the quality and reliability of software delivery. Why it matters Directly measures the success of deployment activities, which is critical for calculating the deployment failure rate and analyzing release stability. Where to get This status is typically recorded in deployment tracking tasks or CI/CD pipeline execution records integrated with ServiceNow DevOps. Examples SuccessFailureCompleted with warnings | |||
| End Time EventEndTime | The exact timestamp indicating when an activity was completed. For instantaneous events, this is the same as the Start Time. | ||
| Description This attribute provides the date and time for when each activity in the development lifecycle was completed. It is particularly useful for activities that have a measurable duration, such as 'Code Review Performed' or 'QA Testing'. In process mining, having both a start and end time allows for precise calculation of activity processing times, distinguishing them from the waiting time between activities. This helps pinpoint whether delays are due to long tasks or long waits for resources. For events that are considered instantaneous, such as 'Build Triggered', the End Time can be the same as the Start Time. Why it matters Enables the precise calculation of activity processing time, which helps differentiate between time spent working versus time spent waiting. Where to get This may need to be derived. It could be the timestamp of the next activity's start time, or it could be sourced from a separate 'end date' field if available in the source system. Examples 2023-10-26T18:05:00Z2023-10-28T11:20:15Z2023-11-02T10:00:00Z | |||
| Planned Release Version PlannedReleaseVersion | The target software release or version in which the development item is planned to be delivered. | ||
| Description This attribute links a development item to a specific, planned release, such as 'Version 2.3' or 'Q4 2023 Release'. It is a key element for project management and release planning. For process mining, this attribute is crucial for the 'Release Plan Adherence Monitoring' dashboard. By comparing the actual completion dates against planned release dates, teams can measure schedule adherence, identify items at risk of missing a release, and analyze the causes of release delays. It provides a direct link between the low-level development process and high-level business objectives. Why it matters Connects development work to specific releases, enabling analysis of schedule adherence and the impact of process delays on release timelines. Where to get This information is typically stored in a 'release' or 'planned_release' field, often referencing a release management table in ServiceNow. Consult ServiceNow DevOps documentation. Examples v3.4.1Q1 2024 ReleaseProject Phoenix Go-Live | |||
| Rework Reason ReworkReason | A classification or description of why a development item required rework after testing. | ||
| Description When an item fails QA or UAT, this attribute captures the reason for the failure. This could be a specific bug category, a misunderstanding of requirements, or an environmental issue. This information provides critical context for the 'Rework and Rejection Flow Analysis' dashboard. Instead of just knowing that rework occurred, analysts can understand why it occurred. This allows for targeted improvements, such as better requirements definition, enhanced unit testing, or more stable test environments, to reduce the overall rework rate. Why it matters Provides qualitative insight into why rework occurs, enabling targeted process improvements to enhance quality and reduce rework loops. Where to get This may be captured in a 'close_notes' field when a test fails, or in a dedicated 'rework_reason' custom field. Consult ServiceNow DevOps documentation. Examples Requirement MisinterpretedRegression BugFailed Performance TestUI/UX Issue | |||
Software Development Lifecycle Activities
| Activity | Description | ||
|---|---|---|---|
| Code Review Performed | This activity indicates the completion of a peer code review, typically associated with a pull or merge request. This event can be captured explicitly through DevOps integrations or inferred from status changes on related records. | ||
| Why it matters This is a critical quality gate. Analyzing its duration helps identify bottlenecks in the review process, which is a common source of delays in the SDLC. Where to get Can be captured from the 'Merged' or 'Completed' event of a Pull Request record in ServiceNow's Git integration, or inferred from a status change on the development item to 'Code Review Complete'. Capture Logged when a Pull Request linked to the work item is merged. Event type explicit | |||
| Deployed to Production | This event marks the successful completion of the deployment to the production environment. It is explicitly captured by ServiceNow DevOps when the CI/CD tool reports a successful pipeline completion. | ||
| Why it matters This is the primary success end-point of the SDLC process. It completes the value stream and is essential for calculating the total cycle time. Where to get Captured from the 'completion_status' of a Pipeline Execution [sn_devops_pipeline_execution] record or its associated Stage Execution Run. A 'Success' status at the end time marks this event. Capture Logged when the production deployment pipeline completes successfully. Event type explicit | |||
| Deployment Failed | Indicates that the attempt to deploy the development item to production was unsuccessful. This is explicitly captured by ServiceNow DevOps when the CI/CD pipeline reports a failure. | ||
| Why it matters This is a critical failure end-point. Analyzing its frequency and causes is key to improving release stability and reducing the deployment failure rate. Where to get Captured from the 'completion_status' of a Pipeline Execution [sn_devops_pipeline_execution] record. A 'Failed' status at the end time marks this event. Capture Logged when the production deployment pipeline reports a failure status. Event type explicit | |||
| Development Item Created | This activity marks the creation of a new development item, such as a story, bug, or epic, within ServiceNow. This event is typically captured explicitly when a new record is inserted into the relevant table, like the Story [rm_story] table. | ||
| Why it matters This is the primary start event for the SDLC process. It allows for the measurement of the total end-to-end cycle time and tracks the initial demand intake. Where to get Recorded in the sys_audit or sys_history_line tables upon creation of a record in a development-related table, such as Story [rm_story], Epic [rm_epic], or Defect [rm_defect]. The creation timestamp is typically on the record itself. Capture Captured from the creation timestamp of the development item record. Event type explicit | |||
| Development Started | This activity marks the point when a developer actively begins coding or implementing the development item. It is typically inferred by a status change on the item to 'In Progress', 'Development', or 'Coding'. | ||
| Why it matters This is a crucial milestone that signals the start of the value-add construction phase. It is essential for measuring developer lead time and code review cycle times. Where to get Inferred from the timestamp when the 'State' field on the development item record (e.g., Story [rm_story]) is updated to an 'In Progress' or equivalent status. Capture Based on the timestamp of a state change to 'In Progress' or a similar value. Event type inferred | |||
| QA Testing Completed | Signifies that the Quality Assurance team has successfully completed their testing activities for the development item. This is typically inferred when the item's state transitions out of a testing phase to a status like 'Ready for UAT' or 'Done'. | ||
| Why it matters This milestone marks the completion of a major quality gate. It is a prerequisite for subsequent stages like User Acceptance Testing or release preparation. Where to get Inferred from the timestamp of a state change from a testing status (e.g., 'In QA') to a post-testing status (e.g., 'Ready for UAT' or 'Resolved'). Capture Based on the timestamp of a state change from 'Testing' to a subsequent state. Event type inferred | |||
| UAT Approved | Indicates that business stakeholders have formally approved the development item after User Acceptance Testing. This is a key milestone inferred from a status change, such as moving from 'In UAT' to 'Ready for Release' or 'Approved'. | ||
| Why it matters This is the final business approval before an item is cleared for production deployment. It is a critical quality and governance checkpoint. Where to get Inferred from a state transition on the development item record that signifies successful completion of UAT. This is logged in the item's activity history. Capture Inferred from a state change from 'UAT' to an approved or release-ready state. Event type inferred | |||
| Build Triggered | This event signifies the start of a CI/CD pipeline build, often triggered by a code commit. ServiceNow DevOps logs this as a pipeline execution, linking it back to the originating development items. | ||
| Why it matters This activity is the bridge between development and automated testing or deployment. Analyzing time between commit and build start can reveal delays in the CI/CD process. Where to get Recorded explicitly in the Pipeline Execution [sn_devops_pipeline_execution] table when a build starts in the integrated CI/CD tool (e.g., Jenkins, Azure DevOps). Capture Captured from the start time of a record in the Pipeline Execution table. Event type explicit | |||
| Code Committed | Represents a developer committing code to a version control system repository that is linked to the development item. ServiceNow DevOps explicitly captures these events from integrated SCM tools like Git or GitHub. | ||
| Why it matters Tracking commits provides granular visibility into development progress and activity frequency. It helps correlate specific code changes to the parent development item. Where to get Captured as an explicit event in the ServiceNow DevOps Commits [sn_devops_commit] table, which is populated by webhooks from the integrated source code management system. Capture Logged when a commit webhook is received from the SCM tool. Event type explicit | |||
| Deployment to Production Started | This activity marks the initiation of the deployment pipeline to the production environment. ServiceNow DevOps captures this as an explicit event when the production stage of a CI/CD pipeline begins execution. | ||
| Why it matters This marks the start of the final, and often most critical, phase of the lifecycle. Tracking this helps in analyzing deployment durations and identifying automation opportunities. Where to get Logged explicitly in the Stage Execution Run [sn_devops_stage_execution] table, filtered for stages related to the production environment. Capture Captured from the start time of a production deployment stage in a Pipeline Execution. Event type explicit | |||
| Design Started | Represents the phase where the technical design or solution architecture for the development item is being created. This is usually inferred from a status or state field on the development item record changing to a value like 'Design' or 'Solutioning'. | ||
| Why it matters Analyzing the duration of the design phase helps identify bottlenecks in requirements translation and solution planning before development work begins. Where to get Inferred from state transitions on the development item record (e.g., Story [rm_story]). Look for changes to the 'State' or a custom 'Stage' field to a design-related value. Capture Inferred from a status change to 'Design' or a similar state. Event type inferred | |||
| Development Item Cancelled | Represents the termination of a development item before completion. This is an alternative end state, typically inferred from the item's state being set to 'Cancelled' or 'Closed Incomplete'. | ||
| Why it matters Tracking cancellations helps identify wasted effort and understand the reasons for scope changes or reprioritization. It provides a more complete picture of all possible process outcomes. Where to get Inferred from the timestamp when the 'State' field on the development item record is updated to a terminal, non-complete status like 'Cancelled'. Capture Inferred from a state change to a 'Cancelled' or equivalent terminal state. Event type inferred | |||
| Prepared For Release | This activity signifies that the development item has passed all quality gates and is packaged into a specific release. It can be inferred when the item is associated with a Release record or its status changes to 'Ready for Deployment'. | ||
| Why it matters This step indicates that an item is technically and functionally complete. The time spent in this state can represent queuing time before a scheduled deployment window. Where to get Inferred from the 'State' field changing to 'Ready for Release' or by tracking when the 'Release' field on the development item record is populated or updated. Capture Inferred from a status change or association with a Release record. Event type inferred | |||
| QA Testing Started | Marks the beginning of the formal Quality Assurance testing phase. This is almost always inferred from the development item's state changing to a value like 'In QA', 'Testing', or 'Ready for Test'. | ||
| Why it matters This activity signals the handoff from development to the QA team. It allows for measuring the duration of the testing phase and identifying testing capacity bottlenecks. Where to get Inferred from the timestamp when the 'State' field on the development item record (e.g., Story, Defect) is updated to a QA-specific status. Capture Based on the timestamp of a state change to 'Testing' or equivalent. Event type inferred | |||
| Rework Identified | Indicates that an issue was found during testing, requiring the item to be sent back to development. This event is inferred by observing a backward movement in the process flow, such as a status change from 'In QA' back to 'In Progress'. | ||
| Why it matters Tracking rework is essential for understanding quality issues and process inefficiencies. A high frequency of this activity points to problems in development or requirements clarity. Where to get Inferred from analyzing the history of the 'State' field in the sys_audit or sys_history_line tables. A change from a later-stage status (e.g., 'Testing') to an earlier one (e.g., 'In Progress') signifies rework. Capture Inferred from a backward status transition, e.g., 'Testing' -> 'In Progress'. Event type inferred | |||
| UAT Started | Represents the start of User Acceptance Testing, where business stakeholders validate the functionality. This event is captured by inferring a status change to 'UAT', 'In UAT', or 'User Acceptance Testing'. | ||
| Why it matters This phase is critical for ensuring the developed feature meets business requirements. Analyzing its duration can reveal issues with user engagement or requirement mismatches. Where to get Inferred from a state transition on the development item record. This relies on the customer's state model including a distinct status for UAT. Capture Inferred from a state change to a 'UAT' status. Event type inferred | |||
Extraction Guides
Steps
- Understand Your State Model: Before creating reports, document the specific values in your development item's state field (e.g., on the Story
[rm_story]or Defect[rm_defect]table) that correspond to the required activities. For example, a state value of 'In Progress' might map to the 'Development Started' activity. - Navigate to Report Creation: Log in to your ServiceNow instance. In the Filter navigator, go to
Reports > View / Runand click theCreate a reportbutton. - Create Report for State Changes: Create the first report to capture status-driven activities. Configure it as follows:
- Report Name:
ProcessMind - State Change Events - Source type:
Table - Table:
Audit [sys_audit] - Type:
List - Configure Columns: Add
Document key,Created on,Table name,Field name,Old value, andNew value. - Filter: Set
Table nameto be one of your development item tables (e.g.,Story), andField nameto be your state field (e.g.,State). Add a date filter on theCreated onfield for the desired time period.
- Report Name:
- Create Report for Item Creation: Create a new report for the initial creation event.
- Report Name:
ProcessMind - Item Creation Events - Source type:
Table - Table:
Story [rm_story](or your primary development item table) - Type:
List - Configure Columns: Add columns for the required attributes like
Number,Created on,Assigned to,Priority,State, etc. - Filter: Apply a date filter on the
Created onfield.
- Report Name:
- Create Report for Code Commits: Create a report for explicit DevOps events related to commits.
- Report Name:
ProcessMind - Commit Events - Source type:
Table - Table:
Commit [sn_devops_commit] - Type:
List - Configure Columns: Add columns like
Work item,Commit time,Author, etc. - Filter: Apply a date filter on the
Commit timefield.
- Report Name:
- Create Reports for Builds and Deployments: Repeat the process from the previous step for the
Build [sn_devops_build]andDeployment [sn_devops_deployment]tables. These tables contain records forBuild Triggered,Deployment to Production Started,Deployed to Production, andDeployment Failedactivities. - Export All Reports: Run each of the created reports individually. For each report, click the context menu icon (three dots or a down arrow) and select
Export > CSVorExport > Excel. Save all files. - Combine and Transform Data: Open the exported files in a spreadsheet program or use a data preparation tool. Manually combine the data from all files into a single sheet. Create the required event log columns (
DevelopmentItem,ActivityName,EventTime, etc.) and map the data from the source columns. For example, map theDocument keyfrom the audit report and theNumberfrom the story report to theDevelopmentItemcolumn. - Map Activity Names: Create the
ActivityNamecolumn by translating the source data. For the state change report, use your documented state model to mapNew valueentries to activity names (e.g., map state 'Testing' to 'QA Testing Started'). For the other reports, assign a fixed activity name for each row (e.g., all rows from the commit export become 'Code Committed'). - Finalize and Save: Add the
SourceSystemandLastDataUpdatecolumns with static values for all rows. Ensure all timestamps are in a consistent format. Save the final combined file as a single CSV, which is now ready to be uploaded to ProcessMind.
Configuration
- Required Tables: The primary tables needed for this extraction are
Audit [sys_audit]for inferred events, your specific development item tables (e.g.,Story [rm_story],Defect [rm_defect]), and the core ServiceNow DevOps tables:Commit [sn_devops_commit],Build [sn_devops_build], andDeployment [sn_devops_deployment]. - Key Filters: The most important filter is the date range, which should be applied consistently across all reports on a timestamp field like
Created on,Commit time, orStart time. For thesys_auditreport, it is critical to filter on theTable name(e.g.,rm_story) andField name(e.g.,state) to limit the data to only relevant state changes. - Date Range Recommendation: It is recommended to extract data for a period of 3 to 6 months to ensure a representative dataset without causing performance issues. For larger systems, consider extracting data in monthly batches.
- State Model Definition: You must have a clear understanding of your organization's work item state model. This is required to correctly map the state values captured in the
sys_audittable to the corresponding business activities like 'QA Testing Started' or 'UAT Approved'. - Prerequisites: Users performing the extraction need the
report_userrole or equivalent permissions to create and run reports. They also need read access to the DevOps and application development tables mentioned above. ServiceNow DevOps plugin must be installed and actively integrated with your SCM and CI/CD tools.
a Sample Query config
/*
This extraction method uses the ServiceNow report builder UI. The following sections describe the configuration for each report that must be created and exported.
The exported data must then be manually combined and transformed into a single event log file.
*/
---
-- REPORT 1: Item Creation Events
---
Report_Name: ProcessMind - Item Creation Events
Source_Table: rm_story
Report_Type: List
Columns:
- Number (maps to DevelopmentItem)
- sys_created_on (maps to EventTime)
- 'Development Item Created' (create a formula or static column for ActivityName)
- Assigned to (maps to AssignedDeveloper)
- Priority (maps to DevelopmentItemPriority)
- State (maps to DevelopmentItemState)
- cmdb_ci (maps to ModuleComponentAffected)
- Type (maps to DevelopmentItemType)
- Assignment group (maps to AssignmentGroup)
Filters:
- sys_created_on ON Last 6 months
---
-- REPORT 2: Inferred State Change Events
---
Report_Name: ProcessMind - State Change Events
Source_Table: sys_audit
Report_Type: List
Columns:
- documentkey (maps to DevelopmentItem)
- sys_created_on (maps to EventTime)
- newvalue (maps to ActivityName, requires translation)
- user (maps to AssignedDeveloper)
ActivityName_Mapping_Logic (Example):
- WHEN newvalue IS '[Your Design State]' THEN 'Design Started'
- WHEN newvalue IS '[Your In Progress State]' THEN 'Development Started'
- WHEN newvalue IS '[Your QA State]' THEN 'QA Testing Started'
- WHEN oldvalue IS '[Your QA State]' AND newvalue IS '[Your In Progress State]' THEN 'Rework Identified'
- WHEN oldvalue IS '[Your QA State]' AND newvalue IS '[Your UAT State]' THEN 'QA Testing Completed'
- WHEN newvalue IS '[Your UAT State]' THEN 'UAT Started'
- WHEN newvalue IS '[Your UAT Approved State]' THEN 'UAT Approved'
- WHEN newvalue IS '[Your Release Ready State]' THEN 'Prepared For Release'
- WHEN newvalue IS '[Your Cancelled State]' THEN 'Development Item Cancelled'
Filters:
- tablename = 'rm_story'
- fieldname = 'state'
- sys_created_on ON Last 6 months
---
-- REPORT 3: Code Commit Events
---
Report_Name: ProcessMind - Commit Events
Source_Table: sn_devops_commit
Report_Type: List
Columns:
- work_item.number (maps to DevelopmentItem)
- commit_time (maps to EventTime)
- 'Code Committed' (create a formula or static column for ActivityName)
- author.name (maps to AssignedDeveloper)
Filters:
- commit_time ON Last 6 months
---
-- REPORT 4: Build Events
---
Report_Name: ProcessMind - Build Events
Source_Table: sn_devops_build
Report_Type: List
Columns:
- work_item.number (maps to DevelopmentItem)
- start_time (maps to EventTime)
- 'Build Triggered' (create a formula or static column for ActivityName)
Filters:
- start_time ON Last 6 months
---
-- REPORT 5: Deployment Events
---
Report_Name: ProcessMind - Deployment Events
Source_Table: sn_devops_deployment
Report_Type: List
Columns:
- work_item.number (maps to DevelopmentItem)
- start_time (maps to EventTime for 'Started' activities)
- end_time (maps to EventTime for 'Completed' or 'Failed' activities)
- state (maps to ActivityName, requires translation)
ActivityName_Mapping_Logic:
- WHEN state IS 'in_progress' THEN 'Deployment to Production Started'
- WHEN state IS 'successful' THEN 'Deployed to Production'
- WHEN state IS 'failed' THEN 'Deployment Failed'
Filters:
- start_time ON Last 6 months
- [Filter for production deployments based on your environment configuration]
---
-- Additional events like 'Code Review Performed' may require a separate report
-- on a table like `sn_devops_pull_request` if available and configured.
--- Steps
- Prerequisites: Ensure you have network access to your ServiceNow instance and have been provided with a dedicated service account with read permissions (the
itilandsn_devops.viewerroles are a good starting point). This user will need access to tables such asrm_story,sys_audit, and thesn_devops_*schema. - Install ServiceNow ODBC Driver: Download the appropriate ServiceNow ODBC driver for your operating system from the ServiceNow support portal. Follow the installation instructions provided.
- Configure DSN: Set up a new System DSN (Data Source Name) on the machine where you will run the query. In the ODBC Data Source Administrator, add the ServiceNow driver and configure it with your instance URL (e.g.,
yourinstance.service-now.com), username, and password. - Connect with a SQL Client: Use a SQL client tool like DBeaver, Microsoft SQL Server Management Studio (using a linked server), or a scripting language like Python with an ODBC library to connect to ServiceNow using the DSN you configured.
- Identify State Model: Before running the query, you must identify the exact values your organization uses for the
statefield on your development item tables (e.g.,rm_story,rm_defect). The provided query uses common examples like 'In Progress' or 'In QA', which you will need to replace with your specific values. - Customize the SQL Query: Copy the provided SQL query into your client. Modify the placeholders at the top of the query, including the start date for the extraction and the specific state values that correspond to your development lifecycle activities.
- Execute the Query: Run the complete SQL query against the ServiceNow database via the ODBC connection. This may take a significant amount of time depending on the date range and volume of data.
- Review the Data: Once the query completes, perform a brief review of the returned dataset. Check for a variety of activities and ensure that key columns like
DevelopmentItem,ActivityName, andEventTimeare populated as expected. - Export to CSV: Export the entire result set to a CSV file. Ensure the file is UTF-8 encoded and that the column headers match the attribute names required by ProcessMind, for example,
DevelopmentItem,ActivityName,EventTime. - Prepare for Upload: Ensure the final CSV file contains no empty rows at the end and that the date format for
EventTimeandLastDataUpdateis consistent and supported by ProcessMind (e.g.,YYYY-MM-DD HH:MM:SS).
Configuration
- Prerequisites: Access to a ServiceNow instance with the DevOps module enabled. A dedicated user account with read permissions for the required tables is necessary. The ServiceNow ODBC driver must be installed and configured on the client machine.
- ODBC Driver Configuration: The connection requires the instance URL, a username, and a password or OAuth token. It is critical to test the DSN connection before attempting to run complex queries.
- Date Range Filtering: The provided query includes a placeholder
s.sys_created_on >= '2023-01-01'to limit the amount of data extracted. It is highly recommended to extract data for a defined period, such as the last 6 to 12 months, to ensure manageable query execution times. - State Model Customization: The query's accuracy for inferred events relies entirely on the state model. You must replace placeholder state values (e.g.,
[Your 'In Progress' State Value],[Your 'In QA' State Value]) with the exact values used in your ServiceNow configuration. These are case sensitive. - Work Item Tables: The query is written for the Story table (
rm_story). If your organization also uses Defects (rm_defect), Enhancements (rm_enhancement), or other task types, you must add them to the initialDevItemsCommon Table Expression (CTE) usingUNION ALL. - Performance: Direct queries against a production ServiceNow instance can impact performance. Running large extractions during off-peak hours is recommended. For very large datasets, consider an incremental extraction strategy based on the
sys_updated_onfield.
a Sample Query sql
WITH DevItems AS (
-- This CTE selects the base set of development items to analyze.
-- Add other tables like rm_defect or rm_enhancement here using UNION ALL if needed.
SELECT
s.sys_id,
s.number,
s.sys_created_on,
s.sys_updated_on,
s.assigned_to,
s.priority,
s.state,
s.cmdb_ci, -- Module/Component Affected
s.sys_class_name, -- Development Item Type
s.assignment_group,
DATEDIFF(second, s.sys_created_on, s.closed_at) AS cycle_time_seconds
FROM rm_story s
WHERE s.sys_created_on >= '2023-01-01' -- *** Placeholder: Set your desired start date ***
),
StateChanges AS (
-- This CTE unnests the audit trail for state changes, which are used for inferred activities.
SELECT
a.documentkey AS item_sys_id,
a.sys_created_on AS change_time,
a.oldvalue,
a.newvalue,
-- *** Placeholder: Define the numeric order of your states to detect rework. Adjust values and names. ***
CASE a.oldvalue
WHEN '1' THEN 1 -- Open
WHEN '[Your 'Design' State Value]' THEN 2
WHEN '[Your 'In Progress' State Value]' THEN 3
WHEN '[Your 'In QA' State Value]' THEN 4
WHEN '[Your 'In UAT' State Value]' THEN 5
ELSE 0
END AS old_state_order,
CASE a.newvalue
WHEN '1' THEN 1 -- Open
WHEN '[Your 'Design' State Value]' THEN 2
WHEN '[Your 'In Progress' State Value]' THEN 3
WHEN '[Your 'In QA' State Value]' THEN 4
WHEN '[Your 'In UAT' State Value]' THEN 5
ELSE 0
END AS new_state_order
FROM sys_audit a
WHERE a.tablename = 'rm_story' AND a.fieldname = 'state'
)
-- 1. Development Item Created
SELECT
i.number AS DevelopmentItem,
'Development Item Created' AS ActivityName,
i.sys_created_on AS EventTime,
'ServiceNow DevOps' AS SourceSystem,
GETDATE() AS LastDataUpdate,
us.name AS AssignedDeveloper,
i.priority AS DevelopmentItemPriority,
i.state AS DevelopmentItemState,
ci.name AS ModuleComponentAffected,
i.sys_class_name AS DevelopmentItemType,
grp.name AS AssignmentGroup,
i.cycle_time_seconds AS DevelopmentItemCycleTime,
CAST(0 AS BIT) AS IsRework
FROM DevItems i
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id
LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id
LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
UNION ALL
-- 2. Design Started
SELECT i.number, 'Design Started', sc.change_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, sc.newvalue, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM StateChanges sc JOIN DevItems i ON sc.item_sys_id = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE sc.newvalue = '[Your ''Design'' State Value]' -- *** Placeholder: Adjust state value ***
UNION ALL
-- 3. Development Started
SELECT i.number, 'Development Started', sc.change_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, sc.newvalue, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM StateChanges sc JOIN DevItems i ON sc.item_sys_id = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE sc.newvalue = '[Your ''In Progress'' State Value]' AND sc.old_state_order < 3 -- *** Placeholder: Adjust state value and order ***
UNION ALL
-- 4. Code Committed
SELECT i.number, 'Code Committed', c.committed_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, i.state, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM sn_devops_commit c JOIN DevItems i ON c.work_item = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
UNION ALL
-- 5. Build Triggered
SELECT i.number, 'Build Triggered', b.start_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, i.state, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM sn_devops_build b JOIN sn_devops_commit_build cb ON b.sys_id = cb.build JOIN sn_devops_commit c ON cb.commit = c.sys_id JOIN DevItems i ON c.work_item = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
UNION ALL
-- 6. Code Review Performed
SELECT i.number, 'Code Review Performed', pr.closed_at, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, i.state, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM sn_devops_pull_request pr JOIN DevItems i ON pr.work_item = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE pr.state = 'merged' -- Or 'closed', depending on process
UNION ALL
-- 7. QA Testing Started
SELECT i.number, 'QA Testing Started', sc.change_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, sc.newvalue, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM StateChanges sc JOIN DevItems i ON sc.item_sys_id = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE sc.newvalue = '[Your ''In QA'' State Value]' -- *** Placeholder: Adjust state value ***
UNION ALL
-- 8. Rework Identified
SELECT i.number, 'Rework Identified', sc.change_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, sc.newvalue, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(1 AS BIT)
FROM StateChanges sc JOIN DevItems i ON sc.item_sys_id = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE sc.new_state_order < sc.old_state_order AND sc.new_state_order > 1 -- Moved to an earlier state
UNION ALL
-- 9. QA Testing Completed
SELECT i.number, 'QA Testing Completed', sc.change_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, sc.newvalue, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM StateChanges sc JOIN DevItems i ON sc.item_sys_id = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE sc.oldvalue = '[Your ''In QA'' State Value]' AND sc.new_state_order > sc.old_state_order -- *** Placeholder: Adjust state value ***
UNION ALL
-- 10. UAT Started
SELECT i.number, 'UAT Started', sc.change_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, sc.newvalue, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM StateChanges sc JOIN DevItems i ON sc.item_sys_id = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE sc.newvalue = '[Your ''In UAT'' State Value]' -- *** Placeholder: Adjust state value ***
UNION ALL
-- 11. UAT Approved
SELECT i.number, 'UAT Approved', sc.change_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, sc.newvalue, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM StateChanges sc JOIN DevItems i ON sc.item_sys_id = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE sc.oldvalue = '[Your ''In UAT'' State Value]' AND sc.new_state_order > sc.old_state_order -- *** Placeholder: Adjust state value ***
UNION ALL
-- 12. Prepared For Release
SELECT i.number, 'Prepared For Release', sc.change_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, sc.newvalue, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM StateChanges sc JOIN DevItems i ON sc.item_sys_id = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE sc.newvalue = '[Your ''Ready for Release'' State Value]' -- *** Placeholder: Adjust state value ***
UNION ALL
-- 13. Deployment to Production Started
SELECT i.number, 'Deployment to Production Started', se.start_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, i.state, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM sn_devops_step_execution se JOIN sn_devops_artifact_build sab ON se.deployable = sab.sys_id JOIN sn_devops_commit_build cb ON sab.build = cb.build JOIN sn_devops_commit c ON cb.commit = c.sys_id JOIN DevItems i ON c.work_item = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE se.stage_name = 'Production' -- *** Placeholder: Adjust stage name ***
UNION ALL
-- 14. Deployed to Production
SELECT i.number, 'Deployed to Production', se.end_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, i.state, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM sn_devops_step_execution se JOIN sn_devops_artifact_build sab ON se.deployable = sab.sys_id JOIN sn_devops_commit_build cb ON sab.build = cb.build JOIN sn_devops_commit c ON cb.commit = c.sys_id JOIN DevItems i ON c.work_item = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE se.stage_name = 'Production' AND se.result = 'SUCCESS' -- *** Placeholder: Adjust stage name and result value ***
UNION ALL
-- 15. Deployment Failed
SELECT i.number, 'Deployment Failed', se.end_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, i.state, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM sn_devops_step_execution se JOIN sn_devops_artifact_build sab ON se.deployable = sab.sys_id JOIN sn_devops_commit_build cb ON sab.build = cb.build JOIN sn_devops_commit c ON cb.commit = c.sys_id JOIN DevItems i ON c.work_item = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE se.stage_name = 'Production' AND se.result = 'FAILURE' -- *** Placeholder: Adjust stage name and result value ***
UNION ALL
-- 16. Development Item Cancelled
SELECT i.number, 'Development Item Cancelled', sc.change_time, 'ServiceNow DevOps', GETDATE(), us.name, i.priority, sc.newvalue, ci.name, i.sys_class_name, grp.name, i.cycle_time_seconds, CAST(0 AS BIT)
FROM StateChanges sc JOIN DevItems i ON sc.item_sys_id = i.sys_id
LEFT JOIN sys_user us ON i.assigned_to = us.sys_id LEFT JOIN cmdb_ci ci ON i.cmdb_ci = ci.sys_id LEFT JOIN sys_user_group grp ON i.assignment_group = grp.sys_id
WHERE sc.newvalue = '[Your ''Cancelled'' State Value]'; -- *** Placeholder: Adjust state value ***