Structure of a data collection rule (DCR) in Azure Monitor

This article describes the JSON structure of DCRs for those cases where you need to work directly with their definition.

• For details about working with the JSON described in this article, see Create and edit DCRs.
• For samples of different scenarios, see Sample DCRs in Azure Monitor.

Properties

The following table describes properties at the top level of the DCR.

¹ This property wasn't created for DCRs created before March 31, 2024. DCRs created before this date required a data collection endpoint (DCE) and the dataCollectionEndpointId property to be specified. If you want to use these embedded DCEs, you must create a new DCR.

Kind

The kind property in the DCR specifies the type of collection that the DCR is used for. Each kind of DCR has a different structure and properties.

The following table lists the different kinds of DCRs and their details.

Overview of DCR data flow

The basic flow of a DCR is shown in the following diagram. Each of the components is described in the following sections.

The full data flow through a DCR follows this sequence: Data sources → Input streams → Data flows → Destinations. Not all DCR types use every element. The following table shows which elements apply to each DCR type.

With multi-stage transformations, the transformations section adds processor pipelines that apply at the data source stage (client-side) or the data flow stage (ingestion-side), depending on where the named transformation is referenced.

Input streams

The input stream section of a DCR defines the incoming data it collects. Two types of incoming streams are possible depending on the data collection scenario. Most data collection scenarios use one of the input streams, while some scenarios use both.

Data sent from the Logs ingestion API uses a streamDeclaration with the schema of the incoming data because the API sends custom data.

Text logs from Azure Monitor Agent (AMA) are an example of data collection that requires both dataSources and streamDeclarations. The data source includes the configuration for connecting to the data source, and streamDeclarations defines the schema of the incoming data.

Data sources

Data sources are unique sources of monitoring data, each with its own format and method for exposing data. Each data source type has a unique set of parameters that you must configure for each data source. The data source typically returns a known data type, so you don't need to define the schema in the DCR.

For example, events and performance data collected from a VM by using the Azure Monitor agent (AMA) use data sources such as windowsEventLogs and performanceCounters. You specify criteria for the events and performance counters that you want to collect, but you don't need to define the structure of the data itself since this schema is known for potential incoming data.

Common data source parameters

All data source types share the following common parameters.

Valid data source types

The following table lists the currently available data source types.

¹ These data sources use both a data source and a stream declaration since the schema of the data they collect can vary. The stream used in the data source should be the custom stream defined in the stream declaration.

Stream declarations

Declare the different types of data you send into the Log Analytics workspace. Each stream is an object whose key represents the stream name, which must begin with Custom-. The stream contains a full list of top-level properties that are contained in the JSON data you send. The shape of the data you send to the endpoint doesn't need to match that of the destination table. Instead, the output of the transform that's applied on top of the input data needs to match the destination shape.

Data types

Assign the following data types to the properties:

• string
• int
• long
• real
• boolean
• dynamic
• datetime

The guid type isn't available in stream declarations. If your source data includes GUID values, declare those columns as string. The Tables API supports the guid type, but the values are actually stored and queried as strings. For more information, see Column data types in Azure Monitor Logs.

Destinations

Include an entry for each destination where you send the data in the destinations section. Match these destinations with input streams in the dataFlows section.

Common destination parameters

Valid destinations

The following table lists the available destinations.

Data flows

Data flows match input streams with destinations. Each data source can optionally specify a transformation and, in some cases, specify a specific table in the Log Analytics workspace.

Data flow properties

Transformations

The transformations section defines reusable named transformations referenced by dataSources and dataFlows. Each transformation defines a pipeline of processors that are applied sequentially to the data. See Multi-stage transformations for conceptual details.

"transformations": [
    {
        "name": "my_transform",
        "headerProcessor": {
            "processor": "header.Syslog",
            "configuration": { }
        },
        "processors": [
            {
                "processor": "filter.Basic",
                "configuration": {
                    "any": [
                        {
                            "all": [
                                {
                                    "columnName": "Facility",
                                    "operator": "==",
                                    "value": "auth"
                                }
                            ]
                        }
                    ]
                }
            }
        ]
    }
]

Transformation object properties

Processor object

Each processor is a declarative building block that describes a specific data transformation operation.

{
    "processor": "{family}.{Name}",
    "configuration": { }
}

Processor types

The following table shows which processors are available on the client side and the ingestion side.

• Client-side header processors are used for client-side transformations assigned to data sources. They require no configuration since the schema of the incoming data is known.
• Ingestion-side header processors are used for ingestion-side transformations assigned to data flows.

Header processors

Header processors receive raw data and convert it into a known schematized tabular format. A header processor must be the first processor in any transformation.

header.Syslog

Client side header for syslog data sources.

header.WindowsEvents

Client side header for Windows event log data sources.

header.WindowsPerformanceCounters

Client side header for Windows performance counter data sources.

header.LinuxPerformanceCounters

Client side header for Linux performance counter data sources.

header.TextLog

Client side header for custom text log files.

header.IISLog

Client side header for IIS log data sources.

header.WindowsFirewallLog

Client side header for Windows Firewall log data sources.

header.StandardStream

Used when the input is a standard stream such as Microsoft-Syslog or Microsoft-Event.

{
    "processor": "header.StandardStream",
    "configuration": {
        "streamId": "Microsoft-Syslog"
    }
}

Output Schema: Same as the corresponding standard Log Analytics table schema. See Log Analytics table reference for schemas by resource type.

header.CustomStream

Used when the input is a custom stream defined in streamDeclarations.

{
    "processor": "header.CustomStream",
    "configuration": {
        "streamId": "Custom-MyStream"
    }
}

After header processors

These processors run after the header processor in a transformation pipeline. Not all processors are available on both stages. Refer to the processor applicability table to determine which processors are supported on the client side, the ingestion side, or both.

filter.Basic

Drops entire records based on condition evaluation. Conditions are structured as OR groups of AND groups.

{
    "processor": "filter.Basic",
    "configuration": {
        "any": [
            {
                "all": [
                    {
                        "columnName": "Facility",
                        "operator": "==",
                        "value": "auth"
                    }
                ]
            }
        ]
    }
}

Each AND group has an all property containing an array of condition objects:

Output schema: Same as input. Records are dropped, not columns.

map.Rename

Renames a column and optionally changes its type.

{
    "processor": "map.Rename",
    "configuration": {
        "all": [
            {
                "columnName": "OldName",
                "nameAs": "NewName",
                "typeAs": "string"
            }
        ]
    }
}

Output schema: Same as input, except for renamed columns and columns with changed types.

map.Drop

Drops one or more columns from the data.

{
    "processor": "map.Drop",
    "configuration": {
        "columnNames": ["Column1", "Column2"]
    }
}

Output schema: Same as input, except for dropped columns.

parse.JsonPath

Parses a JSON-formatted string in a column and extracts specified keys into new columns.

{
    "processor": "parse.JsonPath",
    "configuration": {
        "columnName": "EventData",
        "all": [
            {
                "path": "$.user.name",
                "nameAs": "UserName",
                "typeAs": "string"
            }
        ]
    }
}

Extraction object properties:

Output schema: Same as input, plus new columns for each extracted key.

parse.XmlPath

Parses an XML-formatted string in a column and extracts specified elements into new columns.

{
    "processor": "parse.XmlPath",
    "configuration": {
        "columnName": "RawXml",
        "all": [
            {
                "path": "/Event/System/EventID",
                "nameAs": "EventID",
                "typeAs": "int"
            }
        ]
    }
}

Configuration properties match parse.JsonPath, except path uses XPath syntax (for example, /Event/System/EventID or /Event/EventData/Data[@Name='SubjectUserName']).

Output schema: Same as input, plus new columns for each extracted element.

parse.CEFAttribute

Parses CEF (Common Event Format) data in a column and extracts specified fields into new columns.

{
    "processor": "parse.CEFAttribute",
    "configuration": {
        "columnName": "Message",
        "all": [
            {
                "path": "deviceAction",
                "nameAs": "Action",
                "typeAs": "string"
            }
        ]
    }
}

Configuration properties match parse.JsonPath, except path specifies the CEF key name.

Output schema: Same as input, plus new columns for each extracted field.

aggregate.Basic

Summarizes records using aggregation operators with grouping dimensions.

{
    "processor": "aggregate.Basic",
    "configuration": {
        "batchingSettings": {
            "timeWindow": "5m",
            "maxBatchRows": 1000
        },
        "aggregates": [
            {
                "columnName": "CounterValue",
                "operator": "avg",
                "nameAs": "AvgValue"
            },
            {
                "operator": "count",
                "nameAs": "RecordCount"
            }
        ],
        "dimensionColumns": ["Host", "CounterName"]
    }
}

Aggregate entry properties:

Output schema: Aggregation changes the output schema entirely. The output contains only the aggregate columns and dimension columns. Route aggregated data to a separate custom table.

enrich.DNSLookup

Looks up an IP address in a column and adds a DNS name column.

{
    "processor": "enrich.DNSLookup",
    "configuration": {
        "columnName": "IPAddress",
        "nameAs": "DNSName"
    }
}

Output schema: Same as input, plus new column for the resolved DNS name. DNS resolution is best-effort. If the lookup fails, the output column contains null.

transform.KQL

Applies a custom KQL expression for advanced scenarios. This processor is ingestion-side only and provides limited validation of the KQL. Existing DCRs that use transformKql in data flows have a straightforward migration path with this processor.

{
    "processor": "transform.KQL",
    "configuration": {
        "expression": "source | where SeverityNumber >= 4 | extend EnrichedMsg = strcat(Host, ': ', Message)"
    }
}

Output schema: Defined by the KQL and can't be statically validated.

Related content

• Overview of data collection rules and methods for creating them
• Create and edit data collection rules (DCRs) in Azure Monitor
• Multi-stage transformations in Azure Monitor
• Create a transformation in Azure Monitor
• Collect data from VM clients