Transformations in Azure Monitor

Transformations in Azure Monitor filter or modify incoming data before it's sent to a Log Analytics workspace. Transformations run after the data source delivers the data and before it's sent to the destination. They're defined in a data collection rule (DCR).

Standard transformations use a Kusto Query Language (KQL) statement applied individually to each entry in the incoming data. Multi-stage transformations extend this model with a pipeline of declarative processors, where a KQL query is one of many available processor types in the transformation chain.

The following diagram illustrates the transformation process for incoming data and shows a sample query that might be used. In this sample, only records where the message column contains the word error are collected.

Supported tables

The following tables in a Log Analytics workspace support transformations.

• Any Azure table listed in Tables that support transformations in Azure Monitor Logs. The Azure Monitor data reference also includes whether the table supports transformations and lists other attributes for each table.
• Any custom table created for the Azure Monitor Agent.
• Custom tables with the Auxiliary plan.

Create a transformation

Some data collection scenarios support adding a transformation through the Azure portal, but most scenarios require creating a new DCR using its JSON definition or adding a transformation to an existing DCR. See Create a transformation in Azure Monitor for different options and Best practices and samples for transformations in Azure Monitor for sample transformation queries of common scenarios.

Multi-stage transformations (preview)

Standard transformations apply a single KQL query to incoming data during ingestion. Multi-stage transformations extend this model by letting you define a processing pipeline composed of multiple ordered processing stages, applied as data flows through the system.

With multi-stage transformations, a DCR defines a data processing pipeline rather than a single transformation step. Data flows through the following stages:

1. Data collection. Azure Monitor Agent (AMA) collects data from the resource based on the data source settings in the DCR.
2. Client-side processing. Transformations run locally on the agent before data is sent over the network. The transformation applies to the data in its raw form, which might differ from the standard table representation.
3. Ingestion-time processing. After data reaches Azure Monitor, transformations are applied during ingestion before the data is written to its destination table in Log Analytics. This transformation applies after the data is fully schematized and enriched.
4. Data delivery. Processed logs are delivered to their final destination.

Processors

Transformations in multi-stage DCRs are defined using processors - small, declarative building blocks that each perform a specific type of operation. Processors have the following characteristics:

• Composable. Multiple processors can be chained together in a single transformation.
• Ordered. Processors run sequentially in the order defined in the transformation.
• Stage-agnostic. The same processor type can be used across different data sources, stages, or scenarios, with some limitations during preview.

Every transformation starts with a header processor that converts raw data into a known schematized tabular format. After the header, you can chain additional processors to filter, map, parse, aggregate, enrich, or apply KQL expressions to the data.

The following processor families are available:

For the complete processor reference, including configuration schemas and output schemas, see DCR structure - Transformations.

Client-side vs. ingestion-side transformations

Each transformation is assigned to a specific processing stage. This distinction determines where compute happens, what data is sent over the network, and what costs can be optimized early.

A single DCR can combine both client-side and ingestion-time transformations. The output of the client-side stage becomes the input to the ingestion-time stage automatically.

Multi-stage DCR considerations

• Multi-stage transformations require API version 2025-05-11 or later. The transformations section and the transform property on data sources and data flows aren't recognized by earlier API versions.
• The transform property is mutually exclusive with transformKql per data flow. A DCR can mix old and new data flows across different streams.
• During preview, you must manually coordinate the header of the downstream transformation with the outcome of the upstream transformation. For example, if you apply aggregation to a raw stream of Windows Events, the outcome might not be compatible with the corresponding Event table, and the ingestion-time transformation should begin with a custom stream header.

Design approach for multi-stage processing

Follow these steps when designing a multi-stage DCR:

1. Assess data sources and destinations. Identify your data source types and decide where each lands: the default standard table or a custom table.
2. Identify aggregation needs. Aggregated logs should go to a separate table because their shape differs from the raw form.
3. Plan for differential processing. If you need to process portions of the same logs differently, create multiple data sources of the same type with different collection settings and apply different client-side transformations to each.
4. Author client-side transformations. Use standard streams (Microsoft-*) if the output retains the header schema, or custom streams (Custom-*) if it doesn't. Define custom streams in streamDeclarations.
5. Define data flows. For each stream, create a data flow. Use ingestion-time data flows to split a single stream across multiple destination tables by applying different filter criteria per data flow.

Workspace transformation DCR

Transformations are defined in a data collection rule (DCR), but there are still data collections in Azure Monitor that don't yet use a DCR. Examples include resource logs collected by diagnostic settings and application data collected by Application Insights.

The workspace transformation data collection rule (DCR) is a special DCR that's applied directly to a Log Analytics workspace. The purpose of this DCR is to perform transformations on data that doesn't yet use a DCR for its data collection, and thus has no means to define a transformation.

There can be only one workspace DCR for each workspace, but it can include transformations for any number of supported tables. These transformations are applied to any data sent to these tables unless that data came from another DCR.

For example, the Event table is used to store events from Windows virtual machines. If you create a transformation in the workspace transformation DCR for the Event table, it would be applied to events collected by virtual machines running the Log Analytics agent¹ because this agent doesn't use a DCR. The transformation would be ignored though by any data sent from Azure Monitor Agent (AMA) because it uses a DCR to define its data collection. You can still use a transformation with Azure Monitor agent, but you would include that transformation in the DCR associated with the agent and not the workspace transformation DCR.

¹ The Log Analytics agent has been deprecated, but some environments might still use it. It's only one example of a data source that doesn't use a DCR.

Azure Monitor pipeline transformations

Azure Monitor pipeline data transformations provide similar functionality as client-side transformations in Azure Monitor. Both allow you to apply a KQL query to incoming data to filter or modify that data before it's sent to the next step in the data flow.

Azure Monitor ingestion-time transformations are run after the data is received by Azure Monitor but before it's ingested in the Log Analytics workspace. Azure Monitor pipeline transformations are applied earlier in the data flow, allowing for data shaping and filtering before the data is sent to Azure Monitor. This makes pipeline transformations useful for reducing data volume and network bandwidth when sending data from edge or multicloud environments.

The following table summarizes the key differences between Azure Monitor pipeline transformations and Azure Monitor ingestion-timetransformations:

The data that's ingested into Azure Monitor is a combination of the pipeline transformation and any subsequent Azure Monitor ingestion-time transformations. The only requirement is that the output schema of the pipeline transformation must match the input schema expected by the Azure Monitor ingestion-time transformation. While you can filter data in either transformation, it's generally more efficient to filter data in the pipeline transformations since this reduces the amount of data sent over the network. The schema of the data output by the Azure Monitor ingestion-time transformation must match the schema of the destination table in the Log Analytics workspace.

Cost for transformations

Processing logs (transforming and filtering) in the Azure Monitor cloud pipeline has different billing implications depending on the type of table into which data is being ingested in a Log Analytics workspace.

Auxiliary Logs

Auxiliary Logs assigns a cost for data processed and data ingested into a Log Analytics workspace. The data processing charge applies to all of the incoming data received by the Azure Monitor if the destination in a Log Analytics workspace is an Auxiliary Logs table. This includes the amount of data processed by any Azure Monitor ingestion-time transformations. The data ingestion charge applies only to the data after the ingestion-time transformation delivered to an Auxiliary Logs table. Transformations can either increase or decrease the size of the data.

The following table shows some examples:

See Azure Monitor pricing for prices for log processing and log data ingestion.

Analytics or Basic Logs

For Analytics or Basic Logs, transformations themselves don't usually incur any costs, but the following scenarios can result in additional charges:

• If a transformation increases the size of the incoming data, such as by adding a calculated column, you're charged the standard ingestion rate for the extra data.
• If a transformation reduces the ingested data by more than 50%, you're charged for the amount of filtered data above 50%.

To calculate the data processing charge resulting from transformations, use the following formula:

[GB data dropped by transformation] - ([GB incoming data size] / 2).

The following table shows examples.

To avoid this charge, you should filter ingested data using alternative methods before applying transformations. By doing so, you can reduce the amount of data processed by transformations and, therefore, minimize any additional costs.

See Azure Monitor pricing for pricing for log processing and log data ingestion.

Related content

• Read more about data collection rules (DCRs).
• Create a transformation in Azure Monitor, including multi-stage transformations.
• Structure of a data collection rule (DCR) for the complete JSON schema, including the multi-stage transformations section.
• Create a workspace transformation DCR for data not collected using a DCR.
• Multi-stage transformations announcement on the Azure Observability Blog for background on the public preview.