Edit

Configure data flow endpoints for local storage

To send data to local storage in Azure IoT Operations, you can configure a data flow endpoint. With this configuration, you can specify the endpoint, authentication, table, and other settings.

Prerequisites

  • The Azure CLI version 2.62.0 or newer installed on your development machine. Use az --version to check your version and az upgrade to update if necessary. For more information, see Install the Azure CLI.

  • The Azure IoT Operations extension for the Azure CLI. Use the following command to add the extension or update it to the latest version:

    az extension add --upgrade --name azure-iot-ops
    

Create a local storage data flow endpoint

Use the local storage option to send data to a locally available persistent volume. You can use it to upload data via Azure Container Storage enabled by Azure Arc edge volumes.

  1. In the operations experience, select the Data flow endpoints tab.

  2. Under Create new data flow endpoint, select Local Storage > New.

    Screenshot that shows using the operations experience to create a local storage data flow endpoint.

  3. Enter the following settings for the endpoint:

    Setting Description
    Name The name of the data flow endpoint.
    Persistent volume claim name The name of the PVC to use for local storage.
  4. Select Apply to provision the endpoint.

The PVC must be in the same namespace as DataflowEndpoint.

Supported serialization formats

The only supported serialization format is Parquet.

Use Azure Container Storage enabled by Azure Arc

You can use the local storage data flow endpoint together with Azure Container Storage enabled by Azure Arc to store data locally or send data to a cloud destination.

Important

You must install Azure Container Storage enabled by Azure Arc before you use it with a local storage data flow endpoint.

Local shared volume

To write to a local shared volume, first create a PVC according to the instructions in Local shared edge volumes.

When you configure your local storage data flow endpoint, input the PVC name under persistentVolumeClaimRef.

Cloud ingest

To write your data to the cloud, follow the instructions in Cloud ingest edge volumes configuration to create a PVC and attach a subvolume for the cloud destination that you want.

To configure cloud ingest, your cluster must have secure settings enabled. The cloud ingest feature relies on workload identity federation.

Important

Don't forget to create the subvolume after you create the PVC. Otherwise, the data flow fails to start and the logs show a "read-only file system" error.

When you configure your local storage data flow endpoint, input the PVC name under persistentVolumeClaimRef.

Finally, when you create the data flow, the data destination parameter must match the spec.path parameter that you created for your subvolume during configuration.

Next step