Get events from Azure Event Hubs into Real-Time hub

1. On the Add data page, switch to the Azure tab.

   

2. Hover the mouse over your Azure Event Hubs namespace in the list, and select the Connect button that appears. Alternatively, you can select ... and then Connect from the dropdown menu.

   Use the search bar to quickly find your Event Hubs namespace if you have many Azure resources (or) use filters to filter the list by source type (Azure Event Hubs namespace), subscription, resource group, or region.

   

3. In the Connect data source wizard, on the Configure page, follow these steps:

   1. Select an event hub from the dropdown list. The dropdown is populated with event hubs from the selected Azure Event Hubs namespace.

   2. Select they key name from the dropdown list. The dropdown is populated with key names from the selected event hub.

      

   3. For Consumer group, select $Default or enter the name of a custom consumer group that you have set up for this event hub.

   4. For Data format, select the format of the events in your event hub (for example, JSON, Avro, etc.).

   5. In the Stream details section to the right, follow these steps:

      1. Select the Fabric workspace where you want to create the eventstream.

      2. For Eventstream name, select the Pencil button, and enter a name for the eventstream.

      3. The Stream name value is automatically generated for you by appending -stream to the name of the eventstream. This stream appears on the real-time hub's All data streams page when the wizard finishes.

         

4. Select Review and connect at the bottom of the Configure page.

5. Review the details on the Review + connect page, and then select Connect.

   

1. On the Add data page, switch to the Azure tab.

   

2. Hover the mouse over your Azure Event Hubs namespace in the list, and select the Connect button that appears. Alternatively, you can select ... and then Connect from the dropdown menu.

   Use the search bar to quickly find your Event Hubs namespace if you have many Azure resources (or) use filters to filter the list by source type (Azure Event Hubs namespace), subscription, resource group, or region.

   

3. In the Connect data source wizard, on the Configure page, follow these steps:

   1. Select an event hub from the dropdown list. The dropdown is populated with event hubs from the selected Azure Event Hubs namespace.

   2. Select they key name from the dropdown list. The dropdown is populated with key names from the selected event hub.

      

   3. For Feature level, select Extended features.

      1. For Consumer group, select a consumer group from the dropdown list. The dropdown is populated with consumer groups from the selected event hub.
      2. For Starting position, select the point from which you want to start ingesting events. You can choose to start from the earliest available event, the latest event, or a specific point in time.

   4. In the Stream details section to the right, follow these steps:

      1. Select the Fabric workspace where you want to create the eventstream.

      2. For Eventstream name, select the Pencil button, and enter a name for the eventstream.

      3. The Stream name value is automatically generated for you by appending -stream to the name of the eventstream. This stream appears on the real-time hub's All data streams page when the wizard finishes.

         

   5. Select Next at the bottom of the Configure page.

Schema handling page

1. On the Schema handling page, provide rules to handle events received from the selected event hub, so that the eventstream can apply them correctly. The mapping rules depend on how you model the events.

   If you have one schema that governs all of the events, select Fixed schema.

   If you have multiple schemas that represent the various incoming events, define matching rules to apply your schemas. To choose this mode, select Dynamic schema via headers. Then, specify the rules by using header/value pairs to select each schema. The header is a custom Kafka header property that's part of the event metadata. The value is the expected value for that property.

   

2. Choose schemas by selecting the Add more schema(s) dropdown menu and then choosing one or more existing schemas from the event schema registry. If you don't have schemas to choose from, you can create new schemas from this view. To learn how to define a new event schema, see Create and manage event schemas in schema sets.

   

   If you selected the Choose from event schema registry option, the Associate an event schema pane appears. Select one or more schemas from the registry, depending on your schema matching mode, and then select Choose at the bottom of the pane.

   

3. If you selected the Fixed schema option, you don't need to provide any more rules to match the schema. You can continue to the next step.

   If you selected the Dynamic schema via headers option, specify the Kafka header property and the expected value that maps to the schema. Add more schemas and specify different header properties and/or different values to map to those schemas.

   Note

   When you define the mapping rules, each value of the header must be unique. If you try to reuse a schema, you see a warning message indicating that you might break existing streams. As long as the mapping rules are the same, you can reuse a schema. If this limitation affects your use, reach out to your Microsoft representative to share your feedback. We're actively working on removing this limitation.

   

4. After you map schemas for all expected events, select Next at the bottom of the Schema handling page.

   

5. Select Review and connect at the bottom of the Configure page.

6. Review the details on the Review + connect page, and then select Connect.

1. On Configure connection settings, confirm that Basic is selected for the feature level, and then select New connection.

   

   If there's an existing connection to your event hub, select that existing connection. Then, move on to configuring the data format in the following steps.

   

2. In the Connection settings section, follow these steps:

   1. Enter the name of the Event Hubs namespace.
   2. Enter the name of the event hub.

   

3. In the Connection credentials section, follow these steps:

   1. For Connection name, enter a name for the connection to the event hub.
   2. For Authentication kind, confirm that Shared Access Key is selected.
   3. For Shared Access Key Name, enter the name of the shared access key. For instructions on getting an access key, see Get an Azure Event Hubs connection string.
   4. For Shared Access Key, enter the value of the shared access key.
   5. Select Connect.

   

4. For Consumer group, enter the name of the consumer group. The default consumer group for the event hub is $Default.

5. For Data format, select a data format for the incoming real-time events that you want to get from your Azure event hub. You can select from JSON, Avro, and CSV (with header) data formats.

   

6. On the Source details pane to the right, select the pencil icon next to the source name, and then enter a name for the source. This step is optional.

   

7. Select Next at the bottom of the page.

   

8. On the Review + connect page, review the settings, and then select Add.

   

1. On Configure connection settings, for Choose feature level, select Extended features.

   

   If there's an existing connection to your event hub, select that existing connection. Then, move on to configuring the data format in the following steps.

2. In the Connection settings section, follow these steps:

   1. Enter the name of the Event Hubs namespace.

   2. Enter the name of the event hub.

      

3. In the Connection credentials section, follow these steps:

   1. For Connection name, enter a name for the connection to the event hub.

   2. For Authentication kind, confirm that Shared Access Key is selected.

   3. For Shared Access Key Name, enter the name of the shared access key. For instructions on getting an access key, see Get an Azure Event Hubs connection string.

   4. For Shared Access Key, enter the value of the shared access key.

   5. Select Connect.

      

4. For Consumer group, enter the name of the consumer group. The default consumer group for the event hub is $Default.

   

Stream or source details

1. On the Connect page, follow one of these steps based on whether you're using Eventstream or Real-Time hub.

   • Eventstream:

     In the Source details pane to the right, follow these steps:

     1. For Source name, select the Pencil button to change the name.

     2. Notice that Eventstream name and Stream name are read-only.

   • Real-Time hub:

     In the Stream details section to the right, follow these steps:

     1. Select the Fabric workspace where you want to create the eventstream.

     2. For Eventstream name, select the Pencil button, and enter a name for the eventstream.

     3. The Stream name value is automatically generated for you by appending -stream to the name of the eventstream. This stream appears on the real-time hub's All data streams page when the wizard finishes.

2. Select Next at the bottom of the Configure page.

Schema handling page

1. On the Schema handling page, provide rules to handle events received from the selected event hub, so that the eventstream can apply them correctly. The mapping rules depend on how you model the events.

   If you have one schema that governs all of the events, select Fixed schema.

   If you have multiple schemas that represent the various incoming events, define matching rules to apply your schemas. To choose this mode, select Dynamic schema via headers. Then, specify the rules by using header/value pairs to select each schema. The header is a custom Kafka header property that's part of the event metadata. The value is the expected value for that property.

   

2. Choose schemas by selecting the Add more schema(s) dropdown menu and then choosing one or more existing schemas from the event schema registry. If you don't have schemas to choose from, you can create new schemas from this view. To learn how to define a new event schema, see Create and manage event schemas in schema sets.

   

   If you selected the Choose from event schema registry option, the Associate an event schema pane appears. Select one or more schemas from the registry, depending on your schema matching mode, and then select Choose at the bottom of the pane.

   

3. If you selected the Fixed schema option, you don't need to provide any more rules to match the schema. You can continue to the next step.

   If you selected the Dynamic schema via headers option, specify the Kafka header property and the expected value that maps to the schema. Add more schemas and specify different header properties and/or different values to map to those schemas.

   Note

   When you define the mapping rules, each value of the header must be unique. If you try to reuse a schema, you see a warning message indicating that you might break existing streams. As long as the mapping rules are the same, you can reuse a schema. If this limitation affects your use, reach out to your Microsoft representative to share your feedback. We're actively working on removing this limitation.

   

4. After you map schemas for all expected events, select Next at the bottom of the Schema handling page.

   

Review and connect

1. On the Review + connect page, review the settings, and then select Connect.

   

2. On the Review + connect page, select Add (Eventstream) or Connect (Real-Time hub).