Simulating Azure event hubs functionality end to end with Azure Data Explorer and generate data feature

To evaluate Azure Event Hubs functionality, typically we need to develop an application for data ingestion and another for data consumption

In my previous blog post, I outlined the process of discovering ingested data in Event Hubs using Azure Data Explorer. In this article, I will demonstrate how to ingest data into Event Hubs without writing a single line of code.

Our first step is to navigate to the Event Hubs instance and access the Generate data (preview) feature.

Multiple options are provided to ingest payload either from pre-canned datasets or custom payloads based on a given schema.

Once you click on the send button, the data will be ingested to our even hub instance.

Then, you can easily discover this data in your pre-configured Azure Data Explorer. I have detailed the steps involved in configuring Azure Data Explorer in my previous blog post.

This approach allows you to explore the functionality of Event Hubs without writing any code!

Visualizing ingested events in Azure Event Hub with Azure Data Explorer

In modern cloud-based solutions, event-driven architectures are very common. Microsoft Azure facilitates event processing through Azure Event Hubs, offering essential building blocks to implement scalable solutions capable of processing large volumes of events and data with low latency and high reliability.

Debugging or testing event-based solutions can be challenging due to the nature of their architecture. It requires implementing ingesting applications and consumers to verify the functionality of event-based solutions.

In this article, I will demonstrate how to leverage Azure Data Explorer to visualize ingested data without any delay. With this solution, there is no need to create any custom solutions to view the contents of our Event Hub instance.

First, we need to create our Azure Data Explorer cluster.

Next, we will create a database in Data Explorer.

To connect Data Explorer with Event Hubs, let's enable a managed account at the Data Explorer instance.

After enabling the managed account at the Data Explorer instance, we need to assign the required permissions to the Event Hub.

Now, let's configure Data Explorer for our Event Hub. Navigate to the Event Hub and select the "Analyze data" option.

Let's link our Data Explorer instance

Now that our configuration is complete, let's ingest some data and explore it in Data Explorer.

To explore our data, navigate to Data Explorer and go to the Query section. Then, select the table and run your query to explore the ingested data.

Presentation - Efficient Data Streaming with Event Hubs for Kafka

Recently I did a technical session in Perth Azure User Group at Microsoft Perth. The event was themed as Azure Open Source Night.

My session was on "Efficient Data Streaming with Event Hubs for Kafka"

Following is the presentation I did

Following is the code I demonstrated. In this sample An Azure Function written in .NET 6 pushes events to an event hub, those are consumed by Apache Kafka consumer written in Java. 

Following are the configuration changes you need to perform.

1. In Producer, change the EventHubConnectionAppSetting in local.settings.json file. You can get the connection string by navigating to SAS policies and in the policy you created for sending.

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "EventHubConnectionAppSetting":"CONNECTION"
  }
}

2. In Consumer, change the Event Hubs namespace and connection string in Consumer.config file.You can get the connection string by navigating to SAS policies and in the policy you created for receiving.

bootstrap.servers=EVENTHUBNAMESPACE.servicebus.windows.net:9093
group.id=$Default
request.timeout.ms=60000
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="CONNECTION";

You can get further information from this repository

I loved the event. Following are few snaps.