Azure Events and Messages


In this post, we will try to understand Event and Message handling in Azure.

We will look into Azure Service Bus, Event Grids, Event Hubs, a short demo on Event handling, and monitoring event, and the administrators also should have an understanding of how the events and messages are managed in Azure.

There are three different services that help us work with events messages These services are:

  • Azure Message Service Bus
  • Event Grid
  • Event Hubs

Microsoft designed each one of them to be used for specific scenarios. We will try to understand the differences amongst each of these services, and also understand which service to opt for your need. At times, they can be clubbed together to perform critical business tasks.

eventsmessages

Before we dig deep into each one of them individually, we should understand the below terms.

Dead-Lettering: Azure Service Bus queues and topic subscriptions provides a sub-queue, called a dead-letter queue (DLQ). We do not need to create the dead-letter queue explicitly and cannot be deleted or otherwise managed independently of the main entity.

The purpose of the dead-letter queue is to hold messages that cannot be delivered to any receiver, or messages that could not be processed. Messages can then be removed from the DLQ and inspected separately

At-Least-Once Delivery: In the event that the application crashes after processing the message, but before the CompleteAsync request is issued, the message is redelivered to the application when it restarts. This process is often called At Least Once processing; that is, each message is processed at least once.

Temporal Buffers: Event Hubs are temporal in nature, and contrary to traditional message bot systems, events on an Event Hub are neither acknowledged nor removed by subscribing clients. Instead, events remain intact until a predetermined time, usually 24 hours of access. Event Hubs are a facilitator of big data in that their primary purpose is to move large volumes of data from point A to point B. For example, logging and auditing systems that consist of widely-distributed and diverse data structures that need to be centralized and parsed.

In-order messaging: The first question to be answered is how to get the events to the Azure Function, to begin with. HTTP or Event Grid wouldn’t work, as there is no way to guarantee that message 1 will arrive and be processed by your function before message 2 arrives and is processed. In order to maintain processing order, the order needs to be persisted somewhere that my Azure Function can pull from. Queues work, but if using Azure remember that Azure storage queues do not guarantee to order so you’d need to stick with Service Bus Queues or Topics. For this specific scenario though, Azure Event Hubs is the best fit.

Azure Message Service Bus

It is a brokered messaging system. It stores the messages in the broker, for example, a queue, and waits until the consuming party is ready to receive the messages.

It has the following characteristics:

  1. Reliable asynchronous message delivery (enterprise messaging as a service) that requires polling
  2. Advanced messaging features like FIFO, batching/sessions, transactions, dead-lettering, temporal control, routing and filtering, and duplicate detection
  3. At least once delivery

An Azure Service Bus namespace can be considered as a host for queues holding business-critical jobs. It allows for the creation of message routes that need to travel between applications and application modules

Event Grids

Event Grids are event-driven and enable reactive programming. is an intelligent event routing service that enables you to react to notifications (events) from apps and services. It internally uses a publish-subscribe model, where the Publishers emit events but have no expectation about which events are handled. Subscribers decide which events they want to handle.

Event Grid is deeply integrated with Azure services and can be integrated with third-party services as well. What is does is – It simplifies event consumption and lowers costs by eliminating the need for constant polling. Event Grid efficiently and reliably routes events from Azure and non-Azure resources. It distributes the events to registered subscriber endpoints. The event message has the information you need to react to changes in services and applications. Event Grid isn’t a data pipeline and doesn’t deliver the actual object that was updated.

Event Grid supports dead-lettering for events that aren’t delivered to an endpoint. It has the following characteristics:

  1. Dynamically Scalable
  2. Low Cost
  3. Serverless
  4. At least Once Delivery

Previously we had the support for Blob Storage, Resource Groups, Azure Subscriptions, Event Hubs, and other custom topics. Microsoft Azure team has been working hard to accumulate additional event publishers and event handlers so as to make it more easy for the reactive handling of the events.

functional-model

We now have additional Event Sources added by Microsoft, which can be published to topics (endpoints), and similarly, we have additional event handlers to handle the events from those sources.

Event Hubs

Azure Event Hubs are big data pipeline. It facilitates the capture, retention, and replay of telemetry and event stream data. There can be multiple concurrent sources. They allow the telemetry and event data to be made available to a variety of stream-processing infrastructures and analytics services. It is available either as data streams or bundled event batches. Event Hubs provide a single solution that enables rapid data retrieval for real-time processing as well as the repeated replay of stored raw data.

It has the following characteristics:

  1. Low latency
  2. Capable of receiving and processing millions of events per second
  3. At least once delivery

In certain scenarios, we can combine the services to achieve certain critical business objectives.

You use Event Grid to respond to events in the other services. For an example of using Event Grid with Event Hubs to migrate data to a data warehouse

overview

 

Below is a high-level summary of each of the services, including purpose, type, and when to use each one of them.

servcies

 

Download as PDF

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: