Managing Azure Data Explorer cluster

Services in Azure react to the way they are configured and utilized. They react to the increasing and decreasing demands for the compute power and also comply with security and access configuration. The same is the case with Azure Data Explorer. We will start this module by looking into how database permissions are managed for Azure Data Explorer.

Manage Azure Data Explorer Database Permissions

Azure Data Explorer provides a convenient way to manage database and table level permissions. At the core it uses RBAC (Role Based Access Control) and under the hood AAD users, groups and principals are mapped to roles with different access rights to the underlying resource (database or table). Permissions can be managed both within Azure portal and by using control commands. Here is an extract of all the available database and table roles.

permissions

Health Check of Azure Data Explorer

The metrics within Azure Data Explorer Cluster provide indicators that help in determining the health and performance of the resources within the cluster. You can use the metrics data in creating the dashboards as well as sent alert notifications in case the metric thresholds values are met or exceeded.

As I mentioned that metrics have key indicators, I have outlined them here, which can be put to use to monitor the health and performance of different Azure Data Explorer cluster components.

When creating the chart or the metric dashboard, we need to choose the appropriate key indicator as well as the aggregation type. You can see that in the screenshot.

metrics

For each of the metric indicator we choose, the type of aggregation that we can have is mentioned in the table on the right. Also, for each metric indicator, we have the unit in which the results are returned.

metricunit

Managing Azure Data Explorer Scaling

What will happen to the performance when there is a surge in the requests or underutilization of the cluster?

Azure provides us a flexible way to manage the size of the resources in such cases apart from being handled manually or it can also be kept static based on the foresight of the usage. Since it is difficult to predict the demand with accuracy, it is better to keep the services to auto-scale to respond to the changing performance needs.

There are two ways to do it –

Horizontal scaling, which is also known as scaling out, where the compute instances are automatically increased as per the rules configured during the configuration of the data explorer cluster.There are three ways for scaling out. They are
1. Manual scale, which is manually selecting the cluster’s number of instances.
2. Optimized Autoscale, where we define the minimum and maximum instance count and ADX will handle scale out and scale in automatically when the cluster load is going up or down.
3. Custom Autoscale, which is based on the cluster metrics.
  
  When selecting Custom autoscale, you need to define the scale rules , by specifying the metric source and the criteria for the time aggregation.
  
  Due to the changing nature of cluster load it is recommended to use Optimized Autoscale for automatically managing the cluster size.
Vertical scaling, which is also known as scale up allows changing the cluster’s VM SKU. By changing the VM SKU it is possible to use different SKUs with different proportion between amount of CPU and SSD in the cluster. Scale up helps in optimizing the cluster cost by selecting compute intense workloads or storage intense workloads to fit the specific use case.