Creating ADX Environment Using PowerShell

We can use PowerShell to create the Azure Data Explorer cluster and the related database.

We will be using the new Azure PowerShell Az module. In December 2018, Microsoft released the new PowerShell Az module, which was a huge improvement over the AzureRm module, and we will be working on this new module as this is the new intended way for the developers to work on Azure.

There are a number of benefits of using this new module like

It offers shorter commands
It offers cross-platform support
It also has improved stability

With the Az module, PowerShell can now work on different platforms like macOS, Linux, Windows, and also on embedded platforms. Under the hood, since the Az modules were written in .Net standard, it is now capable of working with PowerShell 5.1 for Windows and PowerShell Core 6.x and later on other platforms.

You can enable the compatibility mode so as to add aliases for AzureRm module. This helps in your transition as a developer from using the old module to the new module by using the Enable-AzureRmAlias. You can enable the alias for specific modules and/or at the current user scope.

By default, the PowerShell gallery isn’t configured as a trusted repository for PowerShellGet and you see the prompt confirming your trust for PSGallery. Else, the installation starts immediately.

Since the Az module is a bootstrapper, it will manage the download and installation of all the service-specific modules for you. You can optionally run the Update-Help command to get all the documentation.

If you have multiple subscriptions, you can get the subscription details using the Get-AzSubscription command, and then in the next step, set the context to a specific subscription using the Set-AzContext command and provide the subscription id.

One important fact that should be taken into consideration before you start working with the PowerShell commands to create and manage the ADX cluster and database is that Microsoft announced that the PowerShell Gallery has deprecated Transport Layer Security (TLS) versions 1.0 and 1.1 as of April 2020. This was done by Microsoft to provide best in class encryption to the customers.

Step-wise commands are listed and explained below:

Use this command to install all the Az modules using PowerShellGet and PowerShell Gallery

Find-Module -Name Az -Repository PSGallery | Install-Module -Verbose –Force
In case you face issues, please use the below command
Find-Module -Name Az | Install-Module -Verbose –Force

You will see the below screen during the installation
installAzmodule

If you wish to gets the list of all the installed Az modules, you can use the below command. This step is optional and is not needed if this is a fresh installation
```
Get-InstalledModule -Name Az -AllVersions | Select Name, Version
```
Connect to your Azure subscription
```
Connect-AzAccount / Login-AzAccount
```
You see the below screen, where you need to enter your credentials

Once the login is successful, you will come back to to the PowerShell command prompt and see the below screen.
In case you have multiple subscriptions, you can get your subscription details using the below command
```
Get-AzSubscription
```
and you can set the context to a particular subscription using the command below, in order to work on a particular subscription
```
Set-AzContext –Subscription ‘<subscription id>’
```
Next step is to install the Kusto modules to work with Azure Data Explorer. Use the below command
```
Install-Module -Name Az.Kusto -AllowClobber
```
You will see the below screen during installation.
Now that the needed AZ and Kusto modules are installed, it is time to start creating the resources. You need to first create the Resource Group. In case you already have the Resource Group, you can skip this step.
```
New-AzResourceGroup -Name <name> -Location ‘<location>'
```
Create the Azure Data Explorer cluster
```
New-AzKustoCluster -Name <name> -ResourceGroupName <rg.name> -Location ‘<location>' -SkuName <sku> -SkuTier <tier> -SkuCapacity <number>
```
-ResourceGroupName – Name of your Resource Group
-SkuName – Name of the SKU that will be used for your cluster
-SkuTier: The tier used to create the ADX Cluster
-SkuCapacity: The number of instances of the cluster

Apart from the above, there are other parameters as well, which can be used to configure your cluster at a granular level. They can be used as per the requirements, but are optional.

You can use this link for detailed information on other parameters that can be used for granular level configurations.

In case you receive the below error –
The subscription is not registered to use namespace ‘Microsoft.Kusto’
use the below command to register the Microsoft.Kusto provider namespace with your subscription
```
Register-AzureRmResourceProvider -ProviderNamespace Microsoft.Kusto'
```
After the cluster creation is complete, create the cluster database for data ingestion to and from ADX.
```
New-AzKustoDatabase -Name <db name> -ClusterName <cl.name> -ResourceGroupName <rg.name> -Kind ReadWrite
```
-ClusterName: Name of the ADX cluster created in the previous step
-ResourceGroupName: Name of the resource group
-Kind: The kind of database to be created

There are other parameters as well, which can be used for more granular level configuration. For more information click here.

Azure Data Explorer Cluster Actions Using PowerShell

Now that the cluster and database creation is complete, you can start using ADX for analytics and ETL jobs. There are other actions as well that can be performed on the Azure Data Explorer cluster and associated database. Some of the most important one are listed below.

Start-AzKustoCluster – The cluster automatically starts when you create it, but you can start the ADX Cluster using Start-AzKustoCluster command if it is in the stop state.
```
Start-AzKustoCluster -ResourceGroupName <rg.name> -Name <cluster name>
```
Stop-AzKustoCluster – To stop the ADX cluster, use the Stop-AzKustoCluster
```
Stop-AzKustoCluster -ResourceGroupName <rg.name> -Name <cluster name>
```

Update-AzKustoDatabase – To stop the ADX cluster, use the Stop-AzKustoCluster

Stop-AzKustoCluster -ResourceGroupName <rg.name> -Name <cluster name>

If you wish to update the soft delete period in days using the database id, you can use the below command

$adxdbid = Get-AzKustoDatabase -ResourceGroupName <rg.name> -ClusterName <cl.name> -Name <db.name>
$softdeletedays = New-TimeSpan -Days <number.of.days>
Update-AzKustoDatabase -InputObject $adxdbid -Kind ReadWrite -SoftDeletePeriod $softdeletedays -Location 'Central US'

You can run the same command using the database name as well.

$softdeletedays = New-TimeSpan -Days <number.of.days>
Update-AzKustoDatabase -ResourceGroupName <rg.name> -ClusterName <cl.name> -Name <db.name> -Kind ReadWrite -SoftDeletePeriod $softdeletedays -Location 'Central US'

Update-AzKustoDataConnection – Use this action to update the existing data connections. Connections can be from EventHub, EventGrid, IoTHub, etc. It can be done using the identity of the connection or by name. In case you want to update the connection with he EventHub, use the below command

Update-AzKustoDataConnection -ResourceGroupName <rg.name> -ClusterName <cl.name> -DatabaseName <db.name> -DataConnectionName <connection.name> -Kind "EventHub" -EventHubResourceId "/subscriptions/<subscriptionid>/resourcegroups/testrg/providers/Microsoft.EventHub/namespaces/myeventhubns/eventhubs/myeventhub" -DataFormat "JSON" -ConsumerGroup '$Default' -Compression "None" -TableName "Events" -MappingRuleName <rule.name>

Update-AzKustoCluster – Use this action to update the ADX Cluster. Again, the updates can be performed either by cluster name or by using the id of the cluster.In order to update the SkuName from ‘Standard_D13_V2’ to ‘Standard_D12_V2’, use the below action
```
Update-AzKustoCluster -ResourceGroupName <rg.name> -Name <cluster.name> -SkuName 'Standard_D12_v2' -SkuTier Standard
```
Add-AzKustoClusterLanguageExtension – This action is used if you want to work with particular languages only. You can use the value parameter with the action to add those language(s) to the cluster.If you want to add Python as a language to work with the ADX cluster, use the below command to add it as a language extension.
```
Add-AzKustoClusterLanguageExtension -ResourceGroupName <rg.name> -ClusterName <cl.name> -Value (@{Name="PYTHON"})
```
While the update is running on the ADX cluster as a result of the actions taken, if you go to Azure portal, under ADX Cluster overview page, you will notice that the state shows to be as “updating” along with a warning sign that “Your Azure Data Explorer cluster is currently in state Updating”.