Creating ADX Environment Using PowerShell

We can use PowerShell to create the Azure Data Explorer cluster and the related database.

Download the PowerShell Az Module

We will be using the new Azure PowerShell Az module. In December 2018, Microsoft released the new PowerShell Az module, which was a huge improvement over the AzureRm module, and we will be working on this new module as this is the new intended way for the developers to work on Azure.

There are a number of benefits of using this new module like

  1. It offers shorter commands
  2. It offers cross-platform support
  3. It also has improved stability

With the Az module, PowerShell can now work on different platforms like macOS, Linux, Windows, and also on embedded platforms. Under the hood, since the Az modules were written in .Net standard, it is now capable of working with PowerShell 5.1 for Windows and PowerShell Core 6.x and later on other platforms.

You can enable the compatibility mode so as to add aliases for AzureRm module. This helps in your transition as a developer from using the old module to the new module by using the Enable-AzureRmAlias. You can enable the alias for specific modules and/or at the current user scope.

By default, the PowerShell gallery isn’t configured as a trusted repository for PowerShellGet and you see the prompt confirming your trust for PSGallery. Else, the installation starts immediately.

Since the Az module is a bootstrapper, it will manage the download and installation of all the service-specific modules for you. You can optionally run the Update-Help command to get all the documentation.

If you have multiple subscriptions, you can get the subscription details using the Get-AzSubscription command, and then in the next step, set the context to a specific subscription using the Set-AzContext command and provide the subscription id.

One important fact that should be taken into consideration before you start working with the PowerShell commands to create and manage the ADX cluster and database is that Microsoft announced that the PowerShell Gallery has deprecated Transport Layer Security (TLS) versions 1.0 and 1.1 as of April 2020. This was done by Microsoft to provide best in class encryption to the customers.

Step-wise commands are listed and explained below:

  1. Use this command to install all the Az modules using PowerShellGet and PowerShell Gallery
    Find-Module -Name Az -Repository PSGallery | Install-Module -Verbose –Force
    In case you face issues, please use the below command
    Find-Module -Name Az | Install-Module -Verbose –Force
    

    You will see the below screen during the installation
    installAzmodule

  2. If you wish to gets the list of all the installed Az modules, you can use the below command. This step is optional and is not needed if this is a fresh installation
    Get-InstalledModule -Name Az -AllVersions | Select Name, Version

    getazversion

  3. Connect to your Azure subscription

    Connect-AzAccount / Login-AzAccount

    You see the below screen, where you need to enter your credentials

    connectAzAccount
    Once the login is successful, you will come back to to the PowerShell command prompt and see the below screen.
    successfullogin

  4. In case you have multiple subscriptions, you can get your subscription details using the below command
    Get-AzSubscription

    and you can set the context to a particular subscription using the command below, in order to work on a particular subscription

    Set-AzContext –Subscription ‘<subscription id>’
  5. Next step is to install the Kusto modules to work with Azure Data Explorer. Use the below command
    Install-Module -Name Az.Kusto -AllowClobber

    You will see the below screen during installation.
    installazkusto

  6. Now that the needed AZ and Kusto modules are installed, it is time to start creating the resources. You need to first create the Resource Group. In case you already have the Resource Group, you can skip this step.
    New-AzResourceGroup -Name <name> -Location ‘<location>'

    resourcegroup

  7. Create the Azure Data Explorer cluster
    New-AzKustoCluster -Name <name> -ResourceGroupName <rg.name> -Location ‘<location>' -SkuName <sku> -SkuTier <tier> -SkuCapacity <number>

    ADXclustercomplete

    -ResourceGroupName – Name of your Resource Group
    -SkuName – Name of the SKU that will be used for your cluster
    -SkuTier: The tier used to create the ADX Cluster
    -SkuCapacity: The number of instances of the cluster

    Apart from the above, there are other parameters as well, which can be used to configure your cluster at a granular level. They can be used as per the requirements, but are optional.

    You can use this link for detailed information on other parameters that can be used for granular level configurations.

    In case you receive the below error
    The subscription is not registered to use namespace ‘Microsoft.Kusto’
    use the below command to register the Microsoft.Kusto provider namespace with your subscription

    Register-AzureRmResourceProvider -ProviderNamespace Microsoft.Kusto'
  8. After the cluster creation is complete, create the cluster database for data ingestion to and from ADX.
    New-AzKustoDatabase -Name <db name> -ClusterName <cl.name> -ResourceGroupName <rg.name> -Kind ReadWrite

    adxdb

    -ClusterName: Name of the ADX cluster created in the previous step
    -ResourceGroupName: Name of the resource group
    -Kind: The kind of database to be created

    There are other parameters as well, which can be used for more granular level configuration. For more information click here.

Azure Data Explorer Cluster Actions Using PowerShell

Now that the cluster and database creation is complete, you can start using ADX for analytics and ETL jobs. There are other actions as well that can be performed on the Azure Data Explorer cluster and associated database. Some of the most important one are listed below.

  • Start-AzKustoCluster – The cluster automatically starts when you create it, but you can start the ADX Cluster using Start-AzKustoCluster command if it is in the stop state.
    Start-AzKustoCluster -ResourceGroupName <rg.name> -Name <cluster name>
  • Stop-AzKustoCluster – To stop the ADX cluster, use the Stop-AzKustoCluster
    Stop-AzKustoCluster -ResourceGroupName <rg.name> -Name <cluster name>
  • Update-AzKustoDatabase – To stop the ADX cluster, use the Stop-AzKustoCluster
    Stop-AzKustoCluster -ResourceGroupName <rg.name> -Name <cluster name>

    If you wish to update the soft delete period in days using the database id, you can use the below command

    $adxdbid = Get-AzKustoDatabase -ResourceGroupName <rg.name> -ClusterName <cl.name> -Name <db.name>
    $softdeletedays = New-TimeSpan -Days <number.of.days>
    Update-AzKustoDatabase -InputObject $adxdbid -Kind ReadWrite -SoftDeletePeriod $softdeletedays -Location 'Central US'

    You can run the same command using the database name as well.

    $softdeletedays = New-TimeSpan -Days <number.of.days>
    Update-AzKustoDatabase -ResourceGroupName <rg.name> -ClusterName <cl.name> -Name <db.name> -Kind ReadWrite -SoftDeletePeriod $softdeletedays -Location 'Central US'
  • Update-AzKustoDataConnection – Use this action to update the existing data connections. Connections can be from EventHub, EventGrid, IoTHub, etc. It can be done using the identity of the connection or by name. In case you want to update the connection with he EventHub, use the below command
    Update-AzKustoDataConnection -ResourceGroupName <rg.name> -ClusterName <cl.name> -DatabaseName <db.name> -DataConnectionName <connection.name> -Kind "EventHub" -EventHubResourceId "/subscriptions/<subscriptionid>/resourcegroups/testrg/providers/Microsoft.EventHub/namespaces/myeventhubns/eventhubs/myeventhub" -DataFormat "JSON" -ConsumerGroup '$Default' -Compression "None" -TableName "Events" -MappingRuleName <rule.name>
  • Update-AzKustoCluster – Use this action to update the ADX Cluster. Again, the updates can be performed either by cluster name or by using the id of the cluster.In order to update the SkuName from ‘Standard_D13_V2’ to ‘Standard_D12_V2’, use the below action
    Update-AzKustoCluster -ResourceGroupName <rg.name> -Name <cluster.name> -SkuName 'Standard_D12_v2' -SkuTier Standard

    updatesku

  • Add-AzKustoClusterLanguageExtension – This action is used if you want to work with particular languages only. You can use the value parameter with the action to add those language(s) to the cluster.If you want to add Python as a language to work with the ADX cluster, use the below command to add it as a language extension.
    Add-AzKustoClusterLanguageExtension -ResourceGroupName <rg.name> -ClusterName <cl.name> -Value (@{Name="PYTHON"})

    While the update is running on the ADX cluster as a result of the actions taken, if you go to Azure portal, under ADX Cluster overview page, you will notice that the state shows to be as “updating” along with a warning sign that “Your Azure Data Explorer cluster is currently in state Updating”.

    adxupdate1

For a more comprehensive list of actions you can visit Mocrosoft docs

Next>> Creating ADX Environment using CLI

Previous>> Creating the ADX Environment Using Azure portal


Part – 1: Data Science Overview

Part – 2: Understanding Azure Data Explorer

Part – 3: Azure Data Explorer Features

Part – 4: Azure Data Explorer Service Capabilities

Part – 6: The Kusto Query Language

Part – 7: Data Obfuscation in Kusto Query Language

Part – 8: Data Ingestion Preparation: Schema Mapping

Part – 9: Overview of data ingestion in Azure Data Explorer

Part – 10: Managing Azure Data Explorer Cluster

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: