locked
How to get Specific Linked Service Type In Azure Data Factory using Power Shell RRS feed

  • Question

  • Hi,

    I am executing below power shell script and getting list of all linked services from azure data factory. I want to extract only specific linked service type from response.

    Below is shell script:

    $azureRmProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile
    $currentAzureContext = Get-AzContext
    $profileClient = New-Object Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient($azureRmProfile)
    $token = $profileClient.AcquireAccessToken($currentAzureContext.Subscription.TenantId)
    $token1 = $token.AccessToken
    $uri = "GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/linkedservices?api-version=2018-06-01"
    $method = "GET"
    $contentType = "application/json"
    $pat = "Bearer $token1"
    
    $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers @{Authorization = $pat } -UseBasicParsing
    write-output $response.Content

    Example:

    In my data factory there are two linked services 1) "type":"AzureDataLakeStore" 2) "type":"AzureKeyVault" and I want to list/extract only AzureDatalakeStore type.

    Below is response from power shell script:

    {"value":[{"id":"/subscriptions/xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.DataFactory/factories/xxxx/linkedservices/xxxx
    xxx","name":"xxxxxx","type":"Microsoft.DataFactory/factories/linkedservices","properties":{"annotations":[],"type":"AzureKeyVault","typeProperties":{"baseUrl":"https://ak
    xxxxxx.vault.azure.net/"}},"etag":"xxxxx"},{"id":"/subscriptions/xxxxx/resourceGroups/xxxxx/providers/
    Microsoft.DataFactory/factories/xxxxx/linkedservices/xxxx","name":"xxxxx","type":"Microsoft.DataFactory/factories/linkedservices","properties":{"an
    notations":[],"type":"AzureDataLakeStore","typeProperties":{"dataLakeStoreUri":"adl://xxxxx.azuredatalakestore.net/","servicePrincipalId":"xxxxxxx
    ","servicePrincipalKey":{"type":"AzureKeyVaultSecret","store":{"referenceName":"xxxxx","type":"LinkedServiceReference"},"secretName":"data-client-secret"},"tenant
    ":"xxxxx","subscriptionId":"xxxxx","resourceGroupName":"xxxxxx"}},"etag":"xxxxxx
    "}]}

    Thanks,

    Brahma

    Friday, May 22, 2020 1:48 PM

Answers

  • The output is JSON which is an object.  Just reference the property you want.

    # example
    $json = @'
    {"value":[{"id":"/subscriptions/xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.DataFactory/factories/xxxx/linkedservices/xxxx
    xxx","name":"xxxxxx","type":"Microsoft.DataFactory/factories/linkedservices","properties":{"annotations":[],"type":"AzureKeyVault","typeProperties":{"baseUrl":"https://ak
    xxxxxx.vault.azure.net/"}},"etag":"xxxxx"},{"id":"/subscriptions/xxxxx/resourceGroups/xxxxx/providers/
    Microsoft.DataFactory/factories/xxxxx/linkedservices/xxxx","name":"xxxxx","type":"Microsoft.DataFactory/factories/linkedservices","properties":{"an
    notations":[],"type":"AzureDataLakeStore","typeProperties":{"dataLakeStoreUri":"adl://xxxxx.azuredatalakestore.net/","servicePrincipalId":"xxxxxxx
    ","servicePrincipalKey":{"type":"AzureKeyVaultSecret","store":{"referenceName":"xxxxx","type":"LinkedServiceReference"},"secretName":"data-client-secret"},"tenant
    ":"xxxxx","subscriptionId":"xxxxx","resourceGroupName":"xxxxxx"}},"etag":"xxxxxx
    "}]}
    '@ | ConvertFrom-Json
    
    # from content
    $response.Content | ConvertFrom-Json
    

    The following shows how to extract the properties

    PS C:\scripts> $json.Value[0]
    
    id         : /subscriptions/xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.DataFactory/factories/xxxx/linkedservices/xxxx
                 xxx
    name       : xxxxxx
    type       : Microsoft.DataFactory/factories/linkedservices
    properties : @{annotations=System.Object[]; type=AzureKeyVault; typeProperties=}
    etag       : xxxxx
    
    
    
    PS C:\scripts> $json.Value[0].type
    Microsoft.DataFactory/factories/linkedservices
    PS C:\scripts> $json.Value[1].type
    Microsoft.DataFactory/factories/linkedservices
    PS C:\scripts>


    \_(ツ)_/

    Friday, May 22, 2020 3:24 PM
  • thank you so much sir helping me on this. It is working as you mentioned.

    I need another help:

    I am getting multiple names for one type, example: "type":"AzureDataLakeStore" having three values I want to export them into individual file instead of single file.

    Below code is writing all final result into single file but I want to export result into individual files:

    $azureRmProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile
    $currentAzureContext = Get-AzContext
    $profileClient = New-Object Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient($azureRmProfile)
    $token = $profileClient.AcquireAccessToken($currentAzureContext.Subscription.TenantId)
    $token1 = $token.AccessToken
    $uri = "https://management.azure.com/subscriptions/******/resourceGroups/******/providers/Microsoft.DataFactory/factories/*****/linkedservices?api-version=2018-06-01"
    $method = "GET"
    $contentType = "application/json"
    $pat = "Bearer $token1"
    
    $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers @{Authorization = $pat } -UseBasicParsing
    
    $json = $response.Content | ConvertFrom-Json
    
    $Counter = $json.value.count
    
    Foreach($Item in $json)
    {
     
     $json1 = $json.Value | Where-Object{$_.properties.type -eq "AzureDataLakeStore"}
    
     $linkedservicename = $json1.name
    
     $OutFilePath = "C:\Users\BTA056\Desktop\adf\adfname\$linkedservicename.txt"
    
     $json1 | ConvertTo-Json | Out-file  -filePath $OutFilePath
    
     $Counter--
    }

    Sunday, May 24, 2020 10:25 AM
  • I got the solution by keeping another Foreach loop function.

    Thank you so much for your help.

    Sunday, May 24, 2020 1:05 PM

All replies

  • The output is JSON which is an object.  Just reference the property you want.

    # example
    $json = @'
    {"value":[{"id":"/subscriptions/xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.DataFactory/factories/xxxx/linkedservices/xxxx
    xxx","name":"xxxxxx","type":"Microsoft.DataFactory/factories/linkedservices","properties":{"annotations":[],"type":"AzureKeyVault","typeProperties":{"baseUrl":"https://ak
    xxxxxx.vault.azure.net/"}},"etag":"xxxxx"},{"id":"/subscriptions/xxxxx/resourceGroups/xxxxx/providers/
    Microsoft.DataFactory/factories/xxxxx/linkedservices/xxxx","name":"xxxxx","type":"Microsoft.DataFactory/factories/linkedservices","properties":{"an
    notations":[],"type":"AzureDataLakeStore","typeProperties":{"dataLakeStoreUri":"adl://xxxxx.azuredatalakestore.net/","servicePrincipalId":"xxxxxxx
    ","servicePrincipalKey":{"type":"AzureKeyVaultSecret","store":{"referenceName":"xxxxx","type":"LinkedServiceReference"},"secretName":"data-client-secret"},"tenant
    ":"xxxxx","subscriptionId":"xxxxx","resourceGroupName":"xxxxxx"}},"etag":"xxxxxx
    "}]}
    '@ | ConvertFrom-Json
    
    # from content
    $response.Content | ConvertFrom-Json
    

    The following shows how to extract the properties

    PS C:\scripts> $json.Value[0]
    
    id         : /subscriptions/xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.DataFactory/factories/xxxx/linkedservices/xxxx
                 xxx
    name       : xxxxxx
    type       : Microsoft.DataFactory/factories/linkedservices
    properties : @{annotations=System.Object[]; type=AzureKeyVault; typeProperties=}
    etag       : xxxxx
    
    
    
    PS C:\scripts> $json.Value[0].type
    Microsoft.DataFactory/factories/linkedservices
    PS C:\scripts> $json.Value[1].type
    Microsoft.DataFactory/factories/linkedservices
    PS C:\scripts>


    \_(ツ)_/

    Friday, May 22, 2020 3:24 PM
  • thank you so much sir helping me on this. It is working as you mentioned.

    I need another help:

    I am getting multiple names for one type, example: "type":"AzureDataLakeStore" having three values I want to export them into individual file instead of single file.

    Below code is writing all final result into single file but I want to export result into individual files:

    $azureRmProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile
    $currentAzureContext = Get-AzContext
    $profileClient = New-Object Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient($azureRmProfile)
    $token = $profileClient.AcquireAccessToken($currentAzureContext.Subscription.TenantId)
    $token1 = $token.AccessToken
    $uri = "https://management.azure.com/subscriptions/******/resourceGroups/******/providers/Microsoft.DataFactory/factories/*****/linkedservices?api-version=2018-06-01"
    $method = "GET"
    $contentType = "application/json"
    $pat = "Bearer $token1"
    
    $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers @{Authorization = $pat } -UseBasicParsing
    
    $json = $response.Content | ConvertFrom-Json
    
    $Counter = $json.value.count
    
    Foreach($Item in $json)
    {
     
     $json1 = $json.Value | Where-Object{$_.properties.type -eq "AzureDataLakeStore"}
    
     $linkedservicename = $json1.name
    
     $OutFilePath = "C:\Users\BTA056\Desktop\adf\adfname\$linkedservicename.txt"
    
     $json1 | ConvertTo-Json | Out-file  -filePath $OutFilePath
    
     $Counter--
    }

    Sunday, May 24, 2020 10:25 AM
  • I got the solution by keeping another Foreach loop function.

    Thank you so much for your help.

    Sunday, May 24, 2020 1:05 PM