Answered by:
How to get Specific Linked Service Type In Azure Data Factory using Power Shell

Question
-
Hi,
I am executing below power shell script and getting list of all linked services from azure data factory. I want to extract only specific linked service type from response.
Below is shell script:
$azureRmProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile $currentAzureContext = Get-AzContext $profileClient = New-Object Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient($azureRmProfile) $token = $profileClient.AcquireAccessToken($currentAzureContext.Subscription.TenantId) $token1 = $token.AccessToken $uri = "GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/linkedservices?api-version=2018-06-01" $method = "GET" $contentType = "application/json" $pat = "Bearer $token1" $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers @{Authorization = $pat } -UseBasicParsing write-output $response.Content
Example:
In my data factory there are two linked services 1) "type":"AzureDataLakeStore" 2) "type":"AzureKeyVault" and I want to list/extract only AzureDatalakeStore type.
Below is response from power shell script:
{"value":[{"id":"/subscriptions/xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.DataFactory/factories/xxxx/linkedservices/xxxx
xxx","name":"xxxxxx","type":"Microsoft.DataFactory/factories/linkedservices","properties":{"annotations":[],"type":"AzureKeyVault","typeProperties":{"baseUrl":"https://ak
xxxxxx.vault.azure.net/"}},"etag":"xxxxx"},{"id":"/subscriptions/xxxxx/resourceGroups/xxxxx/providers/
Microsoft.DataFactory/factories/xxxxx/linkedservices/xxxx","name":"xxxxx","type":"Microsoft.DataFactory/factories/linkedservices","properties":{"an
notations":[],"type":"AzureDataLakeStore","typeProperties":{"dataLakeStoreUri":"adl://xxxxx.azuredatalakestore.net/","servicePrincipalId":"xxxxxxx
","servicePrincipalKey":{"type":"AzureKeyVaultSecret","store":{"referenceName":"xxxxx","type":"LinkedServiceReference"},"secretName":"data-client-secret"},"tenant
":"xxxxx","subscriptionId":"xxxxx","resourceGroupName":"xxxxxx"}},"etag":"xxxxxx
"}]}Thanks,
Brahma
Friday, May 22, 2020 1:48 PM
Answers
-
The output is JSON which is an object. Just reference the property you want.
# example $json = @' {"value":[{"id":"/subscriptions/xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.DataFactory/factories/xxxx/linkedservices/xxxx xxx","name":"xxxxxx","type":"Microsoft.DataFactory/factories/linkedservices","properties":{"annotations":[],"type":"AzureKeyVault","typeProperties":{"baseUrl":"https://ak xxxxxx.vault.azure.net/"}},"etag":"xxxxx"},{"id":"/subscriptions/xxxxx/resourceGroups/xxxxx/providers/ Microsoft.DataFactory/factories/xxxxx/linkedservices/xxxx","name":"xxxxx","type":"Microsoft.DataFactory/factories/linkedservices","properties":{"an notations":[],"type":"AzureDataLakeStore","typeProperties":{"dataLakeStoreUri":"adl://xxxxx.azuredatalakestore.net/","servicePrincipalId":"xxxxxxx ","servicePrincipalKey":{"type":"AzureKeyVaultSecret","store":{"referenceName":"xxxxx","type":"LinkedServiceReference"},"secretName":"data-client-secret"},"tenant ":"xxxxx","subscriptionId":"xxxxx","resourceGroupName":"xxxxxx"}},"etag":"xxxxxx "}]} '@ | ConvertFrom-Json # from content $response.Content | ConvertFrom-Json
The following shows how to extract the properties
PS C:\scripts> $json.Value[0] id : /subscriptions/xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.DataFactory/factories/xxxx/linkedservices/xxxx xxx name : xxxxxx type : Microsoft.DataFactory/factories/linkedservices properties : @{annotations=System.Object[]; type=AzureKeyVault; typeProperties=} etag : xxxxx PS C:\scripts> $json.Value[0].type Microsoft.DataFactory/factories/linkedservices PS C:\scripts> $json.Value[1].type Microsoft.DataFactory/factories/linkedservices PS C:\scripts>
\_(ツ)_/
- Marked as answer by Brahmaiah Takkellapati Sunday, May 24, 2020 10:19 AM
Friday, May 22, 2020 3:24 PM -
thank you so much sir helping me on this. It is working as you mentioned.
I need another help:
I am getting multiple names for one type, example: "type":"AzureDataLakeStore" having three values I want to export them into individual file instead of single file.
Below code is writing all final result into single file but I want to export result into individual files:
$azureRmProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile $currentAzureContext = Get-AzContext $profileClient = New-Object Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient($azureRmProfile) $token = $profileClient.AcquireAccessToken($currentAzureContext.Subscription.TenantId) $token1 = $token.AccessToken $uri = "https://management.azure.com/subscriptions/******/resourceGroups/******/providers/Microsoft.DataFactory/factories/*****/linkedservices?api-version=2018-06-01" $method = "GET" $contentType = "application/json" $pat = "Bearer $token1" $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers @{Authorization = $pat } -UseBasicParsing $json = $response.Content | ConvertFrom-Json $Counter = $json.value.count Foreach($Item in $json) { $json1 = $json.Value | Where-Object{$_.properties.type -eq "AzureDataLakeStore"} $linkedservicename = $json1.name $OutFilePath = "C:\Users\BTA056\Desktop\adf\adfname\$linkedservicename.txt" $json1 | ConvertTo-Json | Out-file -filePath $OutFilePath $Counter-- }
- Marked as answer by Brahmaiah Takkellapati Sunday, May 24, 2020 1:05 PM
Sunday, May 24, 2020 10:25 AM -
I got the solution by keeping another Foreach loop function.
Thank you so much for your help.
- Marked as answer by Brahmaiah Takkellapati Sunday, May 24, 2020 1:05 PM
Sunday, May 24, 2020 1:05 PM
All replies
-
The output is JSON which is an object. Just reference the property you want.
# example $json = @' {"value":[{"id":"/subscriptions/xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.DataFactory/factories/xxxx/linkedservices/xxxx xxx","name":"xxxxxx","type":"Microsoft.DataFactory/factories/linkedservices","properties":{"annotations":[],"type":"AzureKeyVault","typeProperties":{"baseUrl":"https://ak xxxxxx.vault.azure.net/"}},"etag":"xxxxx"},{"id":"/subscriptions/xxxxx/resourceGroups/xxxxx/providers/ Microsoft.DataFactory/factories/xxxxx/linkedservices/xxxx","name":"xxxxx","type":"Microsoft.DataFactory/factories/linkedservices","properties":{"an notations":[],"type":"AzureDataLakeStore","typeProperties":{"dataLakeStoreUri":"adl://xxxxx.azuredatalakestore.net/","servicePrincipalId":"xxxxxxx ","servicePrincipalKey":{"type":"AzureKeyVaultSecret","store":{"referenceName":"xxxxx","type":"LinkedServiceReference"},"secretName":"data-client-secret"},"tenant ":"xxxxx","subscriptionId":"xxxxx","resourceGroupName":"xxxxxx"}},"etag":"xxxxxx "}]} '@ | ConvertFrom-Json # from content $response.Content | ConvertFrom-Json
The following shows how to extract the properties
PS C:\scripts> $json.Value[0] id : /subscriptions/xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.DataFactory/factories/xxxx/linkedservices/xxxx xxx name : xxxxxx type : Microsoft.DataFactory/factories/linkedservices properties : @{annotations=System.Object[]; type=AzureKeyVault; typeProperties=} etag : xxxxx PS C:\scripts> $json.Value[0].type Microsoft.DataFactory/factories/linkedservices PS C:\scripts> $json.Value[1].type Microsoft.DataFactory/factories/linkedservices PS C:\scripts>
\_(ツ)_/
- Marked as answer by Brahmaiah Takkellapati Sunday, May 24, 2020 10:19 AM
Friday, May 22, 2020 3:24 PM -
thank you so much sir helping me on this. It is working as you mentioned.
I need another help:
I am getting multiple names for one type, example: "type":"AzureDataLakeStore" having three values I want to export them into individual file instead of single file.
Below code is writing all final result into single file but I want to export result into individual files:
$azureRmProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile $currentAzureContext = Get-AzContext $profileClient = New-Object Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient($azureRmProfile) $token = $profileClient.AcquireAccessToken($currentAzureContext.Subscription.TenantId) $token1 = $token.AccessToken $uri = "https://management.azure.com/subscriptions/******/resourceGroups/******/providers/Microsoft.DataFactory/factories/*****/linkedservices?api-version=2018-06-01" $method = "GET" $contentType = "application/json" $pat = "Bearer $token1" $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers @{Authorization = $pat } -UseBasicParsing $json = $response.Content | ConvertFrom-Json $Counter = $json.value.count Foreach($Item in $json) { $json1 = $json.Value | Where-Object{$_.properties.type -eq "AzureDataLakeStore"} $linkedservicename = $json1.name $OutFilePath = "C:\Users\BTA056\Desktop\adf\adfname\$linkedservicename.txt" $json1 | ConvertTo-Json | Out-file -filePath $OutFilePath $Counter-- }
- Marked as answer by Brahmaiah Takkellapati Sunday, May 24, 2020 1:05 PM
Sunday, May 24, 2020 10:25 AM -
I got the solution by keeping another Foreach loop function.
Thank you so much for your help.
- Marked as answer by Brahmaiah Takkellapati Sunday, May 24, 2020 1:05 PM
Sunday, May 24, 2020 1:05 PM