none
Remove member from json file using Powers hell RRS feed

  • Question

  • Hi,

    I am new to power shell .

    I have requirement where I want to delete the member from json file  and add with new value.

    my json file looks like as below

    I am using below command to delete  both highlighted member, but it is not removing .

    $clusterName = "trest"
    $BearerToken = "testtoken"
    $WorkspaceID = 'TstWorkspaceID'
    $WorkspaceKey = 'TstWorkspaceKey'

    $configFilePath = "C:\Users\source\repos\CloudHostedPipelines\PowerShell\cluster.spark.json"
    $clusterConfig = Get-Content $configFilePath | ConvertFrom-Json
    #Change here
    $sparkConfig = $clusterConfig | Select-Object -Property spark_env_vars | ConvertTo-Json
    echo "Before"
    $sparkConfig
    echo "After"
    $sparkConfig.PSObject.Properties.Remove("LOG_ANALYTICS_WORKSPACE_KEY")
    $sparkConfig.PSObject.Properties.Remove("LOG_ANALYTICS_WORKSPACE_ID")
    $sparkConfig.PsObject.Members.Remove('spark_env_vars')
    $sparkConfig

    Pleas let me know if some one have idea about it.

    Thanks in advance

     

    Thankx & regards, Vipin jha MCP

    Tuesday, June 18, 2019 9:30 AM

Answers

  • $json = @'
    {
        
        "num_workers": 4,
        "cluster_name": "dev100",
        "spark_version": "5.4.x-scala2.11",
        "spark_conf": {
            "spark.databricks.delta.retentionDurationCheck.enabled": "false",
            "spark.extraListeners": "com.databricks.backend.daemon.driver.DBCEventLoggingListener,org.apache.spark.listeners.UnifiedSparkListener"
            
        },
        "node_type_id": "Standard_DS5_v2",
        "driver_node_type_id": "Standard_DS5_v2",
        "ssh_public_keys": [],
        "custom_tags": {
        },
        "cluster_log_conf": {
            "dbfs": {
                "destination": "dbfs:/cluster-logs/dev100"
            }
        },
        "spark_env_vars": {
            "PYSPARK_PYTHON": "/databricks/python3/bin/python3",
            "LOG_ANALYTICS_WORKSPACE_KEY": "testkey",
            "LOG_ANALYTICS_WORKSPACE_ID": "trestid"
        },
        "autotermination_minutes": 60,
        "enable_elastic_disk": true,
                                    "cluster_source": "UI",
                                    "init_scripts": [
        {
            "dbfs": {
                "destination": "dbfs:/databricks/monitoring-staging/listeners.sh"
            }
        }
        ]
    }
    '@ | ConvertFrom-Json
    $json.spark_env_vars.PsObject.Properties.Remove('LOG_ANALYTICS_WORKSPACE_ID')
    $json | ConvertTo-Json
    


    \_(ツ)_/

    • Marked as answer by Vipin jha Wednesday, June 19, 2019 2:13 PM
    Wednesday, June 19, 2019 1:37 PM
  • To change the value there is no need to remove the property.  Just assign a new vlue.

    $json.spark_env_vars.LOG_ANALYTICS_WORKSPACE_KEY = 'newtestkey'

    That is all you have to do.


    \_(ツ)_/

    • Marked as answer by Vipin jha Wednesday, June 19, 2019 2:13 PM
    Wednesday, June 19, 2019 2:00 PM
  • You cannot remove the property and then assign it.  Don't remove the property.  Just assign a new value.


    \_(ツ)_/

    • Marked as answer by Vipin jha Wednesday, June 19, 2019 2:12 PM
    Wednesday, June 19, 2019 2:06 PM

All replies

    • Edited by jrv Tuesday, June 18, 2019 9:41 AM
    Tuesday, June 18, 2019 9:41 AM
  • Sorry about that ,

    screen shot was for my json file and I pasted the code for Powershell which I am using to remove highlighted member.


    Thankx & regards, Vipin jha MCP

    Tuesday, June 18, 2019 9:47 AM
  • Tuesday, June 18, 2019 9:58 AM
  • I want to Remove "spark_env_vars" property from  json object using powershell.

    Currently I am using below powershell script to remove the property , but it is not removing

    $clusterName = "devstandardspark240"
    $BearerToken = "dapi2057abbef5ff1c4b32efb4773de02250"
    $WorkspaceID = 'TstWorkspaceID'
    $WorkspaceKey = 'TstWorkspaceKey'

    $configFilePath = "C:\Users\vipjha\source\PowerShell\cluster.json"
    $clusterConfig = Get-Content $configFilePath | ConvertFrom-Json

    ########
    #Change here

    $sparkConfig = $clusterConfig | Select-Object -Property 'spark_env_vars' | ConvertTo-Json
    echo "Before"
    $sparkConfig

    echo "After"
    $sparkConfig.PSObject.Properties.Remove("spark_env_vars")
    $sparkConfig.PSObject.Properties.Remove('LOG_ANALYTICS_WORKSPACE_ID')
    #$sparkConfig

    $sparkConfig | Add-Member -NotePropertyName 'LOG_ANALYTICS_WORKSPACE_KEY' -NotePropertyValue $WorkspaceKey
    $sparkConfig | Add-Member -NotePropertyName 'LOG_ANALYTICS_WORKSPACE_ID' -NotePropertyValue $WorkspaceID
    $sparkConfig

    screen shot of my json file is like below


    Thankx & regards, Vipin jha MCP

    Tuesday, June 18, 2019 2:53 PM
  • Please do NOT post code or data as a picture.  Post it using the code posting tool.  We are not going to type all of that in from a picture.


    \_(ツ)_/

    Tuesday, June 18, 2019 4:26 PM
  • Hi Below is my json file 

      

    {

      "num_workers": 4,
      "cluster_name": "dev100",
      "spark_version": "5.4.x-scala2.11",
      "spark_conf": {
        "spark.databricks.delta.retentionDurationCheck.enabled": "false",
        "spark.extraListeners": "com.databricks.backend.daemon.driver.DBCEventLoggingListener,org.apache.spark.listeners.UnifiedSparkListener"

      },
      "node_type_id": "Standard_DS5_v2",
      "driver_node_type_id": "Standard_DS5_v2",
      "ssh_public_keys": [],
      "custom_tags": {},
      "cluster_log_conf": {
        "dbfs": {
          "destination": "dbfs:/cluster-logs/dev100"
        }
      },
      "spark_env_vars": {
        "PYSPARK_PYTHON": "/databricks/python3/bin/python3",
        "LOG_ANALYTICS_WORKSPACE_KEY": "testkey",
        "LOG_ANALYTICS_WORKSPACE_ID": "trestid"
      },
      "autotermination_minutes": 60,
      "enable_elastic_disk": true,
      "cluster_source": "UI",
      "init_scripts": [
        {
          "dbfs": {
            "destination": "dbfs:/databricks/monitoring-staging/listeners.sh"
          }
        }
      ]
    }

    From above json file I want to Remove highlighted element using Poweshel and get back same with new value

    Currently I am using below poweshel code to remove the element but its not working for me

    From above json file I want to remove 

    $clusterName = "dev100"
    $BearerToken = "dsdfsdfasdfdf"
    $WorkspaceID = 'TstWorkspaceID'
    $WorkspaceKey = 'TstWorkspaceKey'

    $configFilePath = "C:\Users\vipjha\source\PowerShell\cluster.json"
    $clusterConfig = Get-Content $configFilePath | ConvertFrom-Json

    ########
    #Change here

    $sparkConfig = $clusterConfig | Select-Object -Property 'spark_env_vars' | ConvertTo-Json
    echo "Before"
    $sparkConfig

    echo "After"
    $sparkConfig.PSObject.Properties.Remove("spark_env_vars")
    $sparkConfig.PSObject.Properties.Remove('LOG_ANALYTICS_WORKSPACE_ID')
    #$sparkConfig

    $sparkConfig | Add-Member -NotePropertyName 'LOG_ANALYTICS_WORKSPACE_KEY' -NotePropertyValue $WorkspaceKey
    $sparkConfig | Add-Member -NotePropertyName 'LOG_ANALYTICS_WORKSPACE_ID' -NotePropertyValue $WorkspaceID
    $sparkConfig


    Thankx & regards, Vipin jha MCP

    Wednesday, June 19, 2019 1:24 PM
  • Why is it so hard for new users to understand how to post code in a forum.  The forum supplies the tool but you newbies don't seem to know how to use it.

    Please edit your post and post code and Json using the tool provided.  You code is hard to read and cannot be correctly copied.


    \_(ツ)_/

    Wednesday, June 19, 2019 1:28 PM
  • $json = @'
    {
        
        "num_workers": 4,
        "cluster_name": "dev100",
        "spark_version": "5.4.x-scala2.11",
        "spark_conf": {
            "spark.databricks.delta.retentionDurationCheck.enabled": "false",
            "spark.extraListeners": "com.databricks.backend.daemon.driver.DBCEventLoggingListener,org.apache.spark.listeners.UnifiedSparkListener"
            
        },
        "node_type_id": "Standard_DS5_v2",
        "driver_node_type_id": "Standard_DS5_v2",
        "ssh_public_keys": [],
        "custom_tags": {
        },
        "cluster_log_conf": {
            "dbfs": {
                "destination": "dbfs:/cluster-logs/dev100"
            }
        },
        "spark_env_vars": {
            "PYSPARK_PYTHON": "/databricks/python3/bin/python3",
            "LOG_ANALYTICS_WORKSPACE_KEY": "testkey",
            "LOG_ANALYTICS_WORKSPACE_ID": "trestid"
        },
        "autotermination_minutes": 60,
        "enable_elastic_disk": true,
                                    "cluster_source": "UI",
                                    "init_scripts": [
        {
            "dbfs": {
                "destination": "dbfs:/databricks/monitoring-staging/listeners.sh"
            }
        }
        ]
    }
    '@ | ConvertFrom-Json
    $json.spark_env_vars.PsObject.Properties.Remove('LOG_ANALYTICS_WORKSPACE_ID')
    $json | ConvertTo-Json
    


    \_(ツ)_/

    • Marked as answer by Vipin jha Wednesday, June 19, 2019 2:13 PM
    Wednesday, June 19, 2019 1:37 PM
  • Thanks you so much for your help sir. One more thing if you have few time.

    My purpose was to Remove and add again with new value .

    Like below

    Before delete

    "spark_env_vars": { "PYSPARK_PYTHON": "/databricks/python3/bin/python3", "LOG_ANALYTICS_WORKSPACE_KEY": "testkey", "LOG_ANALYTICS_WORKSPACE_ID": "trestid" },

    After delete need to add same element with new value like below , can we pass these value from parameter like bewlo

    $WorkspaceID = 'TstWorkspaceID'
    $WorkspaceKey = 'TstWorkspaceKey'

      "spark_env_vars": {
            "PYSPARK_PYTHON": "/databricks/python3/bin/python3",
            "LOG_ANALYTICS_WORKSPACE_KEY": "newtestkey",
            "LOG_ANALYTICS_WORKSPACE_ID": "newtrestid"
        },

    Thank you so much


    Thankx & regards, Vipin jha MCP

    Wednesday, June 19, 2019 1:54 PM
  • To change the value there is no need to remove the property.  Just assign a new vlue.

    $json.spark_env_vars.LOG_ANALYTICS_WORKSPACE_KEY = 'newtestkey'

    That is all you have to do.


    \_(ツ)_/

    • Marked as answer by Vipin jha Wednesday, June 19, 2019 2:13 PM
    Wednesday, June 19, 2019 2:00 PM
  • Hi Sir, Thanks for your quick post .

    But I am getting below error

    Exception setting "LOG_ANALYTICS_WORKSPACE_KEY": "The property 'LOG_ANALYTICS_WORKSPACE_KEY' cannot be found on this object. Verify that the property exists and can be 
    set."
    At line:30 char:1
    + $clusterConfig.spark_env_vars.LOG_ANALYTICS_WORKSPACE_KEY = $Workspac ...
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : NotSpecified: (:) [], SetValueInvocationException
        + FullyQualifiedErrorId : ExceptionWhenSetting


    Thankx & regards, Vipin jha MCP

    • Marked as answer by Vipin jha Wednesday, June 19, 2019 2:13 PM
    • Unmarked as answer by Vipin jha Wednesday, June 19, 2019 2:13 PM
    Wednesday, June 19, 2019 2:05 PM
  • You cannot remove the property and then assign it.  Don't remove the property.  Just assign a new value.


    \_(ツ)_/

    • Marked as answer by Vipin jha Wednesday, June 19, 2019 2:12 PM
    Wednesday, June 19, 2019 2:06 PM
  • Thanks its working now for me, Thank you so much for your quick response.

    I was looking this for last 2 days .

    Thanks again.


    Thankx & regards, Vipin jha MCP

    Wednesday, June 19, 2019 2:13 PM