locked
Powershell script to find duplicate elements in the array. RRS feed

  • Question

  • Hi,

    I am writing a script to find the duplicate elements in the array.

    I am writing a very simple logic.

    $a=@("1","2","3","1","2")

    for($i=0;i -lt $a;$i++)

    {

    for($j=$i+1; j -lt employees.employee.count;$j++)

    {

    if($a[i] -eq $a[j])

    {

    write-host "Duplicate element found" $a[i]

    }

    }

    Actually i am running the logic while remoting on another server.

    it is taking lot of time..

    Is there any better way to find duplicate element in the array using powershell that can improve the performance of code...

     

    Wednesday, December 1, 2010 6:51 AM

Answers

  • Hi,

     

    Try to use select –unique and compare-object:

     

    $a=@(1,2,3,1,2)

    $b=$a | select –unique

    Compare-object –referenceobject $b –differenceobject $a

     

    Best Regards

    Dale


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. ”
    • Marked as answer by Shay Levi Wednesday, December 1, 2010 10:59 AM
    Wednesday, December 1, 2010 9:40 AM
  • Alternate solution, may be faster:

    $a=@(1,2,3,1,2)
    $ht = @{}
    $a | foreach {$ht["$_"] += 1}
    $ht.keys | where {$ht["$_"] -gt 1} | foreach {write-host "Duplicate element found $_" }

     


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "
    • Marked as answer by Dale Qiao Thursday, December 2, 2010 5:40 AM
    Wednesday, December 1, 2010 12:15 PM

All replies

  • Hi,

     

    Try to use select –unique and compare-object:

     

    $a=@(1,2,3,1,2)

    $b=$a | select –unique

    Compare-object –referenceobject $b –differenceobject $a

     

    Best Regards

    Dale


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. ”
    • Marked as answer by Shay Levi Wednesday, December 1, 2010 10:59 AM
    Wednesday, December 1, 2010 9:40 AM
  • What also works is Get-Unique. The only downside is that you need to sort first.

    http://technet.microsoft.com/en-us/library/ee176859.aspx

    $a=@("1","2","3","1","2") 
    
    $b = $a | sort | Get-Unique
    
    
    Edit: oh wait, you were trying to find the duplicate objects. You didn't wanted to only have the unique records. Therefore my solution don't really works.
    Wednesday, December 1, 2010 11:48 AM
  • Alternate solution, may be faster:

    $a=@(1,2,3,1,2)
    $ht = @{}
    $a | foreach {$ht["$_"] += 1}
    $ht.keys | where {$ht["$_"] -gt 1} | foreach {write-host "Duplicate element found $_" }

     


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "
    • Marked as answer by Dale Qiao Thursday, December 2, 2010 5:40 AM
    Wednesday, December 1, 2010 12:15 PM
  • Hi Mjolinor,

    I find your solution as the best one.Many Thanks for that.

    I have one more issue actually....Array is not actually array...It is basically the records of Config xml file...Below is the structure...

    <employees>
     <employee>
                                     <id>101</id>
      <name>Jaspreet</name>
      <age>24</age>
     </employee>
     <employee>
                     <id>102</id>
      <name>Ravi</name>
      <age>26</age>
     </employee>
     <employee>
                         <id>102</id>
      <name>Sumit</name>
      <age>28</age>
     </employee>
    </employees>

    So i want to find duplicate ids..

    i am using the below logic to find duplicate

    $duplicate=0
    for([int] $i=0;$i -lt $e.employees.employee.count;$i++)
     {
        for([int] $j=$i+1;$j -lt $e.employees.employee.count;$j++)
        {
           if($e.employees.employee[$i].id  -eq $e.employees.employee[$j].id)
           {
                write-host "Duplicate item found" $e.employees.employee[$i].id
                $duplicate=1  
           }

       }

     }

        If($duplicate -eq 0)
        {
           Write-host "No Duplicate Entries Found"
        }

     

    Now i want to use above method..how it is possible...to read config file's id as an array and put the above logic of hash table.

    Friday, December 3, 2010 7:32 AM
  • It appears you're not just finding the duplicates, but also reporting which record number in the XML flie the duplicate account resides in.  Do you need to retain that functionality?  The hash table solution I provided won't do that, it will just report which ones had duplicates, but not where in the input they are. 

    I believe it can be made to work either way, but there will be some additoinal processing overhead in keeping track of the record number that may also imact the performace (although I suspect not very much).


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "
    • Proposed as answer by slowscripter Friday, June 28, 2013 9:15 AM
    Friday, December 3, 2010 3:36 PM
  • How about the following:

    >$a=1,1,1,2,3,2,3,4,1
    >($a | group | ?{$_.Count -gt 1}).Values

    Mark

    ____________________

    Mark Michaelis
    www.Intelliect.com/Mark

    Saturday, June 30, 2012 5:07 PM
  • Simple solution to list the duplicates in $e.empoyees.employee shown above:

    $e.employees | Group-Object id | ?{$_.count -gt 1} | %{$_.Group | ft}

    See the grouping info:

    $e.employees | Group-Object ID | ?{$_.count -gt 1} | ft

    Friday, February 17, 2017 3:01 PM
  • This Group-Object solution is GREAT! Love it. I was just attempting to look for some duplicate data and found this.

    I just used in another scenario in User accounts/contacts.

    I tweaked a little to get the multiple values listed under one column, along with the duplicate value and count.

    Select name,MyAttribute,etc|`   #(from AD query)

    WHERE {$_.count -gt 1}|`
    Select -unique -property count,@{Label="MyAttribute";E={$_.Name}},@{Label="DisplayName";E={$_.group.name}}   (Where .NAME is actually the AD name attribute in the grouped data)

    That gives: Count, MyAttribute, DisplayName (with multiple name values)


    Thursday, September 28, 2017 5:55 PM
  • I created this function based on your code.  Has been super helpful!

    FunctionGet-Duplicate($Array)

    {

       

    $Unique=$Array|select-Unique


       

    $Duplicates=(Compare-Object-ReferenceObject$Array-DifferenceObject$Unique|where{$_.sideIndicator -like"<="}).inputobject

       

    $UniqueDuplicates=$Duplicates|Select-Unique


       

    Foreach($Duplicatein$UniqueDuplicates)

        {

           

    [PSCustomObject]`

            @{

                Duplicate

    =$Duplicate


                Amount

    =($Array|where{$_-like$Duplicate}).count

            }

        }

    }

    Friday, March 30, 2018 5:43 PM