Delete the duplicate items in the sharepoint list programatically


  • Hi,

    I have a requirement to traverse through list using sharepoint object model and identify the duplicate items in the list and delete them

    Please suggest the logic.

    Knowledge is power.

    Friday, July 12, 2013 1:03 PM


All replies

  • Check this post from Brian Jackett for a great PowerShell script and explanation on what it is doing.  In this example he is grouping on the "Title" field to check for duplicates.  You would need to modify it for another field or look at multiple fields depending on how you want to identify dupes.

    Here is the blog post:

    Here is the code from the blog post:

    Add-PSSnapin microsoft.sharepoint.powershell 
    $web = Get-SPWeb -Identity "<URL of Site>" 
    $list = $web.Lists["DuplicatesList"] 
    $AllDuplicates = $list.Items.GetDataTable() | Group-Object title | where {$_.count -gt 1} 
    $count = 1 
    $max = $AllDuplicates.Count 
    foreach($duplicate in $AllDuplicates) 
    $ | Select-Object -Skip 1 | % {$list.GetItemById($_.ID).Delete()} 
    Write-Progress -PercentComplete ($count / $max * 100) -Activity "$count duplicates removed" -Status "In Progress" 
    <# # generate 100,000 items in a list using a random single character as the title foreach($x in 1..100000) { $item = $list.AddItem() $item["Title"] = [char](97 + (Get-Random -Maximum 25)) $item.Update() } #> 
    Check the post for a full explanation as well as some helpful links.

    Brandon Atkinson

    Friday, July 12, 2013 1:34 PM
  • Hi,

    COuld you please provide C# logic ?

    Knowledge is power.

    Friday, July 12, 2013 1:37 PM
  • Friday, July 12, 2013 1:44 PM
  • The link provided by you is my post which I didn't get the answer there so created new question :-)

    Knowledge is power.

    Friday, July 12, 2013 1:50 PM
  • Manoj,

    The sample code provided by Hemendra Agrawal in your original post was a decent starting point since he just cooked up the code in Notepad for you. All you have to do was to modify the logic to meet your requirement. Check the below code with modified logic. I have removed the "duplicate" variable assignment outside the IF to meet your requirement. I have also executed this with the data sample that you provided and the results are what it is expected. Please ignore the syntax error in the below code. Hope this helps to solve your issue.

    <OrderBy> <FieldRef Name="Name" /> <FieldRef Name="Priority" Ascending="FALSE" /> </OrderBy>
    string duplicate = null;
    SPListItemCollection items = mylist.GetItems(query);
    if(items.Count > 0)
      foreach(SPListItem item in items)
       string title = Convert.Tostring(item["Name"]);  
       if(title == duplicate)
       duplicate = title;

    Please mark as 'Propose as Answer' or 'Vote' as appropriate.

    Friday, July 12, 2013 2:17 PM
  • Hi Manoj,

    This logic really worked for you.

    i also need to loop through the list items and find the duplicate and delete.

    i stuck at the logic where we assigned duplicate=null 

    and assigning the value of title to duplicate. but it wont work.

    i have the values in title column as

    pof 1, prof 2, prof 3, abb, critical,pof 1, prof 2, prof 3, abb, critical,pof 1, prof 2, prof 3, abb, critical.......

    this logic is not working.

    please help me wit the correct one.


    Wednesday, December 25, 2013 2:54 PM
  • How do I insert this code onto Sharepoint online?
    Friday, June 09, 2017 3:32 PM