none
Capture duplicated records but I lost other columns RRS feed

  • Question

  • Hi all,

    I have this situation in my input file:

    For me a duplicated record is represented by only 3 highlighed columns.

    My target is:

    1. identity the duplicated record and move it into a flat file destination;

    2. load the non-duplucated records into an ado net destination;

    In this moment I'm using TSQL for duplication check, but we need to move this check into SSIS.

    I tried to follow this tutorial:

    https://www.youtube.com/watch?v=O4pf7m-VTSk

    but it consider the entire record as a duplication.

    When I try to use the tutorial above, I lost all other column and I can't map the input flat file columns with my ado net destination:

    Is possible realize this with SSIS?

    Thanks in advance.

    Diego

    Monday, February 11, 2019 1:44 PM

Answers

  • Hello, 

    Please check this Blog by Hans Michiels:

    DETECT DUPLICATES IN A DATA FLOW TASK (SSIS SERIES)

    You can use Script Component to count the occurrence of each row.

    And use Conditional Split to split the rows. 


    MSDN Community Support Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com

    • Marked as answer by d1390 Tuesday, February 12, 2019 10:39 AM
    Tuesday, February 12, 2019 3:29 AM

All replies

  • Hello, 

    Please check this Blog by Hans Michiels:

    DETECT DUPLICATES IN A DATA FLOW TASK (SSIS SERIES)

    You can use Script Component to count the occurrence of each row.

    And use Conditional Split to split the rows. 


    MSDN Community Support Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com

    • Marked as answer by d1390 Tuesday, February 12, 2019 10:39 AM
    Tuesday, February 12, 2019 3:29 AM
  • many thanks Yang.Z.

    Diego

    Tuesday, February 12, 2019 10:39 AM