locked
SharePoint Retention Policy Workflow Association breaking in libraries RRS feed

  • Question

  • Hi, I'm implementing the following solution in SharePoint 2016 (on-prem).

    • Document based Content Type called "Contract" (for argument's sake)
    • The Contract Content Type contains a few date based columns and the customer wants to receive notification emails when these dates are reached
    • I've created a Retention Policy on the Content Type and then added various stages to that policy, each of which trigger a (SharePoint Designer created) custom workflow to send out the emails
    • The Workflows are created using the SharePoint 2010 model, as required to work with Retention policies, and are Globally Published
    • The Content Type is then leveraged in multiple subsites and libraries throughout that single site collection

    All of that is working (after a lot of headaches) but there is one issue that remains which I think may be a bug (or otherwise just a massive annoyance).

    If we update one of the Workflows in SharePoint Designer, and then Publish, the following happens:

    • The new version of the Workflow is automatically applied to the Content Type. The previous version is still visible but is archived, and renamed to include the date it was created. This is fine and expected.
    • The Retention Policy continues to use the previous version of the Workflow. This is fine. We just need to remember to update the Retention Policy when we update the Workflow.
    • However, when we change the Retention Policy to use the new version of the Workflow, this breaks the Retention Policy everywhere it is being used. If you click on "Compliance Details" for any document based on the "Contract" content type, you will see the message "Invalid retention stage defined". This causes the workflow not to be triggered - i.e. broken.
    • The only way I've found to fix this is to go to the Workflow Settings for the Site Content Type and click the option to "Update all related content types with these workflow association settings.".
    • The trouble is this then creates duplicate Workflows with the same name at the child site/library level. The only solution I've found to this is to delete all the relevant workflow associations in each library before pushing the changes down. 
    • Given that these are now seen as new Workflows to the library, a new Workflow Status column gets added to the default View.

    This all means that every time we update the Workflow in SharePoint Designer, in order for that change to take effect in the Retention Policy, we have to delete all of the previous Workflows, everywhere they are being used, before pushing down from the top-level site again. And that will leave us with cluttered "Views" with multiple same-named Workflow status columns. This is not a nice solution.

    What I believe to be a bug is that when you modify the Retention Policy to use the new updated workflow, this should not result in the "Invalid retention stage defined" issue and thus break the policy everywhere it is being used.
     
    If I've got something wrong I will be very glad to hear about it. 
     
    Thanks for your help.

    Friday, September 7, 2018 12:59 PM

Answers

  • I wouldn't argue with the term "flaw".  As I said, this pattern is common in SharePoint in a number of places where they copy some value from a template instead of referencing the template in an ongoing fashion. The problem is that managing all those linkings can become a very time consuming job and that slows down the performance if everytime a change is made a lot of dependencies need to be updated.  For the vast majority once these kinds of things are put into place they don't change.  So managing the referential links would be a drain they don't need. I'm not saying it was the right choice, but there is an argument to be made in favor of doing it that way. Since fixing it would require a major redesign I don't think you'll see the behavior change anytime soon.

    Paul Stork SharePoint Server MVP
    Owner/Principal Architect: Don't Pa..Panic Consulting
    Blog: http://dontpapanic.com/blog
    Twitter: Follow @pstork
    Please remember to mark your question as 'answered' if this solves your problem.

    • Proposed as answer by Sara Fan Tuesday, September 11, 2018 12:41 AM
    • Marked as answer by j.strugnell Tuesday, September 11, 2018 9:29 AM
    Monday, September 10, 2018 7:09 PM

All replies

  • I agree its massively annoying, but it isn't really a bug.  Its just a side effect of the way that two different subsystems interact with each other.  To work the way you describe the retention policy would need to know about all the previous versions of the workflow.  The way it was designed simply isn't sophisticated enough to work at that level.

    Paul Stork SharePoint Server MVP
    Owner/Principal Architect: Don't Pa..Panic Consulting
    Blog: http://dontpapanic.com/blog
    Twitter: Follow @pstork
    Please remember to mark your question as 'answered' if this solves your problem.

    Friday, September 7, 2018 3:27 PM
  • Hi Paul,

    Thank you for responding.

    I don't have a great low-level understanding of how these features work, but the Retention Policy is defined in one place, at the root site, and it is configured to use a global workflow, defined at the root site. The retention policy only needs to know about the workflow it is configured to use, which is globally available, so why can't the lists see it?

    To make it even harder to understand, if you manually run the updated workflow from a Contract document, it has picked up any changes we publish in SharePoint Designer (without needing to push the changes down). It seems like everything is in place for my requirements to work, but it just doesn't. Grrrrrr....

    Saturday, September 8, 2018 10:03 PM
  • The list can't see it because it references the Instance of the workflow.  When you update it then it changes the instance. Publishing the workflow shuffles the names around so the new worklflow has the same name but a new identifier. Since the reference has changed the policy can't see it anymore.  The design is done to keep it simple and it won't allow for modifying the underlying workflow without going back and resetting the reference.

    Manually running the workflow works because Publishing the workflow sets the reference in the Library to the new workflow.  So starting a new instance uses the new workflow.  But workflows that are already running continue to run on the old version of the workflow that has been renamed.  Unfortunately the retention policy isn't running the old workflow so when the reference changes it can't find the new one to start up.


    Paul Stork SharePoint Server MVP
    Owner/Principal Architect: Don't Pa..Panic Consulting
    Blog: http://dontpapanic.com/blog
    Twitter: Follow @pstork
    Please remember to mark your question as 'answered' if this solves your problem.

    • Proposed as answer by Sara Fan Monday, September 10, 2018 1:35 AM
    Sunday, September 9, 2018 1:20 AM
  • Hi j.strugnell,

    If the reply is helpful to you, you could mark the reply as answer. Thanks for your understanding.

    Best regards,

    Sara Fan

    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.


    Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.

    Monday, September 10, 2018 1:35 AM
  • Hi Paul,

    Thanks for taking the time with that explanation. Perhaps "flaw" is a more appropriate word than "bug". It doesn't feel like it would be that much of a stretch to make it work how you would expect it to, but I don't know the full underlying engineering. I am surprised this issue hasn't gained more exposure over the last 8-10 years.

    Thanks again,

    James

    Monday, September 10, 2018 7:51 AM
  • It has been helpful, but I would like to hear from Microsoft as to whether there is anything they can do about it? Or are they no longer maintaining this aspect of SharePoint Server, with the expectation that people will be moving to SharePoint Online and leverage the compliance functionality there and/or MS Flow?
    Monday, September 10, 2018 7:57 AM
  • I wouldn't argue with the term "flaw".  As I said, this pattern is common in SharePoint in a number of places where they copy some value from a template instead of referencing the template in an ongoing fashion. The problem is that managing all those linkings can become a very time consuming job and that slows down the performance if everytime a change is made a lot of dependencies need to be updated.  For the vast majority once these kinds of things are put into place they don't change.  So managing the referential links would be a drain they don't need. I'm not saying it was the right choice, but there is an argument to be made in favor of doing it that way. Since fixing it would require a major redesign I don't think you'll see the behavior change anytime soon.

    Paul Stork SharePoint Server MVP
    Owner/Principal Architect: Don't Pa..Panic Consulting
    Blog: http://dontpapanic.com/blog
    Twitter: Follow @pstork
    Please remember to mark your question as 'answered' if this solves your problem.

    • Proposed as answer by Sara Fan Tuesday, September 11, 2018 12:41 AM
    • Marked as answer by j.strugnell Tuesday, September 11, 2018 9:29 AM
    Monday, September 10, 2018 7:09 PM
  • Hi j.strugnell,

    If the reply is helpful to you, you could mark the reply as answer. Thanks for your understanding.

    Best regards,

    Sara Fan

    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.


    Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.

    Tuesday, September 11, 2018 12:41 AM
  • Thank you again Paul. I'm aware there are 2 Timer Jobs that run to facilitate this functionality. The first one supposedly goes through every item and establishes if any action should be taken on that item, as part of a retention policy. That would have been a good opportunity for the code to resolve any workflow instance integrity issues. 

    It looks like I'm going to have to admit defeat on this one though. 

    Thank you again for sharing your time and expertise. 

    Tuesday, September 11, 2018 9:35 AM