Multiplication of running SCO Runbooks when the HA SQL Cluster changes RRS feed

  • Question

  • I have a System Center Orchestrator 2016 7.3.327.0 environment that runs on a Windows Server 2016 standard.
    The orchestrator database is a SQL Server High Availability Cluster (13.0.5149.0) running on two Windows 2016 standard machines.

    If the primary database is moved from one server to the other, all runbooks currently running in Orchestrator are multiplied. How often the multiplication happens varies from case to case.

    I don't think it is related to orphaned runbooks. (see http://www.techguy.at/check-and-remove-orphaned-runbooks/).  In the case of orphand runbooks, the runbook appears to run, but does not. In my case the runbooks are multiplied, but should not. And they are running till they are finished.

    Example for the multiplication:
    We have a Runbook O, which calls a Runbook O1. This in turn calls a runbook O1a.
    So when the SQL Cluster changes all 3 levels are multiplied at the same time.
    That means there are 5 x O, 5x O1 and 5x O1a - which means that the 5 O call up 5x O1 again, so you have 10X O1. Which again calls O1a with the result that you have 15 O1a.

    So this is an example of such an occurance:

       Any ideas?

    Tuesday, March 17, 2020 3:45 PM

All replies

  • So today we reproduced this behavior. So we have a way to reproduce that.

    We also tried to stop SCO from working before. This did not help, the runbooks multiplied in any case.

    Friday, May 22, 2020 11:45 AM
  • This is not orphaned Runbooks as described here: http://www.techguy.at/check-and-remove-orphaned-runbooks/

    Orphaned runbooks are runbooks that do not run any longer. But these runbooks run

    An (easy) Example:
    So let's say I have a runbook calling a second runbook creating a VM in VMM.
    The standard behavior is, that the runbook is called once and creates one VM. This always works, unless the primary server of the SQL cluster changes.
    When the primary server changes, the first runbook calls the second runbook multiple times (5-8) and so the second runbook runs multiple times, resulting in creating multiple VMs in the VMM.
    Each time we have this issue of runbooks going wild there was the change of the primary server of the SQL cluster.
    I don't have any idea where to look at for this behavior. Any hint would be useful.

    Thursday, June 18, 2020 12:53 PM