none
What does "Locked" mean in PowerShell throttling error message?

    Question

  • We have an application that issues Powershell commands to Exchange Online. We had been running for several days with no problems, but now we've been getting throttling errors for days. See the example cmdlet and error response below. The error says that "Locked: true" and "LockRemaining" is 10+ days.

    Question: what is going here? what is the cause? what causes "Locked" to be true? what does "Locked" mean and how does it originate? can we control this Locked state at all? is "Locked" the cause of the throttling error? 

    PS C:\Users\Administrator> $devices = Get-ActiveSyncDevice -ResultSize 10

    This operation exceeds the throttling budget for policy part 'LocalTime', policy value '3000000',  Budget type: 'PowerShell'.  Suggested backoff time 887405440 ms.
    (PolicyDN: CN=OrganizationThrottlingPolicy88a9d22e-45a8-4b59-83e7-3c6c02f97076,CN=Global Settings, ...;
    Snapshot: Owner:    Sid~XXXXXXXXX~PowerShell~false
    BudgetType:    PowerShell
    ActiveRunspaces:    0/3\
    Balance:    -4.467027E+08/1800000/-3000000
    PowerShellCmdletsLeft:    199/200
    ExchangeCmdletsLeft:    24/25
    CmdletTimePeriod:    5
    DestructiveCmdletsLeft:    120/120
    DestructiveCmdletTimePeriod:    60
    QueueDepth:    50
    MaxRunspacesTimePeriod:    60
    RunSpacesRemaining:    5/5
    LastTimeFrameUpdate:    7/16/2013 8:22:51 PM
    LastTimeFrameUpdateDestructiveCmdlets:    7/16/2013 8:22:51 PM
    LastTimeFrameUpdateMaxRunspaces:    7/16/2013 8:22:51 PM
    Locked:    True
    LockRemaining:    10.06:30:05.4400000


    Also, please note that it appears that as we get the error, the "LockRemaining" time increases raipdly. Initially, when the error first occurred the LockRemaining time was 5 minutes. But over a period of days it has grown to be over 10 days! Apparently we're punished incremently more every time we try to connect.
    • Edited by John24671 Wednesday, July 17, 2013 1:17 AM
    Tuesday, July 16, 2013 9:37 PM

Answers

  • Hi everyone, here's the requested info as the Office365 technical team gives me:

    PowerShellMaxConcurrency                  : 3

    PowerShellMaxTenantConcurrency            : 9

    PowerShellMaxCmdletsTimePeriod            : 5

    ExchangeMaxCmdlets                        : 25

    PowerShellMaxCmdletQueueDepth             : 50

    PowerShellMaxDestructiveCmdlets           : 120

    PowerShellMaxDestructiveCmdletsTimePeriod : 60

    PowerShellMaxCmdlets                      : 200

    Although they say you should open a service request through your portal in case your tenant applies other values different than the default ones as a recommendation.

     

    Regards.



    Alberto Pascual MVP-MCSA-MCITP-MCTS-MCP-O365MS-MCC http://blogs.itpro.es/guruxp

    Monday, August 05, 2013 7:46 AM
  • Hi, good to know you found them useful (it was hard to find), here are the answers to your questions:

    1- the time period is specified in seconds not in minutes so this will allow to run 25 Max ExchangeCmdlets within 5 seconds of timespan.

    2- Same as number 1 but with non-Exchange Cmdlets.

    3- I believe these are Global values so it should be applied per Tenant Wide.

    4- Microsoft controls this but it may change as an exception to a previous petition for something related with these values.

    Regards.


    Alberto Pascual MVP-MCSA-MCITP-MCTS-MCP-O365MS-MCC http://blogs.itpro.es/guruxp

    • Marked as answer by John24671 Tuesday, August 06, 2013 5:28 PM
    Tuesday, August 06, 2013 6:45 AM

All replies

  • Hi, take a look at this thread, it may help you:

    http://social.technet.microsoft.com/Forums/windowsserver/en-US/7688e3df-54d8-497d-84f0-eb76ecd7ebc4/15006704000-ps-c-windows-powershell-runspace-to-exchange-online-budget-exceeded-error

    Regards.


    Alberto Pascual MVP-MCSA-MCITP-MCTS-MCP-O365MS-MCC http://blogs.itpro.es/guruxp

    Thursday, July 18, 2013 6:27 AM
  • That thread doesn't address my question.

    We see that Locked is true and the LockedRemaining time is continually increasing. Why does this happen and how can we get out of this state. It's not reasonable in any way to lock a user for 10+ days.

    Furthermore, how can we determine which throttling policy has been violated? And is there any transparency at all into the Exchange Online throttling state?

    Thursday, July 18, 2013 8:00 PM
  • Hi, what i wanted to show on the above link was that it may be to some kind of missing or wrong code inside your script.

    What i´m suspicious is that maybe you must add some "Remove-PSSession" at the end of the task or script you´re launching, in order to remove the recently created session. It´s always a best practice to remove the created session after you finish the tasks inside powershell as many opened sessions would lead to errors like the one you´re showing :).

    Hope it helps.


    Alberto Pascual MVP-MCSA-MCITP-MCTS-MCP-O365MS-MCC http://blogs.itpro.es/guruxp

    Friday, July 19, 2013 5:57 AM
  • Hi Alberto. No, we remove the session properly.

    According to Get-ThrottlingPolicy there are properties to throttle the number of cmdlets that can be run in a given time period (see PowerShellMaxCmdlets and PowerShellMaxCmdletsTimePeriod). If these are enabled in Exchange Online, then depending on their values, they may cause the problem. Can you tell us what the default values are for these in Exchange Online?

    Also, is there any transparency provided into the actual caused of lock out?

    Friday, July 19, 2013 6:28 PM
  • According to MSDN:

    http://msdn.microsoft.com/en-us/library/exchange/hh881884(v=exchg.140).aspx

    I believe that the defaults for Exchange Online are the same as Exchange Server 2010 or 2013 depending if you have the wave 14 or 15.

    sorry if it´s not what you´re looking for


    Alberto Pascual MVP-MCSA-MCITP-MCTS-MCP-O365MS-MCC http://blogs.itpro.es/guruxp

    Friday, July 19, 2013 8:22 PM
  • Alberto, that link describes the defaults for EWS. As I said, I'm looking for the defaults for the PowerShell throttling properties. PowerShellMaxCmdlets, PowerShellMaxCmdletsTimePeriod, etc.
    Friday, July 19, 2013 8:26 PM
  • sorry for the delay, I've been searching for the specific info you asked for but could not find any, all that I could find is that exonline is way slower than on-prem and that the throttle policy cannot be modified. I've also sent the question to the office365mvp global DL but no one has an idea of it, the only answer I got was a comparison of a script mass made in on-prem (20min) and exonline (8hours).

    I will keep searching for the info.


    Alberto Pascual MVP-MCSA-MCITP-MCTS-MCP-O365MS-MCC http://blogs.itpro.es/guruxp

    Sunday, July 21, 2013 6:39 PM
  • We are seeing the exact same issue.  I would love to see an answer for this.
    Wednesday, July 31, 2013 11:13 PM
  • I'm asking Office365 Product Team what is the default throttling Policy as in ExOl there's no way no know it. When I get something i'll let you know.


    Alberto Pascual MVP-MCSA-MCITP-MCTS-MCP-O365MS-MCC http://blogs.itpro.es/guruxp

    Thursday, August 01, 2013 6:39 AM
  • Hi everyone, here's the requested info as the Office365 technical team gives me:

    PowerShellMaxConcurrency                  : 3

    PowerShellMaxTenantConcurrency            : 9

    PowerShellMaxCmdletsTimePeriod            : 5

    ExchangeMaxCmdlets                        : 25

    PowerShellMaxCmdletQueueDepth             : 50

    PowerShellMaxDestructiveCmdlets           : 120

    PowerShellMaxDestructiveCmdletsTimePeriod : 60

    PowerShellMaxCmdlets                      : 200

    Although they say you should open a service request through your portal in case your tenant applies other values different than the default ones as a recommendation.

     

    Regards.



    Alberto Pascual MVP-MCSA-MCITP-MCTS-MCP-O365MS-MCC http://blogs.itpro.es/guruxp

    Monday, August 05, 2013 7:46 AM
  • Thanks Alberto, that's very helpful.

    I have a couple of follow-up questions on this:

    1. So does this mean you can run no more that 25 Exchange cmdlets within any 5 minute timespan (ExchangeMaxCmdlets=25, PowershellMaxCmdletsTimePeriod=5)?

    2. Does this mean you can run no more that 200 non-Exchange cmdlets within any 5 minute timespan (PowershellMaxCmdlets=200, PowershellMaxCmdletsTimePeriod=5)?

    3. Are the limits per admin account, per IP, global, or something else?

    4. The last comment sounds like some tenants may have different values. Who controls this?

    Regards.

    Monday, August 05, 2013 5:50 PM
  • Hi, good to know you found them useful (it was hard to find), here are the answers to your questions:

    1- the time period is specified in seconds not in minutes so this will allow to run 25 Max ExchangeCmdlets within 5 seconds of timespan.

    2- Same as number 1 but with non-Exchange Cmdlets.

    3- I believe these are Global values so it should be applied per Tenant Wide.

    4- Microsoft controls this but it may change as an exception to a previous petition for something related with these values.

    Regards.


    Alberto Pascual MVP-MCSA-MCITP-MCTS-MCP-O365MS-MCC http://blogs.itpro.es/guruxp

    • Marked as answer by John24671 Tuesday, August 06, 2013 5:28 PM
    Tuesday, August 06, 2013 6:45 AM
  • I have this problem too. And with "PowerShellMaxDestructiveCmdlets" throttle policy also. How does one avoid the Job Failure which seems to be my outcome? Is Start-Sleep inserted into the loop a solution?
    • Edited by dschieber Wednesday, August 14, 2013 10:07 PM
    Wednesday, August 14, 2013 10:06 PM