none
Exchange 2013 Online mode slow performance RRS feed

  • Question

  • Hello,

    We recently moved mailboxes (180 or so) from Exchange 2007 to Exchange 2013.  We are noticing that Outlook navigation seems to be very slow compared to the last server.  Changing folder views, switching from Inbox to Calendar, Deleting items, etc.  Also we used to be able to send 40MB HTML emails very quickly.  This now takes about 5 min just to open a reply. (These are tripwire reports.)  Most users are using Outlook in Online mode on the new server and were using Online mode on the old server

    Environment: Exchange 2013 SP1 on Server 2012r2 Standard (6 cores 24 gigs of RAM, VMWare VM.)  Old server is Exchange 2007 8.3.348.2 on Server 2008x64 Standard SP2 (8 cores 12 gigs of RAM. Physical.)

    I have a ticket open with MS but didn't really get anywhere after 9 hours yesterday.

    Thanks.

    Thursday, June 19, 2014 2:31 PM

Answers

  • OK We found something very very interesting about this. We found a solution that worked for us. Have a look at this other thread:

    http:// community.spiceworks.com/topic/571571-outlook-slow-after-migrating-to-exchange-2013

    Since these changes can affect the entire Workstation, I suggest you try it and read the entire post before doing it.

    TcpAckFrequency:

    http:// support.microsoft.com/kb/328890/fr

    I'm sorry the link are not valid since I'm not able to post it correctly right now but, I hope this will help many other people. Thx.

    • Marked as answer by BriGy86 Wednesday, February 18, 2015 5:07 PM
    Tuesday, September 16, 2014 5:13 PM
  • This is from the SpiceWorks link posted above. I have not tried it though.

    Looks like there is a registry key that can be done on each client or a fix for VMware. https://communities.vmware.com/message/2439033I

    I am going to try rolling the vmwaretools back to 5.1 RU1 and move a problem user to the new server.  If this works I will be marking MasterKic's reply as the answer as well as this one.

    • Marked as answer by BriGy86 Wednesday, February 18, 2015 5:06 PM
    Tuesday, December 16, 2014 3:00 PM

All replies

  • Please post if have any information.

    Philippe

    Friday, June 20, 2014 10:49 AM
  •  My ticket is still open with Microsoft.  After about 15.5 hours with Microsoft we still don't know why Outlook using Online mode is faster to connect to Exchange 2007 than it is to connect to Exchange 2013.
    Friday, June 20, 2014 2:45 PM
  • The first thing that popped into my head was you changed from physical to a VM. Are you sure your VM Cluster is not over provisioned? I see you have 6 Cores assigned to the VM do you know about CPU ready time? Also RAM over provisioning can cause issues as similar to CPU ready. Perhaps its also disk/SAN(if you have one) latency as well.

    RAM Bolloning:  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003470

    CPU ready: http://www.sqlskills.com/blogs/jonathan/cpu-ready-time-in-vmware-and-how-to-interpret-its-real-meaning/

    Hope this helps!

     


    Friday, June 20, 2014 3:06 PM
  • RAM is thin provisioned but 16 gigs of it comes up as private.  57% of 131 gigs of RAM is used on the host.  17% of the CPU is used on the host.  So I don't think it's an over provision issue.
    Friday, June 20, 2014 9:24 PM
  • 1. Ensure you have sized the guest correctly:

    http://blogs.technet.com/b/exchange/archive/2013/05/14/released-exchange-2013-server-role-requirements-calculator.aspx

    2. Verify that the BIOS on the vmhost is set to high performance

    http://blogs.vmware.com/vsphere/2014/01/bios-power-policies-affect-performance.html


    Twitter!: Please Note: My Posts are provided “AS IS” without warranty of any kind, either expressed or implied.

    Saturday, June 21, 2014 1:40 AM
    Moderator
  • As Andy said- size your Exchange using Role Req. Calc.

    On top of that i will make sure that you have not enabled the hyperthreading , if so then you need to disabled.

    and also

    guest virtual machine share a fixed number of logical processors in the physical system. Exchange supports a virtual processor-to-logical processor ratio no greater than 2:1, although we recommend a ratio of 1:1.

    http://technet.microsoft.com/en-us/library/jj619301%28v=exchg.150%29.aspx#BKMK_Prereq


    Where Technology Meets Talent

    Saturday, June 21, 2014 3:37 AM
  • As Andy said- size your Exchange using Role Req. Calc.

    On top of that i will make sure that you have not enabled the hyperthreading , if so then you need to disabled.

    and also

    guest virtual machine share a fixed number of logical processors in the physical system. Exchange supports a virtual processor-to-logical processor ratio no greater than 2:1, although we recommend a ratio of 1:1.

    http://technet.microsoft.com/en-us/library/jj619301%28v=exchg.150%29.aspx#BKMK_Prereq


    Where Technology Meets Talent


    Its ok to hyperthread with virtual. You just don't want to with physical.

    Twitter!: Please Note: My Posts are provided “AS IS” without warranty of any kind, either expressed or implied.

    Saturday, June 21, 2014 3:06 PM
    Moderator
  • Still no luck.  And I've been on the phone with MS each day.

    The VM has been scaled down to 4 CPU's and 18 gigs of RAM.  It is currently only using 13 of the 18 gigs.  I also changed the CPU's to 4 sockets instead of 2 sockets with 2 cores.  We also set the memory reservation to 18 gigs.

    The vCPU allocation is much higher than 2:1, but VMWare handles CPU loads different than Hyper-V.  Currently it is at less than half capacity.  It's using 7,602 MHz out of 8X2.599 GHz

    Wednesday, June 25, 2014 7:17 PM
  • how many mailboxes and what is the avg size of the mailbox?

    do you see in the task manager which application is consuming the resources?


    Where Technology Meets Talent

    Wednesday, June 25, 2014 8:49 PM
  • Still no luck.  And I've been on the phone with MS each day.

    The VM has been scaled down to 4 CPU's and 18 gigs of RAM.  It is currently only using 13 of the 18 gigs.  I also changed the CPU's to 4 sockets instead of 2 sockets with 2 cores.  We also set the memory reservation to 18 gigs.

    The vCPU allocation is much higher than 2:1, but VMWare handles CPU loads different than Hyper-V.  Currently it is at less than half capacity.  It's using 7,602 MHz out of 8X2.599 GHz


    Ok, regardless, the vCPU ratio should not be higher than 2:1. Have you asked VMware about this?

    Twitter!: Please Note: My Posts are provided “AS IS” without warranty of any kind, either expressed or implied.

    Thursday, June 26, 2014 12:17 AM
    Moderator
  • Microsoft.Exchange.Store.Worker is consuming the most RAM at 1.2 and 1.4 gigs (there are 2 of these processes.)

    Mailbox count is 210 and the average size is probably 350 MB.

    What is the reason for the vCPU ratio if systems like VMware do not segregate CPU workload?

    Thursday, June 26, 2014 3:49 PM
  • Its about what is supported and recommended:

    http://technet.microsoft.com/en-us/library/jj619301(v=exchg.150).aspx

    http://www.vmware.com/files/pdf/Exchange_2013_on_VMware_Best_Practices_Guide.pdf

    I would also ensure the host power options are set to "High Performance"


    Twitter!: Please Note: My Posts are provided “AS IS” without warranty of any kind, either expressed or implied.

    Thursday, June 26, 2014 4:49 PM
    Moderator
  • I think disk usage is causing performance issues. Did you check Disk Queue length and Average Disk space usage on Exchange 2007 and Exchange 2013?

    What is the storage that you use on VMware? Try with sharing file on Exchange 2007 and Exchange 2013, and see how much times it takes to copy to client machine.

    I hope that it leads investigation to right direction.


    - Sarvesh Goel - Enterprise Messaging Administrator

    Thursday, June 26, 2014 4:50 PM
  • We are using a Compellent SAN.  We have 1 datastore that houses the C and D drive of our Exchange 2013 server.  These are the only 2 volumes on that data store and it is in tier 1 storage.

    Disk queue length 2013: C=2.3 and lower / D=.1 and lower

    Disk queue length 2007: C and D are barely above 0

    Would setting a CPU reservation equate to correcting the ratio.  Might be easier than flushing the host of most VM's

    Thursday, June 26, 2014 6:08 PM
  • We are using a Compellent SAN.  We have 1 datastore that houses the C and D drive of our Exchange 2013 server.  These are the only 2 volumes on that data store and it is in tier 1 storage.

    Disk queue length 2013: C=2.3 and lower / D=.1 and lower

    Disk queue length 2007: C and D are barely above 0

    Would setting a CPU reservation equate to correcting the ratio.  Might be easier than flushing the host of most VM's

    I Don't think reservations are recommended. I would read that white paper or contact VMware support for the recommended settings.


    Twitter!: Please Note: My Posts are provided “AS IS” without warranty of any kind, either expressed or implied.

    Thursday, June 26, 2014 11:31 PM
    Moderator
  • You may need to run ESXTOP to see where the performance bottleneck is.

    It is clear that issue is most likely with environment outside Exchange. It could be either SAN or VMware.


    - Sarvesh Goel - Enterprise Messaging Administrator

    Friday, June 27, 2014 2:34 AM
  • Thanks for all the replies.

    I have a few things I can try at this point.  I just need to get the ok to do so and schedule them.

    1. Clear the VMhost of other VM's to reduce the vCPU ratio

    2. Try setting the power management to High performance

    3. Disable offload features on the VMXNet3 NIC. My co-worker found this one.  I also noticed when moving the mailboxes it took about 9 hours for 60 gigs.  And both servers have gig connections between them.  Last night when trying to move everyone back the old server the send/receive stats on the new server (2012r2) were about 3.2 MBs

    http://virtualrealization.blogspot.com/2013/11/vmxnet3-and-windows-operating-system.html

     

     

    Friday, June 27, 2014 1:39 PM
  • 60GB in 9 hrs., this is about 15Mbps (or 1.8 MBps). I would still recommend you to check Disk I/O performance especially on VMware. Are the disks on VMware thick or thin?

    - Sarvesh Goel - Enterprise Messaging Administrator

    Friday, June 27, 2014 4:19 PM
  • The disks in VMware are thin provisioned. 

    2013 Server: The SAN volume only the C and D drive for the 2013 server on it.  Both are separate VMDK files

    2007 Server: We use a passthough disk from the SAN.  Meaning we map a drive right to the SAN from the BIOS on the 2007 server. No VHD or VMDK files here.

    Monday, June 30, 2014 8:53 PM
  • We are experiencing the similar issue with Exchange 2013 SP1 on VMware ESXi4 and Outlook 2010.

    OWA runs just fine, but Outlook in online mode is unacceptably slow.

    There is no processor, memory or storage bottleneck. Servers were sized for a greater load and the configuration was checked with Jetstress tool.


    MCSA 2000, MCSA 2003, MCTS

    Friday, July 11, 2014 7:27 AM
  • The disks in VMware are thin provisioned. 

    2013 Server: The SAN volume only the C and D drive for the 2013 server on it.  Both are separate VMDK files

    2007 Server: We use a passthough disk from the SAN.  Meaning we map a drive right to the SAN from the BIOS on the 2007 server. No VHD or VMDK files here.

    BriGi86

    Could you check the performance when you are running Outlook in safe mode?

    outlook.exe /safe


    MCSA 2000, MCSA 2003, MCTS

    Friday, July 11, 2014 8:12 AM
  • We ended up moving all users back to 2007.  We may wait for another service pack to test this again.
    Friday, July 18, 2014 6:05 PM
  • We have many similarities in our environment and yours (VMWare/Compellent/Echange 2010 Online mode/etc.) and are seeing the exact same issues with performance.  The biggest difference that I see is that our Exchange 2010 environment was virtual and not physical servers.  Outlook 2010 and Exchange 2010 in online mode extremely responsive.  Now that we are running Exchange 2013 the experience is much worse.  OWA is very responsive only Outlook has the issue.  We can't run in cached mode because we have a VDI environment.  We've had a case opened with Microsoft for two months plus and are trying to have it escalated to a US based Exchange team.  Would love to share notes with you and find out if you'd be ok with sharing your case number that you had opened. 

    Tuesday, August 19, 2014 2:05 PM
  • Hi everyone,

    Here we have exactly the same thing. We did the same migration from Exchange 2k7 to Exchange 2013. We run on a VMware 5.5 infrastructure. We have a lot of users running on a VM, but I can also tell you that I'm experiencing the problem even on a physical PC. So the problem about the client side infrastructure is maybe not that much related.

     I also found an old thread talking about a known issue with the grouping option in Outlook and online mode, but this is supposed to be only for Exchange and Outlook 2007, so i'm not able to prove this problem is exactly and only related to this issue. however, I tried to disable the grouping option for some folder in my Outlook and I saw an improvement immediately. Here is the thread talking about this issue and the knowledge base associated from Microsoft:

    http: //blogs.technet.com/b/mikelag/archive/2008/08/31/slow-outlook-online-mode-performance.aspx

    http: //support.microsoft.com/kb/948828

    Sorry, I cannot post the real link since my account is not verified yet... simply remove the space.

    If anyone finally found an answer please post-it, thanks.

    Friday, August 22, 2014 5:21 PM
  • We are having the same issue on two properly sized physical Exchange 2013 servers.  We opened a case with Microsoft over six months ago and are still waiting for resolution.

    Tuesday, August 26, 2014 3:35 PM
  • Hi everyone,

    After some search on our environment we found that it seem to come from the connexion type (still not sure at all but...). Exchange 2007 was supporting Tcp/IP connexions with Outlook, but now with Exchange 2013, this isn't possible if i'm not wrong. I did a short test since I still have access to my 2k7 Exchange server. I forced my Outlook connected to our old Exchange server (2k7) to use the HTTP connexion type first to try to reproduce similar condition that we have on our 2013. As we expected we had pretty similar symptoms of slowness in Outlook. Then I turned off the setting to use http first and retry with the same Outlook and I retreived the fast and sweet performance we were having before our migration. I'm not an Exchange expert but this sound for me a good explication (theory). Our next step will be to update our Exchange server with Cumulative update 5 for our Exchange 2013 server. We hope this will help us. If anyone can confirm this theory, it may help Microsoft or other people to find a solution.

    Wednesday, August 27, 2014 7:29 PM
  • Hi there,

    Same here, I've got a multisite architecture and all the users who have been moved from 2007 to 2013 are experiencing the same slowness in online mode. 

    In owa the problem does not exists.

    By proxying the access trough 2013 cas to mailbox on 2007 the issue does not exists.

    This behavior is happening just in coexistence, I did some new installation and the online mode is fast.

    Any idea?

    Thursday, August 28, 2014 7:59 AM
  • Any news?

    Does anybody tried with the CU6?

    Friday, August 29, 2014 7:52 AM
  • I too would be interested if CU6 resolves this problem. I experience this periodically and it is hard to isolate.

    Doug Kinzinger, MCSE

    Tuesday, September 9, 2014 7:30 PM
  • CU6 does not resolve this.
    Take a look at this post:

    http://social.technet.microsoft.com/Forums/office/en-US/0d45b1b0-3047-4666-ad04-217e98ed8823/slow-online-mode-browsing?forum=exchangesvrclients

    • Proposed as answer by MasterKic Tuesday, September 16, 2014 5:06 PM
    • Unproposed as answer by MasterKic Tuesday, September 16, 2014 5:07 PM
    Wednesday, September 10, 2014 9:09 AM
  • OK We found something very very interesting about this. We found a solution that worked for us. Have a look at this other thread:

    http:// community.spiceworks.com/topic/571571-outlook-slow-after-migrating-to-exchange-2013

    Since these changes can affect the entire Workstation, I suggest you try it and read the entire post before doing it.

    TcpAckFrequency:

    http:// support.microsoft.com/kb/328890/fr

    I'm sorry the link are not valid since I'm not able to post it correctly right now but, I hope this will help many other people. Thx.

    • Marked as answer by BriGy86 Wednesday, February 18, 2015 5:07 PM
    Tuesday, September 16, 2014 5:13 PM
  • We are having the EXACT same issue. Exchange 2013 single server environment running on Server 2012 R2, updated to CU6. Running on VMWare 5.5 (latest updates) with VMXNET3 drivers. Compellent SAN. Also, all users migrated from Exchange 2007. Clients are running in ONLINE MODE due to VDI. OWA works flawlessly. All clients experiencing VERY HIGH RPC latency (4000+ ms when running RPCPING).

    Ticket opened with Microsoft over 6 months ago and has been escalated MANY times. Network team points fingers at Exchange, Exchange team points fingers at Outlook team, and Outlook team points fingers back stating it's not their issue.

    FYI, Outlook runs GREAT when installed ON the Exchange server (less than 15ms RPC latency). I've even tried installing Outlook on another VM on the same VMWare host and in the same subnet so that all networking stays on the host.

    Subscribing to this thread...

    Monday, November 17, 2014 10:42 PM
  • This is from the SpiceWorks link posted above. I have not tried it though.

    Looks like there is a registry key that can be done on each client or a fix for VMware. https://communities.vmware.com/message/2439033I

    I am going to try rolling the vmwaretools back to 5.1 RU1 and move a problem user to the new server.  If this works I will be marking MasterKic's reply as the answer as well as this one.

    • Marked as answer by BriGy86 Wednesday, February 18, 2015 5:06 PM
    Tuesday, December 16, 2014 3:00 PM
  • BriGy86: The link to vmware communities you posted is not working. Can you please provide another link?

    EDIT: Found the article in question! https://communities.vmware.com/message/2439033?tstart=0

    Thanks!


    • Edited by JBHorne Friday, December 19, 2014 8:11 PM Update link
    Friday, December 19, 2014 8:04 PM
  • Sorry for the late response.  I have since gotten the tools rolled back

    TAKE A SNAPSHOT!  removing the tools resulted in the old exchange server inoperable for about a day.  Then all of a sudden it started responding (This is after several reboots.)

    After getting them rolled back I ran into some other mail flow issues while testing so I have not completely resolved this issue yet.

    Friday, January 30, 2015 3:07 PM
  • After moving some users to the new machine we are not seeing any issues. (The one with the tools version rolled back.)

    While building the new server I made sure to install the rolled back version of VMware tools and did not get any errors during the install or SP1 install.  The previous server I built had horrendous issues with the install.

    Monday, March 9, 2015 7:16 PM
  • I'm having similar problems with online mode using HTTPS (preparing for our Exchange 2013 migration).   Our Outlook Anywhere and RPC Client Access array host names are the same, so, the suggestion from Brian Day at Microsoft is to put all your clients into HTTPS mode before cutting over DNS to Ex2013.

    If we attempt to have our clients (Outlook 2010 and 2013) connect using HTTPS (Outlook Anywhere) versus MAPI in Exchange 2010, the experience is poor at times.   Exchange 2010 is all physical.    We require online mode because of a VDI environment.    

    I've been working with Microsoft on this issue for two to three months without any progress.  We've ruled a lot out, but, haven't found the issue.      We've tried the TCPAck setting without any change.

    If anyone finds the smoking gun, please post on this.

    Wednesday, April 22, 2015 9:40 PM