locked
App-V 5 Full Infrastructure Apps take long time to stream to the client RRS feed

  • Question

  • Hi was wondering if anyone has the same issue as i am or knows a fix for this, below is my problem and the troubleshooting i have done.

    Overview of problem

     

    App-V 5 apps delivered via App-V 5 full infrastructure take a long time to stream to the client and this means the user has to wait if they try and run an application before it has streamed to the client. Users sometimes have to wait 2 or 3 minutes for an application to stream and this is about 40 times slower than basic SMB and HTTP transfer tests show the system is capable of (see performance results below).

     

    App-V 4.6 apps delivered via App-V 4.6 full infrastructure and HTTP streaming are fine.

     

     

     

    Overview of environment

     

    App-V 5.0 SP1 Full Infrastructure.

    App-V servers are running Server 2012 on Hyper-V 3 or ESX 5.1 with 2 x vCPU and 4GB RAM.

    SQL servers are a SQL 2012 cluster.

    Separate servers for SQL, management, publishing, content and reporting.

    Management, Publishing and Content servers have two servers per role and NLB to provide load balancing. So 7 servers (2 x Man, 2 x Pub, 2 x Content, 1 x Reporting)

    Two further sites with 2 x Pub and 2 x Content each. All publishing servers pointed at the load balance address for management.

    Content delivered via HTTP

     

    Clients are physical desktops and laptops running Windows 7 SP1 x86 and Windows 8 x86

    App-V client is 5.0SP1

    Clients are pointed at their nearest publishing server NLB via a script which looks up the client IP address and uses PowerShell to configure the publishing server

    Content is streamed from the nearest content server NLB by setting the PackageSourceRoot to the nearest content NLB (via the same PowerShell script above).

     

    App-V apps delivered per-user via AD group. One AD group per application. Approximately 200 App-V apps published so far - will eventually reach 400 as we sequence more. About 9000 users.

     

     

     

    Analysis performed so far

     

    Servers not heavily loaded. CPU averages 5%. Lots of RAM free. Very low disk IO. Problem also occurs out-of-hours so we are 99.9% certain that server resources are not a cause.

     

    Streaming performance is the same from all 6 content servers and all 3 NLB addresses (tested by changing the value of PackageSourceRoot). Wireshark used to confirm packages are really streaming from the correct location, enforcing our belief that the problem isn't at the server end (unless all 6 servers are affected).

     

    Streaming via both HTTP and SMB2.1 is approximately the same (tested by changing the value of PackageSourceRoot between http://xxxx and \\server\AppVContent). Wireshark used to confirm we really are using the protocol we think we are using.

     

    All clients exhibit the same behaviour. Issue reported by many users. 5 test PCs chosen at random at all 3 sites confirmed to have the slow streaming problem.

     

    Slow streaming from both Hyper-V and VMware ESX servers.

     

    Client not heavily loaded.

     

    Affects all App-V apps although it obviously affects the larger ones more.

     

    All App-V apps have a Feature Block 1 setup.

     

    If we copy the ".appv" file from the server to the client via either HTTP or SMB then it's reasonably quick (up to 480Mb/s). So we don't believe the network or servers are at fault. For example:

     

    We can copy a 149MB .appv file via SMB from the content server to the client in 5 seconds.

    We copy HTTP download the .appv file from the content server using IE on the client in 5 seconds.

    But if you ask the App-V 5 client to fully download the sequence then it takes 2 - 3 minutes.

    The App-V 4.6 client takes about 8 - 10 seconds to fully download a similar sized application.

     

    App-V 5 publishing works fine - when a new user logs on they get their list of application straight away, it's just the streaming which is slow.

     

    Once the App-V app has streamed locally it runs fine and with a decent performance.

     

     

    Looking at a Wireshark trace of the streaming you can see that the slow performance is due to the transfer stopping and starting a lot. You only notice this when you zoom into the performance graph a fair bit.

    Each time the HTTP server stops sending traffic, it doesn't start again until the client sends a "TCP Window update". Each "stop" is of a different length, but just taking a few from the middle I get 0.06s, 0.11s, 0.13s wasted etc.

     

    cid:image010.png@01CE660D.BC3E2380

     

    I can see that it's the client stopping the transfer by reducing its advertised TCP Window Size. I'll provide an example:

    Server sends 9 x 1514 bytes. Client responds with an ACK and a Window size of 54016 bytes (256x211)

    Server sends 11 x 1514 bytes. Client responds with an ACK and a Window size of 37888 bytes (256x148)

    Server sends 10 x 1514 bytes. Client responds with an ACK and a Window size of 23296 bytes (256x91)

    Server sends 15 x 1514 bytes. Client responds with an ACK and a Window size of 1280 bytes (5 x 256)

    Server stops sending (I'm guessing because the client advertised Window size was less than a single packet's worth of bytes)

    <0.1 seconds passes>

    Client sends a "TCP Window Update" re-advertising a TCP window size of 65536 (256x256).

    Server starts transmitting again

     

    So the way I see this is that the App-V 5 client is controlling the transfer speed by utilising TCP Window flow control. The trace was taken at the client end so there's no room for anything on the network to be fiddling with flow control (and we've confirmed there are no traffic shapers in the loop).

     

    We've also tried streaming directly from the local client by copying some App-V 5 apps down to the client, creating a SMB share on the client and changing PackageSourceRoot to \\localhost\AppVContent (i.e. so we are streaming directly from the client to the client - to remove the network from the equation) and there is only an improvement of 5 to 10 seconds. So we know it's nothing to do with the network or the servers.

     

    We've tried turning off TCP auto-tuning on the client with:

    netsh interface tcp set global autotuninglevel=disabled

    and turning off TCP chimney offloading (which is off anyway because the NIC doesn't support it and Netstat -t output shows "InHost" for offload state for all connections) with:

    netsh int tcp set global chimney=disabled

    and nothing has improved.

     

    So we've now focussed on the extraction of the .appv (ZIP) file on the client.

    Using Windows Explorer it takes 75 seconds to extract the ZIP file

    Using 7ZIP it takes 9 seconds to extract the ZIP file

    Yeah we've always known that the Explorer ZIP engine is terrible. That's why we use 7ZIP or WinRAR on our clients.

     

    So we've started to wonder if the problem with the slow App-V 5 streaming is because the client is downloading the .appv file and extracting it as it goes along in a single thread. If the App-V 5 client is using the same terrible ZIP engine that Explorer does then that would explain the slow performance. The "download" appears to take a long time because the client is using TCP flow control to slow the transfer since it's extracting the .appv file using a very slow ZIP engine and it's all in a single thread.

     

    Saturday, June 22, 2013 12:03 AM

Answers

  • Guys,

    Just wanted to give you a brief update basically close this thread as Answered.

    We had submitted 4 App-V 5 Bugs to Microsoft and these were reproducible and an explanation was given on work around to them. Microsoft sent down a App-v developer to have a look at our problems. They said they will try and include the Bug fixes in SP2 which should be out in a few weeks or they will definitely be included in SP3.

    In regards to the slow streaming it all came down to the Disk IO.

    We found that you could simply enable "Turn off Windows write-cache buffer flushing on the device", then start streaming the app and then disable "Turn off Windows write-cache buffer flushing on the device" immediately after (we don't want to leave it on) and that basically fixed the issue.

    But a normal user would not have permissions to do this, so a code was written to enable and disable this option.

    Apology for not going in detail, like my opening thread, its very late. 

    but if you would like a detailed analysis please message me.

    I would like to Thank the Talented Consultant who designed and implemented are App-V infrastructure who found the bugs and created all the work around and who also emailed the detailed analysis of the problems to Microsoft that got them interested.

    Simon Bond from Ultima Business Solutions.

    Thank you



     

    • Marked as answer by a4r0 Tuesday, September 24, 2013 8:34 PM
    Monday, September 23, 2013 11:28 PM

All replies

  • Hello,

    I would raise the issue with Microsoft. Its a short answer, but the troubleshooting steps that I personally would have performed you have already tested and its quite interesting findings you have.


    Nicke Källén | The Knack| Twitter: @Znackattack

    Saturday, June 22, 2013 2:08 PM
  • Thank you for your response i have already opened a ticket but im not getting passed the second line support. However we have come up with a workout to help us stream FB1 component of a package locally without actually running the app as a temporary fix.

    thank you 

    Saturday, June 22, 2013 2:55 PM
  • Hello,

    Would you mind sharing the setup of the workaround? Is that you simple pre-stream FB1 and therefore increase the perceived performance?


    Nicke Källén | The Knack| Twitter: @Znackattack

    Saturday, June 22, 2013 8:22 PM
  • Firstly hats off, that is some great investigation and probably the most detailed request for assistance I've ever read!

    Having a little hunt around the internet, it would appear Microsoft's implementation of zip handling is not particularly speedy at the best of times. I just did a little quick test on my system in the house. Using a 4TB Hitachi Deskstar 7k4000 I had the following timings when extracting (using Extract All in Explorer and Extract to in WinRAR):

    win64_11gR2_client.zip - 601 Megs archive (1939 files - expands to 638 megs) from one folder to another folder on the same drive. 

    Extracting with WinRAR 5 - 4.32 seconds
    Extracting with Explorer - 15.44 seconds

    Introducing Microsoft Security Essentials (Antimalware Client Version: 4.2.223.0 Engine Version: 1.1.9607.0 Antivirus definition: 1.153.413.0) to the equation is a more interesting read and could be worth consideration.

    Extracting with WinRAR 5 - 6.86 seconds
    Extracting with Explorer - 121.51 seconds
    Extracting with Explorer (ZIP file types excluded in MSE)  - 122.36 seconds
    Extracting with Explorer (Archive scanning disabled in MSE) - 24.16 seconds
    Extracting with Explorer (ZIP file types excluded and Archive scanning disabled) - 23.99 seconds

    I'd imagine if you are using Forefront Client Security the results might well be similar as I think they are based on the same engine. This might well be an additional factor in the slowness you're experiencing. I didnt turn up Microsoft's recommendations for APPV 5.0 AV Exclusions specifically but it might well be the same?.

    Have you tried dabbling with process monitor to look at what's happening when the clients streaming the data, hopefully it'll to be a directory that can be excluded in your AV product as a test.

    If you've already tried any of this or ruled out AV completely then my apologies! I think you might of but I hope it's of some use.

    Good luck with your support call. Please come back here and let us know how you resolved it. I, for one, look forward to finding out!
    Saturday, June 22, 2013 8:28 PM
  • Correct one option we thought of was looking at is having the FB1 of every app package pre-cached on all clients.

    We couldnt do this for the FB0+1+2 of all apps since that's 60GB+ (there are 400ish apps across 9000 users) and most users only use 30 ish apps. But if we pre-stream FB1 only then that's much smaller and would mean faster launch times for a user when they logon to a computer remembering that users roam a lot and very often log into a computer they've never used before. We wanted to make the apps as responsive as possible when we have no idea which apps would be required on a particular machine.

    We came up with a "Superuser" account who can run every app.

    Powershell scheduled task as this user queries each Package and then queries the applications within that package.
    Then it launches the first app it finds and then kills it.
    Then it moves on to the next package.
    It was a bit of a pain because Get-AppVClientApplication can't be linked to Get-AppVClientPackagebecause they don't share any common variables. In order to use Start-AppVVirtualProcess we needed the object from Get-AppVClientPackage and the path of an app from Get-AppVClientApplication so we looked into the AppXManifest.xml XML file to query a list of apps for this particular package.
     
    Regardless, we still need to know why App-V 5 is so much slower than 4.6.

    (ill post the script on the next reply, exceeded the amount of characters i could use)

    Summary:

    The script which enumerates all App-V applications and pre-loads FB1 of each application by launching each application and then killing the app before moving onto the next (since there is no Powershell command to just load FB1).
    It seems stable - I've left it running for a good 50 apps and it works.
    This will allow us to have a scheduled task which runs at midnight (or something) which ensures that ALL apps are loaded to FB1 level.
    FB1 isn't too big and is certainly a LOT less that pre-loading every app which would be 100 GB or something (expanded).

    Once FB1 is pre-loaded, apps will load quicker since they only need FB1 to start.

    This wasnt the best way to do it but the only way we knew a bit mad since it worked by launching and then shutting down each app.

    We then discovered that an alternative seemed to be to read "StreamMap.xml" and pull out the filenames of each filename in FB0 and FB1 and then simply read those files. This caused them to be streamed. The code is finished and it's much faster, simpler and there are a lot of less frogs in the box but it needs to be further tested.

    Apology for the long reply.
    Sunday, June 23, 2013 9:49 AM
  • you can download the script from here

    http://sdrv.ms/1422Czc

    Sunday, June 23, 2013 10:02 AM
  • Thank you for your reply

    Actually the AV might be worth investigating.

    we did disable AV on one box and it didn't help but a little while after removing it I noticed that sccm had reinstalled it.

    we removed AV on my test VM and it didn't help, but I've since done some performance analysis on that VM and it seems quicker.

    I have used ProcMon and it didn't show anything interesting, you could see the App-V files being created in "ProgramData", but nothing else of interest.

    I've performed a load of tests on my PC and a Windows 7 VM and whilst having AV on definitely slows down Windows Explorer extraction of ZIP files, it doesn't seem to have any impact on the speed of loading an App-V package.

    But my laptop and the test VM both use a SSD and everything is quick, so well do some testing on Monday.

    thank you again for your suggestion.


    Sunday, June 23, 2013 10:12 AM
  • Guys,

    Just an update from Microsoft after they clocked up 20 hours they basically said the performance on first launch is by design. 

    I don't see many customers being too happy with the results. Since App-V 5 is a version increment from 4.6 most customers will expect an improved experience and not a step backwards. If it was given a separate product name and marked as v1.0 then fair enough and nobody would have such high expectations.

    • Marked as answer by a4r0 Wednesday, July 10, 2013 3:03 PM
    • Unmarked as answer by a4r0 Tuesday, September 24, 2013 8:34 PM
    Wednesday, July 10, 2013 3:03 PM
  • Guys,

    Just wanted to give you a brief update basically close this thread as Answered.

    We had submitted 4 App-V 5 Bugs to Microsoft and these were reproducible and an explanation was given on work around to them. Microsoft sent down a App-v developer to have a look at our problems. They said they will try and include the Bug fixes in SP2 which should be out in a few weeks or they will definitely be included in SP3.

    In regards to the slow streaming it all came down to the Disk IO.

    We found that you could simply enable "Turn off Windows write-cache buffer flushing on the device", then start streaming the app and then disable "Turn off Windows write-cache buffer flushing on the device" immediately after (we don't want to leave it on) and that basically fixed the issue.

    But a normal user would not have permissions to do this, so a code was written to enable and disable this option.

    Apology for not going in detail, like my opening thread, its very late. 

    but if you would like a detailed analysis please message me.

    I would like to Thank the Talented Consultant who designed and implemented are App-V infrastructure who found the bugs and created all the work around and who also emailed the detailed analysis of the problems to Microsoft that got them interested.

    Simon Bond from Ultima Business Solutions.

    Thank you



     

    • Marked as answer by a4r0 Tuesday, September 24, 2013 8:34 PM
    Monday, September 23, 2013 11:28 PM
  • Tuesday, September 24, 2013 6:49 AM
  • I am using App-v 5.0 SP3 server and client and we are facing the same issue still. Streaming is slow.

    Any workaround for this? any help is greatly appreciated......


    • Edited by Bala3k Sunday, March 8, 2015 10:10 AM
    Sunday, March 8, 2015 10:10 AM
  • Hello,

    Did you read the post marked as answer?


    Nicke Källén | The Knack| Twitter: @Znackattack

    Sunday, March 8, 2015 10:36 AM
  • Below is the Full Explanation of overcoming the slow app-v 5 publishing and streaming issue written by Simon Bond from Ultima Business Solutions

    http://www.algiz-technology.com/overcoming-slow-app-v-5-publishing-and-streaming

     

    Thursday, September 3, 2015 11:46 AM
  • This post should be a must read for anyone that needs to learn now to document and diagnose.
    Wednesday, September 9, 2015 5:35 PM
    Moderator