Orchard on Azure Websites Preview Slow - Anyone Else Experience

Topics: Troubleshooting
Jun 21, 2012 at 5:30 AM

I have recently deployed a test over to the Azure Websites Preview. I believe it will be a good fit for Orchard but I am experiencing serious speed issues.  I currently have the (nearly) identical website (all the same modules) on Azure Cloud Services and that is running great.

Here is the Cloud Services URL (which is fast): http://endlessmountainsolutions.cloudapp.net/

Here is the Websites Preview URL (which is slow): http://emshosting.azurewebsites.net/

I am just wondering if anyone else has tried out the Azure Websites Preview and are experiencing great performance. To be clear, when I have no modules on the Azure Websites Preview it is fast (but I need modules and am running same modules on Cloud Services just fine). Look forward to some thoughts.

Coordinator
Jun 21, 2012 at 5:53 AM

Azure web sites are super-small instances or whatever they're called. They're the equivalent of a very aggressive shared hosting solution. So of course you get what you pay for, like everywhere else. The big advantage though is the elastic scaling: pay more and you'll get more performance.

Jun 21, 2012 at 1:09 PM

I agree, this is why I immediately jumped up to the Reserved site of Size small.  I don't mind paying even more, the issue is one that both the Cloud Services and the Websites Preview are both sitting on a Reserved Small Instance (yet the Cloud Services is much faster).  This makes me believe that the Websites Preview is not as optimized as the Cloud Services.

I implemented Orchard from a source code drop by following this technique: http://docs.orchardproject.net/Documentation/Building-and-deploying-Orchard-from-a-source-code-drop (basically  build "compile;package"  as I don't believe I should be using the Azure compiler approach anymore with Azure Websites).

I also implemented by using the web installer built right into Azure Websites Preview.

Both techniques use a SQL database.

Both resulted in the same (much slower) performance when compared to Cloud Services.

Bertrand, I believe I read that orchardproject.net might be migrating to Azure Websites. How will you do the deployment and has an attempt at this already been made? If so, is the performance good?

Coordinator
Jun 21, 2012 at 5:33 PM

At first I would have blamed the file system but actually you are using SQL Server, so it shouldn't make such a difference. It would be nice if you could share with us the modules you are using so we could try to repro, and give some feedback to the Azure team.

Jun 21, 2012 at 6:37 PM

Hi Sebastien,

I too have concluded it has to be a module that is messing everything up. I am working through each module right now (adding one by one - very painful) to find any that are causing the issue.

I have tried a number of times and the issue seems to be with:

  • Orchard.Email
    • This requires Orchard.Messaging of course (though it seems to be Orchard.Email that caused the issue)

When I add this module, it slows right down. When I remove it, things are back to normal.  I have also gone ahead and stopped and then started the website just to ensure there was not some other latency there due to adding the module. This did not improve things.  It is possible there are other modules but this is the first one that has caused the exact issue I experienced having all the modules moved over.  So you know, I did all this testing on http://emstesting.azurewebsites.net/ 

Two things I need to do:

  1. Fire up a brand new Orchard website and add in Orchard.Email to confirm this is the case.
  2. Go through the remaining modules I have to see if any others cause issue

Please let me know what you find. Perhaps you have an Orchard Web Site instance you can see if the Email is a problem on. Can you think of a reason this module might cause an issue on the Azure setup? I hope I am on the right trail and you can reproduce this also. 

Coordinator
Jun 21, 2012 at 7:02 PM

I have reviews the Email module and there is nothing inside which should cause this. I have removed some unnecessary code from the constructor though just in case. The change is on 1.x now.

Jun 21, 2012 at 7:32 PM

Ugh, this is where it gets ugly. I removed all the other modules and just left Orchard.Email and it is not causing the issue.  This (to me) means one of two things:

  1. There is some conflict going on with Orchard.Email
  2. OR adding Orchard.Email tipped the scales on too many modules for Azure to handle
    1. This seems VERY unrealistic

I am leaning towards number 1 and will have to add the other modules back in and see if I can find the issue.  It is just weird that it would only show up on Azure (and not on my localhost). I hope I can figure this out as I want to start using Azure.

I will document what I find.

Jun 21, 2012 at 11:14 PM

I must say, the one thing about the Azure Web Sites that is going to be hard to overcome is how long it takes the site to recompile after a module is added, removed, or updated. I obviously don't know what is going on behind the scenes, but there are other web hosting companies that are much faster at this...

Back to the discussion at hand.  Here is the list of modules that work:

  • CKEditor
  • CloudConstruct.AppleTouchIcon
  • Contrib.Cache
  • Contrib.ContentReference
  • Contrib.KeepAlive
  • Contrib.RewriteRules
  • Contrib.Taxonomies
  • Downlpay.Audit
  • EMS.PageTitleOverride
  • FeaturedItemSlider
  • Iroo.VersionManager
  • koosfiz.WebSiteOwner
  • Lucene
  • Markdown
  • NogginBox.BingMapList
  • NogginBox.MailChimp
  • NogginBox.Tagged
  • Nwazet.Commerce
  • Nwazet.ZenGallery
  • Orchard.Alias
  • Orchard.Autoroute
  • Orchard.Blogs
  • Orchard.Comments
  • Orchard.ContentTypes
  • Orchard.Email
  • Orchard.Fields
  • Orchard.Forms
  • Orchard.ImportExport
  • Orchard.Indexing
  • Orchard.jQuery
  • Orchard.Media
  • Orchard.MediaPicker
  • Orchard.Messaging
  • Orchard.Packaging
  • Orchard.Pages
  • Orchard.Projections
  • Orchard.PublishLater
  • Orchard.Recipes
  • Orchard.Roles
  • Orchard.Rules
  • Orchard.Scripting
  • Orchard.Setup
  • Orchard.Tags
  • Orchard.Tokens
  • Orchard.Users
  • Orchard.Warmup
  • Orchard.Widgets
  • S.H.Robots
  • TinyMce
  • UpgradeTo14
  • WebAdvanced.Sitemap

However, if I add "Vandelay.Industries" onto this stack, I drag. Problem is, I don't really believe it is "Vandelay.Industries" (i.e. don't hate me Bertrand) but I don't know where else to turn other than posting up the stack here to see what your thoughts are from the list above. I will be removing "Vandelay.Industries" and swapping in another module to learn more, but for now want to get you this list in case you can think of something else to look into.

Jun 23, 2012 at 5:01 PM

Well this is a frustrating experience :) but I don't know how else to determine what module (if any) is slowing Azure Web Sites down (if you have an alternate suggestion, I am all ears).

The frustration comes because it still does not seem to be around any one module (or any module in general).  It also seems intermittent. The website can be slow, then I remove a couple of modules and it gets fast. You would think then that the module was the culprit. However, I then add these modules back in and it still is fast. How did removing them and then adding them back in cause an improvement.

The one thing great about life is that every frustration is a great time to learn!

More and more I am convinced it has something to do with the compilation process versus any one module. For example, if I look at the App_Data > Dependencies folder, insice the dependencies.xml I see the following:

-<Dependency> 
<ExtensionId>Orchard.ContentTypes</ExtensionId>
<LoaderName>PrecompiledExtensionLoader</LoaderName>
<VirtualPath>~/Modules/Orchard.ContentTypes/bin/Orchard.ContentTypes.dll</VirtualPath>
<Hash>49110e7434bb5146</Hash>
</Dependency>


-<Dependency>
<ExtensionId>Orchard.Email</ExtensionId>
<LoaderName>DynamicExtensionLoader</LoaderName>
<VirtualPath>~/Modules/Orchard.Email/Orchard.Email.csproj</VirtualPath>
<Hash>69de2f89b54c6256</Hash>
</Dependency>

I am wondering what makes Orchard to treat one as PrecompiledExtensionLoader and the other as DynamicExtensionLoader. Seems to me that Precompiled would be the fastest (but I don't know exactly what goes on behind the scenes with the above xml interpretation).

Also, the Dependencies module sometimes contains the module dll files and sometimes it doesn't.  Am I wrong to assume that Orchard will manage it's own dependencies folder?  Could this be the problem where sometimes Orchard is doing dynamic compilation rather than reading from the dll? 

I am very interested to hear from someone who is successful on deploying to Azure without speed issues and a lot of modules. I am also interested in finally understanding how the dependencies folder works, I thought I knew, now I am not so sure. Thanks for your help. 

 

 

Jul 19, 2012 at 1:13 AM

I've had all sorts of performance trouble with Orchard on AWS, resulting in the site going down 10-20-30 times a day. But today I disabled the Email module and haven't had any hiccups since then. Fingers crossed this is the solution...

Jul 19, 2012 at 2:02 AM

Hello wictor,

I am in the process of finalizing my transition to Windows Azure and and am finally pleased. I had help from AJ at Microsoft and we were able to establish a technique that works very well. I am trying to finish up documentation of this but the major point (that sebastienros points out in another thread) is to go to the Orchard.Web/Config folder and find the "Sample.HostComponents.config" and change the filename to "HostComponents.config". This is necessary on Azure Websites because of the fact that dynamic extension loading is constantly looking for changes in your code and rebuilding the project (it is worse on Azure Websites due to the nature of how it is set up with your code base on a separate file server as the compiled application (as I understand it).

Here is the quick summary of what I have observed as well as the final area I am working on:

  • There are two approaches to getting increased performance on Windows Azure websites (and likely any other host that is used):
    • Publish and FTP all the files over to the web server AND include the HostComponents.config file (as mentioned above)
      • The config file prevents dynamic compilation from occurring (i.e. fiels are no longer monitored to check for updates and thus not recompiled on the server)
    • OR Publish and FTP all the files over to the web server EXCEPT ".cs" and ".csproj" files
      • Because there are no ".csproj" files there is nothing being monitored so, effectively, this turns off dynamic compilation
    • None of this really matters too much until you get a number of modules, then you can really see Windows Azure Websites drag if you do not utilize one of the above techniques

The final item I am checking on is the technique to modify a module locally and bring it back on the server in order fo the website to be updated WITHOUT having to stop and restart the website (early tests showed that without the dyncamic compilation the dll's were not registered - or something like that - so the changes in code were not observed).

I hope this helps you out - it made all the difference for me.  I actually believe Windows Azure Websites (using the above technique) is now faster that Windows Azure Hosted Services. Love to hear how it turns out for you.

 

Jul 19, 2012 at 10:17 AM

Hi, I've tested the HostComponents.config option. But some modules and my theme (custom views) does not render correctly, since I do the changes directly in AWS compared to publishing the compiled solution.

Jul 19, 2012 at 12:59 PM

Yes, that would be problematic (from what I know) as the changes you are making would be ignored (i.e. not recompiled).  If there was a page you could visit that force the recompilation (or a button in the admin side) that would be ideal. Perhaps someone else can chime in on this regard and indicate whether that is a possibility.

Jul 20, 2012 at 10:45 PM

Hi, I just created a test Orchard site yesterday. And so far I have been largely disappointed with the startup speed as opposed what widely percieved experience that the whole site is slow. If I leave it alone for a few hours I can expect to wait up for 45 secs for the site to come back to life. Once the site loads (or compiles???) it is okay. I say okay as opposed fine as I know it is running in a shared hosted environment and so I'm not expecting blistering fast response times.

I am also developing an orchard application as an Azure cloud service and I know from compiling orchard that the startup is slow as there are a hell of a lot of assemblies involved. But once it's running it's generally okay. So that's what points to imagine that this is what's happening behine the scenes when an instance is unloaded. So Microsoft....can you confirm that what exactly is going on with these shared websites.

The other point I would make is in relation to Jao28 suggestion to disable dynamic compilation. Now I've only been working with Orchard for about a month and the reason I've chosen it is because of that same feature - dynamic compilation. So excuse my ignorance but if dynamic compilation is turned off what will happen when I create a new content type on the fly? Add a field? Change a route? Add dynamic properties to a view? Is dynamic compilation not essential for this things to work?

Coordinator
Jul 21, 2012 at 1:06 AM

I have been running Orchard on AWS before it's publicly released and always found it pretty responsive. It's just since a few weeks that I heard about those issues from Jao28, and I also saw AJ from the Azure team (technically I also am in the Azure team) working on it. We came to the conclusion that using FileSystemWatcher could not be supported on AWS, so we need to provide a different way to detect changes and trigger dynamic compilation. I will work on that ASAP. For the moment the only way is to use WebMatrix locally and deploy when dev is done. Actually it should not be an issue to turn dynamic compilation off on a production site. That's how I am using it actually.

Jul 21, 2012 at 2:14 AM

AWS is responsive, once it's up and running. The dynamic compilation is what's causing the trouble. For now I reduced the number of modules and the site don't crash/lock that often.

I tried turning off the dynamic compilation using the HostComponents.config method, but then some modules stopped working and my widgets didn't show.

Jul 23, 2012 at 8:42 AM
Edited Jul 23, 2012 at 8:42 AM

Equally, I've found it fairly responsive on a reserved instance (www.experiencethestory.co.uk) once up and running.  I tried the KeepAlive module to stop it from dying, but Azure seemed to kill it when there weren't any requests within some period regardless.  I then set up an account on pingdom to keep watch on it, and it would seem that the regular requests from pingdom have been enough to keep it alive and responsive.

Jul 23, 2012 at 12:59 PM

Hi DavidCornish,

I have had good success with KeepAlive module. Did you go to the settings (located under Settings --> General and make sure to "Enable keep alive behavior" as well as set the "Url to request from this server" to be your url? Likely you did but worth mentioning just in case. Having those set keeps everything running smoothly for me (and hopefully for you).

My only complaint with AWS (other than having to turn off dynamic compilation) is a Visual Studio complaint. I want to use te Web Deploy method but it seems like this ignores all of the modules (possibly a visual studio issue??) and only migrates over the Orchard.Web folder.  When you use Local publish it works fine (go figure).  Also, I need it to ignore App_Data/Sites folder as well as Media folder but I cannot figure out how to do this either. 

Can anyone come to my rescue and inform me if they are utilizing Web Deploy with their new AWS site as that will save a LOT of time.

Jul 23, 2012 at 2:41 PM

sebastienros,

I see you made a tweak to Orchard.proj file to remove obj folder for WebDeploy. Unfortunately I am a newbie in using WebDeploy though I have opened up the Orchard.proj file to see all the configurations that are established already. In short, it is possible I am using the wrong method to accomplish WebDeploy (though I don't yet know the correct process).  Currently in Visual Stuio I right click on Orchard.Web project and choose Publish.  This then let's me utilize web deploy (but as mentioned above ignores all of the modules when using web deploy so does not work). 

Can you tell me how to utilize the Orchard.proj file to accomplish a successful WebDeploy (or point me in the direction of some documentation)?

Coordinator
Jul 23, 2012 at 7:37 PM

I usually create the web deploy package by right-clicking the web project, and I deploy that as a second step. I've successfully deployed to Azure web sites this way.

Coordinator
Jul 24, 2012 at 4:29 PM

@Bertrand, my Visual Studio install is also showing the same issue. I think it's because of a service pack which introduced new WebDeploy features, but I don't know the answer right now.

@jao28, is your WebDeploy dialog looking like the new VS2012 design ?

Jul 26, 2012 at 8:10 PM

@sebastienros, yes, it has the new VS2012 design (and was the result of a service pack as I recall). I really would like to use the new WebDeploy but cannot figure it out. Here is some info on my Visual Studio:

Microsoft Visual Studio 2010 Professional
Version 10.0.4.0219.1 SP1Rel

Microsoft .NET Framework
Version 4.0.30319 SP1Rel

Look forward to hearing your thoughts.