Azure Web Sites file system seems to affect files changes detection

Topics: Troubleshooting
Jun 2, 2013 at 12:39 AM
Sorry for my english, I am french. I like the concepts of Orchard and Azure Web Sites and most of the time my sites work well that's why I want to continue. But sometimes I experience very long startup and request responses

Please, for more details take a look on this thread that I have post on the AWS forum : I tried to give some useful infos talking about what I have understood on dynamic compilation, extensions loading and monitoring, components you can disable...

Effectively, without dynamic compilation and extensions monitoring, a site can work well for a while, but sometimes the same symptoms reappear. That's why I have done some logging with log4net. In log files I have seen other problems, same symptoms but not related to dynamic compilation of *.cs

Sometimes, on startup there are multiple extensions reloading because of the loss of cached data that use cache dependencies on files changes. And there are many Razor recompilations of *.cshtml that use also dependencies on files changes. So, for testing, in web.config I have done one option to not use CacheDependency with the HostingEnvironment.Cache, even I lost some other dynamic ressources. And one option to not add VirtualPathDependency to the RazorBuildProvider

whereafter there was no more multiple extensions loading, and Razor JIT compilation seemed to be less frequent. But, in log file I saw that a *.cshtml file is still dependent on itself and can be still compiled more than once

Finally, some unexpected Recycling seem to be due to the same kind of problem but at the IIS level, like after a web.config change. Especially, this happen after a first very long startup, so the following request execute another application startup, and so on

All these problems seem to have one thing in common, some operations based on files changes are triggered although no file has been modified. So, during some period you have very long startup and successive Recycling. This could explain many other problems as, for example, Contrib.Cache issues on AWS that I have seen on another discussion. I tested it, during period without files dependencies problem, a site work well and Contrib.Cache work fine also

So, I suspect a problem with the Azure Web Sites file system that affects the files changes detection by ASP.NET objects. Maybe, idem at the IIS level

I know that Sebastien Ros has a blog on AWS and that he do some investigations about Orchard on Azure Web Sites. Of course, I am not sure of my analysis, but I hope it helps to find the real cause

Thanks for your feedback
Jun 3, 2013 at 1:15 AM
I suspected that there could be the same kind of problem at the AppDomain level because I also have sometimes unexpected Recycling like after a web.config file modification. Particulary during first startup and following requests

Recently, I was able to make logging also in global.asax.cs to capture Application_Start and Application_End event with the ShutdownReason that specifies why the AppDomain class shut down

So, after a test site was updated, I just have tried some requests and that is what I got

11:08:55 Application_Start
11:09:08 Application_End HostingEnvironment
11:09:24 Application_Start
11:09:36 Application_Start
11:09:37 Application_End HostingEnvironment
11:10:02 Application_Start
11:10:03 Application_End ConfigurationChange
11:11:21 Application_Start
11:11:22 Application_End ConfigurationChange
11:11:33 Application_Start
11:11:34 Application_End ConfigurationChange
11:11:46 Application_Start
11:11:47 Application_End ConfigurationChange
11:12:00 Application_End ConfigurationChange
11:12:30 Application_Start

HostingEnvironment : The hosting environment shut down the application domain
ConfigurationChange : A change was made to the application-level configuration file

So, one of my problem is the same as I have seen in another thread on Azure Web Sites, like something is modifying the web.config file. Maybe, my other problems are the result of this one

Jun 3, 2013 at 6:29 PM
The Azure team is currently investigating these issue with my help. We'll keep you posted, it's also an issue for us.
Jun 3, 2013 at 7:34 PM
Thanks a lot
Jun 6, 2013 at 5:08 PM
I'm also encountering this problem on Azure Websites. I'm working on a Orchard site for a client and I'm wondering if this is something that is likely to be fixed in the near term.
Jun 6, 2013 at 5:53 PM
In the very short term actually. If you are on the East US DC then it should be ok now. The other DCs will be updated today. Please let me know if your issues are fixed then.
Jun 6, 2013 at 8:57 PM
Good news. I'll tell you soon if it is ok for me
Jun 12, 2013 at 10:02 PM
I still have the same symptoms, my DC is West Europe. So I put another post on the thread mentioned above, where I said that I also had these AppDomain ShutdownReason:
BinDirChangeOrDirectoryRename : A change was made to the Bin folder or to files in it
CodeDirChangeOrDirectoryRename : A change was made to the App_Code folder or to files in it
ResourcesDirChangeOrDirectoryRename : A change was made to the App_GlobalResources folder or to files in it

Even if I do not have any App_Code and App_GlobalResources directories

Jun 17, 2013 at 8:41 PM
Hi, Sebastien

Please, do you know if this issue is fixed on West Europe DC ? I ask this because last week you said it was. Otherwise, I can wait. Today I had the same problems on one site while another with the same code on another subscription (but same DC) works very well. Take a look on this example where the site is quite immediately recycled on startup

06:15:21 Application_Start
06:15:21 Application_End HostingEnvironment
06:15:22 Application_Start
06:15:22 Application_Start
06:15:22 Application_End HostingEnvironment
06:15:23 Application_End ConfigurationChange
06:15:23 Application_Start
06:15:23 Application_End ConfigurationChange
06:15:23 Application_Start
06:15:23 Application_End BinDirChangeOrDirectoryRename
06:15:36 Application_Start
06:15:37 Application_End HostingEnvironment

Jun 17, 2013 at 9:26 PM
Did you see this log today ? My statement was wrong, they were able to fix East US the same day as I mentioned, on West US the day after, and the rest of the world later. And by later I assume it has been done last WE. I will ask for an official answer.
Jun 17, 2013 at 10:30 PM
Yes, I saw this log today and others of the same kind every day of the last week, except for 1 day this WE. I am quite relieved when you say : " and the rest of the world later ". It is OK if I just have to wait a little more

Jun 18, 2013 at 12:24 AM
Have been told "right now", so let's wait another 24h. I can assure you the difference is impressive though, my monitor charts are just flat to a very low level of CPU and a flat 0 in terms of HTTP errors. Worth the wait, even though it should have been the default since day 1.