Content Manager throws errors when too many requests

Topics: Administration, Core, Customizing Orchard, General
Aug 13, 2014 at 8:44 PM
We've been using orchard for a few months and have about ~800 concurrent users. I like the cms but I like using it as a framework even more. Currently we have only one problem.

When too many people request information through nhibernate it returns a 404 error. We have implemented caching to reduce this but we still get errors sometimes. When the the site autoscales it also gives us errors until it's cached. I found some forums with the same question but no solution. This seems to big of a problem to have gone unfixed for so long.

Any suggestions or solutions would be helpful in finally solving this problem.

Aug 13, 2014 at 8:53 PM
You think you get 404s because of NHibernate ?

With output cache on you should be able to handle hundreds of requests per second, which would translate to thousands of concurrent user sessions. If you think NHibernate is the bottleneck maybe it's actually the SQL queries themselves. Are these users hitting your own queries ? I would suggest you to trace and analyze the queries. Sometimes some optimizations improve the site a lot.

Also, are you using 1.8.1 ? There have been huge database improvements in 1.8, and some more in 1.8.1.

Otherwise please provide more details about how we can reproduce your scenario in order to find out what can be done.
Aug 13, 2014 at 9:11 PM
Edited Aug 14, 2014 at 4:28 PM
I did some stress testing(using jmeter) and got these results:
So, here's an example of getting a list of content used on the site.
_contentManager.GetMany<part>("list of ids", VersionOptions.Published, QueryHints.Empty).ToList();
The errors I received:

Orchard/ContentManagement/DefaultContentMangager.cs   Line: 359  
return contentItemVersionCriteria.List<ContentItemVersionRecord>()
An exception of type 'System.InvalidOperationException' occurred in NHibernate.SqlAzure.dll but was not handled in user code. There is already an open DataReader associated with this Command which must be closed first.

Also got the error on line 243
private ContentItemVersionRecord GetVersionRecord(VersionOptions options, ContentItemRecord itemRecord) 
I also found that the error originated when I called
_contentManager.Get<...Part>(itemId, VersionOptions.Published);
So everything seems to work fine locally but when put under stress it breaks.

Note: Orchard is is up to date.

I noticed that when I query my tables for information that are not connected to Orchard parts I don't get errors but when contentManager gets multiple requests to get a part it starts to break and cause a 404 error.

If anyone has used contentManager to get a part and tested it under heavy load it would be cool if you posted an example of your code. Also, ways to go around content manager to get parts if it's possible would be helpful.
Aug 26, 2014 at 11:06 AM
Following this post seems problem could come because context is reused.

I'm not an Orchard expert but I will take a look to Content Manager to try to understand what could be happening,
Aug 27, 2014 at 7:52 AM
I have been taking a look on DefaultContentManager.cs and what I see is GetManyImplementation uses _sessionLocator to get NHibernate session. The point to take into account is _sessionLocator will return same session for every request it will receive within the same DefaultContentManager instance. And following post I mentioned we know NHibernate can throw the exception if two queries use same session.

So it will be interesting to see the complete code of your stress test to find out what's happening.
Do you share _contentManager between all your threads?