Max concurrent connections / Request Per Second / Benchmark 1.7?

Topics: Core, General, Troubleshooting
Jul 8, 2013 at 3:07 PM
How many max concurrent connections have you configured at IIS?

We put it now at 250 since putting it on virtual limitless, it was very easy to take down our site (just do 1000 concurrent requests using AB ...)

I wonder what the other Orchard users put their limit to, or who isn't aware how sensitive orchard is for trying to let it handle just 1000 connections in one second?

Also, if they did test their site, how much request per seconds did you see 'max'?

We are running into more and more issues with Orchard where it seems like it just isn't made for larger websites :/

Also: Did anyone try to benchmark Orchard 1.6 vs Orchard 1.7 (well, vs trunk that is)?
Coordinator
Jul 8, 2013 at 5:15 PM
Are you using the Cache module ?
Have you tried Drupal or Wordpress without the cache module on ? Same issue, you can't handle this kind of traffic without any output caching solution.

And maybe you need more that just the Output Cache module, a reverse proxy actually, like ARR or NGinx/Varnish.
Jul 9, 2013 at 6:43 AM
We're using a custom caching module that we wrote (offering donut hole caching). And I don't expect it to handle 1000 rps but the problem is that Orchard doesn't have a 'queue' option.

If you put the IIS connection limit to 1000, and someone does 1000 requests, Orchard will try to execute ALL requests at same time. Even though our cached pages do not take long to render, trying to execute 1000 requests in one second... Well, it kinda 'locks' our site.

A simple AB benchmark with 1000 concurrent connections was sufficient to 'take down' (as in make it load forever) our website. We 'evaded' the issue for now by configuring a connection limit, but it be nice that Orchard itself also have an option that defines how many concurrent requests can be handled at the same time.
Developer
Jul 9, 2013 at 11:02 AM
Edited Jul 9, 2013 at 11:06 AM
IIS will try to execute as many concurrent requests as the resources allow to and it actually should. I can't see no real point in limiting the number of concurrent requests - most of the time you need to squeeze the most. This limit would actually be a workaround to hide the underlying concurrency problems, not a solution.

The real question is what happens during the request itself that results in taking down the site? Database access, custom code etc. This all can result in deadlocks, thread starvation, timeouts, database concurrency issues and other things. Performance tuning is a very broad subject and there is no silver bullet. It highly depends on a scenario.

First, you should take a look at the database - this is the common bottleneck. Try eg. changing isolation mode to ReadUncommitted (if you haven't done so yet).
Second thing is session - try turning it off. Session usage is a huge perf killer as it results in forcing synchronous request execution by locking threads on certain resources. You can turn session off by putting SessionState: disabled in modules' manifest files (or in RouteDescriptor definition). This way requests to all routes declared inside a given module will not use session at all. This has drawbacks (like INotifier and all things dependent on Session or TempData won't work)
Jul 9, 2013 at 11:19 AM
Database should not be the issue.

Also, it IS a bad idea.. It is no use trying to execute 1000 at the same time when you (for example) only have 4 cores...

If you have an Orchard site where I may run AB against for testing purposes, that be great ;)
Developer
Jul 9, 2013 at 11:49 AM
Then put some reasonable limit. Request threads are scheduled by IIS worker process based on the available resources.
If the site gets unresponsive after putting that kind of load it may be due to various factors (IO locks, lack of memory etc.). Try attaching a remote debugger and see what's going on. Also, try putting the same load on a vanilla Orchard instance and compare.
I wonder what the other Orchard users put their limit to, or who isn't aware how sensitive orchard is for trying to let it handle just 1000 connections in one second?
1000 connections/s is not just. It depends on how much your machine can actually handle. And like Bertrand wrote - you won't be able to handle this kind of load without any tweaks (unless you have a very powerful machine).
Jul 9, 2013 at 11:59 AM
I don't want to handle that kind of load, I just don't want Orchard to blow up when someone 'tries' to connect that many times...

And if you wonder: it had a lot of threads all 'stuck' @ autofac... It had 300 threads at one point when I tried to connect with 1000 concurrent connections.

I did not have evil intent, just to test, but if it is this easy to take down an orchard site...

"Then put some reasonable limit." Give me the option @ Orchard to define such a limit? A limit so that Orchard does not blindly try to process all incoming requests, even when it is already overloaded :/
Jul 9, 2013 at 12:13 PM
Edited Jul 9, 2013 at 12:20 PM
Playing with a production site is not in the best professional usage list, especially when heavilly loaded.

If not easy to limit Orchard transactions (it deeply depends on the data structures imbrication you have created using content items/parts), and if using SQL Server, you may limit the number of DB transactions and optimize its index startegy (Orchard heavily uses one main index source to generate all the content item indexes then all the related parts indexes, this index should not generate page/table escalation lock, etc).
And concerning threads, that's an IIS limitation, this strange concept of 'application pools'...

The only solution I see is having as many server instances (VM) as necessary may be with dedicated hardware and .... network.
Jul 9, 2013 at 12:18 PM
Well, afaik Orchard accepts all incoming requests 'manually' and then launches some async err stuff that then does all the processing.

Wouldn't it be possible to add an option to the core where Orchard still accepts all incoming requests BUT only processes X (or Y per CPU) max requests at the same time?

Only when a request is fully processed, another pending request is processed.

As it is now (or looks) it just launches the moment the request is received, not caring at all on how much it can actually handle.

All put in short: IMHO, it is VERY easy at the moment to take ANY orchard site down (or make it load 'forever', also counts as 'down' imho) if they not limit the concurrent connections. And even if it IS limited, it might still make it inaccessible when a moron is simply doing plenty of requests using AB.

During our own tests, I only had to run AB for 2 seconds with 1000 concurrent connections. Result: After 10min the site still was 'loading' for me, and I had to kill the application pool...

Our site was down with just 2 seconds of effort! No need to keep connections open or anything.
Jul 9, 2013 at 12:32 PM
Edited Jul 9, 2013 at 12:36 PM
An Admin will use performance counters and trigger some external action when the fixed threshold is reached.
It is very problematic to have an application controlling itself.

This kind of destructive test is so easy, recently using a simple fluke tester and an IP multicast we shutted down a POC of a great city subway management: 10 seconds.... all the internet technology is 'fragile', it is very strange when we remember that it was a military project made to be resilient :)
Developer
Jul 9, 2013 at 12:42 PM
Edited Jul 9, 2013 at 12:43 PM
AimOrchard wrote:
"Then put some reasonable limit." Give me the option @ Orchard to define such a limit? A limit so that Orchard does not blindly try to process all incoming requests, even when it is already overloaded :/
It's IIS, not Orchard responsibility to limit the incoming requests. Request will still be processed and thread will still be created. Even if we implement this limiting feature inside Orchard, short-circuit all requests past the limit and eg. return an error page, some processing will still happen. So it will still be possible to take down the site with a finite number of requests (like any other page).
And if you wonder: it had a lot of threads all 'stuck' @ autofac..
That's something worth digging into. Could you please provide a stacktrace of the hanging thread? This may indicate that Orchard was not able to create a lifetime context (either work or shell one) for some reason. Have to ask - are you sure that none of your custom components does any processing in constructors (besides simple field assignments)? Ctors should be free from any processing that may result in lock or an exception being thrown, otherwise Autofac won't be able to construct the dependency tree.
If not easy to limit Orchard transactions (it deeply depends on the data structures imbrication you have created using content items/parts), and if using SQL Server, you may limit the number of DB transactions and optimize its index startegy (Orchard heavily uses one main source to generate the content item index then all the related parts indexes, this index should not generate page escalation lock, etc).
Have you tried turning NHibernate caching on? And, as I wrote before - for best performance you have to change isolation mode to ReadUncommitted (inside ITransactionManager) to avoid locking.
The only solution I see is having as many server instances (VM) as necessary may be with dedicated hardware and .... network.
Having separate machines for testing/staging environments is a good practice anyway - you should have those.
Jul 9, 2013 at 12:50 PM
Edited Jul 9, 2013 at 12:52 PM
Caching is good when you have a high level of read related to create/update, but as soon as something is updating things turn badly, especially when using relationnal concept where everything is 'related' there is a domino effect through all the caches chain (IIS, .NET, Orchard, SQL Server, OS, Hard drives, Network proxies and firewalls, etc.).
Jul 9, 2013 at 12:57 PM
Edited Jul 9, 2013 at 12:58 PM
May be the idea is to have a module providing Orchard Management/Performance counters interfaced with windows infrastructure.
May be it already exists ?
Jul 9, 2013 at 1:03 PM
@pszmyd As far as I can see, Orchard accepts 'any' connection 'async', meaning that IIS won't cap at all since Orchard keeps accept 'everything' connection, without limit. From IIS point of view, there is nothing to limit, except the total amount of active connections. Or tell me where I can tell IIS to never create more than 64 threads for processing incoming requests? Since I haven't found any such option...

And the thread wasn't really stuck, just extremly slow (guess what happens when 300+ threads try to load all dependencies using autofac? Thing get slow...)

And we reverted back from ReadUncommitted to ReadCommitted, as that gave issues with signals if 'other visitors' replenish the cache between triggering the signal and committing the transaction. If it is such a holy grail for Orchard, make it default and make sure that all modules work with it? (Not saying it is)

We test locally and have a dedicated external development machine for larger tests: ofc we don't just 'apply' live...
Developer
Jul 9, 2013 at 1:12 PM
Edited Jul 9, 2013 at 1:12 PM
Caching is good when you have a high level of read related to create/update.
Which is the case of 99% of websites...
May be the idea is to have a module providing Orchard Management/Performance counters interfaced with windows infrastructure. May be it already exists ?
Don't know of any. If we'd need to measure database traffic and other generic stuff (like req/s) then I don't see the point of implementing custom counters - there are existing ones for this. Orchard is an ordinary ASP.NET MVC app after all. Custom counters would be perfect for Orchard-specific things (eg. event bus message throughtput, content items loaded/s).
Jul 9, 2013 at 1:25 PM
Edited Jul 9, 2013 at 1:29 PM
May be Orchard is an ordinary ASP.NET MVC App but it certainly have to enter the world of Administration by another team than the dev team if it wants to gain large audience.

Building the 'SharePoint like' Management infrastructure is certainly not adapted, but in Windows world, managing app goes through WMI and Performance Counters.
For sure windows provides plenty of detail counters, from the HD read access to the page swap, but something more adapted to Orchard would be appreciated.
Autofac and its threads are a very good exemple: if you see that the number or the completion time increase anormally you may trigger an alarm, this Windows does not provides.
Another exemple is the number of cache hits in Orchard, and the number of ContentItem creation/update/publish/ etc.

I even found somebody sharing this idea here :)
https://orchard.codeplex.com/workitem/19151
Developer
Jul 9, 2013 at 1:36 PM
Guess who was that;)
I like perf counters a lot and agree with you that Orchard-specific counters would be great to have. Just not to duplicate what's already there.
Jul 15, 2013 at 6:40 AM
Edited Jul 15, 2013 at 6:40 AM
Still I have yet to see any benchmark results? ;)

We get around 20 req/sec at the moment. We cache alot but we use donut caching so some things are never cached (like the 'user' box).

Query wise, every request still does around 10 queries : like 2 for the 'content', rest are 'site settings' that are getting queries every request.

FYI, tested like so:

ab -n 100 -c 10 http://website.tld/
Jul 16, 2013 at 10:51 AM
Edited Jul 16, 2013 at 10:56 AM
We succeeded in adding the code required to make these 2 options work:
    <add key="Orchard:MaxConcurrentRequests" value="24" />
    <add key="Orchard:MaxEnqueuedRequests" value="256" />
Still wish Orchard would include it in the core, instead of us having to patch it (up).

Performance is now improved, we have an upper limit and we also remove dead requests from the queue to prevent wasted CPU cycles (so if the client disconnects between the request being queued and being executed, it isn't executed at all)

We can now get 80 req/sec @ one of our cached pages, 2 weeks ago we could do 14, yesterday we could do 30.

edit: One of our other pages can handle 345 req/sec now :)
Coordinator
Jul 16, 2013 at 11:24 PM
Sounds like a great improvement, congrats. I'm a little puzzled by this however, as it would be the first time that I see IIS performance defaults that are worth changing. Sébastien, any thoughts?
Coordinator
Jul 16, 2013 at 11:55 PM
I don't think he's talking about changes to IIS configuration but actually some changes in Orchard itself to prevent it from answering some requests after a specific amount of them. Like a counter in the BeginRequest/EndRequest and just return if the threshold is reached.

And honestly I don't understand everything with thread limits and the issues you are describing, I might have to learn more on the subject or ask for a friend to help in order to give a recommendation or take further steps.
Jul 17, 2013 at 5:43 AM
Edited Jul 17, 2013 at 5:45 AM
Well we now have a concurrent execution limit and a max enqueued requests limit.

So the max concurrent execution limit is hit, it'll start processing the list of enqueued requests (not exceeding the concurrent execution limit).

An additional check there (as stated) is that it skips enqueued requests where the connection has been lost.

Maybe in other cases this could be bad, but our case is: if there is no-one listening for the result, there is no need to execute the request.