Typical IIS environment for live Orchard site?

Topics: General, Installing Orchard
Sep 3, 2013 at 8:37 AM
Hi,

We're ready to deploy our first Orchard site to our production machine. We own the webserver (no shared hosting).

What would be a typical IIS environment for a live Orchard site? I would describe the website as medium-sized with several visitors at the same time during peak moments. It seems to me I need to define an application pool with multiple worker processes.

However, session is used at several places (for example, a wizard-style registration for an event). Using multiple worker processes implies we should move to the session to a state server or SQL Server. I assume Orchard is designed for this? Everything that the core code stores in the session is marked as serializable?

Secondly, the current wesbite has one bottleneck.

We have created an IActionProvider to send emails to a lot of users. Next, we defined a rule on the publication of a certain content item to trigger this action. The event gets triggered, but no other visitors can enter the website while the action is running.

I assume this problem will be solved by using multiple worker processes, but maybe there is a better solution in this particular case? Starting a new thread in code, use delayed actions, etc...?

Thanks for any advice, Orchard folks!
Coordinator
Sep 3, 2013 at 4:41 PM
Orchard doesn't care about session: that's a service that is offered one level down, by ASP.NET MVC.

Because of the design of the HTTP protocol, it's generally a bad idea to have a web server run long processes, and preferable to move those to out of process, queued services, but sometimes you don't have a choice. In those cases, applications often use one of the threads in their pools to run scheduled tasks. Don't create your own thread, it is usually a bad idea to upset the automatic management of the thread pool. Orchard provides an API for scheduled tasks. Is this what you're using? It shouldn't block other threads in any way, unless you have a really weird IIS configuration with only one thread in the pool.
Sep 3, 2013 at 5:25 PM
I played around with IIS today and it seems like Orchard doesn't use the ASP.NET session at all. Good!

The task is not scheduled in our code, if that's what you mean. Instead the task is executed by a trigger on the publication of certain content items in the dashboard. As long as the task is running, the website stays unresponsive. This needs to be fixed.

Maybe the API you mentioned is what we need. Are there docs/codesamples that can get me started?
Coordinator
Sep 4, 2013 at 12:10 AM
Well, by default TempData is using Session and I'm sure it's used in a couple of places, so technically...

What you describe is problematic: instead of doing the work right away on the published event (which will block the thread where it happens, and any request that comes onto the same thread afterwards, until it's done), you should schedule a task, using the proper API. Look for usage of IScheduledTask for examples.
Developer
Sep 4, 2013 at 2:33 AM
Edited Sep 4, 2013 at 2:33 AM
Actually Orchard uses session in exactly one place iirc - the notifications. It uses TempData to store messages to display between requests. If you can opt out from using default Orchard notifications, you can simply disable the session at all. That should give some performance boost under high loads.
Developer
Sep 4, 2013 at 10:18 AM
TempData is also used in Comments.
Sep 4, 2013 at 12:36 PM
We are using notifications, but all used classes are serializable.

So I switched from InProc to StateServer and everything seems to run fine (after a few modifications in my own code).

As for the long running task: we now run the code as a 'delayed action' and this seems to solve our problem.

Thanks Bertrand, Pjotr and Zoltan for your help!
Sep 4, 2013 at 1:01 PM
BertrandLeRoy wrote:
Orchard doesn't care about session: that's a service that is offered one level down, by ASP.NET MVC.

It shouldn't block other threads in any way, unless you have a really weird IIS configuration with only one thread in the pool.
And what about pending requests being blocked since the transaction isn't completed until the mail sending is done?
Coordinator
Sep 4, 2013 at 7:16 PM
AimOrchard wrote:
And what about pending requests being blocked since the transaction isn't completed until the mail sending is done?
Never send e-mail from the main thread, do it from a scheduled task. If core code is doing that, it should be fixed.
Sep 18, 2013 at 2:50 PM
Thanks for the great advice.

As Netwave said, we now run the code as a 'delayed action' which solves our responsiveness problem.

We created a rule that waits a fixed time and then starts another rule. This other rule then sends out the emails. (through an IActionProvider )

This approach works perfect.
For small number of emails: no problem.
but when the number of mails is bigger, the process takes longer until completion. (couple of hours )

The process does everything perfect until the end, but on completion, the process starts over from scratch.

We do see the scheduled action in Scheduling_ScheduledTaskRecord does not get removed after this long transaction.

Thanks for any advice,
Coordinator
Sep 18, 2013 at 4:58 PM
How many e-mails are we talking about? I mean, there is a scale where you need to use proper mailing-list software.
Sep 19, 2013 at 8:32 AM
We send emails to a selection of the user database. (those who opted in)

I understand this shouldn't be functionality in the core of a CMS, however we can leverage a lot in the emails, by using the framework (layout, content, linking, etc)

Our concern now, is how to prevent Orchard of re-executing the delayed task.

Any ideas?
Coordinator
Sep 19, 2013 at 7:43 PM
(I would consider something like Mailchimp, for which there is a module on the gallery, for any mass e-mailing)

So your task actually crashes? What's the exception? Did you consider queuing each e-mail, and treating each e-mail sending so that when the task restarts it doesn't re-send what's not in the queue anymore because it's already been sent?
Sep 19, 2013 at 9:59 PM
I'm assuming the transaction times out because the task is taking too long.
Coordinator
Sep 20, 2013 at 2:01 AM
If that's the case, the job should be chopped into smaller chunks.
Sep 20, 2013 at 8:55 AM
The task doesn't crash, it completes nicely.

However after the completion he restarts.

The only error logged (not at the time of the restart is):
NHibernate.Util.ADOExceptionReporter - Transaction (Process ID 93) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Coordinator
Sep 20, 2013 at 5:52 PM
Well, leaving a deadlock doesn't exactly qualify as completing nicely ;) I think you need to find what that deadlock is caused by exactly, because it's obviously causing a transaction to be rolled back.
Sep 24, 2013 at 5:15 PM
As suggested, we removed the mass-mailing functionality to another service.
However we used the same logic for a content publishing service (when a certain Newsitem gets published, we wait for 1 minute and then send notifications through IMessageManager to all those who subscribed to a topic. A topic in this case is represented by a Tag, so users have a Tag Part and can add subscriptions to those tags. When a newsitem with a tag gets published they receive notifications.

Yesterday on the production server we had similar behaviour as above.
Record in Scheduling_ScheduledTaskRecord does not get removed and task restarts.

Error log had following message:
2013-09-23 18:13:54,272 [56] NHibernate.AdoNet.AbstractBatcher - Could not execute query: INSERT INTO Orchard_Framework_ContentItemVersionRecord (Number, Published, Latest, Data, ContentItemRecord_id) VALUES (@p0, @p1, @p2, @p3, @p4); select SCOPE_IDENTITY()
System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource
1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at NHibernate.AdoNet.AbstractBatcher.ExecuteReader(IDbCommand cmd)
ClientConnectionId:0dea8577-639a-4f1b-b840-027790390049
2013-09-23 18:13:54,319 [56] NHibernate.Util.ADOExceptionReporter - Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
2013-09-23 18:13:54,319 [56] NHibernate.Util.ADOExceptionReporter - The wait operation timed out
2013-09-23 18:13:54,319 [56] Orchard.Exceptions.DefaultExceptionPolicy - An unexpected exception was caught
NHibernate.Exceptions.GenericADOException: could not insert: [Orchard.ContentManagement.Records.ContentItemVersionRecord][SQL: INSERT INTO Orchard_Framework_ContentItemVersionRecord (Number, Published, Latest, Data, ContentItemRecord_id) VALUES (?, ?, ?, ?, ?); select SCOPE_IDENTITY()] ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception: The wait operation timed out
--- End of inner exception stack trace ---
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource
1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at NHibernate.AdoNet.AbstractBatcher.ExecuteReader(IDbCommand cmd)
at NHibernate.Id.IdentityGenerator.InsertSelectDelegate.ExecuteAndExtract(IDbCommand insert, ISessionImplementor session)
at NHibernate.Id.Insert.AbstractReturningDelegate.PerformInsert(SqlCommandInfo insertSQL, ISessionImplementor session, IBinder binder)
--- End of inner exception stack trace ---
at NHibernate.Id.Insert.AbstractReturningDelegate.PerformInsert(SqlCommandInfo insertSQL, ISessionImplementor session, IBinder binder)
at NHibernate.Persister.Entity.AbstractEntityPersister.Insert(Object[] fields, Object obj, ISessionImplementor session)
at NHibernate.Action.EntityIdentityInsertAction.Execute()
at NHibernate.Engine.ActionQueue.Execute(IExecutable executable)
at NHibernate.Event.Default.AbstractSaveEventListener.PerformSaveOrReplicate(Object entity, EntityKey key, IEntityPersister persister, Boolean useIdentityColumn, Object anything, IEventSource source, Boolean requiresImmediateIdAccess)
at NHibernate.Event.Default.AbstractSaveEventListener.SaveWithGeneratedId(Object entity, String entityName, Object anything, IEventSource source, Boolean requiresImmediateIdAccess)
at NHibernate.Event.Default.DefaultSaveOrUpdateEventListener.SaveWithGeneratedOrRequestedId(SaveOrUpdateEvent event)
at NHibernate.Event.Default.DefaultSaveOrUpdateEventListener.EntityIsTransient(SaveOrUpdateEvent event)
at NHibernate.Event.Default.DefaultSaveOrUpdateEventListener.OnSaveOrUpdate(SaveOrUpdateEvent event)
at NHibernate.Impl.SessionImpl.FireSave(SaveOrUpdateEvent event)
at NHibernate.Impl.SessionImpl.Save(Object obj)
at Orchard.ContentManagement.DefaultContentManager.Create(ContentItem contentItem, VersionOptions options) in c:\VS2012\Orchard\1.6\vfb.be\src\Orchard\ContentManagement\DefaultContentManager.cs:line 487
at Orchard.Core.Contents.Controllers.AdminController.CreatePOST(String id, String returnUrl, Action1 conditionallyPublish) in c:\VS2012\Orchard\1.6\vfb.be\src\Orchard.Web\Core\Contents\Controllers\AdminController.cs:line 235
at lambda_method(Closure , ControllerBase , Object[] )
at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary
2 parameters)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary2 parameters)
at System.Web.Mvc.ControllerActionInvoker.<>c__DisplayClass13.<InvokeActionMethodWithFilters>b__10()
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func
1 continuation)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func1 continuation)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func
1 continuation)
2013-09-23 19:06:34,053 [51] NHibernate.Util.ADOExceptionReporter - Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
2013-09-23 19:06:34,053 [51] NHibernate.Util.ADOExceptionReporter - The wait operation timed out
2013-09-23 19:06:34,053 [51] NHibernate.Event.Default.AbstractFlushingEventListener - Could not synchronize database state with session
NHibernate.Exceptions.GenericADOException: could not execute batch command.[SQL: SQL not available] ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception: The wait operation timed out
--- End of inner exception stack trace ---
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior
Coordinator
Sep 24, 2013 at 5:25 PM
A simple insert should clearly not time out. Something is wrong with the connection to your database server. I think you may have a connection leak somewhere. I'd watch performance counters for that. If you see your connections steadily rise, then there's your problem.