Web farm

Topics: Installing Orchard
Apr 19, 2011 at 7:32 AM

We're implementing Orchard in a web farm but we're having issues with Microsoft web farm framework replicating across the farm. Has anyone else implemented a web farm and please could you share your distribution method? Any ideas or suggestions welcome.

Coordinator
Apr 19, 2011 at 5:50 PM

We haven't tested it yet using the web farm framework, I will investigate ASAP.

Coordinator
Apr 20, 2011 at 12:35 AM

I have investigated, and right now there is a show stopper which is the replication of the Media folder. This can be easily solved by a module, which would save the content in a shared folder. Other solutions could be to use db or Azure or S3 to store those files. I will implement it as a module on the gallery.

However I would be interested in knowing what issues you faced with the web farm framework.

Thanks
Sebastien

Apr 27, 2011 at 4:56 AM

We have completed setup of Orchard CMS site on cluster. It was working fine till Session State configuration was not added. After adding session state server to the site, it started failing with following exception. I searched source code of Orchard but could not find dll mentioned in the error.

===============

Server Error in '/' Application.


Unable to serialize the session state. In 'StateServer' and 'SQLServer' mode, ASP.NET will serialize the session state objects, and as a result non-serializable objects or MarshalByRef objects are not permitted. The same restriction applies if similar serialization is done by the custom session state store in 'Custom' mode.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Web.HttpException: Unable to serialize the session state. In 'StateServer' and 'SQLServer' mode, ASP.NET will serialize the session state objects, and as a result non-serializable objects or MarshalByRef objects are not permitted. The same restriction applies if similar serialization is done by the custom session state store in 'Custom' mode.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.


Stack Trace:

 

[SerializationException: Type 'Szmyd.Orchard.Modules.Menu.Models.CounterRecord' in Assembly 'App_Web_n2kin3n2, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' is not marked as serializable.]

   System.Runtime.Serialization.FormatterServices.InternalGetSerializableMembers(RuntimeType type) +12472287

   System.Runtime.Serialization.FormatterServices.GetSerializableMembers(Type type, StreamingContext context) +361

   System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitMemberInfo() +413

   System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitSerialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter, SerializationBinder binder) +556

   System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Write(WriteObjectInfo objectInfo, NameInfo memberNameInfo, NameInfo typeNameInfo) +969

   System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteArrayMember(WriteObjectInfo objectInfo, NameInfo arrayElemTypeNameInfo, Object data) +516

   System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteArray(WriteObjectInfo objectInfo, NameInfo memberNameInfo, WriteObjectInfo memberObjectInfo) +1050

   System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Write(WriteObjectInfo objectInfo, NameInfo memberNameInfo, NameInfo typeNameInfo) +534

   System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize(Object graph, Header[] inHeaders, __BinaryWriter serWriter, Boolean fCheck) +1016

   System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph, Header[] headers, Boolean fCheck) +319

   System.Web.Util.AltSerialization.WriteValueToStream(Object value, BinaryWriter writer) +1469

 

[HttpException (0x80004005): Unable to serialize the session state. In 'StateServer' and 'SQLServer' mode, ASP.NET will serialize the session state objects, and as a result non-serializable objects or MarshalByRef objects are not permitted. The same restriction applies if similar serialization is done by the custom session state store in 'Custom' mode.]

   System.Web.Util.AltSerialization.WriteValueToStream(Object value, BinaryWriter writer) +2209943

   System.Web.SessionState.SessionStateItemCollection.WriteValueToStreamWithAssert(Object value, BinaryWriter writer) +49

   System.Web.SessionState.SessionStateItemCollection.Serialize(BinaryWriter writer) +721

   System.Web.SessionState.SessionStateUtility.Serialize(SessionStateStoreData item, Stream stream) +342

   System.Web.SessionState.SessionStateUtility.SerializeStoreData(SessionStateStoreData item, Int32 initialStreamSize, Byte[]& buf, Int32& length, Boolean compressionEnabled) +99

   System.Web.SessionState.OutOfProcSessionStateStore.SetAndReleaseItemExclusive(HttpContext context, String id, SessionStateStoreData item, Object lockId, Boolean newItem) +3628548

   System.Web.SessionState.SessionStateModule.OnReleaseState(Object source, EventArgs eventArgs) +929

   System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +80

   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +266

 


Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.1

 =============================

Apr 27, 2011 at 8:02 AM
ShirishSharma wrote:


Stack Trace:

 

[SerializationException: Type 'Szmyd.Orchard.Modules.Menu.Models.CounterRecord' in Assembly 'App_Web_n2kin3n2, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' is not marked as serializable.]

   

 

 

That first line of the stack trace tells you the problem.

The Advanced menu module is storing its view counter record in session but it's not serializable.

You should report this to Piotr (the module author) so he can look into it.

Apr 27, 2011 at 11:16 AM

Thanks for following it.

Do you have contact informaton f Piotr? Also do you mean the this is the only problem which makes Orchard to restrict to work under load balance?

Thanks and Regards,

Shirish Sharma

 

 

Apr 27, 2011 at 11:24 AM

Well I haven't tested and I don't know what else might be in session. But the fact that that's the first session object that throws an error, not anything in Orchard core, is a good sign.

The system is designed with deployment to cloud services like Azure in mind so there is plenty of support for distributed operation. This case is just one specific module problem tripping up session state serialization.

Piotr's website is http://www.szmyd.com.pl/ but you can also contact any module owner directly through their module page on the Gallery website.

Apr 27, 2011 at 1:26 PM

Thanks, this is helpful.

Some of the quick observations under web farm:

1. Some abnormal behaviour of login/logout: Some times it does not accept the credentials.

2. A very strange string appends on every url: http://mysite.com/(S(c5bjx33fx414oj0dblpkadtl)).

3. Some times suddenly "Access denied" message appears on the admin screen.

4. Not able to create a user from admin (DashBoard), after filling up the form and clicking on the button, nothing happens after postback.

5. Not able to change the Settings > User.

I am testing other things as well, but the above mentioned things are the part of core Orchard.

Could you please help here if you have any info regarding these?

Thanks and Regards,

Shirish Sharma

 

Coordinator
Apr 27, 2011 at 5:29 PM

About login and access denied problems, you do have a machine key defined in web.config that is the same on all the farms's servers, right? If not, that's what you need to fix.

The strange string comes from your activating cookieless sessions. I don't think you should do that.

Apr 28, 2011 at 1:07 PM

Problem of strange string is solved.

We put  same <MachineKey>  in web.config file, on all the web servers.

We have also used state server.

<sessionState mode="StateServer" stateConnectionString="tcpip=ServerName:42424" cookieless="false" timeout="20"></sessionState>

<machineKey validationKey="7A3EE509D0E868141FDFF4DEB5335359B2C310772077C275AB82AFB03264BDD4139ADD36D41259D721CBEA87DE04DA4D80C4AD22078D439932B999BDA5FAE37A" decryptionKey="0E63AEF50F81FF95F0EE695F262DD0488FD1A4F1991A92404662F7D053118B97" validation="SHA1" decryption="AES" />

Above two settings are put in the top level root web.config file.

While testing i found following problems:

1. Site does not allow to log in some times, after clicking on log in button, home page is shown as the nonlogged in user.

2. While logged in as the admin, and doing admin activities suddenly "Access is denied" error is shown

3. Some time while logging in, nothing happens, but on clicking the orchard logo, page refreshes and user is shown as the logged in.

4. While doing admin activities, some times following error is shown:

"

401 - Unauthorized: Access is denied due to invalid credentials.

"

Also a crash error is shown while loggin in, but it happens some times.

Do you have any insight for it?

Thanks and Regards,

Shirish Sharma

 

 

Apr 28, 2011 at 1:23 PM

follwoing error is shown while clicking on log in button:

 

Server Error in '/' Application.

Length of the data to decrypt is invalid.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Security.Cryptography.CryptographicException: Length of the data to decrypt is invalid.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.


Stack Trace:

[CryptographicException: Length of the data to decrypt is invalid.]
   System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) +12521039
   System.Security.Cryptography.CryptoStream.FlushFinalBlock() +53
   System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Boolean useValidationSymAlgo, Boolean useLegacyMode, IVType ivType) +331
   System.Web.Security.MachineKey.Decode(String encodedData, MachineKeyProtection protectionOption) +162
   System.Web.Helpers.AntiForgeryDataSerializer.Deserialize(String serializedToken) +90

[HttpAntiForgeryException (0x80004005): A required anti-forgery token was not supplied or was invalid.]
   System.Web.Helpers.AntiForgeryDataSerializer.Deserialize(String serializedToken) +676
   System.Web.Helpers.AntiForgeryWorker.Validate(HttpContextBase context, String salt) +194
   Orchard.Mvc.AntiForgery.AntiForgeryAuthorizationFilter.OnAuthorization(AuthorizationContext filterContext) in d:\TeamCity\Projects\Orchard-Default\src\Orchard\Mvc\AntiForgery\AntiForgeryAuthorizationFilter.cs:38
   System.Web.Mvc.ControllerActionInvoker.InvokeAuthorizationFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor) +156
   System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName) +691
   System.Web.Mvc.Controller.ExecuteCore() +162
   System.Web.Mvc.ControllerBase.Execute(RequestContext requestContext) +305
   System.Web.Mvc.<>c__DisplayClassb.<BeginProcessRequest>b__5() +62
   System.Web.Mvc.Async.<>c__DisplayClass1.<MakeVoidDelegate>b__0() +20
   Orchard.Mvc.Routes.HttpAsyncHandler.EndProcessRequest(IAsyncResult result) in d:\TeamCity\Projects\Orchard-Default\src\Orchard\Mvc\Routes\ShellRoute.cs:148
   System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +453
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +371



Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.1

<!-- [CryptographicException]: Length of the data to decrypt is invalid. at System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) at System.Security.Cryptography.CryptoStream.FlushFinalBlock() at System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Boolean useValidationSymAlgo, Boolean useLegacyMode, IVType ivType) at System.Web.Security.MachineKey.Decode(String encodedData, MachineKeyProtection protectionOption) at System.Web.Helpers.AntiForgeryDataSerializer.Deserialize(String serializedToken) [HttpAntiForgeryException]: A required anti-forgery token was not supplied or was invalid. at System.Web.Helpers.AntiForgeryDataSerializer.Deserialize(String serializedToken) at System.Web.Helpers.AntiForgeryWorker.Validate(HttpContextBase context, String salt) at Orchard.Mvc.AntiForgery.AntiForgeryAuthorizationFilter.OnAuthorization(AuthorizationContext filterContext) in d:\TeamCity\Projects\Orchard-Default\src\Orchard\Mvc\AntiForgery\AntiForgeryAuthorizationFilter.cs:line 38 at System.Web.Mvc.ControllerActionInvoker.InvokeAuthorizationFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor) at System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName) at System.Web.Mvc.Controller.ExecuteCore() at System.Web.Mvc.ControllerBase.Execute(RequestContext requestContext) at System.Web.Mvc.MvcHandler.<>c__DisplayClass6.<>c__DisplayClassb.<BeginProcessRequest>b__5() at System.Web.Mvc.Async.AsyncResultWrapper.<>c__DisplayClass1.<MakeVoidDelegate>b__0() at Orchard.Mvc.Routes.ShellRoute.HttpAsyncHandler.EndProcessRequest(IAsyncResult result) in d:\TeamCity\Projects\Orchard-Default\src\Orchard\Mvc\Routes\ShellRoute.cs:line 148 at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) --><!-- This error page might contain sensitive information because ASP.NET is configured to show verbose error messages using <customErrors mode="Off"/>. Consider using <customErrors mode="On"/> or <customErrors mode="RemoteOnly"/> in production environments.-->
Coordinator
Apr 28, 2011 at 6:11 PM

Something is still wrong in your farm setup. Could be one of the machines using a different key or a slightly different configuration. Hard to say from just these symptoms.

On a side note, you'll want to set-up a new machine key: you have effectively just given any hacker a way to hack into your site. Machine keys are confidential information that you should never expose publicly.

Apr 28, 2011 at 6:22 PM

Thanks for replying.

I also think the same way that something is wrong with my farm set up.

About machine key: it is of the development machine, and will not be exposed to the world. But yes you are right, i should not make it public. I will definately change it.

I will keep posting my testing results here, thank for following this thread.

Thanks and Regards,

Shirish Sharma

 

 

Apr 29, 2011 at 3:12 PM

One thing i noticed that "Enable 32 bit application" was set to "False". Now it is changed to "True". And one of the site is working fine in the cluster.

Other site is not working after setting app pool to "Enable 32 bit application= true", it works well when "Enable 32 bit application=true". I am still looking in to this issue.

I think the issues which i mentioned can be related to that.

Thanks and Regards,

Shirish Sharma

 

May 4, 2011 at 2:25 PM

We were able to fix the issue of running site under 32 bit mode. But we are still facing the issue of "Access denied" and other which i mentioned above.

Any body havinga pointer on it or your expirence with Orchard under web farm.

Thanks and Regards,

Shirish Sharma

 

Coordinator
May 5, 2011 at 12:08 AM
Edited May 5, 2011 at 5:29 PM

Quick recap on the current web farm situation:

* We did design everything with web farms in mind
* We only test on Azure at this moment
* Known farm hosting caveats are:
** media folder: we have an extensibility point so that this is replaceable, for example in Azure we use blob storage.
** Session state: the web farm needs to be configured for shared session state. This has the consequence that objects put in session by modules should be serializable.
** Machine key: the machine key must be set to  the same value on each machine of the farm.
** Cache invalidation: if a module relies on cache invalidation signals, those signals won't propagate across the farm and some servers may retain some stale entries longer than they should.

We very much want this to work as easily as possible. In the 2.0 time frame, we'd like to have better support out of the box for this type of situation and better guidance in documentation.

In the meantime, we'd like to help as much as possible. For that, we should probably switch to e-mail so we can take a closer look at what's going on. Please send me e-mail at bleroy at you know where, and we'll get this to a satisfactory resolution. 

Jan 14, 2012 at 8:08 PM

Is it more stuff to think of?

like Lucene, should it be runed on only one server and if not can the search index between the server risk to be different then? the index is saved in files right?