Azure Cache Modules Error

Topics: Administration, General
Jun 3, 2014 at 9:42 PM
We have an Orchard install set up in an Azure Website (installed via Azure Gallery) and I've just enabled the Azure output and database caching modules. The cache is configured in Azure and the AppSettings added to the web.config.

Whilst it works for a while, I'm now frequently getting the following which I can only resolve by recycling the app pool (by way of a web.config change - don't have access to the management portal). This appears to only occur on some instances within the farm as whilst it my error for one it will work for someone else.

Any guidance on this is much appreciated as this is going to be a very high volume site so caching is going to be vital.
Server Error in '/' Application.An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was fullDescription: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full

Source Error: 

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace: 

[SocketException (0x2747): An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full]
   System.Net.Sockets.Socket.DoMultipleAddressConnectCallback(Object result, MultipleAddressConnectAsyncResult context) +1844178
   System.Net.Sockets.Socket.DoDnsCallback(IAsyncResult result, MultipleAddressConnectAsyncResult context) +34
   System.Net.Sockets.Socket.BeginConnect(String host, Int32 port, AsyncCallback requestCallback, Object state) +271
   Microsoft.ApplicationServer.Caching.SocketConnectionFactory.BeginConnect(String host, Int32 port, AsyncCallback callback, Object state, TimeSpan timeout) +469
   Microsoft.ApplicationServer.Caching.TcpClientChannelFactory.BeginConnect(String host, Int32 port, AsyncCallback callback, Object state) +119
   Microsoft.ApplicationServer.Caching.TcpClientChannel.ConnectAsync() +368
   Microsoft.ApplicationServer.Caching.SocketClientChannel.TryGetTcpChannelUntilTimeout(TcpClientChannel clientChannel, ITcpChannel& chnl) +41
   Microsoft.ApplicationServer.Caching.SocketClientChannel.Send(EndpointID endpoint, IVelocityRequestPacket packet) +188
   Microsoft.ApplicationServer.Caching.SocketClientChannel.Send(EndpointID endpoint, ICreateMessage message) +112
   Microsoft.ApplicationServer.Caching.SimpleSendReceiveModule.Send(EndpointID endpoint, RequestBody request, Int32 retries) +58
   Microsoft.ApplicationServer.Caching.SimpleSendReceiveModule.SendMsgAndWait(EndpointID endpt, RequestBody reqMsg, TimeSpan requestTimeout, IRequestTracker& tracker) +428
   Microsoft.ApplicationServer.Caching.SocketClientProtocol.SendMessage(EndpointID endPt, RequestBody request) +52
   Microsoft.ApplicationServer.Caching.DataCacheFactory.EstablishConnection(IEnumerable`1 servers, RequestBody request, Func`3 sendMessageDelegate, DataCacheReadyRetryPolicy retryPolicy) +253
   Microsoft.ApplicationServer.Caching.<>c__DisplayClass5.<Initialize>b__2(RequestBody req) +83
   Microsoft.ApplicationServer.Caching.SocketClientProtocol.SendReceive(IVelocityRequestPacket request, Func`2 delegate, EndpointID& destination) +34
   Microsoft.ApplicationServer.Caching.SocketClientProtocol.Initialize(IEnumerable`1 servers) +367
   Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCache(String cacheName, CreateNewCacheDelegate cacheCreationDelegate, DataCacheInitializationViaCopyDelegate initializeDelegate) +400
   Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCache(String cacheName) +61
   Orchard.Azure.Services.Caching.CacheClientConfiguration.CreateCache() +259
   Orchard.Azure.Services.Caching.Database.AzureCacheProvider.Start(IDictionary`2 properties) +200
   NHibernate.Impl.SessionFactoryImpl..ctor(Configuration cfg, IMapping mapping, Settings settings, EventListeners listeners) in c:\Users\sebros\Documents\My Projects\nhibernate-core\src\NHibernate\Impl\SessionFactoryImpl.cs:179
   NHibernate.Cfg.Configuration.BuildSessionFactory() in c:\Users\sebros\Documents\My Projects\nhibernate-core\src\NHibernate\Cfg\Configuration.cs:1269
   Orchard.Data.SessionFactoryHolder.BuildSessionFactory() in c:\Orchard\src\Orchard\Data\SessionFactoryHolder.cs:90
   Orchard.Data.SessionFactoryHolder.GetSessionFactory() in c:\Orchard\src\Orchard\Data\SessionFactoryHolder.cs:68
   Orchard.Data.SessionLocator.EnsureSession() in c:\Orchard\src\Orchard\Data\SessionLocator.cs:117
   Orchard.Data.SessionLocator.Demand() in c:\Orchard\src\Orchard\Data\SessionLocator.cs:42
   Orchard.Data.SessionLocator.For(Type entityType) in c:\Orchard\src\Orchard\Data\SessionLocator.cs:36
   Orchard.ContentManagement.DefaultContentManager.GetManyImplementation(QueryHints hints, Action`2 predicate) in c:\Orchard\src\Orchard\ContentManagement\DefaultContentManager.cs:326
   Orchard.ContentManagement.DefaultContentManager.Get(Int32 id, VersionOptions options, QueryHints hints) in c:\Orchard\src\Orchard\ContentManagement\DefaultContentManager.cs:160
   Orchard.ContentManagement.DefaultContentManager.Get(Int32 id, VersionOptions options) in c:\Orchard\src\Orchard\ContentManagement\DefaultContentManager.cs:131
   Orchard.ContentManagement.DefaultContentManager.Get(Int32 id) in c:\Orchard\src\Orchard\ContentManagement\DefaultContentManager.cs:127
   Orchard.Security.Providers.FormsAuthenticationService.GetAuthenticatedUser() in c:\Orchard\src\Orchard\Security\Providers\FormsAuthenticationService.cs:132
   Orchard.Security.CurrentUserWorkContext.<Get>b__0(WorkContext ctx) in c:\Orchard\src\Orchard\Security\CurrentUserWorkContext.cs:13
   Orchard.Environment.<>c__DisplayClass7`1.<FindResolverForState>b__5() in c:\Orchard\src\Orchard\Environment\WorkContextImplementation.cs:37
   Orchard.Environment.WorkContextImplementation.GetState(String name) in c:\Orchard\src\Orchard\Environment\WorkContextImplementation.cs:28
   Orchard.WorkContext.get_CurrentUser() in c:\Orchard\src\Orchard\WorkContext.cs:58
   Orchard.Security.Authorizer.Authorize(Permission permission, IContent content, LocalizedString message) in c:\Orchard\src\Orchard\Security\Authorizer.cs:72
   Orchard.Security.Authorizer.Authorize(Permission permission) in c:\Orchard\src\Orchard\Security\Authorizer.cs:60
   Orchard.Security.SecurityFilter.OnAuthorization(AuthorizationContext filterContext) in c:\Orchard\src\Orchard\Security\SecurityFilter.cs:24
   System.Web.Mvc.ControllerActionInvoker.InvokeAuthorizationFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor) +97
   System.Web.Mvc.Async.<>c__DisplayClass21.<BeginInvokeAction>b__19(AsyncCallback asyncCallback, Object asyncState) +743
   System.Web.Mvc.Async.WrappedAsyncResult`1.CallBeginDelegate(AsyncCallback callback, Object callbackState) +14
   System.Web.Mvc.Async.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout) +128
   System.Web.Mvc.Async.AsyncControllerActionInvoker.BeginInvokeAction(ControllerContext controllerContext, String actionName, AsyncCallback callback, Object state) +328
   System.Web.Mvc.Controller.<BeginExecuteCore>b__1c(AsyncCallback asyncCallback, Object asyncState, ExecuteCoreState innerState) +25
   System.Web.Mvc.Async.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState) +30
   System.Web.Mvc.Async.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout) +128
   System.Web.Mvc.Controller.BeginExecuteCore(AsyncCallback callback, Object state) +556
   System.Web.Mvc.Controller.<BeginExecute>b__14(AsyncCallback asyncCallback, Object callbackState, Controller controller) +18
   System.Web.Mvc.Async.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState) +20
   System.Web.Mvc.Async.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout) +128
   System.Web.Mvc.Controller.BeginExecute(RequestContext requestContext, AsyncCallback callback, Object state) +420
   System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.BeginExecute(RequestContext requestContext, AsyncCallback callback, Object state) +16
   System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__4(AsyncCallback asyncCallback, Object asyncState, ProcessRequestState innerState) +52
   System.Web.Mvc.Async.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState) +30
   System.Web.Mvc.Async.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout) +128
   System.Web.Mvc.MvcHandler.BeginProcessRequest(HttpContextBase httpContext, AsyncCallback callback, Object state) +369
   System.Web.Mvc.MvcHandler.BeginProcessRequest(HttpContext httpContext, AsyncCallback callback, Object state) +48
   System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData) +16
   Orchard.Mvc.Routes.HttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData) in c:\Orchard\src\Orchard\Mvc\Routes\ShellRoute.cs:152
   System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +301
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +155


Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.19453
That only occurs when accessing the admin, trying to view the front-end results in server error 404 (opposed to an orchard 404).
Jun 3, 2014 at 10:32 PM
Looking in the logs there's loads of the following entries:
2014-06-03 21:21:36,542 [308] NHibernate.Util.ADOExceptionReporter - (null) - Cannot insert the value NULL into column 'ContentItemRecord_id', table 'mywebsite.dbo.Orchard_Framework_ContentItemVersionRecord'; column does not allow nulls. UPDATE fails.
and
2014-06-03 21:21:36,542 [308] NHibernate.Event.Default.AbstractFlushingEventListener - (null) - Could not synchronize database state with session
Potentially related is we disabled the feeds and container module?
Jun 4, 2014 at 4:59 PM
Seemingly this was caused by a database locking issue, restarting the app pool only recovered things for so long but in the end killing all database connections (by means of exporting a backup from Azure) did the job. Fingers crossed this was a one-off!
Marked as answer by DanMoov2 on 6/4/2014 at 8:59 AM
Coordinator
Jun 12, 2014 at 6:19 PM
Actually this is not a database locking issue but the Azure Cache API which is faulty. Yesterday some blogs on weblogs.asp.net were down because of that, and I have a fix.
This is documented here: http://blogs.msdn.com/b/cie/archive/2014/04/29/cache-retry-fails-what-next.aspx

I am about to push this to 1.8.x
Jun 14, 2014 at 11:49 AM
Good to know thanks. Seemingly this hasn't caused us an issue since but during our investigations we did turn off high-availability on the cache service and disabled the azure database caching module (relying on output caching seems to be performant enough). Will keep an eye out for the 1.8.x update though.