NH3 and multiple web servers

Topics: Administration, Core, Customizing Orchard, General, Installing Orchard, Troubleshooting, Writing modules
Aug 9, 2012 at 3:12 PM


I was reading the meeting notes from the 18th and saw that there's a branch for NH3. Last time I tried out database caching it failed because of the use of multiple servers (the writes weren't being synced between the servers) and it caused all sorts of corruption to core Orchard tables in the DB.

Is it safe to use if I'm using multiple servers?

And as an aside, is there a way to configure NH to allow for write caching with multiple servers?

Aug 9, 2012 at 5:40 PM

Caching implementations can be done as module. Right now there is only one which is based on SysCache and thus won't work on farms. But this allows you to create one for Memcache for instance, which would work. It should be pretty straightforward, copy paste the SysCache module and add whatever is needed, like Memcache binaries and optionally some configuration settings. Note that you can also configure specific services using config/HostComponents.config if you don't want to have to expose it in the UI.

Aug 9, 2012 at 7:37 PM

But is the NH3 implementation safe to use on multiple servers? It's really really hard to test for issues :)

Aug 10, 2012 at 12:05 AM

I have blindness :) I see now where you say it won't work on farms. 

I've looked into this and would like to use the AppFabric cache implementation based on how easily it works with Azure.

I'll give it a try. Do you have any hints on the best way to test if it's working? 

Aug 10, 2012 at 12:53 AM

Ideally, you RDP to two instances, that you test locally. You update some content on one node, refresh the page on this node. Then you refresh the page on the second node, there should not be any request, though the content should be fresh.

To check how many requests, enable the Mini Profiler module which is preent by default with the 1.x branch.

Aug 10, 2012 at 4:16 PM


I think I've completed the module, but I'm struggling to enable it when using Azure dev instances. I can't enable SysCache either. Any clues?

I have also installed in on a live Azure instance and I don't think it's working :( Add fields to a type and some don't 'stick', and sometimes i get a page not found, which seems like classic not using one cache in a farm!

Aug 10, 2012 at 11:58 PM

Hmn. Odd things are happening here. Is your implementation of NH3 complete Sebastien?

What I'm seeing:

1. Install a fresh copy of Orchard, run it in the Azure emulator (same thing happens when uploading it to live site)

2. Everything works as it should

3. Add the new caching module, but don't enable it

4. Everything stops working. Can't add a new content item, can't add a new content type. Also, can't enable the module. The notification says that it has been enabled, but the module still says 'Enable' next to it. Also no access to the module via the debugger.

5. Even if I remove the module and make sure that its binaries and the Azure binaries are removed from the Orchard.Azure.Web bin folders and recycle the roles I can't add a new page (or any other content type). It seems like database writes are going into a black hole.

Is nhibernate picking up the fact that the module's dll implement's its caching interface and using it somehow? The Azure deployment is slightly different than the normal deployment as all the binaries are in one folder?

Any help greatly appreciated!

Aug 11, 2012 at 11:05 PM

Okay... I'm less worried now :)

I installed Orchard, fresh copy using SQL CE. (Mainly to codegen a new module).

I enabled CodeGen, and I was curious about my previous failures so then I enabled Mini Profiler. Then I added a page just title and menu and refreshed the browser a few times to see what the average time was. Around 200ms shown in the profiler output.

Next I enabled SysCache, and added a new page (just title and menu) and had a look on the browser. Still around 200ms, and nothing seemed different in the profiler. Didn't think much of it, but thought I'd ask a question here to see what I should be seeing there.

Then I thought, let's try a more DB intensive type, so I created a new type, called TestType, and I started adding boolean fields to it. First one I got the notification message saying that the field had been added, and I added the next but as I pressed submit I noticed that I didn't see any fields. When the page loaded there were no fields in the type! I tried adding some others (a content picker and a date time), but same result. Again, it's like writes are going into a black hole. But apparently not all of them? The type was created successfully, just adding fields seemed to stop things from working. I can create new items of the type, but can't add fields to the type.

My aim tonight was to put together a second level cache that uses the Azure Shared Cache (the names change too many times! This is the managed one that you set up in the dashboard and you get a hostname and port) to replace my failed implementation but now I'm not sure that mine has failed, as I'm getting the same issues with the SysCache module. So I'll put my other module on hold while this clears up.

I'm very busy this week, but if you need me to help with this I think I'm in an okay starting position now. I've done a load of reading about nhibernate over the past few days :)

Aug 11, 2012 at 11:08 PM

PS If all goes well and these bugs can be worked out I am planning on having a module for each of the Azure caching solutions ready for 1.6.

Both will be heavily based on this: http://www.sharpcoder.co.uk/post/2012/01/11/NHibernateCachesAppFabric-Documentation.aspx but I'll wrap a module around them.

Aug 12, 2012 at 12:06 PM

Nice looking forward to it!

Aug 13, 2012 at 5:43 PM

There might be some issues with the cache isolation level, I will try to tweek it to see how it goes, thanks for the repro.

There might also be some issues with the cached mappings. I'll keep you posted.

Aug 13, 2012 at 7:46 PM

With the latest changeset everything seems to work fine now. Enabling and disabling SysCache shows the expected behavior instantaneously.

Could you give it a new try ?


Aug 17, 2012 at 6:03 PM

Okay. I think I have it working when using a dedicated cache role and the memcached provider. it's not a simple module implementation unfortunately (but because it's azure I'm not that concerned by that fact). I will write up instructions and post a link here sometime soon and would appreciate some help testing. It's one of those things where it could be broken and I never know because I can't test two things at once :)

I've also had to recompile the memcached provider to work with orchard. I have sworn a lot regarding log4net and their keys.

Aug 19, 2012 at 1:42 PM

Small setback... I can't get a dedicated cache role to work using a live Azure sub. It seems to work fine using the emulator. I'm not sure if this is a VS 2012 issue or something else, but I can't even get it to work with a new project with only a cache role.

I tried again with a shimmed version of the on-role cache and I have the same issues that I've had before. When adding a type, getting: 


And when adding fields to an already created type they are not being added.


I'm using the 1.x branch from Friday.


Any suggestions as to what is going wrong? When I manually add a key to the localhost memcache on one role it is instantly visible on the other role's localhost memcache.

Right now I'm not sure if this is something to do with the caching provider implementation or the actual caching implementation, and I'm not sure how to find out!


Aug 20, 2012 at 3:09 PM

Can you share your code? I'd be happy to be another pair of eyes on it. I've been working on an implementation using Azure Preview Caching.

Aug 20, 2012 at 10:15 PM

TBH there isn't much to my code. I'm using the memcached second level cache provider from the nhibernate contrib project, and it uses memcachedotnet client. I've had to recompile both to work with log4net shipped with orchard.

The actual module is just a copy of the SysCache module with the provider swapped.

The only time I've had it working is when I have had 1 single instance of a cache running, but that has been using the emulator. I'm not convinced that the Azure Caching (Preview) works as advertised... 1. I can't deploy a dedicated cache role (I still haven't tried from VS 2010, but I've tried all sorts of combinations from VS2012) and 2. why does it work with one instance and not 2? I think that the two different instances are doing something in the background that is getting in the way somehow. The documentation is light, but it does mention that the shim uses different servers based on a hash of the key. I have a feeling that there's something not working 100% there.

I'll share what I have when I get to work tomorrow.

Aug 21, 2012 at 11:30 PM

Today's update... There was an issue with the pack command which was stopping the dedicated cache from working. I uploaded a new deployment with a single cache and I had the same issue as above (can't create content types). Just could not create content types, always received the 'Type has been created. Not Found' combo.

No cache was enabled. Which is surprising.

I couldn't enable the memcache module, as pressing 'enable' on any module was giving the message that the module had been enabled, but was not actually enabling the modules.

Will continue investigations...

Sorry about the lack of code, had a busy day with getting ready for a big release. I'll let you all know about it tomorrow when it's public.

Aug 21, 2012 at 11:32 PM

I've just checked the database, and the caching module is enabled in the Settings_ShellFeatureStateRecord table. It is showing as not enabled on the Modules page.

Aug 22, 2012 at 1:17 AM


So I think that the issue is with Orchard.

I now have a single cache, and things are still screwy with creating/modifying content types.

If I create a content type on one server, it is not visible on the other server until the cache expires or I create a new content type.

Same with fields. If I create a field on one server, only visible on the other server when I add a field on that server.

Seems like Orchard is caching more than it says on the tin. If nhibernate is querying the cache each request, then it would know that the new object was in place, but it doesn't seem to until it does a write of a similar type. I haven't properly investigated this, but this is where I'd start looking.

@sebastienros can you point me in a good direction to do further investigation here? You do say that adding fields/types etc should be done on a dev server, but we have done it on our live servers a number of times. I'd suggest as a first step a 'refresh' in the dashboard that refreshed any caches that may be in operation or if that didn't work then restarted the orchard session/shell.

PS I've upvoted some issues. I can easily repro the same behaviour described here *without* using the cache. Just a vanilla installation of orchard won't let me see a new content type on another server till restart the server/create a similar item on the server.

Aug 23, 2012 at 12:13 AM

I've been adding types a lot today :)

I think there's something not quite right in this method from ContentDefinitionManager.cs:106

        public void StoreTypeDefinition(ContentTypeDefinition contentTypeDefinition) {
            Apply(contentTypeDefinition, Acquire(contentTypeDefinition));

The Acquire is not getting anything back. What's really strange is that this is still in the same session that has added the content item. It's definitely not storing the type definition either.

What's the most odd is that I can't replicate this every time. Sometimes the site works really well. I added 10 types from each of the instances and they all worked. The next time (after restarting the emulator) I couldn't add one. Maybe restarting the instances is what makes it break. I'll investigate more tomorrow...

The worrying thing is that I can't create content either. I get a not found page. And when debugging the type is not coming back from the query.

The module is now here: https://bitbucket.org/Webmoco/azurememcached. There isn't much to it now, it autodiscovers any caches with endpoints named 'memcache_default'. To do that check the 'Enable Caching' checkbox in the role properties screen, and then create an internal endpoint. I've included the memcache from NHibernate.Caches.MemCached but changed it to autodiscover the endpoints (they change between live and the emulator). The bundled MemCached.ClientLibrary has also been rebuilt to use the version of log4net bundled with Orchard.

If anyone would like to investigate the strange issues please do :) I'm dying to get this working!

Aug 26, 2012 at 7:53 PM


It's not just my module. Misery loves company :)

There's a repro at this workitem http://orchard.codeplex.com/workitem/18985

At this point I'll say that my Distributed Signals module and the Azure On-Role 2nd level cache module are both working, but will do more testing tomorrow. If I'm happy with the tests then I'll polish and release them with 1.6. In fact the signals module can be released now if there isn't anything using non-string signals. The clock based signals are based on time anyway, and shouldn't need to be triggered across instances.