Synchronization problems between azure instances

Topics: Customizing Orchard, General, Troubleshooting
Oct 1, 2014 at 9:18 AM
Using Orchard 1.8.1, we deployed our site in an Azure cloud service composed of 2 role instances. Lately we ran into synchronization problems when activating a module: it seems the module is activated only on one of the two instances (pressing ctrl-F5 on the modules admin page confirms that, the module switches between activated and deactivated).

In detail, we shipped a new version of our application including a new module. When deployed, we activated the module in the admin, and discovered that the new module's pages resulted in http 404 errors half of the time (the behavior seems to altern between 404 and 200, especially when hitting ctrl-F5).

The problem disappears when we restart each azure instance, but reappears each time we activate a new module.

If it can help, here is a list of activated modules in our setup (most of them were activated by default with the standard orchard installation, the ones we activated manually are in bold if I remembered them correctly...). All others are not activated or are custom modules.

  • Alias
  • Autoroute
  • Blogs
  • Containers
  • ContentTypes
  • Pages
  • Projector
  • Publish later
  • Taxonomies
  • Tokens
  • Workflows
  • Forms
  • Fields
  • Microsoft Azure Media Storage
  • Microsoft Azure Services
Input Editor
  • Orchard.ContentPicker
  • TinyMce
  • Media Library
  • Media Processing
  • Email Messaging
  • Tags
  • Gallery
  • Packaging
  • Packaging commands
  • Microsoft Azure Database Cache
  • Output Cache
  • Lightweight scripting
  • Scripting
  • Comments
  • Feeds
  • Widgets
  • all Core modules
Oct 1, 2014 at 11:42 AM
That is how things work with Orchard + Azure Cloud Services.

By default there is no synchronization between Orchard instances.
There are some modules, and some projects which are trying to solve that problem.

My suggestion, scale down your instances to 1 before you make module or content definition changes.
Then scale back up.
Oct 1, 2014 at 12:40 PM
We use quite successfully to overcome this problem. We had to rewrite it to use a request filter instead of a background task to do tenant restarts, but we contributed that improvement back to the original source and I believe it's in there now.
Oct 1, 2014 at 1:59 PM
I thought there was a bug that prevented Orchard Distributed Events from handling shell events properly.
Oct 1, 2014 at 2:57 PM
There still is such a bug but it only prevents shell restarts to be invoked from background tasks; however the current implementation doesn't use background tasks for this any more.

In Orchard there are at least a dozen different features breaking when you go multi-node: some of these can be fixed by using a different built-in implementation for some services, like Azure Media Storage. Most of the issues are not yet solved OOTB, but almost all of the remaining ones to my knowledge are solved with modules of the Hosting Suite.
Oct 1, 2014 at 4:47 PM
Edited Oct 1, 2014 at 4:48 PM
Worth mentioning there is a new Orchard.MessageBus module in the feature/messagebus branch which does that too, but using a message bus instead of a poll so the changes are near real-time across instances. It currently works using SQL Server Service Broker or Redis. And it's extensible so you can reuse the message bus for other purposes. And if you use the Redis solution you can also use the same Redis server for your shared Output Cache store, all for only 8$ per month on Azure using the smallest instance (250 MB).
Oct 1, 2014 at 5:48 PM
Whenever Piedone mentions the 'Hosting Suite' ™ I hear lazer sound effects in my head. ;)
Oct 1, 2014 at 5:54 PM
Hosting Suite - The Imperial Edition: providing the enterprise-level features for making crucial systems redundant on a moon-sized space station. Because you don't want exhaust port hatches to go out of sync and keep open, right?
Hosting Suite - The Rebel Edition: if you don't distribute workload among hundreds of small fighters how are you going to fight moon-sized space stations? The right tool for coordinating the Alliance.

light sabres crossing