Multi-tenancy and restarts

Topics: Core
Apr 4, 2013 at 7:11 PM
Edited Apr 4, 2013 at 7:13 PM
Hi, quick question: why does enabling a feature in one tenant cause all other tenants to restart too?

I know how this happens, but why? Tenants are separate in terms of enabled features and the resulting Autofac container configuration, so there should be no need, or am I missing something? Don't need to say that this is a performance killer.

Here's the how, for the record:
  1. Enabling a feature changes a tenant's ShellDescriptor
  2. ShellDescriptorManager fires IShellSettingsManagerEventHandler .Saved or .Changed
  3. DefaultHostLocalRestart handles the event by touching hrestart.txt
  4. DefaultOrchardHost detects that and calls DisposeShellContext
  5. DisposeShellContext terminates and disposes all tenants
Developer
Apr 4, 2013 at 10:03 PM
The point of multi-tenancy is to have multiple independent Orchard sites running from the same application, i.e. you have as single application serving multiple websites. Now enabling or disabling features restarts this only application, that's why you see it for all tenants.
Also this is why there can be no module or theme installed for one tenant but not for the other ones (actually showing them or having them enabled is a different question).
Apr 5, 2013 at 7:33 AM
I know the purpose that multi-tenancy serves, we are using it quite extensively in our web application :)

Can you please explain what you meant by:
Now enabling or disabling features restarts this only application, that's why you see it for all tenants.
Enabling or disabling a feature restarts all tenants - that's the observation, and that's what's in the code, as described in my first post.

The real question is why do we need to restart all tenants when only one is changed? Why not only restart the one affected by the change? This is what DefaultOrchardHost does, tracking a list of tenants to restart. The hrestart.txt business that I described in something additional that happens separately. Is it intentional? If so, what purpose does it serve?

Also, I do understand that tenants share available modules and themes. I am asking specifically for enabled ones.
Coordinator
Apr 5, 2013 at 7:35 AM
It's the application that needs to be restarted. The tenants are within the application, so they restart at the same time it does.
Apr 5, 2013 at 7:42 AM
I am still going to be asking why like a 5-year old ;) Why does the entire application need to restart when enabling a feature in one tenant? Which in my understanding is just more-or-less about reconfiguring the shell Autofac container of that tenant?
Coordinator
Apr 5, 2013 at 7:45 AM
"just more-or-less"... famous last words.
Apr 5, 2013 at 7:48 AM
Edited Apr 5, 2013 at 7:49 AM
Well, to be honest, "more-or-less" was only a figure of speech. Studying and debugging the code I haven't found the "less" part.

Could you then perhaps explain what else there is to consider? I really want to understand the reasoning, the cases when this is important.
Coordinator
Apr 5, 2013 at 7:51 AM
Sébastien could probably give you a better explanation than I could, but I have a better idea: try to make it work without an application restart. Worst case, you'll find out exactly why first-hand without having to take our word for it. Best case, you fix it.
Apr 25, 2013 at 8:38 PM
Thank you TripleEmcoder for bringing up this issue. I have been having a terrible time with tenants dropping out because I am working on another tenant. You inspired me to dig into the code and see why/when the "DisposeShellContext" get's called/triggered. The answer will surprise you:
  • If you go into a Content Type on the Admin panel and mess with Placement that will trigger off an "IshellSettingsManagerEventHandler.Saved(ShellSettings settings)" which causes ALL of the tenants to restart
  • If you enable / disable a module then that will trigger off "IShellDescriptorManagerEventHandler.Changed(SHellDescriptor descriptor, string tenant)" which causes ALL of the tenants to restart
  • Here is the real fun one, if you enable / disable a module on one tenant and go to another tenant and click a link (anything to force a page reload) and keep clicking you can indefinitely trigger off the DisposeShellContext() and all of the tenants will remain down indefinitely - sounds like a bad denial of service scheme
  • The last one gets triggered from Orchard.Specs.Bindings.OrchardSiteFactory GivenIHaveEnabled(string name) method. It appears to be called when you install a new module and that makes sense to me that this would cause all of the tenants to be restarted since, at this time, all of the tenants share the same pool of modules
I think like you that in I do not see why the first two should require ALL of the tenants to be restarted. This bogs down the system and causes frustration. Also, if you are lucky enough to restart a tenant while the other tenants are busy, you can encounter bullet three above and find yourself indefinitely restarting. All this being said, I have put the fact that I don't think the tenants need to always be restarted to the test by making the following changes:
        void IShellSettingsManagerEventHandler.Saved(ShellSettings settings) {
            // EMS CHANGE - start
            //TouchFile();
            // EMS CHANGE - end
        }

        void IShellDescriptorManagerEventHandler.Changed(ShellDescriptor descriptor, string tenant) {
            // EMS CHANGE - start
            //TouchFile();
            // EMS CHANGE - end
        }
I have been using this for a bit and the only thing that is "broken" so far is that you cannot use "Placement" in the content type. To fix this you would need to uncomment the "IShellSettingsManagerEventHandler.Saved(ShellSettings settings)". I never use "Placement" so I am not sure if I will keep that commented out or not. Ideally I may see if I can just call that one tenant to be restarted instead of triggering off all the tenants to be restarted. Other than that, it is fantastic disabling a module in one tenant while working on another. I am only (so far) doing this in localhost. Would love if someone else would give this a try and confirm they too are not having any issues (other than the "Placement" one I mention above). Look forward to your comments.
Apr 25, 2013 at 9:21 PM
One point of clarification to my above observations, my "Placement" doesn't seem to be working with or without "IShellSettingsManagerEventHandler.Saved(ShellSettings settings)" being commented out or not (so something else is going on). In short, I cannot say whether or not it is actually needed but so far having good experiences having the above two lines commented out.
Apr 29, 2013 at 8:46 AM
We're also running with some of those restarts commented out and nothing bad seems to surface. We have also noticed the triggers around Content Types the studying the code, but we aren't using Orchard's Content Type system at all, so can't say anything about the influence on that.
Aug 19, 2013 at 7:25 PM
For anyone who stumbles upon this in the future, it seems that as of 1.7 those restarts (i.e. TouchFile()) are needed. I cannot actually determine why they are needed, but I know there are problems with modules enabling / disabling if they are commented out. Unfortunately, this seems to return multi-tenancy back to a slower state as, when one tenant enables a module, it bogs down other tenants who are trying to do a page load at the same time. Alternative solutions to improving this area are appreciated.
Feb 25 at 12:15 PM
jao28 wrote:
  • Here is the real fun one, if you enable / disable a module on one tenant and go to another tenant and click a link (anything to force a page reload) and keep clicking you can indefinitely trigger off the DisposeShellContext() and all of the tenants will remain down indefinitely - sounds like a bad denial of service scheme
I know it is a bit late, but have you reported this issue yet? As that seems to be a big problem when you do this on a live site that gets hits constantly!
Feb 25 at 8:36 PM
AimOrchard,

As far as I can recall, this was never submitted as an issue. Since there was a lot of work done to Orchard.Environment.Configuration.ShellSettingsManager and Orchard.Environment.DefaultOrchardHost on the 1.x branch I am not sure if this is still an issue or not (i.e. restarting shells when people are hitting it on a busy site). I will agree that if it is, it is a very problematic issue though I am hoping some of this is in the past. Can you share your observations? Are you on 1.x and have you noticed any difference? I just moved to 1.x last evening (on some live sites) so I will know more as time goes on.
Developer
Feb 25 at 8:46 PM
I don't think this is an issue on 1.x but please confirm.
Coordinator
Feb 25 at 10:38 PM
No more issues in 1.x, you can create tenants concurrently, without even interrupting other ones. Same thing for features.
Feb 25 at 11:21 PM
Now that just made my night! I have only tested it on features and on those it worked great! I had a number of sites that needed modules disabled so I went ahead and hit them with overlapping disable events and nothing locked up on me - very impressed. If it works this good with creating / enabling / disabling tenants Orchard has reached an all new high.
Feb 27 at 6:03 PM
Yeah, I've upgrade to 1.8 as well and multi-tenancy is working so much better. This is a big deal for us. Well done Sebastien!