Max Number of Multi-Tenants

Topics: Administration, Installing Orchard
Aug 16, 2011 at 4:52 PM

I was curious if anyone knew the maximum number of multi-tenants that can be created without greatly affecting performance.

I built/manage around 20 websites on a off-site hosting environment.  The sites use an CMS system that I built a long time ago.  I was thinking of rebuilding these sites using Orchard.  Each site is a different client, with their own domain name.  My thought was to load Orchard on the server and setup multi-tenants, one for each client.

Looking at the documentation it looks like it wouldn't be a problem to assign each domain to a tenant, but I did not see how many tenants can reasonably be added.

Will this work or should I use more instances of Orchard?  Especially if I get lucky and was able to add another 20 clients...

Thank you for any advice.

Aug 16, 2011 at 5:29 PM

Hi,

I don't think there is a limit to how many you can add and the amount your thinking of adding wouldn't affect performance.

I would suggest though, that if they are seperate clients to use seperate instances.  This way you'll have more flexability when it comes to enhancements etc. Also, if a client wants to move away from you and take there website then keeping them seperate will make this process a lot easier.

thanks

Steve

 

Aug 16, 2011 at 6:11 PM

That helps.  Thanks.

Aug 17, 2011 at 1:58 PM

I've got 27 tenants running in one installation.  Here are my current thoughts on multi-tenant scalability:

  • It's looking like memory is the main limiting factor on scalability.  With 27 tenants running, IIS is running right around 850MB of memory.  This is of course dependent on what modules you run, etc.
  • The start-up time of the application scales with the number of tenants you have running.  I don't have the scaling factor off the top of my head, but it can take over a minute or two to start-up an Orchard installation with 27 tenants.  I'm not taking advantage of the warmup module at this time though.
  • The default of performing background tasks every minute may be a bit too often as you add more tenants to the site (depending on what modules you're using).  The iteration time is currently hard-coded into SweepGenerator.cs in the Orchard.Framework project.
  • I'm using MySQL as my database.  By default, NHibernate opens 1 connection per tenant to the database.  I'm hitting what appears to be a provisioning / setup bug when I add and setup a new site that causes database connections to multiply from each tenant and also almost doubles IIS memory usage.  I haven't had time to root cause the issue yet, but my current best practice is to only provision a new tenant when I have time to recycle IIS after the new tenant setup.  Obviously this needs to be investigated soon.

I hope that helps, and I hope to report more to the community as I learn more.

-Steve

Coordinator
Aug 17, 2011 at 7:23 PM

The provisioning bug you are describing has been fixed in 1.2 ... and I hope you are not on 1.2 yet then ;)

We should also push some patches which will improve greatly memory usage. We found an issue on how FileSystemWatchers were used, and comsuming too much memory.

Whateber, very nice feedback, thanks.

Aug 17, 2011 at 8:51 PM

I'm on 1.2.41.  I've got to step through the setup of a new tenant to see where everything goes crazy to better point you in the right direction.  Hopefully I can get to it later tonight and report back (it is definitely reproducible at this point in my current environment.  Need to see if I can reproduce in a vanilla install).  I'm happy to test any patches that would reduce memory or help be a test case for you in larger tenant scenarios. 

Our plan is to scale the platform to host 70 or 80 sites by the end of the year and continue growth at that pace beyond that.  We'll be exploring the limits of the current code base and techniques to grow the platform (load balancing, etc.) along the way.

Has anyone explored only loading tenants into memory as they're first requested (instead of all at once on global app start)?  I could see a really nice use case for that if we load balance an orchard instance across multiple servers utilizing host name affinity (request for tenant www.example.com always goes to node 1, unless that node's unavailable, etc.).

Thanks!

Aug 18, 2011 at 1:30 AM

smeyers, this information is very helpful!  Any more information you can post about multi-tenant setup and administration is greatly appriciated.

Aug 18, 2011 at 7:34 PM
sebastienros wrote:

The provisioning bug you are describing has been fixed in 1.2 ... and I hope you are not on 1.2 yet then ;)

We should also push some patches which will improve greatly memory usage. We found an issue on how FileSystemWatchers were used, and comsuming too much memory.

Whateber, very nice feedback, thanks.

Sebastian,

I'm able to replicate the memory spike and database connection proliferation issue upon tenant add in both my development and production environments (Orchard 1.2.41).  The problem is happening during the AddPOST method of the AdminController in Orchard.Multitenancy.  I'm guessing that it's something triggered (or NOT triggered) by DisposeShellContext() in DefaultOrchardHost.cs.  Is it necessary to dispose of all running shells when a new shell is added?  Is that what is really going on here, and if so, is it possible that we're duplicating shells in memory (not removing the old ones, and reloading all the contexts again after dispose)?  That could explain the memory spike and the multiplication of database connections... 

I'd like to try to replicate this with a pure vanilla install to give you a repro, but in the mean time, any thoughts?

Thanks!

-Steve

Coordinator
Aug 18, 2011 at 7:43 PM

Yes, it has to recreate all shells. By design.

The memory leak could be demonstrated and analyzed using JetBrain's memory profiler. You can open a bug for it, and if you have time, check what classes are taking so much memory. You can do a memory dump too.

Aug 18, 2011 at 11:24 PM

Hey Steve,

Are you running a different database for each tenant?

Do you plan to upgrade all of your tenants at the same time for a new version of orchard?

Have you many modules over and above the modules provided by the install?

Are you running on a shared host or are you running your own server?

Regards

Aug 20, 2011 at 9:38 PM

On databases:

Yes, running a database for each tenant.  For this particular implementation, we decided that it was worth the headache so that we can do tenant-level backup and restores without having to worry about row-level backups, etc.  Plus, it'll be a bit easier to scale out MySQL across different servers without having to get into clustering for the near term.  We modified the Orchard.Multitenancy module to provision a new database and database user for the tenant when we add a tenant.  I'm REALLY looking forward to the cloud database market to mature and allow me to think less about this piece of the puzzle.

On upgrades:

Yes, we'd like to push new versions of code out to all clients at once.  In order to do that, we're looking to build a few administrative views and services that will allow us to view the status, versions, etc. of modules on each tenant as well as being able to update all tenants at once for particular module.  However, we see the Modules feature as a great way to differentiate service offerings from one tenant to another, so we don't expose that functionality to the end client.

On modules:

I'm running around 10 extra modules to handle things like contact forms, twitter feeds, calendaring, more theming options, etc. for now.  I can see that growing a bit as we add new features to the offering.

On hosting:

We're currently using Rackspace's Cloud Servers offering so that I can spin resources up and down as I need them.  So, it's not a traditional shared host where I get a small amount of resources, but I am sharing the hardware right now with other folks.  Our plans are to move to a Rackspace Private Cloud when the workload and business case supports it.

Hope that helps!
-Steve

Aug 22, 2011 at 5:12 PM

I've created an issue to track the provisioning bug I'm running into as well as taken memory dumps for someone who can analyize:

http://orchard.codeplex.com/workitem/18093

Aug 23, 2011 at 11:26 AM
We modified the Orchard.Multitenancy module to provision a new database and database user for the tenant when we add a tenant.

Would you mind sharing that code, maybe as an alternative Multi-Tenancy module? We're looking into doing the exact same thing for pretty much the same reasons.

Regards, Oliver
Aug 23, 2011 at 4:43 PM
_oliver_ wrote:
We modified the Orchard.Multitenancy module to provision a new database and database user for the tenant when we add a tenant.
Would you mind sharing that code, maybe as an alternative Multi-Tenancy module? We're looking into doing the exact same thing for pretty much the same reasons.
Regards, Oliver

Hi Oliver,

We've just got some prototype code running now, so it's not mature enough to become a module just yet.  I'm happy to share our current method though to get you started. 

Before we get started, some disclaimers!

  1. This is prototype code, not production, so it's probably not something you want to copy and paste into your solution.
  2. It's a good idea to copy any Orchard modules that you're going to be modifying and work on them instead of working on the standard distribution ones.
  3. Now that we've got this running, we really need to sit back and look at the entire deployment strategy to see if this is the right direction or not.

First, we modified the TenantService.cs file to create the database, create the database user and generate the connection string.  To do this, we modified the CreateTenant method as follows:

 

public void CreateTenant(ShellSettings settings) {
            _shellSettingsManager.SaveSettings(createMySqlTenantDatabase(settings));
        }

private ShellSettings createMySqlTenantDatabase(ShellSettings settings)
        {
            using (new TransactionScope(TransactionScopeOption.Suppress))
            {
                string tenantDatabaseName = "t_" + settings.Name.ToLowerInvariant();

                // First check to see if the database exists.  We want to fail safe and throw an error instead of deleting or overwriting it.
                string sql = "SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = '" + tenantDatabaseName + "';";

                ISession session = _sessionFactoryHolder.GetSessionFactory().OpenSession();

                IQuery sqlQuery = session.CreateSQLQuery(sql);

                if (sqlQuery.List().Count > 0)
                {
                    throw new ConfigurationException("The database " + tenantDatabaseName + " already exists.  Please select a different tenant name or remove the existing database.");
                }

                // Create the database
                sql = "CREATE DATABASE " + tenantDatabaseName + ";";
                sqlQuery = session.CreateSQLQuery(sql);
                sqlQuery.ExecuteUpdate();

                // Create a new user with permissions only to the new tenant database.

                //TODO: bullet proof database user name creation to ensure uniqueness (should check db for exisiting name and increment if necessary)

                string tenantDbUsername = settings.Name.Length > 16 ? settings.Name.ToLowerInvariant().Substring(0, 16) : settings.Name.ToLowerInvariant();
                string tenantDbUserPassword = PasswordService.Generate();

                sql = string.Format("CREATE USER '{0}'@'localhost' IDENTIFIED BY '{1}';", tenantDbUsername, tenantDbUserPassword);
                sqlQuery = session.CreateSQLQuery(sql);
                sqlQuery.ExecuteUpdate();

                sql = string.Format("GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,ALTER ON {0}.* TO '{1}'@'localhost';", tenantDatabaseName, tenantDbUsername);
                sqlQuery = session.CreateSQLQuery(sql);
                sqlQuery.ExecuteUpdate();

                sql = string.Format("CREATE USER '{0}'@'%' IDENTIFIED BY '{1}';", tenantDbUsername, tenantDbUserPassword);
                sqlQuery = session.CreateSQLQuery(sql);
                sqlQuery.ExecuteUpdate();

                sql = string.Format("GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,ALTER ON {0}.* TO '{1}'@'%';", tenantDatabaseName, tenantDbUsername);
                sqlQuery = session.CreateSQLQuery(sql);
                sqlQuery.ExecuteUpdate();

                sql = "FLUSH PRIVILEGES;";
                sqlQuery = session.CreateSQLQuery(sql);
                sqlQuery.ExecuteUpdate();

                settings.DataProvider = "MySql";
                settings.DataConnectionString = string.Format("Server={0};Database={1};Uid={2};Pwd={3};",
                                                    _dbHost,
                                                    tenantDatabaseName,
                                                    tenantDbUsername,
                                                    tenantDbUserPassword);

                session.Close();
            }
            return settings;
 }

Key for us here was learning about "using (new TransactionScope(TransactionScopeOption.Suppress))".  Without that, we had a lot of trouble trying to open our own connection to the database.  I learned this in this thread.

We also simplified the Add.cshtml file in Orchard.Multitenancy to:

@model Orchard.MultiTenancy.ViewModels.TenantAddViewModel

@{ Layout.Title = T("Add New Tenant").ToString(); }

@using (Html.BeginFormAntiForgeryPost()) { 
    @Html.ValidationSummary()
    <fieldset>
        <div>
            <label for="@Html.FieldIdFor(m => m.Name)">@T("Name")</label>
            @Html.TextBoxFor(m => m.Name, new { @class = "text" })
        </div>        
    </fieldset>
    <fieldset>
        <button class="primaryAction" type="submit">@T("Save")</button>
    </fieldset>
 }

The last thing we did was simplify the Orchard.Setup views so that we only have to add a site title when initially setting up a tenant (no need for any of the database stuff).

I hope that helps a little with organizing your thoughts.  Now if we can just get around this bug and figure out how to not impact other tenants as we add new tenants, we'll be golden.

-Steve

 

 

Coordinator
Aug 23, 2011 at 5:25 PM

Wouldn't it be insteresting to create a module which would plug on a specific event of the multi-tenancy, and create the db on purpose ? Or to make the multi-tenancy extensible through a driver ?

Aug 24, 2011 at 11:27 AM
Hi Steve,

thanks a lot for sharing this with us. We're not quite there yet with our own development so I'm afraid I can't give any feedback right now (except for Thank You!) but once we come around this corner I'll be updating this thread with any ideas/findings we make.

Cheers, Oliver
Aug 31, 2011 at 10:32 PM
Edited Aug 31, 2011 at 11:52 PM
smeyers wrote:

I've got 27 tenants running in one installation.  Here are my current thoughts on multi-tenant scalability:

  • <items deleted>

I hope that helps, and I hope to report more to the community as I learn more.

-Steve

Steve,

Can you elaborate on the hardware requirements and the Internet connection speed/bandwidth consumption based on your experience?

Thanks,

Mario

Oct 5, 2011 at 8:48 AM

Just created site with 47 tenans. First start - about 5 minutes. "Connecton pool limit exceeded" error after installing module.

Coordinator
Oct 5, 2011 at 8:54 AM

Is that on 1.3?

Oct 12, 2011 at 9:03 AM

Ouch! I had no idea the Orchard multi-tenant module would be hitting the limit of tenants it can support at such a low number.  Are these websites all quite busy traffic wise?

We're hoping to use orchard (and are quite a long way along the process) to support more like hundreds of clients if not more (although they are all likely to be low traffic sites).

I'm not sure where we'll go from here.  Does it scale better if you create a new orchard site per tenant rather than using multi-tenancy (I understand periodic warmup will probably be required for each our clients as they have low traffic).

Oct 12, 2011 at 2:25 PM

@grokmonsieur, I wouldn't throw in the towel just yet.  I've been working with Sebastien over the last couple of months on getting over multitenancy scaling issues, and Sebastien has made huge strides in updating performance and scalability in this area.  I've got 55 low traffic sites running in one multitenant instance on Orchard 1.2.41 with some additional scalability patches.  We're averaging around 800MB of memory usage (although Sebastien still thinks we have room to improve there) and I haven't seen any database issues since Sebastien's patches.  We're using the latest MySql Connector for .Net and a slightly modified MySql module from the Orchard gallery.

All the patches I'm referring to look to be in Orchard 1.3, and I'm in the process now of moving our code base over to that.  Hopefully I'll have some additional information by the end of the day tomorrow.  If you're not on 1.3 yet, get there as soon as you can for scaling purposes.

I'm guessing that the overhead of setting up a separate Orchard site for each of you customers wouldn't buy you that much in terms of scalability.  If we all work together, I'm confident we can get to hundreds of tenants on the same node.  My guess is that the physical memory of your server will be the limiting factor at this point.

I'm happy to share whatever other experiences I have that would help the community grow from a multitenancy point of view.

-Steve

Oct 12, 2011 at 2:35 PM

47 tenants on 1.2 version. Version 1.3 was released during tests :)

Does 1.3 have improvements in multi tenancy?

Oct 12, 2011 at 2:39 PM

@grokmonsieur, there are no traffic. Site just launched. IIS process consumed about 500MB RAM.

Oct 12, 2011 at 4:14 PM
gandjustas wrote:

47 tenants on 1.2 version. Version 1.3 was released during tests :)

Does 1.3 have improvements in multi tenancy?

Big, big improvements in 1.3.  I wasn't able to scale past 33 or so clients without it due to some memory leaks and dependency injection configurations on 1.2.41.  I'm not fully on 1.3 just yet, but I have the post 1.2.41 multitenancy patches installed in my current environment.

Oct 13, 2011 at 9:49 AM
Edited Oct 13, 2011 at 12:47 PM

If there are still some memory leaks in 1.3 then would frequent app pool recycling help a little.  It would be good if anyone running multi-tenant could share their experience of how frequently they choose to recycle currently.

It sounds like if we're happy to run it on a server that has at least a reasonable amount of ram (say 4GB to 8GB) which we're bound to need for multi-tenancy then we would be able to support (very, very roughly!) 250 to 500 clients currently on v.1.3 and potentially more as improvements are made.

Out of interest, we've only talked about ram.  I imagine a lot of caching goes on in orchard which would cause ram usage to be quite but cpu usage should be lower as a lot of it should be cached so most of the time it's not really doing any computations.  For anyone that is already running their multi-tenant project live / has got on to perf testing by now, is this assumption correct?

I'll say this much though... even in debug mode it seems a lot faster in v1.3!

Oct 13, 2011 at 3:08 PM

The leaks I was referring to were seen during the addition and setup of new tenants.  I haven't run into any memory leaks on running tenants.  We only recycle our app pool once a day (around 3:00 AM when no one is on the platform).  We have a availability service pinging the site every 15 minutes, so the app pool is recycled and reloaded before anyone hits the site.  That way, no one sees the start-up penalty (we're still doing dynamic compilation of the modules and themes until we can find time to pre-compile everything).

The CPU on the site is pretty minimal most of the time.  The CPU gets hammered during a new tenant set up though (we've got a decent sized default recipe).  Your mileage may vary.

Oct 13, 2011 at 5:10 PM

Thanks, I've only been doing some very simplistic performance tests using perfmon today but I think I can see the same as you.

The CPU was hammered on tenant setup -- our recipe is also a fair size I suppose with the standard features being enabled plus 3 others of our own, addition of a couple of pages and layers and spitting some widgets into those layers.

I wasn't really trying to test it specifically at the time but I think it was still consuming extra ram after the tenants were set up and after recycling the app pool (and then hitting each of the tenant sites) the usage seem less.  It also seems to drop back to around the original amount of RAM that was required to run a single site after about 10 minutes - not sure how that happened and hitting the pages on the tenant sites didn't seem to budge it at all... curious (maybe some IIS caching?...).

I'm going to up it to about 50 clients tomorrow and then up to 100 and compare the two (if I don't cause everyone to raise their arms in anger before I get that far - this dev box isn't dedicated to stress testing).  So far though, I'm less concerned than I was and at a rough guess of 15 to 20MB per client site it's not exactly looking expensive currently.  Don't suppose you have a script to create tenants you want to share do you ;-)

Oct 13, 2011 at 5:48 PM

Right now, I'm adding tenants one at a time through the admin, so I don't have anything handy.  But, check out the Orchard.exe command line tool.  There are existing commands to create a new tenant and set that tenant up (look in the commands folder of both the Orchard.Multitenancy and Orchard.Setup projects).  You could probably work up a quick batch file to bust out whatever you need from those.

Oct 14, 2011 at 8:26 AM

Sounds like a plan!

Coordinator
Oct 14, 2011 at 4:29 PM

Here is a little piece of the file I am using to drive the performance tests on multi-tenancy

C:\> orchard.exe

Then 

setup /SiteName:Default /AdminUsername:admin /AdminPassword:demo123! /DatabaseProvider:SqlCe /DatabaseConnectionString:feature enable Orchard.MultiTenancy

tenant add a /Host:a.127-0-0-1.org.uk
setup /t:a /SiteName:a /AdminUsername:admin /AdminPassword:password123! /DatabaseProvider:SqlCe /DatabaseConnectionString:

tenant add b /Host:b.127-0-0-1.org.uk
setup /t:b /SiteName:b /AdminUsername:admin /AdminPassword:password123! /DatabaseProvider:SqlCe /DatabaseConnectionString:

...

You can even define which recipe to use in the setup commande, using /Recipe:recipe-name

Oct 14, 2011 at 5:40 PM

Nice discussion, guys. Sorry, I have been away, was busy. I have to catch up.

Oct 17, 2011 at 11:09 AM

Hi Sebastien,

I'm not seeing the "setup" command in v1.3 orchard.exe?

Coordinator
Oct 17, 2011 at 5:39 PM

It's in though ;) But you the Setup module is disabled once the tenant is "setup", which means it can only be used before the setup. Be sure that the tenant you are calling the command on is still not configured.

Oct 18, 2011 at 8:32 AM
Edited Oct 18, 2011 at 8:37 AM

Ha - that's why then :-).  Well my tests have had to be incredibly rough because the testing environment here is so useless for performance testing that I've had to use my dev machine which is clearly a bad test and meant I was twiddling my thumbs but it gets me a ballpark on ram usage.  

Anyhow, for me the first client costs roughly 256MB (continuously in use) and each 50 thereafter costs about the same again (so each subsequent client is about 5 to 6MB).  This however has to include running selenium webdriver and firefox so please take it with a large dosage of salt.

You'd have to add your hosts entries yourself to use the following bat but here's a snippet of what I've been using:

 

ECHO OFF

cd c:\windows\system32\inetsrv
FOR /L %%G IN (1,1,50) DO appcmd set site /site.name: orchardstresstest /+bindings.[protocol='http',bindingInformation='*:80:%%G.MYDOMAINNAME']

cd C:\Websites\Orchard\Orchard.Web.1.3.9_\Orchard\bin
FOR /L %%G IN (1,1,50) DO orchard tenant add Tenant%%G /Host:%%G.MYDOMAINNAME
REM -- didn't work for me but might just be me -- FOR /L %%G IN (1,1,50) DO orchard setup /t:tenant%%G /SiteName:tenant%%G /AdminUsername:admin /AdminPassword:password /DatabaseProvider:SqlServer /DatabaseConnectionString:MYCONN_STR

ECHO "Press any key to continue"
pause

 

To set up the tenants I recorded my actions in selenium, exported the test case to c# and used selenium webdriver (with a bit of extra code / tweeking) to loop over the 50 tenants.

I then recorded hitting all pages on a tenant in selenium and exported that to c# and used webdriver again to hit all tenants in a single firefox browser instance.

I should mention that this is v1.3.9 and I was still having to recycle the app pool every few tenants because of the apparent memory leak.  For now, when we go live, we'll probably need to re-assess typical memory usage and then set the app pool to recycle at a set memory limit and at least every night at an appropriate time.

Thanks for everyone's help.  Hope that helps someone.

Developer
Mar 24, 2012 at 11:02 PM

Pretty much awesome thread! Has anyone new experience with hosting many tenants, perhaps with 1.4?

Jun 1, 2012 at 2:39 AM
Edited Jun 1, 2012 at 2:40 AM

I'm sporting 33 tenants at the moment on 1.4.2, on my way to 50 tenants over the next couple weeks. Using about 560mb so far and no issues in sight :)

Coordinator
Jun 1, 2012 at 5:48 AM

would you give us the website urls or a description ? maybe in private ?

Jun 13, 2012 at 2:11 PM

Great thread! I'm in the very early stages of discovering Orchard but have come from a DNN environment where I've had many Portals/Child-portals in one implementation. I'm encouraged by the recognised performance and scalability or Orchard to-date.

If anyone can give examples of high-traffic Orchard sites out there it'd be well received by the whole community (and beyond!).

Jun 15, 2012 at 12:04 AM

What do you consider high-traffic?  > 1000 per day?

Jun 28, 2012 at 1:57 PM

Yes... Anything like that I'd consider high-traffic for these early days of Orchard.