Overusing resources

Topics: Troubleshooting
Mar 26, 2011 at 6:37 PM

The issue with excessive resource usage on the server has finally come to a head.  GoDaddy has now put a cap on my utilization and restarts my app pool whenever it gets hit.  My site is effectively dead now:

==========================

Support Staff Response

Dear Sir/Madam,

Regarding your hosting account ariankulp.com:

It has come to our attention that your ariankulp.com is causing an overload of the shared resources on this server. This, in turn, affects the server performance for other customers.We have determined that your application pool is using up to 1GB of memory per node. We have set your Private Memory Limit to recycle at 200000KB. Please review your code to determine what is causing this over-utilization. Continued over-utilization of the shared resources of the server may  result in action including, but not limited to, the disabling of the  offending services/processes or the suspension of services for that  domain.

Unfortunately, we are not able to troubleshoot custom scripting and therefore, we are not able to provide specific solutions.We were not able to determine the exact files that are causing this issue.

If you have questions about this matter, you may contact us by replying to this email or by calling 480-505-8871.

We appreciate your cooperation in this matter.

Hosting Support Team

========================

I can't get into any page of the site now.  Are there settings I can make that will reduce utilization?  Is there a module that might be the culprit (I can provide my list if there aren't ones you know off the top of your head)?  Are there log files in any of the folders on the site?  I can't seem to find any.  I just don't even know where to start.

Mar 26, 2011 at 7:42 PM

Which modules are you using and which version of Orchard? There are tonnes of modules, I doubt anybody would be able to go through every single one to look for problems!

Coordinator
Mar 26, 2011 at 10:01 PM

1GB is an impressive amount of memory... Could you please provide as many details as possible about your setup? Installed modules, approximate number of items in your database, etc.?

Another thing you could try is to disable some module to see if it makes a difference. My guess would be that one of the modules you have is leaking memory like crazy. But it's just a guess.

If we don't find anything obvious, we might even try to run your site locally and try to figure out what happens.

Mar 27, 2011 at 12:06 AM

Can I disable without being able to access the site?  I know I can just remove the folders in the Modules folder, but I don't want to destabilize anything.  When I attempt to hit my site now I simply get a timeout.  These are my modules:

  • Contrib.AntiSpam
  • Contrib.ChoiceList
  • Contrib.DateTimeField
  • Contrib.FeedsAggregator
  • Contrib.Hyperlink
  • Contrib.ImageField
  • Contrib.KeepAlive
  • Contrib.Stars
  • Contrib.Taxonomies
  • Contrib.Voting
  • CyberStride.Contacts
  • EmailObfuscator
  • EWSNoggin.ContactUs
  • Iroo.PackageManager
  • Lucene
  • NGM.BlogML
  • Orchard.Blogs
  • Orchard.Comments
  • Orchard.ContentTypes
  • Orchard.Email
  • Orchard.Indexing
  • Orchard.jQuery
  • Orchard.Lists
  • Orchard.Media
  • Orchard.Messaging
  • Orchard.Migrations
  • Orchard.Modules
  • Orchard.Packaging
  • Orchard.Pages
  • Orchard.PublishLater
  • Orchard.Roles
  • Orchard.Scripting
  • Orchard.Search
  • Orchard.Setup
  • Orchard.Tags
  • Orchard.Themes
  • Orchard.Users
  • Orchard.Widgets
  • SH.GoogleAnalytics
  • SH.Robots
  • Themes
  • TinyMce
  • Vandelay.Favicon
  • Vandelay.TagCloud
Mar 27, 2011 at 1:51 AM

I would suggest first backing up the entire site, then start deleting module folders. You can always reupload if anything properly breaks (but it shouldn't). In terms of "destabilizing anything" it sounds like things aren't exactly stable at the minute!

From looking over that list, you've got quite a few modules installed. The more modules you have, the more resources the application will use, because there are more view override locations to look in etc. So you should probably trim things down a bit and see how things go, then re-enable things one by one and see how stability goes.

I'd also specifically suggest removing the following at least to begin with;

- Contrib.FeedsAggregator

I don't know how many feeds you're aggregating but that could be a lot of background processing.

- Contrib.Taxonomies

I've noticed the Taxonomies Widget causing errors on my installation, I think it's related to uncategorized content but I'm not sure.

- CyberStride.Contacts

- EWSNoggin.ContactUs

You don't need two contact page modules - remove one of these.

- Iroo.PackageManager

This could be doing some heavy processing (only a guess)

- Lucene / Orchard.Search / Orchard.Indexing

Depends on how much content you've got. Search indexing can be a pretty complex process.

- Contrib.KeepAlive

How often are you pinging the site? Since it's having trouble serving even one request, I don't think you need to be giving it even more work at this stage.

On top of that, start disabling *anything else* non-essential. The idea here is to get your installation to a working state, and then add stuff one by one.

Another way you could try is to back everything up, and then start with a *clean* Orchard installation, and work up from there.

Finally - you didn't say what version of Orchard you have. Is it 1.0.20, or are you using a recent dev build?

Mar 27, 2011 at 3:53 AM

It's the official v1 release.  I'll start disabling modules and see if I see anything at all (even an error would be a start!).  Thanks for looking through this!

Mar 27, 2011 at 7:57 AM

I've gotten the site responding again.  Incredibly long responses with timeouts still, but that's what's been.  :-(

I disabled about five modules (had to delete them since I couldn't actually disable with every site page erroring out!).  Now we'll see what I can re-enable.  I added around ten modules, and the rest are default ones.  Is this really a big installation?  I had the impression that the strength of the platform was mixing and matching lots of modules.  It might be good to be able to get resource info on modules (like Chrome does with extensions).  I hope we can get some fixes for perf soon!

Coordinator
Mar 27, 2011 at 8:05 AM

No, we're just trying to determine what's wrong here: if there is a module leaking memory, we need to find which one.

Mar 27, 2011 at 6:00 PM

Exactly which modules have you disabled so far?

From having a look at your site, what I immediately notice is that you have quite a lot of content and tags. I'm justing looking here for things that might differentiate it from other Orchard sites that are running well. Could be scalability problems with specific modules or Orchard components. For instance, search and indexing could be exponentially consuming resources as content grows. Just ideas.

Are you still using the FeedsAggregator? You didn't respond to my question as to how many feeds are being aggregated. (And also are they also being indexed?) Remember feeds can potentially be huge volumes of data

I'd strongly suggest that you bare down to a totally minimal install; I mean remove *everything* - Voting, Stars, Tag Cloud, Search. Then start adding things in one by one until you hit problems.

Finally, I noticed something weird - the "Older" link at the bottom of your home page has the following URL:

http://ariankulp.com/?2536d891-9216-4357-a98d-ff6c0265f91b=System.Web.Mvc.DictionaryValueProvider%601[System.Object]&page=2

That looks pretty unusual and if it's being generated by the Blogs module that needs looking into; not great to be exposing MVC class names in what should be a straightforward paging URL!

BTW: I wasn't suggesting that Orchard shouldn't be able to handle that many modules; it's just that you have there over 40 separate components all engaged in some variety of crosstalk, and since many of them are unofficial / beta / early versions of software there is bound to be some friction!

Mar 28, 2011 at 2:56 AM

Currently disabled:

  • Contrib.FeedsAggregator (I had two feeds aggregated: MSDN Coding 4 Fun and Clint Rutkas.  I wanted a merged view in the sidebar but failed at getting that to work)
  • Contrib.KeepAlive (since the site takes a minute to spin-up and often has the app pool recycled, I wondered if this would help that...  I also have my Home Server HTTP pinging it once per hour)
  • Contrib.Taxonomies (not really using)
  • Iroo.PackageManager (helps to rollback and install updates I think)
  • NGM.BlogML (not needed since the site has been fully imported by now of cours)

I don't think my site has very much content for a blog.  I don't write that many posts!  I do have a custom type used for my Projects (Projects link).  There are a few custom fields in that type.  I had originally disabled Lucene but reenabled it.  Having it disabled didn't seem to make a difference.

As for troubleshooting, I'm afraid of breaking the site if modules are missing.  For example, if I'm using Stars on a page and I remove that module, what happens?

 

Mar 28, 2011 at 4:17 AM

KeepAlive - yes that might help avoid cold starts. But since the problem we're dealing with here is regular page access, better remove anything that's causing extra load on the system. KeepAlive is regularly pinging the site and running a background process, that's extra load that we don't need.

Amount of content - well, I just saw quite a lot of tags in that tag cloud. That implies you've been posting a reasonable amount. Certainly there's more than any site I'm currently running on Orchard.

I didn't say just disable Lucene - disable Search, Indexing and Lucene. I'd also disable that tag cloud. Get rid of anything unneccessary until you've got it working acceptably.

As far as I've seen, if you disable a feature the database records will stay in there, so when you re-enable you won't have lost any data. If you're worried, just back up your database before trying anything.

Apr 8, 2011 at 6:05 PM

I've played around with things a bit as I've had time, and it appears that the big culprits are search and indexing (Lucene is still enabled, not sure the point though).  With those disabled, there's a huge difference in responsiveness (actually being usable!).  Indexing in the background might be too aggressive.  I'm not sure, but I can't affort to re-enable it.

Coordinator
Apr 8, 2011 at 8:00 PM

Oh, this one seems to be for me ...

The Indexing has been completely rewritten in 1.1. Might be worth trying the new code. I will check this locally to see if I can reproduce. RandomPete (Pete ?) you might want to stop answering on the forum and testing it while I am fixing another bug ;) Might you ?

By the way we have several web sites running and using those modules without issue. Have you tried rebuilding the index ?

You can also try to get the new versions for Orchard.Indexing and give it a try, but locally as it could have you in trouble with Godaddy again if it doesn't work.

Apr 8, 2011 at 8:17 PM

Hi, I'm not sure what you mean - atkulp was having the issue, I was just suggesting some ways he might narrow down the cause. But my last post on this thread was over a week ago, and atkulp only just responded... Sounds like 1.1 might fix his situation anyway :) I certainly haven't experienced any problems with indexing.

Coordinator
Apr 8, 2011 at 8:21 PM

Pete, you need to know that Sébastien has a peculiar sense of humor. By all means, don't stop responding to forum threads. Please ;)

Apr 8, 2011 at 8:39 PM

Ha, well yes I thought that perhaps, it was just a little confusing as this thread was from a week ago ... But being British my sense of humour is also a bit odd so I quite understand :)

Anyways I seriously hope I'm not giving you guys too many headaches ;) ... I'm aware I've been perhaps a bit vocal on a project to which I'm still a relative newcomer ...

Coordinator
Apr 8, 2011 at 8:40 PM

oooh, nonononono. Keep it coming. I really appreciate the help.

Apr 8, 2011 at 8:52 PM

Ok, no worries :) Glad to be of assistance!

Apr 12, 2011 at 1:49 AM

I tried re-enabling Indexing but leaving Search off, but it goes back to an unuseable site.  If I try to go to Search Index to force a re-index, I get an exception "Root element is missing."    I then re-enabled Search as well, but I still can't manage Search Index anymore:

[XmlException: Root element is missing.]
   System.Xml.XmlTextReaderImpl.Throw(Exception e) +73
   System.Xml.XmlTextReaderImpl.ParseDocumentContent() +4076618
   System.Xml.XmlTextReaderImpl.Read() +145
   System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options) +44
   System.Xml.Linq.XDocument.Load(String uri, LoadOptions options) +65
   Lucene.Services.LuceneIndexProvider.GetLastIndexUtc(String indexName) +99
   Orchard.Indexing.Services.IndexingService.Orchard.Indexing.Services.IIndexingService.GetIndexEntry() +119
   Orchard.Indexing.Controllers.AdminController.Index() +31
   lambda_method(Closure , ControllerBase , Object[] ) +40
   System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters) +17
   System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters) +188
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters) +27
   System.Web.Mvc.<>c__DisplayClass15.b__12() +56

 I'm definitely interested in a 1.1 release, but I don't think I want to built it myself.  I'll leave Search/Index disabled for now I guess.

Apr 12, 2011 at 1:51 AM

Wow.  And there it is!  No sooner do I hit send, and a message appears in my Inbox alerting me to a new release!  Hopefully I'll have time to upgrade tonight.

YAYAYAYYAYAY!!

Apr 16, 2011 at 1:47 AM

Please let us know how it goes I was just about to switch from WordPress to Orchard on godaddy and this would be a problem.

Thanks