Orchard performance issue in Azure

Topics: Core, General, Troubleshooting
Feb 15, 2012 at 11:35 PM
Edited Feb 15, 2012 at 11:36 PM

I have just gone live with a client's site (http://www.mvpind.com/) built atop Orchard 1.3.10, hosted in Azure accessing a DB in SQLAzure (both in North Central US).

I've turned on caching and set a default cache timeout of 3600s.

However, even with caching enabled, I am seeing page GET's take between 5 and 12s to return. Here, for example is a net trace of loading the homepage of the site that takes almost 10s!

Shouldn't this be considerably faster if the page is in cache (which it is - I checked on the cache stats page in the dashboard and when I view source I can see the generated caching info appended to the bottom of the page's HTML.

TIA.

Coordinator
Feb 16, 2012 at 12:56 AM

It should be faster in all cases. For example, the Orchard web site is running on Azure and it's pretty fast. Same for the gallery. Something else is going on here.

Feb 17, 2012 at 3:22 AM

I've just installed Orchard on Azure in SE Asia yesterday.  Some thoughts here:  http://stackoverflow.com/questions/9303194/orchard-performance-on-azure/9305866#9305866   Site is going very well.

Coordinator
Feb 17, 2012 at 4:54 PM

There must be another module causing this. Could you give us the list on the enabled modules ? Does it repro on your localh machine ? There is a new version of the profiler for Orchard on the gallery, you might want to give it a shot locally.

There is also a new version of the cache module, with cache-control management. Please upgrade to this one.

Feb 17, 2012 at 9:14 PM

The enabled modules installed into this site are:

Content:
Blogs
Containers
Content Types
Import Export
Lists
Pages
Publish Later
Vandelay Favicon

Content Publishing:
Remote Blog Publishing
XmlRpc

Hosting:
Warmup

Input Editor:
Media Picker
TinyMce

Media:
Image Gallery
Media

Navigation
Tags

Packaging:
Gallery
Package Updates
Packaging
Packaging Commands

Performance:
Cache

Scripting:
Lightweight Scripting
Scripting

Social:
Comments

Syndication:
Feeds

Widget:
Page Layer Hinting
Widgets

Core:
Everything except Setup and Title

All modules (other than cache) are up to date according to the "Modules | Installed" list.

Perf on my local machine is a little sluggish but NOWHERE near as pronounced as when hosted in Azure (small instance). I've cleared the decks today to really dig into this perf issue.

I'll also try making the project full-trust (since we're not co-hosting), will update the updated cache module and examine perf using the new profiler.

Updates to follow ...

Coordinator
Feb 17, 2012 at 9:36 PM

Ideally we could do some profiling on this. One thing that may make a little difference is to fix a nasty bug in the Mello Image Gallery module. In the ImageGalleryService constructor, there is a call to check media folders that can take a long time. Moving that call out of the constructor can save quite some time on each request:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Web;
using ICSharpCode.SharpZipLib.Zip;
using Mello.ImageGallery.Models;
using Orchard;
using Orchard.Data;
using Orchard.FileSystems.Media;
using Orchard.Localization;
using Orchard.Media.Models;
using Orchard.Media.Services;
using Orchard.Validation;
using Orchard.UI.Notify;

namespace Mello.ImageGallery.Services {
    public class ImageGalleryService : IImageGalleryService {
        private const string ImageGalleriesMediaFolder = "ImageGalleries";
        private const int ThumbnailDefaultSize = 100;
        private const bool DefaultKeepAspectRatio = true;

        private readonly IMediaService _mediaService;
        private readonly IThumbnailService _thumbnailService;
        private readonly IRepository<ImageGallerySettingsRecord> _repository;
        private readonly IRepository<ImageGalleryImageSettingsRecord> _imageRepository;
        private readonly IRepository<ImageGalleryRecord> _imageGalleryPartRepository;
        private readonly IOrchardServices _services;
        private readonly IStorageProvider _storageProvider;

        private readonly IList<string> _imageFileFormats = new[] { "BMP", "GIF", "EXIF", "JPG", "PNG", "TIFF" };
        private readonly IList<string> _fileFormats = new[] { "BMP", "GIF", "EXIF", "JPG", "PNG", "TIFF", "ZIP" }; 

        //TODO: Remove Image repository as soon as it can cascade the saving
        public ImageGalleryService(IMediaService mediaService, IRepository<ImageGallerySettingsRecord> repository,
                                   IRepository<ImageGalleryImageSettingsRecord> imageRepository, IThumbnailService thumbnailService,
                                   IRepository<ImageGalleryRecord> imageGalleryPartRepository, IOrchardServices services,
                                   IStorageProvider storageProvider)
        {
            _storageProvider = storageProvider;
            _services = services;
            _imageGalleryPartRepository = imageGalleryPartRepository;
            _repository = repository;
            _mediaService = mediaService;
            _imageRepository = imageRepository;
            _thumbnailService = thumbnailService;
        }

        private void EnsureMediaFolder() {
            if (!_mediaService.GetMediaFolders(string.Empty).Any(o => o.Name == ImageGalleriesMediaFolder)) {
                _mediaService.CreateFolder(string.Empty, ImageGalleriesMediaFolder);
            }
        }

        public IEnumerable<string> AllowedFileFormats
        {
            get { return _fileFormats; }
        }

        public IEnumerable<Models.ImageGallery> GetImageGalleries() {
            EnsureMediaFolder();
            return _mediaService.GetMediaFolders(ImageGalleriesMediaFolder).Select(CreateImageGalleryFromMediaFolder);
        }

        public void CreateImageGallery(string name) {
            EnsureMediaFolder();
            _mediaService.CreateFolder(ImageGalleriesMediaFolder, name);
        }

        public void DeleteImageGallery(string name) {
            var gallerySettings = GetImageGallerySettings(GetMediaPath(name));

            foreach (ImageGalleryImage image in GetImageGallery(name).Images) {
                DeleteImage(name, image.Name, GetImageSettings(gallerySettings, image.Name));
            }

            if (gallerySettings != null)
                _repository.Delete(gallerySettings);
            _mediaService.DeleteFolder(GetMediaPath(name));
        }

        public void RenameImageGallery(string imageGalleryName, string newName) {
            string mediaPath = GetMediaPath(imageGalleryName);
            _mediaService.RenameFolder(mediaPath, newName);

            ImageGallerySettingsRecord settings = GetImageGallerySettings(imageGalleryName);
            if (settings != null) {
                settings.ImageGalleryName = newName;
                _repository.Update(settings);
            }

            IEnumerable<ImageGalleryRecord> records = _imageGalleryPartRepository.Fetch(partRecord => partRecord.ImageGalleryName == imageGalleryName);

            foreach (ImageGalleryRecord imageGalleryRecord in records) {
                imageGalleryRecord.ImageGalleryName = newName;
                _imageGalleryPartRepository.Update(imageGalleryRecord);
            }
        }

        public void UpdateImageGalleryProperties(string imageGalleryName, int thumbnailHeight, int thumbnailWidth, bool keepAspectRatio) {
            var imageGallery = GetImageGallery(imageGalleryName);
            var imageGallerySettings = GetImageGallerySettings(imageGallery.MediaPath);

            if (imageGallerySettings == null) {
                CreateImageGallerySettings(imageGallery.MediaPath, thumbnailHeight, thumbnailWidth, keepAspectRatio);
            }
            else {
                imageGallerySettings.ThumbnailHeight = thumbnailHeight;
                imageGallerySettings.ThumbnailWidth = thumbnailWidth;
                imageGallerySettings.KeepAspectRatio = keepAspectRatio;

                _repository.Update(imageGallerySettings);
            }
        }

        public ImageGalleryImage GetImage(string imageGalleryName, string imageName) {
            string imageGalleryMediaPath = GetMediaPath(imageGalleryName);
            ImageGallerySettingsRecord imageGallerySettings = GetImageGallerySettings(imageGalleryMediaPath);

            MediaFile file = _mediaService.GetMediaFiles(imageGalleryMediaPath)
                .SingleOrDefault(mediaFile => mediaFile.Name == imageName);

            if (file == null) {
                return null;
            }

            return CreateImageFromMediaFile(file, imageGallerySettings);
        }

        public Models.ImageGallery GetImageGallery(string imageGalleryName) {
            EnsureMediaFolder();
            if (imageGalleryName.Contains("\\") || imageGalleryName.Contains("/"))
                imageGalleryName = GetName(imageGalleryName);

            var mediaFolder = _mediaService.GetMediaFolders(ImageGalleriesMediaFolder).SingleOrDefault(m => m.Name == imageGalleryName);

            if (mediaFolder != null) {
                return CreateImageGalleryFromMediaFolder(mediaFolder);
            }
            return null;
        }

        public int CountImages(IEnumerable<string> galleries) {
            return galleries.Sum(g => {
                                     var folder =
                                         _mediaService.GetMediaFolders(ImageGalleriesMediaFolder).SingleOrDefault(m => m.Name == GetName(g));
                                     return _mediaService.GetMediaFiles(folder.MediaPath).Count();
                                 });
        }

        public void AddImage(string imageGalleryName, HttpPostedFileBase imageFile) {            
            AddImage(imageGalleryName, imageFile.FileName, imageFile.InputStream);
        }

        public void AddImage(string imageGalleryName, string fileName, Stream imageFile){
            if(!IsFileAllowed(fileName, true)) {
              throw new InvalidOperationException(string.Format("{0} is not a valid file.", fileName));
            }

            // Zip file processing is different from Media module since we want the folders structure to be flattened
            if (IsZipFile(Path.GetExtension(fileName))) {
              UnzipMediaFileArchive(imageGalleryName, imageFile);
            }
            else {
              _mediaService.UploadMediaFile(GetMediaPath(imageGalleryName), fileName, imageFile, false);
            }
        }

        public void UpdateImageProperties(string imageGalleryName, string imageName, string imageTitle, string imageCaption) {
            UpdateImageProperties(imageGalleryName, imageName, imageTitle, imageCaption, null);
        }

        private void UpdateImageProperties(string imageGalleryName, string imageName, string imageTitle, string imageCaption, int? position) {
            var image = GetImage(imageGalleryName, imageName);
            var imageGallery = GetImageGallery(imageGalleryName);

            var imageGallerySettings = GetImageGallerySettings(imageGallery.MediaPath);

            if (imageGallerySettings.ImageSettings.Any(o => o.Name == image.Name)) {
                var imageSetting = imageGallerySettings.ImageSettings.Single(o => o.Name == image.Name);
                imageSetting.Caption = imageCaption;
                imageSetting.Title = imageTitle;
                if (position.HasValue)
                    imageSetting.Position = position.Value;
                _imageRepository.Update(imageSetting); // TODO: Remove when cascade is fixed
            }
            else {
                var imageSetting = new ImageGalleryImageSettingsRecord {Caption = imageCaption, Name = image.Name, Title = imageTitle};
                if (position.HasValue)
                    imageSetting.Position = position.Value;
                imageGallerySettings.ImageSettings.Add(imageSetting);
                _imageRepository.Create(imageSetting); // TODO: Remove when cascade is fixed
            }

            // TODO: See how to cascade changes          
            _repository.Update(imageGallerySettings);
        }

        private ImageGallerySettingsRecord GetImageGallerySettings(string imageGalleryName) {
            if (imageGalleryName.Contains("\\") || imageGalleryName.Contains("/"))
                imageGalleryName = GetName(imageGalleryName);
            return _repository.Get(o => o.ImageGalleryName == imageGalleryName);
        }

        private ImageGalleryImageSettingsRecord GetImageSettings(ImageGallerySettingsRecord imageGallerySettings, string imageName) {
            if (imageGallerySettings == null || imageGallerySettings.ImageSettings == null)
                return null;
            return imageGallerySettings.ImageSettings.SingleOrDefault(o => o.Name == imageName);
        }

        private ImageGalleryImageSettingsRecord GetImageSettings(string imageGalleryName, string imageName) {
            var imageGallerySettings = GetImageGallerySettings(GetMediaPath(imageGalleryName));

            return imageGallerySettings.ImageSettings.SingleOrDefault(o => o.Name == imageName);
        }

        private Models.ImageGallery CreateImageGalleryFromMediaFolder(MediaFolder mediaFolder) {
            var images = _mediaService.GetMediaFiles(mediaFolder.MediaPath);
            ImageGallerySettingsRecord imageGallerySettings = GetImageGallerySettings(GetName(mediaFolder.MediaPath)) ??
                                                              CreateImageGallerySettings(mediaFolder.MediaPath, ThumbnailDefaultSize,
                                                                                         ThumbnailDefaultSize, DefaultKeepAspectRatio);

            return new Models.ImageGallery
                   {
                       Id = imageGallerySettings.Id,
                       LastUpdated = mediaFolder.LastUpdated,
                       MediaPath = mediaFolder.MediaPath,
                       Name = mediaFolder.Name,
                       Size = mediaFolder.Size,
                       User = mediaFolder.User,
                       ThumbnailHeight = imageGallerySettings.ThumbnailHeight,
                       ThumbnailWidth = imageGallerySettings.ThumbnailWidth,
                       Images = images.Select(image => CreateImageFromMediaFile(image, imageGallerySettings)).OrderBy(image => image.Position),
                       KeepAspectRatio = imageGallerySettings.KeepAspectRatio
                   };
        }

        private ImageGallerySettingsRecord CreateImageGallerySettings(string imageGalleryMediaPath, int thumbnailHeight, int thumbnailWidth,
            bool keepAspectRatio) {
            ImageGallerySettingsRecord imageGallerySettings = new ImageGallerySettingsRecord
                                                              {
                                                                  ImageGalleryName = GetName(imageGalleryMediaPath),
                                                                  ThumbnailHeight = thumbnailHeight,
                                                                  ThumbnailWidth = thumbnailWidth,
                                                                  KeepAspectRatio = keepAspectRatio
                                                              };
            _repository.Create(imageGallerySettings);

            return imageGallerySettings;
        }

        private ImageGalleryImage CreateImageFromMediaFile(MediaFile mediaFile, ImageGallerySettingsRecord imageGallerySettings) {
            if (imageGallerySettings == null) {
                throw new ArgumentNullException("imageGallerySettings");
            }

            var imageSettings = GetImageSettings(imageGallerySettings, mediaFile.Name);
            bool isValidThumbnailSize = imageGallerySettings.ThumbnailWidth > 0 &&
                                        imageGallerySettings.ThumbnailHeight > 0;
            Thumbnail thumbnail = null;

            if (isValidThumbnailSize) {
                thumbnail = _thumbnailService.GetThumbnail( _storageProvider.Combine(mediaFile.FolderName, mediaFile.Name),
                                                              imageGallerySettings.ThumbnailWidth,
                                                              imageGallerySettings.ThumbnailHeight,
                                                              imageGallerySettings.KeepAspectRatio);
            }

            return new ImageGalleryImage
                   {
                       PublicUrl = _mediaService.GetPublicUrl(Path.Combine(mediaFile.FolderName, mediaFile.Name)),
                       Name = mediaFile.Name,
                       Size = mediaFile.Size,
                       User = mediaFile.User,
                       LastUpdated = mediaFile.LastUpdated,
                       Caption = imageSettings == null ? string.Empty : imageSettings.Caption,
                       Thumbnail = thumbnail,
                       Title = imageSettings == null ? null : imageSettings.Title,
                       Position = imageSettings == null ? 0 : imageSettings.Position                       
                   };
        }

        private string GetMediaPath(string imageGalleryName) {
            return _storageProvider.Combine(ImageGalleriesMediaFolder, imageGalleryName);
        }

        private string GetName(string mediaPath) {
            return mediaPath.Split(new[] { '\\', '/' }).Last();
        }

        public void DeleteImage(string imageGalleryName, string imageName) {
            var imageSettings = GetImageSettings(imageGalleryName, imageName);
            DeleteImage(imageGalleryName, imageName, imageSettings);
        }

        public string GetPublicUrl(string path) {
            return _mediaService.GetPublicUrl(path);
        }

        public bool IsFileAllowed(string fileName, bool allowZip) {
            return (IsImageFile(fileName) || (allowZip && IsZipFile(Path.GetExtension(fileName)))) && _mediaService.FileAllowed(fileName, allowZip);
        }

        public bool IsFileAllowed(HttpPostedFileBase postedFile) {
            if (postedFile == null)
            {
              return false;
            }

            return IsFileAllowed(postedFile.FileName, true);
        }

        private bool IsImageFile(string fileName)
        {
          string extension = Path.GetExtension(fileName);
          if(extension == null)
            return false;
          extension = extension.TrimStart('.');

          return _imageFileFormats.Any(o => extension.Equals(o, StringComparison.OrdinalIgnoreCase));
        }        

        private void DeleteImage(string imageGalleryName, string imageName, ImageGalleryImageSettingsRecord imageSettings) {
            if (imageSettings != null) {
                _imageRepository.Delete(imageSettings);
            }
            _mediaService.DeleteFile(GetMediaPath(imageGalleryName), imageName);
        }


        public void ReorderImages(string imageGalleryName, IEnumerable<string> images) {
            Models.ImageGallery imageGallery = GetImageGallery(imageGalleryName);
            int position = 0;

            foreach (string image in images) {
                ImageGalleryImage imageGalleryImage = imageGallery.Images.Single(o => o.Name == image);
                imageGalleryImage.Position = position++;
                UpdateImageProperties(imageGalleryName, imageGalleryImage.Name, imageGalleryImage.Title, imageGalleryImage.Caption,
                                      imageGalleryImage.Position);
            }

            foreach (ImageGalleryImage imageGalleryImage in imageGallery.Images.Where(o => !images.Contains(o.Name))) {
                imageGalleryImage.Position = position++;
                UpdateImageProperties(imageGalleryName, imageGalleryImage.Name, imageGalleryImage.Title, imageGalleryImage.Caption,
                                      imageGalleryImage.Position);
            }
        }

        // TODO: Submit a path to Media module to make this method public?
        /// <summary>
        /// Determines if a file is a Zip Archive based on its extension.
        /// </summary>
        /// <param name="extension">The extension of the file to analyze.</param>
        /// <returns>True if the file is a Zip archive; false otherwise.</returns>
        private static bool IsZipFile(string extension)
        {
            return string.Equals(extension.TrimStart('.'), "zip", StringComparison.OrdinalIgnoreCase);
        }

        /// <summary>
        /// Unzips a media archive file flattening the folder structure.
        /// </summary>
        /// <param name="imageGallery">Image gallery name.</param>
        /// <param name="zipStream">The archive file stream.</param>
        protected void UnzipMediaFileArchive(string imageGallery, Stream zipStream)
        {
            Argument.ThrowIfNullOrEmpty(imageGallery, "imageGallery");
            Argument.ThrowIfNull(zipStream, "zipStream");

            var fileInflater = new ZipInputStream(zipStream);
            ZipEntry entry;

            while ((entry = fileInflater.GetNextEntry()) != null)
            {
                if (!entry.IsDirectory && !string.IsNullOrEmpty(entry.Name))
                {
                    // skip disallowed files
                    if (IsFileAllowed(entry.Name, false))
                    {
                        string fileName = Path.GetFileName(entry.Name);

                        try
                        {
                            AddImage(imageGallery, fileName, fileInflater);
                        }
                        catch(ArgumentException argumentException)
                        {
                            //if (argumentException.ParamName == entry.Name) {
                            if(argumentException.Message.Contains(fileName)) {
                                _services.Notifier.Warning(new LocalizedString(string.Format("File \"{0}\" skipped since it already exists.", fileName)));
                            }
                            else {
                                throw;
                            }
                        }
                    }
                }
            }
        }
    }
}

Feb 18, 2012 at 12:12 AM

Thanks for the tip Bertrand. I've applied your fix and am uploading a new build as I type. I'll let you know what perf looks like afterwards.

Question though: The perf improvement above should only apply to requests for the image gallery pages, right? Orchard doesn't construct a new ImageGallery service unless the visitor is actually asking to view an Image Gallery does it?

Coordinator
Feb 18, 2012 at 12:16 AM

It could, as for dependency resolutions, or any filter. A lot of objects a created but it's cheap, what is expensive is when those objects do weird stuff in the constructors. Such behavior should be delayed until the real usage, for instance by using Lazy<T>(delegate)

Feb 18, 2012 at 12:23 AM
sebastienros wrote:

There is also a new version of the cache module, with cache-control management. Please upgrade to this one.

Hey Sebastian. I tried to upgrade to the 1.0 release of the caching module, but it tells me that it doesn't know what an IAliasAspect is!

Is the 1.0 caching module built against an unreleased version of Orchard?

Coordinator
Feb 18, 2012 at 12:27 AM

Oops ! Developed it under 1.x ... should check if it's worth making it work for 1.3 as we will soon have vnext ready. 

Feb 18, 2012 at 1:07 AM

Phew! Glad its not just me that does this ;) Even if v.next is pretty imminent, I think it'd be worth porting to 1.3.x too since a lot of current sites will be running on 1.3 and may need to wait a little before porting to 1.4.

Thanks for all your hard work and support - you guys rock :)

Feb 18, 2012 at 9:14 AM

I confirm that the Image Gallery module is quite nasty on Azure... For each image displayed on a gallery, it will download the image on the server to check if the thumbnail must be generated. I am not sure that it's good practice with images on the local HD, but I am sure that it's very nasty with images on blob storage.

Feb 18, 2012 at 5:18 PM
Laere wrote:

I confirm that the Image Gallery module is quite nasty on Azure... For each image displayed on a gallery, it will download the image on the server to check if the thumbnail must be generated. I am not sure that it's good practice with images on the local HD, but I am sure that it's very nasty with images on blob storage.

Yeah, I've been looking into the image gallery module and am not happy with how it works. I'll be making substantial changes to it shortly ;)

Feb 18, 2012 at 5:41 PM

PROBLEM SOLVED!! You're gonna love this ;) :

Compare the screenshot above with the new screenshot I just took a moment ago of the new and fixed site:

The page request completion has dropped from 9.5s to 05.s!!!!! Yep, half a second!!!

Admittedly, this is with caching turned on and the server and client cache populated. If I evict the page from the server cache, clear the client cache and then refresh, the page request takes 1.5 - 2.5s. This is still well within our performance expectations!!

So, what was the fix? Well, to be honest, I am not entirely sure. But after installing the Mini Profier module and examining the traces it produced, I found that there were a TON of unexpected DB accesses and widgets and shapes being rendered that weren't even on the page!! I figured that the best thing to do was to clear-down and rebuild from scratch. So I deleted all my locally built binaries and re-built all of Orchard. Note: If you do this, some modules ship dependencies in the Bin folder (Mini profiler, I'm looking at you ;)) so you'll have to replace those dependencies after deleting all your bin and obj folders. 

Running Orchard.Web locally (with server caching turned off), I noticed a significant perf improvement - 350ms on average for most pages!!

I then built and ran Orchard.Azure and tested locally in the Azure emulator. Again, perf was off the charts compared to before.

So I then deployed to Azure and was astounded to see the post-initial-request perf drop from several seconds per page request to sub-second!!

I'm still not happy with the perf of the image gallery so will be working on fixing that, but the perf of the rest of the site is now AWESOME ... even on a small-instance Azure web role! :D

Hope this helps anyone else out there running into perf issues!

Coordinator
Feb 18, 2012 at 6:21 PM

You could get even better results if you had the cache v1.0 work on Orchard 1.3 and turn the Max Age property to some descent value. Would use IIS kernel cache instead of Orchard server one, and would blow everything.

Also, you can use the Combinator module to reduce the number of requests. Bundling module, works on Azure too.

Feb 20, 2012 at 6:05 PM

Thanks for the tips Sebastian. Since this is a fairly simple site and the traffic volumes aren't particularly high, I'll defer bundling and combining until later.

If I get some free cycles, I'll take a look at fixing up the 1.0 Caching module to run on 1.3 ... after I've taken a look at fixing the Image Gallery module ;)

 

Feb 22, 2012 at 1:04 AM

Digging into the results from the mini-profiler:

Is there any way to eliminate the (many) SQL calls to gather comments and tags?

I have modified our theme's placement.info to exclude them from being displayed:

  <Match DisplayType="Summary" ContentType="BlogPost">
    <Place Parts_Comments="-" />
    <Place Parts_Comments_Count="-" />
    <Place Parts_Tags="-" />
  </Match>

But to no aviail - they still show up in the MiniProfiler trace.

Coordinator
Feb 22, 2012 at 5:09 PM

This is fixed in 1.x

Feb 22, 2012 at 7:05 PM
sebastienros wrote:

This is fixed in 1.x


AWESOME :) I know Bertrand mentioned that 1.4 is coming really soon now - do you have guidance for the ETA?

Coordinator
Feb 22, 2012 at 9:00 PM

Next Wednesday.