Very slow browsing with mega items

Topics: Administration, Customizing Orchard, Troubleshooting, Writing modules
Dec 3, 2011 at 4:44 PM

We built a migration module to migrate old data from a very old and very complex web-app db to a new instance of Orchard, mapped to content types, taxonomies, users, etc.. We have about 10.000 taxonomy terms in 11 trees and about 500.000 record in users, content type items, etc...

Before migration, browsing taxonomies and editing content-types were very good, but after the migration, the browsing is very very very slow.

We worked in this migration module for about 3 months, so is it was a waste or we shall do something to have a good performance in this mega-data app?

Also, does archiving items help?

 

Thanks in advance!

Coordinator
Dec 3, 2011 at 6:37 PM
There is little optimization in the database currently. This would need to be profiled to identify the bottlenecks. In particular, you probably need to find the most commonly used queries, analyze their plan and create indices accordingly.

Note that there is a performance working group for 1.4 that would be very interested in whatever findings you make.

Sent from my TI-99/4A

From: moh_kin
Sent: 12/3/2011 9:46 AM
To: Bertrand Le Roy
Subject: Very slow browsing with mega items [orchard:281701]

From: moh_kin

We built a migration module to migrate old data from a very old and very complex web-app db to a new instance of Orchard, mapped to content types, taxonomies, users, etc.. We have about 10.000 taxonomy terms in 11 trees and about 500.000 record in users, content type items, etc...

Before migration, browsing taxonomies and editing content-types were very good, but after the migration, the browsing is very very very slow.

We worked in this migration module for about 3 months, so is it was a waste or we shall do something to have a good performance in this mega-data app?

Also, does archiving items help?

Thanks in advance!

Coordinator
Dec 4, 2011 at 5:45 AM

You should use the dev branch of taxonomy, it's much faster.

Dec 4, 2011 at 6:15 AM

Thanks guys, @sebastienros: The migration from taxonomy 0.9 to the dev branch is a mess :( could you please provide a cleaner one, coz we already migrated data to the old taxonomy.

Coordinator
Dec 5, 2011 at 4:32 PM

There is no migration path from the two versions, completely different. 

Dec 5, 2011 at 5:33 PM
sebastienros wrote:

There is no migration path from the two versions, completely different. 

So in the future, you will support the dev branch or there will be another new branch with no migration also :(

Coordinator
Dec 5, 2011 at 6:08 PM

There will be a 2.0 version of Taxonomy for Orchard 1.4+, and the 0.9 will still be available for 1.0-1.3.

Orchard 1.4 checks for compatible modules before installing it also. A migration path though would be to do an Export of the data, then clear everything from the Taxonomy module, install the new one and do an import. I still have to test if it works, but it should not be difficult to have it working. Even If I have to make an update to the 0.9 version.

Dec 8, 2011 at 2:43 AM
Edited Dec 8, 2011 at 2:47 AM

As Bertrand said, with that amount of content items you will need to create indexes in the database. None of the orchard database table have indexes (other than primary keys) by default. I am about to import about 9,000 items from my existing database that i'm plugging into Orchard so I may have to start creating indexes soon. There are some obvious ones you can tell will be needed for any site with a lot of users. For example adding one for username on the user table would help for the authentication query to verify user login credentials. I found it odd at first that the system so far has been created like this, because in most of my development I tend to add at least the more obvious indexes up front, while my tables are being created. 

Dec 10, 2011 at 9:23 AM
TheMonarch wrote:

As Bertrand said, with that amount of content items you will need to create indexes in the database. None of the orchard database table have indexes (other than primary keys) by default. I am about to import about 9,000 items from my existing database that i'm plugging into Orchard so I may have to start creating indexes soon. There are some obvious ones you can tell will be needed for any site with a lot of users. For example adding one for username on the user table would help for the authentication query to verify user login credentials. I found it odd at first that the system so far has been created like this, because in most of my development I tend to add at least the more obvious indexes up front, while my tables are being created. 

Thanks TheMonarch for the great reply, I'm working also on some of the indexes, and please provide the community with a blog/article about the must-have indexes.