This project is read-only.

Bulk Insert In nhibernate orchard cms

Topics: Administration, Core, Customizing Orchard, General, Jobs, Troubleshooting, Writing modules
Aug 25, 2015 at 7:29 AM
hi all ,
I have problem with inserting large number of data in orchard cms with nhibernate .

let me say : I have a background task that read data from external source every 15 min and insert into one of my table in foreach loop , record count is about 10000 and more!
but when this task start , all of request in my web site get timeout error and my site is down!

now my questions :
1- How could I do bulk insert in orchard with nhibernate , I should say I have been tryed _orchardServices.ContentManager.Clear();
but with this solution , inserted record publish status get false !!! and also it dose not do so much with performance !!!
2 - is there a solution to avoid lock in foreach loop ! or Should I use some change in my code?
Aug 26, 2015 at 9:49 PM
The way we handled a similar issue with our import scripts was to do things in batches.

Something like this:
var batchCount = 0;
var maxCount = 10;

foreach (var item in importItems)
    if (batchCount > maxCount)
        _transactionManager.RequireNew(); // _transactionManager comes from injecting ITransactionManager in your constructor
        batchCount = 0;
This at least helps with the nasty locking issue.
Sep 1, 2015 at 10:38 AM
@PacmanDave thanks for your replay !
I set :

I uesd the solution that you mentioned too , but again it dose not meet my request !

I need for some thing like table type in sql server , which I could insert more than 100000 record in less than one minute!!!

and also again request get time out when two or more select request sent to the server in the same time !!!

also I want to know if there is solution to get rid of inserting record in contentItemRecord table and record table?
in some situation like news module and so on , the database grows unexpectedly and we have redundant data ?
Sep 1, 2015 at 1:02 PM
This unfortunately is beyond my knowledge then :(

The only thing I would suggest if the running in less than a minute requirement is due to the task scheduler running every minute and causing collisions is to use the JobsQueueService. That way your scheduled task can just kick off a job in the background rather than holding everything up.

Here is an example:
_jobsQueueService.Enqueue("IMyImportHandler.Import", new { }, 0);
Sep 1, 2015 at 3:25 PM
@PacmanDave thanks again for your quick reply ! something that I 've forgot to say is that I use scheduled task and jobs are running in background tasks , but when tasks start my request for page in front end stopped with time out!!!
and when I stopped background task ,I could get my query result , however it's response is too slow!