Bulk record Insert in Database

Topics: Administration, Core, Customizing Orchard, Jobs, Writing modules
Feb 1, 2013 at 6:10 AM
i have created a module for upload .csv file and the file records are going to insert into the table name "DeviceRecords", this file will have a huge records its around 50000, my problem is orchard does not support BulkCopy method, but i tried _orchardService.ContentManager.Flush(); _orchardService.ContentManager.Clear(); after 100 records to be inserted but the datas are inserted in un order wise. hence i want to know about how to insert bulk record into database. if its possible with orchard it would be more helpfull
Feb 10, 2013 at 9:39 PM
The main problem of bulk operations like these is that they will by default happen in a single transaction. If you look at the import code, you'll see what I mean. One thing you can do is break the work down into smaller batches and schedule each.
Dec 28, 2014 at 10:13 AM
Edited Dec 28, 2014 at 10:15 AM
hi all,
I use Orchard 1.8 and I could not find Flush() ! I want to bulk insert content part record ,
or Is there another solution for bulk insert in this version ?
Jan 6, 2015 at 5:13 AM
Edited Jan 6, 2015 at 5:24 AM
In previous versions (e.g 1.6) there is a contentManager.Flush() that do a repository.Flush() that do a session.Flush(), and a contentManager.Clear() that clear internal collections and the current session. In 1.6, look at the ImportContentSession.cs for an example that use both of them

Since 1.7.x (maybe not 1.7), and with 1.8.1 there is no more contentManager.Flush(). If you want, you can do the same by using the repository directly. But, take a look at DataRecipeHandler.cs for an example that only use contentManager.Clear(), but also transactionManager.RequireNew() that do a session.Transaction.Commit() (that do a Flush() before in NHibernate). Note that here you create a new transaction and you need to inject an ITransactionManager dependency

I've read in a discussion that there is a better performance if you only use Clear() without RequireNew()(not a big difference). But, as it is mentioned in the same discussion, if there are a lot of items it's better to use a new transaction on each batch size

Finally, in 1.8.x, contentManager.Clear() do nothing, but transactionManager.RequireNew() completely close and dispose the session. See in DataRecipeHandler.cs of this version, you will see that only transactionManager.RequireNew() is called. If in 1.8.1 you use Clear() and RequireNew(), it will be compatible with the current 1.8.x version