Errors on rebuilding indexes

Topics: Core, Troubleshooting
Jan 15, 2015 at 12:50 PM
Hi

We're having issues when trying to rebuild our indexes. Nothing wants to get indexed and we are getting the following errors:
2015-01-15 00:00:32,191 [35] Orchard.ContentManagement.DefaultContentManager - Default - ArgumentException thrown from IContentHandler by Orchard.Core.Common.Handlers.CommonPartHandler
(null)
System.ArgumentException: An item with the same key has already been added.
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary2.Insert(TKey key, TValue value, Boolean add)
at System.Collections.Generic.Dictionary
2.Add(TKey key, TValue value)
at Orchard.ContentManagement.DefaultContentManagerSession.Store(ContentItem item) in c:\work\website\Code\src\Orchard\ContentManagement\DefaultContentManagerSession.cs:line 12
at Orchard.ContentManagement.DefaultContentManager.Get(Int32 id, VersionOptions options, QueryHints hints) in c:\work\website\Code\src\Orchard\ContentManagement\DefaultContentManager.cs:line 231
at Orchard.ContentManagement.DefaultContentManager.Get(Int32 id, VersionOptions options) in c:\work\website\Code\src\Orchard\ContentManagement\DefaultContentManager.cs:line 131
at Orchard.ContentManagement.DefaultContentManager.Get(Int32 id) in c:\work\website\Code\src\Orchard\ContentManagement\DefaultContentManager.cs:line 127
at Orchard.ContentManagement.ContentGetExtensions.Get[T](IContentManager manager, Int32 id) in c:\work\website\Code\src\Orchard\ContentManagement\ContentExtensions.cs:line 155
at Orchard.Core.Common.Handlers.CommonPartHandler.<>c__DisplayClassa.<LazyLoadHandlers>b__8() in c:\work\website\Code\src\Orchard.Web\Core\Common\Handlers\CommonPartHandler.cs:line 146
at Orchard.Core.Common.Utilities.LazyField1.GetValue() in c:\work\website\Code\src\Orchard.Web\Core\Common\Utilities\LazyField.cs:line 24
at Orchard.Core.Common.Utilities.LazyField
1.get_Value() in c:\work\website\Code\src\Orchard.Web\Core\Common\Utilities\LazyField.cs:line 10
at Orchard.Core.Common.Models.CommonPart.get_Owner() in c:\work\website\Code\src\Orchard.Web\Core\Common\Models\CommonPart.cs:line 17
at Orchard.Core.Common.Handlers.CommonPartHandler.<.ctor>b__2(IndexContentContext context, CommonPart commonPart) in c:\work\website\Code\src\Orchard.Web\Core\Common\Handlers\CommonPartHandler.cs:line 63
at Orchard.ContentManagement.Handlers.ContentHandler.InlineStorageFilter1.Indexing(IndexContentContext context, TPart instance) in c:\work\website\Code\src\Orchard\ContentManagement\Handlers\ContentHandler.cs:line 192
at Orchard.ContentManagement.Handlers.StorageFilterBase
1.Orchard.ContentManagement.Handlers.IContentStorageFilter.Indexing(IndexContentContext context) in c:\work\website\Code\src\Orchard\ContentManagement\Handlers\StorageFilterBase.cs:line 116
at Orchard.ContentManagement.Handlers.ContentHandler.Orchard.ContentManagement.Handlers.IContentHandler.Indexing(IndexContentContext context) in c:\work\website\Code\src\Orchard\ContentManagement\Handlers\ContentHandler.cs:line 345
at Orchard.ContentManagement.DefaultContentManager.<>c__DisplayClass99.<Index>b__97(IContentHandler handler) in c:\work\website\Code\src\Orchard\ContentManagement\DefaultContentManager.cs:line 851
at Orchard.InvokeExtensions.Invoke[TEvents](IEnumerable1 events, Action1 dispatch, ILogger logger) in c:\work\website\Code\src\Orchard\InvokeExtensions.cs:line 17
2015-01-15 00:00:32,508 [35] Orchard.Exceptions.DefaultExceptionPolicy - Default - An unexpected exception was caught
(null)
NHibernate.ObjectNotFoundException: No row with the given identifier exists[Orchard.ContentManagement.Records.ContentItemRecord#2896]
at NHibernate.Impl.SessionFactoryImpl.DefaultEntityNotFoundDelegate.HandleEntityNotFound(String entityName, Object id)
at NHibernate.Proxy.AbstractLazyInitializer.CheckTargetState()
at NHibernate.Proxy.AbstractLazyInitializer.Initialize()
at NHibernate.Proxy.AbstractLazyInitializer.GetImplementation()
at NHibernate.Proxy.DefaultLazyInitializer.Intercept(InvocationInfo info)
at ContentItemRecordProxy.get_ContentType()
at Orchard.ContentManagement.DefaultContentManager.Get(Int32 id, VersionOptions options, QueryHints hints) in c:\work\website\Code\src\Orchard\ContentManagement\DefaultContentManager.cs:line 227
at Orchard.ContentManagement.DefaultContentManager.Get(Int32 id, VersionOptions options) in c:\work\website\Code\src\Orchard\ContentManagement\DefaultContentManager.cs:line 131
at Orchard.Indexing.Services.IndexingTaskExecutor.<BatchIndex>b__0(ContentItemVersionRecord versionRecord)
at System.Linq.Enumerable.WhereSelectListIterator2.MoveNext()
at System.Linq.Enumerable.<DistinctIterator>d__81
1.MoveNext()
at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
at Orchard.Indexing.Services.IndexingTaskExecutor.BatchIndex(String indexName, String settingsFilename, IndexSettings indexSettings)
at Orchard.Indexing.Services.IndexingTaskExecutor.UpdateIndexBatch(String indexName)
at Orchard.Indexing.Services.UpdateIndexScheduler.UpdateIndex(String indexName)
at Orchard.Events.DelegateHelper.<>c__DisplayClass2b
2.<BuildAction>b__2a(Object target, Object[] p) in c:\work\website\Code\src\Orchard\Events\DelegateHelper.cs:line 116
at Orchard.Events.DefaultOrchardEventBus.TryInvokeMethod(IEventHandler eventHandler, Type interfaceType, String messageName, String interfaceName, String methodName, IDictionary2 arguments, IEnumerable& returnValue) in c:\work\website\Code\src\Orchard\Events\DefaultOrchardEventBus.cs:line 81
at Orchard.Events.DefaultOrchardEventBus.TryInvoke(IEventHandler eventHandler, String messageName, String interfaceName, String methodName, IDictionary
2 arguments, IEnumerable& returnValue) in c:\work\website\Code\src\Orchard\Events\DefaultOrchardEventBus.cs:line 67
at Orchard.Events.DefaultOrchardEventBus.TryNotifyHandler(IEventHandler eventHandler, String messageName, String interfaceName, String methodName, IDictionary`2 eventData, IEnumerable& returnValue) in c:\work\website\Code\src\Orchard\Events\DefaultOrchardEventBus.cs:line 53
2015-01-15 00:00:32,523 [35] Orchard.Exceptions.DefaultExceptionPolicy - Default - An unexpected exception was caught
(null)

Please help
Coordinator
Jan 15, 2015 at 5:49 PM
Can you check the table Orchard_Indexing_IndexingTask and delete any row with a NULL content item id (last column). I saw that on a customer's website, but was fixed in 1.x. If that's the case, please report which version you are using.
Jan 16, 2015 at 6:12 AM
Hi Seb.

There is no NULL's in the last column.

We are using 1.x which we've last upgraded to in December 2014. But we've upgraded 1.7 to 1.x at that point. Not sure if that might've cause some trouble.
Jan 17, 2015 at 7:58 AM
By reading the trace, we are in the BatchIndex() method context (in IndexingTaskExecutor.cs), not in the "IndexingMode.Update" part where indexing tasks are implemented, but in the "IndexingMode.Rebuild" part where all content items are loaded for indexing... So, maybe there are some issues with the IndexingTaskRecord Table, but first, after analysing the trace, it seems that the issue is on the ContentItemRecord and VersionRecord tables

In the first issue, between the two dated line, in the context of the BatchIndex() method via ExtractDocumentIndex(), the content handlers are invoked for indexing, and we go until DefaultContentManagerSession.Store() In DefaultContentManagerSession.cs, the failing line is this one. Note that this line isn't present in 1.7.x
_itemByVersionNumber.Add(Tuple.Create(item.Id, item.Version), item);
The exception is about a duplicate dictionary key, here that means you have 2 items (or more) with the same Id and Version. So, you have to check in the ContentItemVersionRecord table if you have 2 rows with same values, this in the ContentItemRecord_id column and in the Number column. If you have such a duplicate (id & version) with one Published and/or Latest and another not Published neither Latest, you can try to delete this last one, but do a backup before

In the second issue, we pass through BatchIndex(), then, we see the use of Select(), Distinct() and List(). So, in IndexingTaskExecutor.cs, the failing line is
.Select(versionRecord => _contentManager.Get(versionRecord.ContentItemRecord.Id, VersionOptions.VersionRecord(versionRecord.Id)))
Then, we go until DefaultContentManager.Get(), ContentItemRecordProxy.get..., where NHibernate fails with "No row with the given identifier ". So, you have to check in the ContentItemVersionRecord table if you have one value (or more) in the ContentItemRecord_id column that you don't retrieve in the Id column of the ContentItemRecord table. You can try to delete such rows, but do a backup before

Regards
Jan 17, 2015 at 10:36 PM
I could simulate your issues

In the ContentItemVersionRecord table, with 2 rows associated with the same item (same ContentItemRecord_id value) but 2 different version numbers, I've put the same version value in the Number column. Then, in my log file, I've got the exact same first part of your trace

In the ContentItemVersionRecord table, by putting in the ContentItemRecord_id column a value that don't exist in the Id column of the ContentItemRecord table, I've got the exact same second part of your trace

All of that after an index rebuild command

Regards
Jan 19, 2015 at 12:19 PM
Thanks for the help jtkech.

Seem to have fixed the second issue, but still getting the first issue, below is the latest log errors I get:
System.NullReferenceException: Object reference not set to an instance of an object.
at Orchard.Blogs.Handlers.BlogPostPartHandler.GetItemMetadata(GetContentItemMetadataContext context)
at Orchard.ContentManagement.Handlers.ContentHandler.Orchard.ContentManagement.Handlers.IContentHandler.GetContentItemMetadata(GetContentItemMetadataContext context)
at Orchard.InvokeExtensions.Invoke[TEvents](IEnumerable1 events, Action1 dispatch, ILogger logger)
2015-01-19 14:05:03,521 [8] Orchard.ContentManagement.DefaultContentManager - Default - NullReferenceException thrown from IContentHandler by Orchard.Blogs.Handlers.BlogPostPartHandler
http://website/Admin/Indexing System.NullReferenceException: Object reference not set to an instance of an object.
at Orchard.Blogs.Handlers.BlogPostPartHandler.GetItemMetadata(GetContentItemMetadataContext context)
at Orchard.ContentManagement.Handlers.ContentHandler.Orchard.ContentManagement.Handlers.IContentHandler.GetContentItemMetadata(GetContentItemMetadataContext context)
at Orchard.InvokeExtensions.Invoke[TEvents](IEnumerable1 events, Action1 dispatch, ILogger logger)
2015-01-19 14:05:05,830 [5] Orchard.ContentManagement.DefaultContentManager - Default - ArgumentException thrown from IContentHandler by Orchard.Core.Common.Handlers.CommonPartHandler
http://website/Admin/Indexing/Rebuild/ContentItems System.ArgumentException: An item with the same key has already been added.
at System.Collections.Generic.Dictionary2.Insert(TKey key, TValue value, Boolean add)
at Orchard.ContentManagement.DefaultContentManagerSession.Store(ContentItem item)
at Orchard.ContentManagement.DefaultContentManager.Get(Int32 id, VersionOptions options, QueryHints hints)
at Orchard.ContentManagement.ContentGetExtensions.Get[T](IContentManager manager, Int32 id)
at Orchard.Core.Common.Utilities.LazyField
1.GetValue()
at Orchard.Core.Common.Handlers.CommonPartHandler.<.ctor>b__2(IndexContentContext context, CommonPart commonPart)
at Orchard.ContentManagement.Handlers.ContentHandler.Orchard.ContentManagement.Handlers.IContentHandler.Indexing(IndexContentContext context)
at Orchard.InvokeExtensions.Invoke[TEvents](IEnumerable1 events, Action1 dispatch, ILogger logger)
2015-01-19 14:05:05,877 [5] Orchard.ContentManagement.DefaultContentManager - Default - ArgumentException thrown from IContentHandler by Orchard.Core.Common.Handlers.CommonPartHandler
http://website/Admin/Indexing/Rebuild/ContentItems System.ArgumentException: An item with the same key has already been added.
at System.Collections.Generic.Dictionary2.Insert(TKey key, TValue value, Boolean add)
at Orchard.ContentManagement.DefaultContentManagerSession.Store(ContentItem item)
at Orchard.ContentManagement.DefaultContentManager.Get(Int32 id, VersionOptions options, QueryHints hints)
at Orchard.ContentManagement.ContentGetExtensions.Get[T](IContentManager manager, Int32 id)
at Orchard.Core.Common.Utilities.LazyField
1.GetValue()
at Orchard.Core.Common.Handlers.CommonPartHandler.<.ctor>b__2(IndexContentContext context, CommonPart commonPart)
at Orchard.ContentManagement.Handlers.ContentHandler.Orchard.ContentManagement.Handlers.IContentHandler.Indexing(IndexContentContext context)
at Orchard.InvokeExtensions.Invoke[TEvents](IEnumerable1 events, Action1 dispatch, ILogger logger)
2015-01-19 14:05:06,079 [5] Orchard.ContentManagement.DefaultContentManager - Default - ArgumentException thrown from IContentHandler by Orchard.Core.Common.Handlers.CommonPartHandler

Thanks in advance
Jan 20, 2015 at 3:39 AM
Edited Jan 20, 2015 at 7:38 AM
Good for the second issue, for the 1st one check again your table to see if there are some duplicates. But maybe you have a lot of content items...

Contact me directly by email, different things can be made

After a backup, we can try to clear the indexing table, but normaly, on a rebuild all existing indexing tasks are marked to be skipped. We can try to delete an IndexName.settings.xml file, but normaly this is done by the rebuild action... One thing we can do is to specify "Not Indexed" in some content types to see the failing ones...

The best would be to have a context where I can use a debugger with some breakpoints...

Regards

Update:

About the first exceptions in your last trace that refers to a null reference on "BlogPostPartHandler.GetItemMetadata", this seems to not be related to the indexing feature, this even it's in the dashboard indexing page (because of this url: .../Adim/Indexing in your trace)

This seems to happen when one of the current displayed shape uses a BlogPost part that is not defined. The only way I found to reproduce it is by attaching the BlogPost part to a content type of which an instance is currently displayed and where the blog post part isn't defined. For example, by attaching this part to the "Page", or by making the BlogPost type creatable, so it appears on the dashboard "New menu"... So, check if you have done such things with the BlogPost type and / or part

Regards
Jan 20, 2015 at 7:18 AM
Thanks jtkech

We've managed to track it down to BlogPost content type being the failing one.

The exception is:
2015-01-20 08:34:24,167 [14] Orchard.ContentManagement.DefaultContentManager - Default - NullReferenceException thrown from IContentHandler by Orchard.Blogs.Handlers.BlogPostPartHandler
http://website/admin System.NullReferenceException: Object reference not set to an instance of an object.
at Orchard.Blogs.Handlers.BlogPostPartHandler.GetItemMetadata(GetContentItemMetadataContext context)
at Orchard.ContentManagement.Handlers.ContentHandler.Orchard.ContentManagement.Handlers.IContentHandler.GetContentItemMetadata(GetContentItemMetadataContext context)
at Orchard.InvokeExtensions.Invoke[TEvents](IEnumerable1 events, Action1 dispatch, ILogger logger)
2015-01-20 08:34:26,102 [14] Orchard.ContentManagement.DefaultContentManager - Default - NullReferenceException thrown from IContentHandler by Orchard.Blogs.Handlers.BlogPostPartHandler

Within that function there is a blogPost variable and the BlogPart is null, anyway of fixing that in the database or a workaround?

Thanks in advance.
Jan 20, 2015 at 7:45 AM
Edited Jan 20, 2015 at 8:17 AM
How, I was updating my previous post while you was writing your last post. So, see my update above. Briefly, it seems that you have attached the BlogPostPart to a content type, and an instance of this type has this part not defined. So you have to define it or remove the BlogPostPart from this type in the "Content Definition" page

Another possibility: If I make the BlogPost type creatable, it appears on the "New Menu", and because the BlogPost has a BlogPostPart, I get a lot of the exact same issue...

I will see your last post more in depth tomorrow

Regards
Jan 20, 2015 at 7:40 PM
For the last issue, if you don't find which instances of which content types have a BlogPostPart with a null BlogPostPart.BlogPart. A possible workaround is to use this test in BlogPostPartHandler.cs => GetItemMetadata()
        if (blogPost == null || blogPost.BlogPart == null) return;
And use this test in BlogPostPartDriver.cs => Display()
        if (displayType.StartsWith("Detail") && part.BlogPart != null) {...}
Regards
Marked as answer by rudolphk on 1/21/2015 at 1:59 AM
Jan 21, 2015 at 9:59 AM
Thank you jtkech.

We've going for the approach of changing the BlogPostPartHander, not ideal as an upgrade will overwrite this change, but it work for now.

We couldn't find any other content type except the Blog Post that contains the Blog Part.

Thanks for your help, much appreciated.