True confession time. I’m learning Orchard by practicing on a live patient, er, client. Let’s say I have an instance of Orchard 1.51 up and running in staging with a fair amount of client content, a specific customized third party theme, and a set of modules, some deployed and some installed but not enabled. Let’s say I’ve played with both the WebMatrix approach and the Source enlistment approach but this running site is based on a full source set up.

Let’s say the running instance was successfully migrated from 1.4 to 1.51. Let’s say I activated Custom Forms for 1.51 but, in learning how to get it working correctly, I deleted a database record or two directly [no migration, no positive control or recordkeeping, pure hack-and-chop methodology]. I’ve been planning to put this project into TFS Online, but I haven’t done so yet. Also, I know I need to use migrations, but I haven’t up till now.

Let’s say that my local dev copy of the project stopped compiling a week or two ago and I’ve kept modifying the staging instance via the dashboard and via direct editing of .css files and .cshtml files. Let’s say my custom forms module is enabled but I get this error when I hit “Custom Forms” in the dashboard:

Line 57:<td>

Line 58: <input type="hidden" value="@Model.CustomForms[formIndex].CustomForm.Id" name="@Html.NameOf(m => m.CustomForms[formIndex].CustomForm.Id)"/>

Line 59: <input type="checkbox" value="true" name="@Html.NameOf(m => m.CustomForms[formIndex].IsChecked)"/>

Line 60: </td>

Source File: c:\Websites\TechWise\Modules\Orchard.CustomForms\Views\Admin\Index.cshtml Line: 58

 So here is my situation. I have two problems. I need to fix this error. But the larger context is that I need to gain DevOps control over this beast on a forward going basis. My question is what are the best practices here? I would love to start fresh with a fresh source code install in my dev environment, set up source control and a structured deployment process.

The hard part comes in understanding how to separate and preserve two kinds of data that reside in the current staging database:

1. Actual client data. I know that a lot of it is in Common_BodyPartRecord, but of course, a lot of it is all over the place.
2. Site structuring metadata. Obviously the selected theme subdirectory is a treasure trove of this, but there is a lot more all over the place.

So, can I harvest and rebuild or do I really need to extract all relevant data from the existing structure “by hand” and rebuild and re-inject all the data?

Thanks,

-          Mike

Ps. If I learn a good DevOps pattern I will document it up for the community.

Can you describe more what you mean by "site structuring metadata"? Examples? 

Two examples would be the contents of:

   -  Settings_ContentTypeDefinitionRecord
   -  Settings_ShellFeatureStateRecord

You can export all the data and settings you mentioned via the Import & Export tool.

There might be better ways of doing this, but without knowing all the details of your site, one way to do it would be to:

1. Back up production/staging websites and databases
2. Start fresh with a new source enlistment, using proper source control methods, 
3. Build the site barebones without any custom or 3rd party modules and deploy it to a test environment
4. Start adding all the modules you use to the mix, make sure you can build and deploy with all the modules you need
5. Make sure you are able to check out everything from source control and build everything from scratch every time. 
6. Start importing the data from your production or staging environments via recipes and/or Import/Export, until your fresh version of the site mimics your production/staging ones perfectly. 
7. Back up production/staging websites and databases
8. Deploy the source controlled version in place of your messy staging/prod ones. 

Thank you for this great feedback and task list.  No wonder they call you The Monarch.

I'm stepping through the stages outlined above and writing notes to share later.  I've hit my first significant question:

How do Orchard people typically handle database source control?

In my case, I'm using TFS Online for source code control but I do not have SQL wired to TFS.  I have an instance of the Orchard DB on SQL on my dev machine and a separate SQL server that supports the staging/live environments.  Keeping the DBs in Sync is easy when all changes are flowing in one direction.  However, once the site is in staging and editors are changing data while I might be changing DB schemas [using migrations] on the dev side, what is the easiest way to maintain synchronization?

That one is a bit hard. What I did, before my site was live we had a staging environment where content editors got to start creating/editing content. It was pretty annoying, but I'm not sure of a better way to do it. In my development environment, I had scripts to destroy and rebuild my orchard site from scratch using a custom recipe. Then I had data recipes that I kept in source control. I would periodically export the data from the staging environment into my source controlled recipes. This works best if you don't allow people to modify the content type definitions or other settings in the production/stating/whatever environment. 

The database schema itself shouldn't be a problem to keep in sync because you control that from Migrations.cs, which is in source control. 

OK, I'm making progress.  The export/import module works well.  One tip I have is that I'm not exporting and importing all in one file.  I'm doing it in several chunks - data from groups of associated modules.  For instance.  I did the blog related modules as one ex/im,  I did pages as one ex/im, and I did HTML widgets as one ex/im.

Now, here is my latest hurdle.  This may be more of a Visual Studio question than an Orchard question but here goes.   I'm using Visual Studio 2012.  I added some modules to my "new" site from within Orchard, via the gallery, on my dev machine.  I look in Visual Studio and the new modules do not appear in the "Visual Studio folder" with the other modules.  One cannot invoke "show all files" from the "Visual Studio folder".  I look under Orchard.Web and I see them as non-included files under modules at that location on the VS 2012 interface.  Now, I could "include" them from this location.  I hesitate to do this because all the modules I already have in the project are representing themselves in Visual Studio as being in included in the "Visual Studio folder" and not-included under Orchard.Web/modules.  I know why the modules are under Orchard.web in the file system, but I still don't know what to do here.  I'm trying to rebuild my site in a clean manner and I am stymied here.  Another way of asking this question is: how did the already existing modules in Orchard get expressed in Visual Studio 2012 as being in the "Visual Studio folder" and greyed out under the Orchard.Web project.  What metadata/config expresses this?

thanks,

Mike

I think you are saying that the modules you added to your site are not showing up in the Visual Studio Solution. Specifically, under the "Modules" folder within the solution. 

If so, then you can fix this by right clicking the "Modules" folder (the one at the top of the solution, *not* the one inside Orchard.Web, that you have to do "show all files" to see), and select "Add Existing Project". Navigate to the .csproj of the module within the Orchard.Web\Modules\...\ folder. 

Yes, I also manage my source controlled recipe/exports data using a single file for each content type. One thing that's really annoying about this is that when I export from my production environment, and try to compare that .xml to an export from another environment, the XML tags in each export are ordered differently each time, so it's hard to diff the two exports. I sometimes need to diff them because sometimes I have modified the content type definitions and want to manually edit the xml, so it becomes hard to find what to edit. 

If I ever have to do something like the above again I'll look for an external program that will sort the tags within an XML file. 

That's what I'm saying -- in Visual Studio 2012 you CAN'T right click on the upper Modules folder to get "show all files".  Am I wrong about this?  If I'm right, do I need to hack the new module's .csproj file to create correct settings?

Thanks for this great feedback on managing ex/im.

Yes, you can. I didn't say to do show all files, the instructions where:

right-click the "Modules" folder --> Add... --> Existing Project

Yes, of course! Thanks.  Please pardon my stupidity.   :-)

Progress.  However, there is always a next hurdle.  I have a custom theme that we purchased from Bind.pt.  When I try to create a nupkg from the theme to package it for importing to the rebuilt site, the codegen works.  However, on the step of importing the theme to the new site from the local computer, this is what is reported:

=================================================

Successfully added 'Orchard.Theme.twg1 1.1.0' to C:\Websites\MyWebSite\

Package uploading and installation failed.

=================================================

I only get about 5K (\filters\layoutfilter.cs and the twg1.csproj file) in the nupkg file. The csproj file is only 25 lines.  The actual csproj file on the source is 400 lines.

I tested copying "Contoso" from the old instance to the new, using the same "orchard>package create Contoso c:\temp" codegen syntax.  that package imported perfectly.

I've either got the twg1 theme in a non-conforming state or bind.pt has a mechanism to prevent repackaging their theme (twg is derived from Bind.pt Flexo).  I'll contact Bind.pt, but I'm inclined to think that this is a code/config problem on my side.

The theme runs "correctly" on the old site.

I'm not sure. I build & deploy all modules I use from source, so I don't use packages. Someone else might be able to help with this. 

mcullina wrote:

Successfully added 'Orchard.Theme.twg1 1.1.0' to C:\Websites\MyWebSite\

Package uploading and installation failed.

My next step on this was easy.  As you can see from above, the theme installed - but all the content failed to transfer (it failed to load into the nuget package).  I simply copied the theme's folder contents over under the theme's directory.   It immediately worked.

Now, my next question is back on the topic of export/import.  Most of my imports went perfectly: blog, pages, menus, etc.  However, I had a problem with HTML Widgets.  My export.xml file is perfect.  I think that I may have made the mistake of importing the data before having set up the layers needed.  I think that this has orphaned this data.  I could re-import and perhaps I will do this, but I want to understand what's going on more precisely.  Therefore, I turned to my existing site and I find that I'm confused about how the system knows which widgets are activated for a given layer.

I know that the widgets are stored in Orchard_Widgets_WidgetPartRecord.  I know that the layers are stored in Orchard_Widgets_LayerPartRecord.  I thought that one or more of the "Containers" tables would provide the linkage between the two tables, but they do not. 

I looked at:

\src\Orchard.Web\Modules\Orchard.Widgets\Handlers\LayerPartHandler.cs and \src\Orchard.Web\Modules\Orchard.Widgets\Views\EditorTemplates\Parts.Widgets.WidgetPart.cshtml

which led me to:

\src\Orchard.Web\Modules\Orchard.Widgets\Models\WidgetPart.cs

but I don't understand how this works in this context:

<summary>
The layerPart where the widget belongs.
</summary>

public LayerPart
LayerPart {
get { return this.As<ICommonPart>().Container.As<LayerPart>(); }
set { this.As<ICommonPart>().Container = value; }
}

How does the Layer know that a given HTML Widget has been assigned to one of its zones?

thanks,

Mike

The Layer doesn't have to know, because the widget knows to what zone it has been assigned to. If the Layer did want to know, it would simply ask the Widget. It would go something like this: "Hey, Widget! What zone you at?" Or, in C#:

IEnumerable<WidgetPart> widgetParts = _widgetsService.GetWidgets(layerIds: activeLayerIds.ToArray());

            // Build and add shape to zone.
            var zones = workContext.Layout.Zones;
            foreach (var widgetPart in widgetParts) {
                var widgetShape = _contentManager.BuildDisplay(widgetPart);
                zones[widgetPart.Record.Zone].Add(widgetShape, widgetPart.Record.Position);
            }

See "WidgetFilter.cs" for how this works exactly.