Re-thinking the Orchard data access layer/API

Topics: Announcements, Core
Oct 28, 2014 at 8:11 PM
On today's Community Meeting we started the discussion (that we already had shortly before) on what to do with Orchard's data access layer and data access API.

For those not very familiar, a recap: in the core of Orchard's database access is NHibernate. However NHibernate is abstracted away under multiple layers, so what you actually access are the following APIs: IContentManager (talking about the Get/Many methods), IContentQuery, IRepository<T>, IHqlQuery. All of these have various limitations. We came to the conclusion multiple times that the best would be to build a new, only, generic data access API instead, probably by surfacing NHibernate.

Furthermore we seem to have performance issues in regards to NHibernate as well (which are not necessarily performance issues of NHibernate itself).

So the discussion is on: what do you want to see from a new Orchard data API? What do you like or hate today?

Sebastien will come and share his notes from the meeting.

I personally think that one of the biggest advantage of NHibernate is HQL, a database-agnostic SQL-like query language. It's awesome because you almost never want to use it, because keeping to ORM is enough; but in certain edge cases you have a tool that you can use instead of resorting to SQL (what in case of Orchard, due to multiple DB engines being supported, is an absolute no-go).

Some food for thought: promises (almost) database-agnostic SQL query generation.
Oct 29, 2014 at 1:31 AM
Edited Oct 29, 2014 at 8:32 AM
What would we need:
  • store parts on a plugable file system using an int index inside a transaction ( see update ), the int index being auto increment ( see index generator)
  • retrieve these parts without generating any lock using the index, not necessarily inside a transaction
  • update these parts with a transaction to keep integrity. In case of race conditions use a timer to return an error and rollback any prev update/create in same transac.
  • a db server side lock manager working on indexes and using timer.
  • a db server side transaction manager accepting several transactions concurrently in the same http request. Managing commit, rollback and abandon, managing hierarchy in nested transactions. Managing transactions for non http request.
  • a query language based on linq because already in c# and easy to plug on a db system. Preferably to direct SQL.
  • server side ( db server ) procedures callable by name with parameters in order to avoid bringing back all data in the client ( orchard server) just for a filter. Allowing cleanup and complex tasks as storage initial allocation, ETL. Managing some king of pagination on query multiple with or without locks.
  • a server side ( db server ) profiler
  • optional indexes on text and date fields.
  • global index generator to generate unique values for several clients (orchard servers).
  • a backup/ restore interface
  • restful API
  • voice command in v2, direct mind image to code in v3 :)
Oct 29, 2014 at 7:19 PM
Well... if we're already actually building a completely new abstraction, personally I would not mind seeing NHibernate thrown out in favor of the new lightweight, modular, extensible and portable EF7.

I also have my suspicions around actual real-world usage of anything other than MSSQL... :) But that's just a hunch - perhaps more folks use SQLite, Oracle, MySQL etc than I would have thought?
Oct 29, 2014 at 9:09 PM
@CSADNT: I'd like to see world peace, unicorns and reasonably priced printer ink too :-). Joke aside: as far as I understand you'd like to see a radically different data storage mechanism (you used the expression "file system" so I'm not entirely sure). Would you explain why do you see a custom data storage mechanism as the way to go instead of using an SQL database?

@Decorum: does entity framework support a database agnostic mechanism for doing multiple operations on the database server in one query? An example: update a counter by one; in simple ORM terms this is a get, then an update, but in SQL this is a single update.

Regarding other databases: I personally never had and probably never will use anything else than SQL Server and the lightweight database option bundled with Orchard (currently SQL CE) but I fully agree that Orchard should support more than one database engines. Currently it doesn't need a lot of effort to keep the different providers up to date and somebody wanted Orchard to run on e.g. MySQL badly enough to implement MySQL support - this shows something. Also being able to use an embedded database is awesome: now we have SQL CE, probably we'll (also) have SQLite but nevertheless we'd need at least two providers anyway.
Oct 29, 2014 at 10:19 PM
Edited Oct 29, 2014 at 10:37 PM
Yes -I try to use each word in correct place, even if it seems sometimes strange :)-
I think the problem of orchard is in using an ORM because trying to map objects to sql is perverting the original concept of contentitems aggregating parts with properties and fields.
It is not only the oversized time and the memory spent and used to map orchard objects to something totally hierarchical but also the thinking which are perverted by this SQL target.
Why do we need all these query tools (HQL, Criteria, direct SQL, etc) ? Only because there are these sql tables on disk.
Orchard uses less than 10% of what SQL systems are offering (no views, no sps, no replication, no log file, etc.)
Our mind is perverted by it.
I think it would be more simple to have a stored structure corresponding to orchard concept. This is also why I speak of a plugable file system, to be able to run orchard on the one the envionment offers: Windows file system, but also linux, mobiles, cloud storage, etc.
The file system should be plugable, not the DB which in turn plug the filesystem. Less layers.
Yes I think this community should have the 'courage' to try something really new as the aggregate with IOC idea has been on its birth.
For me this new concept would be creating its own 'storage server', how to call it, able to run only orchard required operations in each os where runs Orchard. And they are not so numerous (compared to the hell of what are doing jewels as Oracle or SQL Server, but not needed and complicating the life of an orchard dev).
Writing a lock manager and a transaction manager when you don't want to manage everything as is doing an sql DB is not so complicated today in .net, writing a REST layer in the server to communicate with external world and able to run automated tasks with parameters on request also.
I estimate that 70 % of code samples to use are already present in orchard.
Oct 29, 2014 at 10:35 PM
We could split the conversation into two:
  • what does the storage should look like if v2
  • what is the best data layer we can have in v1, using the current storage
Christian's comments make sense for v2, but I am trying to solve some issues in the v1 scope.
Oct 30, 2014 at 11:22 AM
What about some baseline common sense and focusing not on what the developers of Orchard want, but the users who have to work with what you come up with?

IHQL - how many developers do you think speak HQL compared to LINQ?

.NET already has perfect abstractions built in for quite some time. IQueryable and LINQ.

I am all in favour of exposing some lower level access, but please do that in the standard .NET way so that those of us who have been using the language can just sit down and write some queries ;)

LINQ has a lot of advantages - among them that people actually know it.

I also question the use of anythign excel SQL Server in production - realistically. I see all this "abstract the db away" but at the same ime the orchard database is quite full of bad abstractions and the lack of any server side functionality is having a performance impact.
Oct 30, 2014 at 12:20 PM
@CSADNT: I see your points. I don't know if writing our own data storage from scratch is something we could realistically aim for. It's a different question whether we want to stick to SQL: this came up before too. The general standpoint was that SQL is a widely adopted and trusted technology, while others like document DBs are less accessible (both technologically and in enterprise politics) despite a document data model would be a natural fit for Orchard. Nevertheless we could explore the idea of choosing an abstraction of the data access that would ultimately allow you to use an SQL DB as well as something else for storage. But again I don't know if this is feasible or possible in a way that prevents us from implementing the wrong level of abstractions again.

@Sebastien: I thought this was only about v2, didn't know you had something about 1.x. Please open a new discussion for the latter one then.

@NetTecture: how about noticing that this is a public forum thread that is specifically here so everybody (including "users" with what I believe you mean developers who build on Orchard but not Orchard itself) can share their opinion before any decision is being made? You telling that the developers of Orchard (what, you have to see are not just those who have commit rights but everybody, since Orchard is open-source; yes, you too!) are only focusing on their own good is not only downright insulting but also shows that you lack the knowledge that literally everybody who commits to Orchard also builds applications on Orchard for clients.

You misunderstood what I've written about HQL. With NHibernate the point in my opinion is not that you'd have to use HQL (nobody said that) but besides all the ORM functionalities (including LINQ to NHibernate) for the edge cases you'd have the option to resort to HQL instead of SQL when the OR abstraction doesn't cut it (and there are cases like this). We're specifically talking about simplifying the data access and removing abstractions of abstractions because there are multiple APIs currently that each have their own drawbacks and limitations and provide unnecessary abstractions.

As I said before, if you say that we should only support SQL Server then you automatically say that we shouldn't support embedded databases like SQL CE. Any Orchard developer can tell that using an embedded DB radically simplifies development in cases where you're rapidly changing your code (Want to reset? Copy-paste.). Also not having to set up (or even understand) SQL Server (or any external DB server) significantly lowers the barrier of entry for Orchard since you can just run it OOTB. Not to mention that there are people who use it in production for low-traffic sites.

So dropping support for multiple DB engines is short-sighted in my opinion. Aren't people using MySQL under Orchard? 81 discussions and 30 issues mentioning MySQL suggests otherwise. And don't forget that MySQL support wasn't something forced into Orchard, it was a community effort coming from the outside, just as SQLite (it was done but is not integrated into the current branches) or Oracle (which is something rather experimental and seems that Orchard couldn't really support Oracle DB even if we wanted to due to technical limitations).

Orchard is not just a software that people use but also a platform that they develop on. Any fundamental change will impact everybody relying on this platform for their own services. Orchard's core value that also makes it unique is its unmatched flexibility that allows it to support scenarios nobody dreamt of initially (this careful engineering makes extremes like running Orchard console applications or cloud workers possible: Take flexibility away, make restrictive decisions because you personally don't see the point of flexibility and we end up with Just Another CMS.
Oct 31, 2014 at 4:49 PM
Whatever happens, I'm pretty sure nHibernate is what has to go.
I'm trying to do something that should be really simple, but nHibernate is what hurts me again and again.

After all the problems I encountered here

Trying to use the APIs and eventually just trying to create my own HQL.
Both of those efforts were met with failure.

Now I'm trying to create my own SQL and I can't do that even.

nHibernate uses a colon to prefix sql parameters, but you can't escape it.
In case you actually did want a colon, you can go screw.
Oct 31, 2014 at 6:04 PM
Edited Oct 31, 2014 at 6:06 PM

My opinion is best option would be to move to Entity Framework and to a library like the one mentioned by Piedone for direct SQLs.
I think the wish of continuing using HQL is not enough reason to maintain whole NHibernate thing.

Related to my preference of Entity Framework over NH reason is I think first one is taking the right direction on its vNext version giving support to NoSql dbs. It ends with an old problem: ORM = easy programing but poor performance (due to impedance mismatch). Now the impedance mismatch completely depends on the kind of database you select not on the fact of using an ORM. ORM now is simply a lingua franca for projects that need to support a wider range of diferents dbs. In fact, it won't be strange in the future we find ourselves using one nosql db and an sql one within the same orchard app. In that situation it would be very convenient to have the same data access API to work with both.
Oct 31, 2014 at 7:20 PM
@jersiovic can you point to resources elaborating plans of Entity Framework supporting any non-SQL databases? I'm not questioning you telling the truth but I can't really find any (if you don't count this: and I'd love to learn more.

Technological considerations aside I'd prefer Entity Framework just because it seems to have support from Microsoft for the foreseeable future while NHibernate's future seems less dynamic (one key difference is async what is already supported by EF but there are seems to be no plans to integrate it into NHibernate; this illustrates the difference). For NHibernate vs EF see:
That said we'll surely find shortcomings of EF that would make such a migration worse, it being already hell of a work (and breaking a lot of 3rd party code, so a hell of a work for the whole community). Such a breaking change would also mean that since upgrading to the new Orchard version would be hard we'd have to support older versions longer. With this we risk a fragmentation of the platform like it happened with Joomla.

Obviously changing the ORM is huge work and a very impactful decision, so I'll oppose it for the good of the platform as long as it is not crystal clear that a) staying with NH is a disadvantage compared to moving to a different ORM (or generally: data access layer) and b) the suitable new candidate is better than NHIbernate or at least doesn't require us to do any big compromises in every field we care about.
Oct 31, 2014 at 7:47 PM
Edited Oct 31, 2014 at 8:11 PM
So if this thread is not reserved on solving 1.x problems as it seems, I would say that the inner question would be:
  • should we offer a migration path from 1.x to next ?
My personal answer is NO, because
  • too much energy lost in this writing
  • too many new bugs
  • penality introduced on new system not suffering from actual NH but having to keep compatible
Still personal thinking:
1) if the next version is NH based, it will be less interesting.
2) if next version uses an ORM, it will suffer from the same ORM inherent problems as NH, what I see from EF 6 and next 7 versions has same pbs as NH, only progress is that actually MS supports its dev....but should we base our choice on the fact that MS supports it ?
Nov 1, 2014 at 12:27 AM
Edited Nov 1, 2014 at 12:37 AM
Well, the description of the project in Github say they want enable access to data across relational and non-relational stores.
It is an strong statement in the description of the project to be a secondary objective

Furthermore, you can see there some project notes designing one non-relational provider for Redis,-2014#ef7-redis-provider

Indeed following my experience working on multiplatform projects I can say It is not something to worry about. In my case I have worked in multiplatform project which had to run initially on the one hand in .Net and SQL server and on the other hand in Windows Phone 7 and SQL CE for windows phone 7 (which forced you to access it through a kind of mini ORM, so SQL was not available). I decided to abstract my access layer: repositories and UoW and allow in domain layer only LINQ but not technology dependent of the access data layer. The abstraction was very similar to what EF offers but I was not sure if in the future I would migrate to Nhibernate so I prefered to abstract it. The implementation of the UoW was interchangeable between the one for SQL CE ORM or the one based on EF using IoC

After that we find performance issues with SQL CE ORM. So, we decided to change to nosql database: SIAQODB an embeded db which offers a Linq API. An this is the important point of this story: I managed to make a implementation of my UoW easier than I ever had though: Tracking entities was not difficult and the fact it offers LINQ API save me from implementing any kind of maping between domain entities and stored data. So the conclusion for me is the same EF7 devs arrived: an ORM does a hard work for reducing the gap between objects and relational dbs but gets not so good performance, however it is far easier to implement this mapping with Nosql dbs which indeed will perform better. This also will help to normalize access to all these plethora of new nosql dbs.

So maybe you don't find between the providers designed by the moment for EF7 anyone for the nosql db you would like. However I think this is only a matter of time, specially if there is a LINQ API for the db you want. In that case I think you won't need to be an expert to create your own provider.

A little sample that illustrates what I mean:

Regarding other advantages of EF 7 over NH: Its support async await is a strong difference, support for Winphone and win8 apps also is very strong for me for developing standalone apps based on orchard. If I can run Orchard on devices using it as develop platform library it is likely I will use it for a bigger number of projects than now. Currently I only use it for web projects. Finally EF 7 supporting NoSql dbs removes a great number of the issues people blames to ORMs, because indeed those problems are due to the mapping mismatch between object model and relational dbs.

But main advantage I see is that NH forces you to select between ORM or NoSQL (maybe your own db system), however EF7 seems to attempt to let you plug anything that fits with an object model. So for me EF helps to focus Orchard community on how next version of Orchard will be instead of redo the wheel creating an own db. And if it is still interesting in the future then make this new db with an own provider for EF
Nov 1, 2014 at 7:56 AM
Edited Nov 1, 2014 at 8:29 AM
Interesting experience, especially for mobile. But I still can't imagine orchard running on a mobile because many other things are missing (IE or Owin, DNS are the main). Having a web site running on a mobile should be an interesting object to enhance local navigation without servers and without ... operators, nice utopia but....
Back to the subject, I don't really see the interest of asynch data access for a web site. I have tryed asynch ADO.NET and it is not especially usefull?
And concernig the ORM, it only makes sense if you target a set of preexisting DB Engines, reducing everything to the common denominator for all those engines. Looking to actual NH it is so difficult to have something working equal between even the SQL Server familly, not to speak of My SQL, Oracle has never been contacted, Postgress I don't think so. So the question is Why an ORM when you use SQL Server in 90% of cases and with only 10% of its functionalities. Is the result worth the burned energy.
EDIT: For EF7, I follow the actual dev and it has nothing really breaking from last version, even concerning integration of IOC. And many EF6 features has been abandonned. I am not sure the engine has a long futur.
Nov 1, 2014 at 9:09 AM
Well where you see an utopia I see a chance, navigation in wp8/win8 apps uses URLs and Orchard architecture allows replace components so maybe there is a path using alternate components that replace Orchard not available ones for navigation in a mobile environent in the future. Indeed I didn't mention it should work in client-server way, it would work as a standalone app. But I agree this is a bit off topic at this thread and a chance of seeing it implemented a bit far in time.

Related to interest in async await I think that if you are interested on performance then you are interested on async await or at least on other previous .NET asynchronous methods (those others are the hard way in comparison to asyn await). Take a look to this post

I agree with you that an ORM is a common denominator but the point is how much you cover with this common denominator. If the part you need to code specifically for each different db you want to support is small then is worth the cost you pay for supporting current DBS and future ones. Specially when DBS available are not only SQL dbs as years ago. Now the dbs available are very different so the role of a common denominator is even worth.

In my case I was able to fit a nosql db which appeared after EF was there. Indeed if Orchard finally uses EF7 I would give a try to a db like Brightstardb to see if it improve things or other which could appear in the future. Can you say the same?
In the project I mention for example I found Siaqodb allowed me to improve dramatically performance of Include nested entities operations. An operation available also with entity framework + sqlserver but that has poor performance due to arquitectural limitations of relational dbs. So this is a sample that shows how abstracting DAL can help you harness future DBS. I know that is neither a rule for all future nor existent DBS but if we are talking of DBS oriented to object or near approaches probabilities are greater.

Related to what to expect with EF7 I have pinged someone involved in the development of EF7 to take a look to this thread. Maybe he can answer some of the doubts arised about future of EF which I cannot answer because I don't have knowledge of which is the state of the project
Nov 1, 2014 at 9:59 AM
Edited Nov 1, 2014 at 10:08 AM
I understand your points, but related to async db calls, even the article you mentionned says
Database – worth using async, but not a lot of gain for simple accesses
The overhead of using async is small: in my real world tests something like five milliseconds, maybe ten if the context is large. Running side-by-side testing on an ASP.NET MVC5 shows a small difference between sync and async method for small database accesses, i.e. they approximately took about the same time and there was little gain on scalability. This is because, for small database accesses, most of the time is spent computing rather than waiting.
and my personnal tests on sqlserver 2012 (sorry no numbers) and a real application provide no clear advantage ... but code harder to debug.
The real time shift in orchard db access is to avoid bringing to the webserver all the data from the dbserver just to filter it and send back to client only 1% of the moved data. This kind of task is the traditionnal pitfall of all ORMs and to be efficient, nothing better than do the filter job directly on the dbserver.

Is being able to use one either db present on an environment easier/better than to have a dedicated open source 'storage' source code that you can build and use in this environment?
Would building the source code of such a dedicated and efficient storage be a larger effort than adapting to an ORM and then to all these db systems? I am not sure.
Nov 1, 2014 at 11:00 AM
Edited Nov 1, 2014 at 11:17 AM
Yes you are right what async/await impact doesn't look as good as it was promised.

What I don't see as clear as you is this:
"The real time shift in orchard db access is to avoid bringing to the webserver all the data from the dbserver just to filter it and send back to client only 1% of the moved data. This kind of task is the traditionnal pitfall of all ORMs and to be efficient, nothing better than do the filter job directly on the dbserver."

What kind of filterings is Orchard doing in memory instead of in server? The reason is current lack of mechanism from current data access abstraction added by Orchard? A lack on Nhibernate? Or a lack also in other ORMs available? Are you talking about complex queries not simpler ones, isn't it? Something that LInQ2SQL (select, join, where and include) cannot do properly. Can you share of a concrete example to grasp better available solutions? Maybe changing strategy of how data is stored problems disappear.

Build an own storage db that scales properly in the cloud and fits Orchard needs is an interesting option but it seems a hard work one. When you start a path by your way usually lot of non expected problems appear that others has solved. I don't say no to this. But maybe is starting from the scratch too many things to have success in near time.

Maybe we should focus on which exactly services this DAL would offer to upside layers and then evaluate the cost and limitations of implement them with each available option: NH, EF, own db, ...
Nov 1, 2014 at 11:14 AM
Edited Nov 1, 2014 at 12:02 PM
Related to how much NH is a problem for Orchard one question.
Why to declare a ContentPart in Orchard you need to declare its related ContentPartRecord instead of having only the class mapped to db and a configuration class to determine which fields have to be stored in db? It is sth imposed by NH or is it a design decision in Orchard?

I ask this because repeating similar code for those two classes is something I don't enjoy too much ;)

Edit: I would prefer that by default all are fields stored in db and in migrations have the possibility of setting which fields have to be ignored.
In relation with this what do you think of the availability at runtime of a tool which generates code and compile it dynamically for content parts classes even in relation between various parts. Sth similar to what EF Model First does with its model tool but specialized in Orchard. I say it now cause in some way is sth to take in mind when designing the DAL If this is sth we are interested in.
Nov 16, 2014 at 11:44 PM
Edited Nov 16, 2014 at 11:45 PM
@jersiovic: improvements in this space would be nice, but it's also worth calling out that you don't need a record backing class for your parts. this is something that is not well discussed but certainly simplifies life if you don't need a custom table.
Nov 17, 2014 at 12:02 AM
On the main concern of this thread... I think the team really needs to evaluate how this platform is trying to mature. There are a lot of good points about ORM and other techniques but I think the team needs to think about simplifying and by focusing this platform for the platform they wish to serve the most.

Stop and look around - where are your biggest customers? what do they need in order to continue using the application? what are they using?

I can't tell you how many times I've walked into an engagement to find some software that is abstracted and overly designed to fit into every possible socket possible. The exercise of flexibility is super cool when you're a baby project. You need someone to find your app useful. As the app gains maturity, though, it's best to transition away from the gadgets.

Think about this way... how many of you have ever bought a fancy electrical adapter that has the slide out plugs so that you can carry one adapter to 12 countries and always have the right plug? Neat concept. However, most times they are super ineffective. Everyone dreams they are going to go 12 countries and this box will always be best thing since sliced bread. Then reality sets in ... they are bulky. often times they are not reliable. they don't hold up to high loads. they have crappy fuses (if at all) that you can only find in 1 country anyways... and finally you realize that you won't ever need more than 1 maybe 2 adapters for a single trip and in the end you figure out that if you buy the right 1 adapter for your use, you get much more use out of it (and it's more compact, holds more loads, easier to use). What happens? The expensive "do it all" adapter is thrown away and you get yourself some simple reliable adapters.

If I were to look at the platform today, it seems this was always built to be a MSSQL app. If you find your user base is primarily sitting on that technology stack, leverage that as much as possible. Get rid of gimmicky do-it-all layers. :)
Nov 17, 2014 at 1:02 PM
I agree with your vision Maurice about simplification but if you also imply that Orchard should be MSSQL only, then I don't agree for the same reasons I already expressed above.

.NET, especially as the recent complete open-sourcing shows, is moving towards being truly multi-platform. Orchard in its core design is multi-platform as well (and already runs on Linux with Mono, albeit with some tweaks). Taking a step back and stripping down the data access layer to only support Microsoft's SQL implementation would be something completely backwards IMO.

Also, Orchard was always built to be an SQL app, but not specifically an SQL Sever one: SQL CE (which, although being developed by MS, is a slightly different SQL dialect than SQL Server with much different limitations) was always built in. Also keep in mind that even if we would drop everything and have only support for SQL Server there is still SQL Azure, which isn't 100% SQL Server. So again, if we only wanted to support everything Microsoft only (what I oppose) then we'd still have to support at least two SQL variations.
Nov 17, 2014 at 8:58 PM
Piedone, the idea that I was trying to drive home is that you can either be generically OK for everyone or decide to be really good for someone.

When I mentioned MSSQL, I wasn't referring to just a version but the entire stack. I've spent the better part of 15 years building apps on every msft sql box imaginable (wmsde, express, standard, ee, ce, azure), so I'm fully aware of the differences. Even working within the msft stack, it's hard for an app developer to get the right mix of features lined with the underlying platform. However, if you can start to light up features as you move up to more powerful versions, that's where an app really starts to gain traction. To me that is the value of selecting a platform. Yes, we can deploy with CE, but when your move Azure SQL you get X... when you move to Standard/EE, you get X, Y and Z.

Personally, I mostly don't care if the team picks the msft platform or some other platform. I do think they should focus and make themselves really good on a platform. That will help elevate this product from neato to indispensable.
Nov 18, 2014 at 5:11 AM
Hi Maurice, you said
"... really good on a platform. That will help elevate this product from neato to indispensable"
Related to this, can you provide an example of CMS that thanks to be dependent of a db has been indispensable?
In the CMS world success seems to come when a product gives end users what they need very easy. And yes it should perform well but it can be achieved with different data access technologies so in this case data access technology uses doesn't seem the key feature for success.

If we understand Orchard also as a develop platform then data technology is relevant for developers who want to use it for their projects. Do you think to be tied to a db will make a develop platform indispensable? In my opinion the key feature for a develop platform to be successful and durable in time is flexibility and to keep update to last technologies in time. Within this flexibility I also expect to find space to customize data access for optimizing it as much as I can for a concrete need in a concrete project where I'm using it.
Nov 18, 2014 at 8:56 PM
I think it all comes down to how you plan to view/use Orchard.

If I look at your question, I could walk away with the notion that you are saying data store features are irrelevant. If that's the case, then I completely disagree. Every database product out there has different offerings from free to premium. Moving up that stack gives you more features that help either the product or help the business. This is why the major db vendors of the world are successful with their products. Yes, you can get free SQLCE or SQLE but if my business wants to improve the application (computing power, indexing, io, etc.) or address different business needs (redundancy, failover, etc), then there are different versions that I can acquire to help me achieve those goals. This is how an app becomes indispensable. The app doesn't believe it's the end all and instead utilizes the platform to deliver more.

To look at it a different way - If I put my dev hat on, I can envision Orchard as being the sandbox that allows me to test new ideas. As a I dev, I realize there are projects to push the boundaries and there are projects to allow me to get other work done. Right now, I hear (and I'm part of the crowd) folks asking the team to look at growing Orchard so that it can become a really good platform to get work done. It's pushed the boundaries and won recognition. Now it needs to refine and continue pushing in focused areas. I'm the dev that wants a better platform that makes good decisions to help this grow. I'm not the dev that wants to keep this as an experimental sandbox always trying to prove it can do everything.

Now... if I put my admin hat or try to view the platform through the eyes of a client who wants functionality, I would want Orchard to be much more stable and really make the most of what it uses. I don't really care if it can plug into 12 different stores. I want it to plug into my store and do it well. If I can upgrade the data store independent of the app, then I can begin to open up more opportunities because new functionality lights up. This way I can say this application is indispensable to my company.

To circle back to the intent of this thread: I'm in agreement with several comments already. Unless you can prove a need to change, the cost to change might be too much to bear. If the decision to change is made, then I believe that it's ok to focus on a technology stack/platform and simplify things. There are a lot of ways this product can win with that approach and that's what I've been trying to advocate in my past comments.
Nov 19, 2014 at 2:41 PM
Another interesting open source DB to have in the radar:
Nov 23, 2014 at 6:43 AM
@CSADNT yesterday I attended to a talk related with problems you will find when you move your apps to Azure. There they explained us that between other things they talked of the importance of async await in the cloud. The reason is that if you rent a VM in the cloud with a processor/memory specifications similar to one physical machine you own you will notice important differences in performance. They showed that Azure VM performance for monothread actions performed pretty bad and same operations using async await worked pretty well. However in the physical machine the monothread results wasn't bad in comparison to multithread ones.

So it will explain why the tests of the guy of the post I linked previously in this thread showed no improvement for async operations. Because he used for the tests a physical machine: "an Intel i7 processor clocked at 3.31GHz. The database is localDb running SQL server 2012". So the async/await thing I still think it is something it should be taken into account for vNext.
Nov 23, 2014 at 7:39 AM
We can hear so many talks in the programing area today...
I have deployed and I am managing, since nearly 2 years, 22 servers in azure, most part in Cloud Service, some in Vms. They are installed in 15 azure regions. Some are pure owin self hosted net 4.51 REST engines, some hybrid Web/REST using MVC, others are Orchard 1.8.
All are using cache, storage, SQL azure and many azure features.
They process a great number of very small db access using direct sync For some tasks they use EF 6.
My try with async proved me it was nok for this profile of app.
So my answer is: you need to bench it yourself, there is no generic answer. It depends your app ... and also the azure region where you run, azure datacenters are not all equal.
Nov 23, 2014 at 9:57 AM
jersiovic wrote:
@CSADNT yesterday I attended to a talk related with problems you will find when you move your apps to Azure. There they explained us that between other things they talked of the importance of async await in the cloud. The reason is that if you rent a VM in the cloud with a processor/memory specifications similar to one physical machine you own you will notice important differences in performance. They showed that Azure VM performance for monothread actions performed pretty bad and same operations using async await worked pretty well. However in the physical machine the monothread results wasn't bad in comparison to multithread ones.

So it will explain why the tests of the guy of the post I linked previously in this thread showed no improvement for async operations. Because he used for the tests a physical machine: "an Intel i7 processor clocked at 3.31GHz. The database is localDb running SQL server 2012". So the async/await thing I still think it is something it should be taken into account for vNext.
We have a large app running Orchard 1.7.2 in azure VMs, pointing at SQL Azure. Its not performing great (I'm sure a some of that is our problem. But it is really enhanced on the cloud VMs. My local laptop returns a set page consistently in 600-800ms (pointing at the same SQL azure db). The azure VMs (A2s currently) take 2700ms to return exactly the same! The only difference remaining is the fact that it is on a VM. There are two iroonies here - (1) the latency from my machine to the azure db is significantly higher than the VMs, and (2) We barely use the orchard db, as all our data is in external APIs.

If what jersiovic said is true, then good async await is critical. Personally I'd prefer Orchard to run really well on one tech stack, than poorly on twenty. I'd also agree with comments that Linq is the defacto mechanism to get data in .NET, and it would seem sensible to use that as the query layer, but then, I've never needs to deliberately make a query in orchard ever.
Nov 23, 2014 at 10:47 AM
Edited Nov 23, 2014 at 10:54 AM
One of my server is running Orchard and it receives frequent sollicitations (REST calls dozen by second) from the Microsoft Azure Store platform where our products are sold. Each time one of our users connects to azure portal and open the Add-on page orchard receives REST requests to auth the user and to provide running details, some of the requests have a very small delay (200ms ) to be answered, in case of no answer Azure Store replicates the request....hard to live.
My problem here is that NH is killing me. Not Orchard but NH. NH is slow and its API very limited.
Concerning the SQL Azure delay, I have noticed that:
  • SQL Azure behaves differently depending on its hosting datacenter
  • Regularly (weekly,daily ?) SQL Azure network connection experiments a failure: my servers have all automatic ping sent from different continents each 2 minutes (azure feature), and I have a DB call associated, and this shows me the network connection PB, it is a DNS pb, during a small intervall, the DNS is no more solved.
    I spent hours trying to find the NH pb for this incident, but my opinion is that for this NH is not the cause of the pb, but NH is totally unable to restart something on such condition (or I would have to re-develop large parts of orchard code and use very specific parameters very unlikely to be acceptable in production).
I am now thinking to move my platform facing azure store to something more specific leaving Orchard because NH is so 'everywhere' in Orchard's code...
Nov 24, 2014 at 2:34 PM
Regarding async: one huge current drawback of NHibernate is the lack of async support although there are efforts for it (see:
Nov 25, 2014 at 5:20 AM
Edited Nov 25, 2014 at 5:26 AM
@CSADNT regarding Aync/Await performance I want to work in a post with results of different tests looking for borderlines in async/await use in the cloud. It would be nice to have a table that show time diffs in different scenarios to take the right decision.

Regarding your problems with Nhibernate, on the one hand could be they caused by what Sebastien commented: that currently Orchard uses NHibernate always with tracking entities enabled? It hits performance unnecessarily when you are only reading data, and you don't have any intention of writing changes.. In my apps I use two kinds of Units of work and repositories. ReadOnly ones and normal ones. In that way it is easy for a programmer know what is the intent of a code when he/she see it.
Maybe it is something similar applicable for next Data access version of Orchard?

On the other hand, the other problem with NH is the work it has to do for mapping queries to SQL. My answer to this is that EF will improve this mapping because we should have the ability of plug-in other DBs more objected oriented for which those mapping should be a piece of cake, not penalizing as much performance as object-relational mapping.
Nov 25, 2014 at 7:30 AM
Edited Nov 25, 2014 at 7:32 AM
Thanks, but I will go on other challenges if I got time, my thinking being that in very near futur we will not need all these sexy layers of code named ORM and DB engines because there will no more be hard drives.

We all know this:

and the kind of question this guy is interestingly developing is already obsolete

Concerning my own app context config I evoked previously, I am always writing something when accessing DB...and I think that writing dozen lines and classes, folding the code because NH is poor is not a way to stay open to evolutions (why all these double classes, why these unique transactions, these limited mappings, etc.)

Wishing the best to Orchard community which is plugged in NH and MS as a bee which has fallen in the honey of the hive.
Nov 25, 2014 at 8:36 AM
@CSDANT - Erm... just because there will be no more HDD, people will still be using Solid State... so you will still have a DB or Storage of some type.

Just because we use NH now, what makes you think we will use it in the next major release of Orchard? At the moment the VNext implementation has No data layer - 0% of a data layer, zip... none, nadda.. not even a Library referencing a data-layer. This is because conversations like the on Zoltan and co are bringing up are about what to do for pre-2.0 and what to do for 2.0.

And yes, we have roots in MS... it would be stupid not too.
Nov 30, 2014 at 8:51 AM
Edited Nov 30, 2014 at 10:47 AM
Did you already make a firm decision on data access strategy for vnext?

I still would like to keep nHibernate because I have some positive experience with it and no any experience with EF.
Not sure how EF7 compares to NH4, but for earlier EF versions there were a lot of NH features missing in EF.
Here are some disappointments about EF7:

Not sure which CMS to select, this one or N2 which uses NH too. Btw, N2 admin GUI looks better for me.
I need a base for my own web project, would like to avoid maintaining commodity login, forgot, pages, etc.

Though Orchard seems to have more developers for better progress in the future.
Nov 30, 2014 at 6:06 PM
@jersiovic: although I can't reply to that Daniel Dabrowsky ( commented before that Orchard uses NHibernate in a sub-optimal way. I asked him to participate in this discussion.

@sanyock We are nowhere near a decision yet. For the next (1.9) version Orchard will stick to the current data API (or something compatible) for sure.
Dec 2, 2014 at 1:55 PM
Edited Dec 2, 2014 at 1:56 PM
Async access to DB can give us totally nothing if the code is running synchronous anyway. EF async feature shows that its speeds up only data modification ... maybe because its speeds up "dirty checking" in EF only?

ORM is tool which helps us to change data. This is not the best tool for "reporting" purposes like just reading data across many tables to display something. No matter if it is NH, EF, Linq2Sql and so on.

@jersovitc: Try to find complicated query in Orchard and try to implement in in EF.

In enterprise application, using ORM modifying data is quite easy but sooner or later you need to reach for pure SQL .... session.CreateSqlQuery("SELECT ....."). Of course there is one BUT (beside sql dialect in different db's)... Unit of Work. In Orchard, Developer cannot control whole "request pipeline". Orchard has many extension points. Let's assume such situation.

We have 2 modules A and B ... both of them use entity type ContentItemRecord
  1. Transaction begins
  2. Module A reads some records with some WHERE statement ... SQL select query goes (NH keeps all fetched entities in 1st level cache... in its session)
  3. Module A modifies only one of fetched record.
  4. Module B in other extension point wants to display some of the records using WHERE statement ... Does it get modified data in point 3 ?
NH helps to make it consistent. When some module wants to query type of entity which is already in 1st level cache by default and in Orchard it calls flush and makes dirty checking. This flushing can be controlled, example
        public void Use_default_FlushMode_makes_flush_before_select()
            // Arrange
            var t = new Thing { Id = 1, Name = "Foo" };

            // Act
            var results = this.session.CreateSQLQuery("SELECT * FROM Thing")

            // Assert
            Assert.That(results, Is.Not.Empty);

        public void Use_commit_FlushMode_makes_NO_flush_before_select()
            // Arrange
            this.session.FlushMode = FlushMode.Commit;
            var t = new Thing { Id = 1, Name = "Foo" };

            // Act
            var results = this.session.CreateSQLQuery("SELECT * FROM [Thing]")

            // Assert
            Assert.That(results, Is.Empty);
Why is it a problem? Because everything is a ContentItem and is connected to ContentItemRecord. The more content items we load into NH Session by many queries the more is dirty checking has to be made before every query. Try to run MiniProfiller and you will notice how many queries are done during one request ... Layers, MenuItems, Widgets, Images, Users, Roles. These all are taken from on a table and this is one type of entity "ContentItemRecord"

Do you guys still thinking that changing NH to EF makes any difference? ... No fu@#@# way

It will be not any better with EF because it can be fixed and NH is far more extensible than EF to do it. Controlling flushing, event handling user types and so on

I have not much time to explain it everything in one post right now because it takes too much time for me .. the other this which slows down the performance are:
  1. Too much dirty checking (as I've just mentioned)
  2. Dirty checking is too slow. I've got idea how to speed it up. I've already told about it to Sebastien .. and waiting for write access to repo coz PR takes too long time.
  3. Locks and deadlocks because of ReadCommited isolation level.
  4. Inconsistent data when using ReadUncomited in web farms. Jobs for example.... TaskLease doesn't help when isolation level is ReadUncommited .. it just doesn't work.
I'm acctually handling Internal Website with registered 200.000 users on web farm of 3 servers. It has dozens of Orchard Modules ... So there is no chance for OutputCaching or SysCaching coz every visitor must be authenticated. I would never switch to other CMS than Orchard in that situation ... same with NH.

... to be continued ... some day
Dec 2, 2014 at 2:07 PM
Interesting insights Daniel. So this means that you have some fixes in mind that could be applied already to optimized data access?
Dec 2, 2014 at 2:10 PM
Some ready fixes in mind to fix even 1.8.x (I'm the most interested in) and some idea which must be discussed with steering committee because are more strategical.
Dec 3, 2014 at 4:33 AM
Edited Dec 3, 2014 at 4:41 AM
Technological considerations aside I'd prefer Entity Framework just because it seems to have support from Microsoft for the foreseeable future while NHibernate's future seems less dynamic (one key difference is async what is already supported by EF but there are seems to be no plans to integrate it into NHibernate; this illustrates the difference). For NHibernate vs EF see:
NHibernate is very mature now and it is mostly ported from Java world hibernate which was under development for a longer time and is mature too of course.
NH is well supported by hazzik, say when a new framework version compatibility needed.
Low amount of github commits is an indication of nhibernate maturity rather than its lack of development drive.
That said we'll surely find shortcomings of EF that would make such a migration worse, it being already hell of a work (and breaking a lot of 3rd party code, so a hell of a work for the whole community). Such a breaking change would also mean that since upgrading to the new Orchard version would be hard we'd have to support older versions longer. With this we risk a fragmentation of the platform like it happened with Joomla.

Obviously changing the ORM is huge work and a very impactful decision, so I'll oppose it for the good of the platform as long as it is not crystal clear that a) staying with NH is a disadvantage compared to moving to a different ORM (or generally: data access layer) and b) the suitable new candidate is better than NHIbernate or at least doesn't require us to do any big compromises in every field we care about.
EF constantly changes and it is less mature than nHibernate and we've already seen many times when MS stopped development of a data access method making it legacy like DAO, ADO. That is why one of the reason I prefer nHibernate for data access now and CSLA for juggling GUIs:

I guess, NH is a more community supported project and even if hazzik would eventually stop working on it, community would still be able to support it while it is not clear regarding EF even if EF is open sourced. There are a lot of companies with deep pockets who rely on nHibernate in their projects and they will always support (say compatibility) NH for sure anyway.
Dec 3, 2014 at 5:47 PM
rodpl wrote:
Async access to DB can give us totally nothing if the code is running synchronous anyway. EF async feature shows that its speeds up only data modification ... maybe because its speeds up "dirty checking" in EF only?
The idea would be to use async code to harness the async call in Orchard 2.x.
Correct me if I'm wrong but if your code is async then any operation you run at the same time the read or write in the db is happening should speed up (not write or read itself) at least if your server is running in the cloud.
ORM is tool which helps us to change data. This is not the best tool for "reporting" purposes like just reading data across many tables to display something. No matter if it is NH, EF, Linq2Sql and so on.

@jersovitc: Try to find complicated query in Orchard and try to implement in in EF.
I'm aware of that, SQL queries produced by an ORM won't be optimal and will be expensive of compile. Your statement is true if the underlying technology is an SQL db. Can you say it is true for all kinds of NOSql dbs? For example a Graph db accessible trough a Linq based API. I can't
Do you guys still thinking that changing NH to EF makes any difference? ... No fu@#@# way
I agree performance problems won't improve just changing to EF. As you pointed, there is room for improvement on 1.8.x and 1.x simply optimizing how the ORM is used.
I see interesting to move to EF in 2.x version due to its promise of coverage of NoSQL DBS not for performance improvements in a Entity2SQL scenario. Why then? because after filling the room for improvement optimizing the use of the ORM if you want to continue optimizing data access performance next thing I would like to see is bypassing the Entity2SQL effort ORM has to do. At this point two options: To use sql directly as we all do when we need it. Or to use an ORM which can use a NOSQL db for which the mapping is not a problem. Specially if do you prefer dynamic Linq for your queries than concatenate strings. It is obvious the advantages for developers of using dynamic LINQ. Problem is dynamic linq queries force the ORM to recompile almost every time the query, because to cache precompiled dynamic queries is a problem . So the bottle neck is in the compilation of queries = entity2sql. If this work is much faster with a NoSQL db I would like to have this option open in orchard 2.x simply using the right ORM as an abstraction to help me change from SQL2NOSQL.

The point is if you show me Nosql support and async await support also is in the roadmap of NH then I don't have any special preference between NH or EF. Problem is I don't have clear signals about this. It would be a pity to have a brand new Orchard 2.x version anchored to a technology that does not evolves at the same time db technologies are evolving.
Dec 3, 2014 at 7:04 PM
Som new about the future of EF

One question regarding this. Does NHibernate support or plan to support CoreCLR?
Dec 3, 2014 at 7:43 PM
Async I/O in your code won't speed up request execution (unless you also parallelize your code) but will improve the throughput of the server (i.e. the number of concurrent requests the server can serve will increase) due to freeing up threads to do actual work instead of waiting for an I/O operation to be completed. Since the number of threads in the thread pool (and there are multiple layers here, like the ASP.NET thread pool and the IIS thread pool; I don't know about these in detail but you can find good explanations) is limited unblocking threads can, depending on the server machine, dramatically increase throughput.

I have to say that although I'd love to see a data access layer that lets you change the data store from SQL to graph DBs or document DBs I personally doubt that this is possible without compromises that will prevent taking advantage of the benefits of each. I only know to an extent about graph DBs besides SQL, and they operate with vastly different concepts and toolbox. I can't imagine, but I'd be really happy to be proven wrong, that you can write e.g. a LINQ query to fetch some data and this will use both SQL Server and Neo4j optimally.
Dec 8, 2014 at 9:01 AM
@Piedone you are right. I didn't explain pretty well. But that was what I was trying to say with "any operation you run at the same time". Having a thread waiting for the I/O is a waste CPU. This CPU time can be used for other things so in general it speeds up server operation.

Well regarding your doubts regarding differences between graph DBs and SQL. I agree they are different. The point OO programing is nearer to graph dbs than to relational ones.
My personal experience with Siaqodb (an OO database + document db) showed me it is possible. A key point for this is the database developer has a real intention of offering a good implementation of its LINQ API as the Siaqodb developers had.

A sample of how important is this is watching at the differences between Neo4j and Orientdb. Both are graph databases (well the second one is a graph + document database). While Neo4j simply create a new language to query its database. Orientdb take the time to offer a SQL extension to access their API. If you take a look to BrightStardb you see that they have take the time to offer an EF based API so things should be easier with them (Problem with that db is it is not as mature as first two).

Look at this sample of Neo4J extracted from their documentation :
MATCH (movie:Movie {title:{title}})
 OPTIONAL MATCH (movie)<-[r]-(person:Person)
 RETURN movie.title as title,
                role:r.roles}) as cast LIMIT 1
It matches the movie by title, then optionally finds all people working on that movie and returns the movie title and a crew-list consisting of a map with person-name, job identifier derived from the relationship-type and optionally a role for actors. Don't you agree it is something expresable with LINQ?

This query from OrientDB also extracted from their documentation sure will sound you very familiar :
WHERE like '%Saint%"' and
    ( = 'Italy' or = 'France' )
It is almost SQL, but what is best it is directly translatable from LINQ (No need of joins). As you see if the developer of the db puts effort on using and extend an existent query language things can be easy.

However, there is other queries in OrientDB that I agree that are out of the scope of LINQ like this one
traverse friends from #10:1234 while $depth <= 3 strategy BREADTH_FIRST
This query assuming #10:1234 is the Record Id of the record to start traversing get all the friends up to the third level of depth using the BREADTH_FIRST strategy:

Well here is where "problems" come. @Piedone I suppose this is what you were pointing. Using LINQ we can use Include to get child entities and even we can control the depth of the tree (in a way not as elegant as using an scalar) but we cannot set the strategy to retrieve those children.

At this point I recognize LINQ is not as expressive as a GraphDB language. But what I can say is all you can express with LINQ is easily expressible in a Graphdb language.

Problem of limitation of expressivity of LINQ can be overcome extending it. But it will be a next step that will happen in the future if finally there is a real need of it.
The main point now is all you can do with LINQ looks can be done easily by a GraphDb and if those queries to be done in a SQL db need a Join in theory it will perform better in a GraphDb. So at this point why not to stay near of this chance moving to EF. I don't say we have to change SQL Server. I would continue using SQL Server but at the same time I would start testing the new possibilities it opens with other dbs.

Days ago after the announcement of the priorities of the Entity Framework project team I felt dissapointed due to the fact they delay the efforts on NoSQL to put it in finishing the SQL part. Furthermore they only mentioned document dbs not graph dbs. So, I contacted with OrientDB author to ask him for the availability of a LINQ API for his db and for their disposition to develop a provider for EF7 (we don’t have to wait for MSFT). The point is I was so happy to see how engaged he looks with the idea. It is not strange because it fits with their idea of offering an SQL based APIs. Because they seems have understood they have to make it easy to devs to migrate from most extended technology, now relational dbs, to their db. He asked me I send an issue, so as soon I send the issue I will post it here. In that way you can see yourselves how this idea evolves. It will be a good way of see how crazy is what I’m pointing ;)

Regarding the sample you ask me I can provide you a sample that for the same LINQ query fech some data optimally from SQLServer and for Siaqodb (it offers a LINQ API). The only thing I need is you allow me to abstract the DbContext.
Dec 8, 2014 at 12:57 PM
EF7 looks interesting, though I am always concerned about how to handle their version of a "Context".

If someone wants to start sketching the datalayer out, maybe take a fork of vnext and see what you come up with.

To be honest, there are many things that we need to think about when it comes to data access, with the API being the main part - if you look at the Orchard.Data and Orchard.ContentManagement namespaces, there is a huge about of stuff going on, how would it look if you started from scratch?
Dec 10, 2014 at 1:11 AM
That is a massive thread since last time I checked, so apologies if I'm repeating stuff that's been said already.

Someone said the word "migration path". I think we can get away with a lot of breaking changes if we do have such a path, which will free us to make the right choices. I'm firmly convinced that that migration path is export/import. It has some limitations today (media) but I'm working on closing those gaps. We should have a rock-solid feature in time for 2.0.

The rest of the discussion seems to be a NH vs. EF fight. How about we look a little beyond ORMs? There are some who think ORMs were a mistake. There are those who think storage and querying should be separated. Talking about querying, SQL is not the only game in town: map/reduce is another solution that has proven its efficacy. In other words, at least for this discussion, let's not restrict our thinking to "older" technology, and instead let's keep our options and our minds open.
Dec 14, 2014 at 7:45 PM
Bertrand, the more I think about it, the more I like the idea of separating
storage and querying. If my understanding is correct the abstraction would be:
  • define fields, parts and content types
  • define map/reduce indexes
  • query against the indexes
As Sebastien suggested in, a
SQL-based implementation of this could use:
  • the ideas behind yessql to store content fields/parts/types, as well as storage
    for indexes
  • a micro-ORM like dapper for SQL access
  • a SQL dialect abstraction to maintain multiple RDBMS support
Other implementations for document/nosql databases would be fairly natural.

So... what could an API look like? I can suggest a few starting points:

a) Unify the UI and code when it comes to index and query definition.
Right now we can define dynamic content types and parts, and these are the
same things that we can work with in code. Indexes (and their associated map
and reduce actions) and queries would be available in the UI directly, and
developers would work directly with these entities through code as well.

b) Redefine/enhance the capabilities of fields:
  • they don't have to be just for storage, they can define rich data types.
    Why not use them throughout Orchard (e.g. forms, layouts, etc)
  • the storage aspect should support scalar, composite, collection (typed
    and untyped) and reference (other content identifier) 'primitives'.
  • based on those primitives, any number of rich field types could be coded
    fairly easily. These could be standard with Orchard.
  • field instances could be constructed in the UI, including composite and
    collection fields
  • fields continue to support aspects such as import/export, full-text indexes
    and display/editor shapes
c) Parts are similar to what they are now:
  • they are used to extend the behavior of content types (is-a)
  • in contrast, composite fields (has-a) would be a distinct alternative that
    remove the temptation to mis-use parts
  • in many cases, part development could be simplified: declare
    the fields that the part uses, and by default let the fields implement
    storage, display/editing, import/export, full-text indexing etc - then
    concentrate on coding the logic of the part. You could still override this
    default behavior if you wanted.
d) Define fields for parts, content types and composite fields either in the UI
or in declarative code. The code approach would probably look like migrations
do now, except without records.

e) Similarly, define indexes in the UI or declarative code. This could be part
of the same migrations above. A question: how similar are map/reduce indexes
to full-text indexes? They may be able to use a lot of the same plumbing.

f) Queries could also be created in the UI or in code. This is a separate step
as parts, content types and indexes would need to be in place first. If done
right queries as declared entities may allow batching queries (reducing
chattiness with the db server).

g) As this is all very dynamic, code generation could be used to make this more
friendly to devs who like static type checking. To toss a few ideas out
  • a developer could "import" interfaces for content parts and types. This
    would flatten field objects into properties with .NET native types.
  • interfaces for indexes could be imported to enable querying through LINQ
    or other means.
  • perhaps even map-reduce actions could be defined through statically-typed
    code (anonymous types?)
Dec 15, 2014 at 7:07 PM
Thanks for taking the time to write this. One thing that confuses me however is your notion of composite fields. How is this different from a part that you add from the dashboard and that you build out of fields?
Dec 15, 2014 at 11:11 PM
BertrandLeRoy wrote:
Thanks for taking the time to write this. One thing that confuses me however
is your notion of composite fields. How is this different from a part that
you add from the dashboard and that you build out of fields?
The main difference is that a part can only be added to a content type once. From my understanding - and I can't remember who said it first - a content part is designed to add behavior to a content type ("is-a"), and a field is designed to add data to a content type ("has-a").

The fields I'm talking about are a bit of a departure from Orchard's current fields (which IMO are constrained by the current data access layer). Maybe "fields" isn't the best name, but I'm thinking of self-contained data types that can be arranged in any number of ways. For example:
GeoLocation: composite-of:
    Latitude: number field
    Longitude: number field

Address: composite-of:
    Location: GeoLocation field
    Line1: text field
    Line2: text field
    City, State, Country, PostalCode etc.

Company: content type with:
    Title: Title part
    Addresses: collection-of Address
Each field type knows how to serialize itself using storage primitives (standard .NET scalar types, collection-of, composite-of), with these values stored within an XML or JSON document for the content item.

An example Company content item could be like the following JSON:
    _id: 10
    Title: {
        Text: "Microsoft"
    Addresses: [
            Location: {
                Latitude: 47.6395831,
                Longitude: -122.128381
            Line1: "1 Microsoft Way",
            City: "Redmond",
            State: "WA"
            Location: {
                Latitude: 49.2569777,
                Longitude: -123.123904
            Line1: "858 Beatty Street",
            City: "Vancouver",
            State: "BC"
When creating parts, composite fields and collection fields you wouldn't need to define serialization logic if you didn't want to. The Title above is a good example; it's the TitlePart that happens to have a "Text" text field. You could still override the serialization logic if you wanted, e.g. if you'd like to save some space:
Title: "Microsoft"
The same principle applies to other aspects such as import/export, drivers for display/editors and index support (generating scalar values for both full-text and map/reduce). As a developer, you could use the defaults that have already been implemented at the field level, and override only when you need to. In this way a lot of the plumbing work involved with writing parts and composite types could be avoided.

Not to get too side-tracked, but another huge benefit of this approach is regarding localization. Imagine a LocalizedText field. Storage could look like:
SomeText: {
    en: "This is English",
    fr: "Ce est le français"
(Indexing support would spit out values for both the culture and text, which queries and full-text searches could subsequently use).

As a developer or admin user - I can choose what gets localized at the field level, and best of all, I don't have to code it. :-)
Jan 4, 2015 at 6:42 AM
If someone is interested on follow how things evolve here is the issue someone at OrientDB asked me to add to their github repository.

I'm aware it is a bit off-topic now that the thread is focused in a very interesting topic after @BertrandLeroy and @brentbysouth entries. It is just I said I will share the link to the issue and here it is.
Jan 19, 2015 at 9:12 PM
Hey All,

I have started to sketch out a EF7 branch on the Vnext stuff in GitHub ( - this by no means that Orchard will use EF7 in the future, its just a Proof Of Concept to see if what we have been talking about in this thread, along with a few other ideas can be done using EF7.

The first steps are to create something very generic, that allows the Demo module to store and retrieve a record on the fly without adding DbSets to the context and also to setup the records with a migration, after that, then I wonder where it can go.

I am concerned by the performance of EF7, however I like that the api's are alot lighter, so I wonder if this might be a more generic DAL than EF used to be.

@jersiovic I like the idea of EF7 and OrientDB wrapper, maybe we can conciser that as an abstraction.

Jan 19, 2015 at 9:19 PM
One of the main things I like about EF7 is that it is built to be cross platform. Just Sayin.... (Not that other providers cannot support that later too.)
Jan 20, 2015 at 5:20 AM
Great! I will take a look to that branch

I see your point cross platform db is a very interesting point. But nowadays we are living very innovative times in DBS world so things are far from stable. I don't want Orchard to support lots of DBS but I think nowadays is interesting you can say at some point without pain "hey Orchard changes its prefered EF provider to this db that gives me better performance for this thing".

For example recently I met another interesting hybrid db (sql-nosql graph oriented db) ArangoDb written in C++. I also asked them to provide LINQ provider and EF7 provider. Which of those new bds are more suitable for Orchard? Which will provide a better provider for EF7? I don't know so if we choose one, in the future it is easy we change our decision.

By the moment I have received from Siaqodb an statement saying they are going to make an EF7 provider, from Orientdb as you see in a previous comment I shared they looks interested on a LINQ provider but now they have their focus on rewriting their SQL provider (if I understood it right) and from ArangoDb I still haven't received an answer.

If I were to develop an EF7 provider for a NOSQL db with a LINQ API I would take the in-memory provider as the code base for my db own provider, which is included in this version of EF. So the work would consist on change its code to store changes in the db instead of in memory.

The point is that in my opinion the "most concrete level of abstraction" we should use while we can and while other dbs offer their providers should be the EF7 in memory provider instead of the SQL Server provider in order to maintain the cross-platform db option available.

Jan 20, 2015 at 5:21 AM
Edited Jan 20, 2015 at 3:38 PM
Jan 20, 2015 at 10:15 AM
Good news, ArangoDb is currently working on a LINQ provider (indeed they have a first Alpha)
Jan 25, 2015 at 6:13 AM
Hibernate OGM also pursues same objective, supporting NoSql.

Problem is this project in java is in development, so when will we have an NHibernate C# version?
Jan 27, 2015 at 10:13 PM
@Jetski5822 I tried to open the Brochard solution with VS2015 Preview however it returns me errors like this one for all the projects:

src\OrchardVNext\OrchardVNext.Framework.kproj: The application which this project type is based on was not found. Please try this link for further information:

Do I need to install something extra?
Jan 27, 2015 at 10:47 PM
Edited Jan 27, 2015 at 10:54 PM
@jersiovic Um, not that I am aware of. The project solution should just open up.

Do you have the latest Visual Studio 2015?
Jan 28, 2015 at 5:15 AM
Sorry, it was my fault I only installed core components. After install web components it worked
Jan 28, 2015 at 8:33 AM
Edited Jan 28, 2015 at 8:34 AM
The problem now is I'm not able to build the project.

I get lots of missing assembly reference errors.
Automatically check for missing packages during build in Visual Studio is enabled.

An screenshot of the Visual Studio window after compilation:!9217&authkey=!ABMGP_fVMDDoKN4&v=3&ithint=photo%2cpng

I'm new with VS2015. Any idea of what could be happening?
Jan 28, 2015 at 10:14 AM
Jan 28, 2015 at 6:42 PM
Yes it was the the problem, now it works.
Thank you!
Jan 28, 2015 at 11:05 PM
Edited Jan 28, 2015 at 11:07 PM
Well it works using command "k web"

However if I try to debug the solution running it in IIS Express when I access with the browser I get following error: "Couldn't determine an appropriate version of runtime to run"

I've found an issue in aspnet project related to this

I also tried to run it from the console and attach its process but without success. So I will wait for further news
Jan 29, 2015 at 11:06 PM
Awesome, glad I could help! - Can we move the VNext debugging stuff over to the Brochard Issue log? as I feel we are getting off topic here :) N
Jan 31, 2015 at 6:26 AM
Edited Jan 31, 2015 at 6:31 AM

In Tuesday meeting I heard reasons to do not use EF that looks initially to me weak if they are not supported with concrete facts. So I think it would be good we have a discussion of each one of those disadvantages if those are real disadvantages analyzing their real impact on Orchard.
I added those two issues to Brochard gitub project that maybe should have been better to add in this thread because they are food for discussion.

An extra point I understood in this meeting was that Orchard bottleneck are 1-1 relations so EF has nothing to do helping at that? I understood it right or it was my limited Spanish ear for the Shakespeare language? ;)
Feb 8, 2015 at 9:33 AM
Maybe you are interested to know that further disussion related to this topic is happening in this issue thread at Brochard Github page