Graph databases

Topics: Writing modules
Developer
Sep 12, 2012 at 12:55 PM

The ones who attended Harvest may remember that Todd Rockhold proposed the idea of discussing the usage of the Neo4j graph database in Orchard for the hackathon. We got a bunch of ideas there and I also did some research on the subject of using graph databases in Orchard, here are the compiled findings:

  • .NET API to wrap the REST calls: http://hg.readify.net/neo4jclient/overview
  • Heroku cloud hosting with Neo4j add-on: https://addons.heroku.com/neo4j AFAIK it's possible to have a 36$/month account, add a Neo4j DB to it and expose its content through RESTful webservices.
  • Company behind Neo4j: http://www.neotechnology.com
  • Other .NET drivers for the RESTful API: https://bitbucket.org/2hdddg/neo4restnet and http://neo4jrestsharp.codeplex.com/ (both seem abandoned), https://github.com/ahzf/Neo4Sharp
  • A port to .NET, work-in-progress, but they're doing it: http://neo4net.codeplex.com/
  • Running on Azure (like in a virtual server): http://blog.neo4j.org/2011/02/announcing-neo4j-on-windows-azure.html and also http://www.neotechnology.com/2012/06/case-study-neo4j-in-a-net-world/
  • Compiling Neo4j to .NET (outdated): http://www.thewebsemantic.com/2010/06/03/neo4j-on-net-3-5/
  • An ASP.NET MVC sample of the Australian guys' client: https://bitbucket.org/romiko/frictionfree
  • Compiling Neo4j to .NET is apparently possible with IKVM.NET

I may like to use a graph database, at least as an option to drive Associativy knowledge graphs. One important aspect to keep in mind for this is to only use a graph database if it's a possible option for a wider audience, including the ones using some kind of hosting, but not being able to make changes on a server-level. This leaves the following options:

  1. Use a graph database that runs on .NET and is usable directly from a web application (i.e. it's in a dll and the database itself is stored in a file, so the whole DB is self-contained).
  2. Use a DB that is possible to find decent hosting for and then use it through web services.

While Neo4j seems like an excellent choice, also featuring a RESTful interface that has a great .NET driver too, it isn't easy to run it anywhere. The only hosting option I could find was Heroku and I'm unsure about how it could be used in a self-contained (embedded) way; however, it might be possible. It's definitely possible to use it e.g. in a Java application, and it's apparently also possible to compile it somehow to a .NET dll. This way, you could essentially use the same classes, same methods as in the Java example from .NET.

What do you think?

Coordinator
Sep 12, 2012 at 5:02 PM

It seems a little esoteric to me, and I'm not sure I understand how this is relevant to Orchard. Beyond that, what is the licensing (their site doesn't seem to have a search engine)?

Developer
Sep 12, 2012 at 7:56 PM

It's just about using a graph database for the underlying storage in an Orchard module.

AFAIK for non-commercial use they community edition is freely usable.

Coordinator
Sep 12, 2012 at 9:26 PM
Edited Sep 12, 2012 at 9:27 PM

What's the point? (I'm being totally serious here) And don't answer: "to use a graph database".

Oh, and with just that: "for non-commercial use they community edition is freely usable", you just killed it. This is precisely the reason why we can't use Raven.

Developer
Sep 12, 2012 at 9:54 PM

Yes, but I don't mean anything getting into the core package, just being able to use it from a module. Licensing then is the responsibility of the module's author and the module's users. But this really isn't about licensing only and also not just about Neo4j, it's just that this was the DB mentioned on Harvest and I haven't found any suitable alternatives yet.

So the point isn't to change anything in Orchard itself, but about seeking options when it comes to cases where an Orchard module could benefit from being able to use a graph database. As mentioned I'd try it with Associativy, just to experiment, but Todd wants to do something serious with graph databases (and I'm not being mysterious here, but I can only vaguely remember he mentioned some big-data-processing his company does).

Coordinator
Sep 12, 2012 at 10:37 PM

Oh, ok then. But that should work today. Can't see why not.

Developer
Sep 18, 2012 at 10:56 AM
Edited Sep 18, 2012 at 11:00 AM

Yes, but since this is about using a Java database engine in conjunction with Orchard I was asking about others about their opinions or experiences.

BTW Tatham Oddies, the guy (and his company) behind the REST wrapper sent me a link to his talk about hosting Neo4j in Azure and about the project aimed to make this easier.

I've modified the in-memory graph storage of Associativy to use the Orchard cache (didn't know you can simply change a value of a cache entry without clearing and re-setting it) so this should work fairly well for reasonably sized graphs.

Developer
Oct 1, 2012 at 4:35 PM

I've implemented Neo4j support for Associativy: Associativy now has Neo4j support! Through Neo4jClient it's entirely possible to use the Neo4j graph database!

Developer
Oct 4, 2012 at 3:55 AM
bertrandleroy wrote:

What's the point? (I'm being totally serious here) And don't answer: "to use a graph database".

Oh, and with just that: "for non-commercial use they community edition is freely usable", you just killed it. This is precisely the reason why we can't use Raven.

Ability to keep and, more important, query complex relations between content items in realtime. I can imagine a module that would allow users to easily build relations between items (and/or set up some rules that would create such relations based on events) and then create queries over that graph like Projections do. Pretty powerful tool for e-commerce solutions, social networks etc. 

Developer
Oct 4, 2012 at 6:34 PM

Actually that's exactly what Associativy does: it's for creating, editing and querying connections between content items (and of course meanwhile does a bunch of other things, but essentially it's about this).

Oct 4, 2012 at 7:14 PM

Just curious, If you are looking at options that aren't fully FOSS, why did you choose neo4j instead of RavenDB? 

Developer
Oct 4, 2012 at 8:20 PM

I specifically wanted a graph database, as what needs to be stored here is a graph. And given that with the need to use it from Orchard Neo4j looked like the best option.

Developer
Oct 5, 2012 at 1:07 AM
Edited Oct 5, 2012 at 1:09 AM

@Piedone What the Associativy is? Is it a generic Orchard module, or a whole application (customized Orchard instance)? The former would be great to have, if it does what you wrote about:)

@TheMonarch It's apples and oranges. The only common thing for RavenDB and Neo4J is that those are both 'NoSQL' databases. Neo4J is not a document database. Its main purpose is to store items (nodes) and relations (vertices) between those items and allow users to query that dataset (graph) using DSLs like Cypher and/or Gremlin. The emphasis is mainly on the second part - queries - and their performance.

Developer
Oct 5, 2012 at 7:57 AM
Edited Oct 5, 2012 at 8:27 AM

Associativy is a collection of modules usable in any Orchard instance, but there's one core module that has all the graph manipulation APIs. It's just that there's a complete Orchard source in the main repo, including the modules as subrepos there as well, for the ease of the development of Associativy itself; you don't necessarily need to use that though. But this might be confusing for others as well, so I'll write about it.