REST API Design

Topics: General
Coordinator
Aug 8, 2014 at 7:11 PM
Because it's somewhere on the road-map and I already thought about the subject, here are some random thoughts about the REST API module.

Goals

  • some APIs and best practices to build REST APIs for Orchard
  • default implementations for major modules (content, taxonomy, query, search, ...)
  • target websites using client-side, server-side , native apps.

Ideas

  • Needs to work with multi-tenancy by providing the tenant url as the base url
  • Cache
  • Etag ?
  • Limit results, to prevent miss-use of the service (100?)
  • Returns Exports like information
  • Common media formatter for JSON
  • Security: API key, shared key
  • Configurable CORS

Content Manager API

/documents/id1,id2,id3
  • Returns the specific ids in order
    /documents/type/Product/page/1?take=10
PUT
Take a document and use Import

Queries API

/query/name
  • Returns the result of the query with the specific name
    /query/name/page/1
Add a new filter which can take REST parameters

Lucene API

/api/search?q="type='Product' AND product-name='foo'

Content Definition API

/type/Product
  • Display name, stereotype. parts
    /fields/Product
  • All the fields of the Product part

General guidelines:

http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api

http://localhost:30321/OrchardLocal/api/search?q=title:'tesla' http://localhost:30321/OrchardLocal/api/Orchard.RestApi/ContentItems/16

Some usage examples

Multi-tenant solution to host several content repositories with only one instance

http://osmek.com
https://prismic.io/

Windows Phone, Windows 8, iPad apps using the same content as the website. Or even with no website.
Developer
Aug 8, 2014 at 7:14 PM
Didn't you have a fork or branch somewhere?
Coordinator
Aug 8, 2014 at 7:30 PM
Yes I had started something, which is public: https://orchard.codeplex.com/SourceControl/network/forks/sebastienros/Backup?branch=feature/rest

I will start one on the main repos, but first talk with the Halo team who took my work farther.
Aug 8, 2014 at 8:12 PM
Bearer Authentication for security
Aug 11, 2014 at 5:48 PM
Just a point a view to be discussed ...
Why not define a common routing template as in standard url apicontrollers
~/api/{moduleIdentifier}/{action}/{parameter}?

{action} may be reused -and should be as much as possible- to facilitate api usage in an homogeneous way.
{parameter} should be then a string optional and relative to the action.

The idea is to refine a common way to use the api whatever the module is really called.
I think specific user parameters should be avoided as much as possible because these need to be parsed into the action api controller to be applied properly.

Most scenarios in syndication and/or mobile devices' consumption are to "query" the server while providing filters to avoid heavy traffic.
In some basic use-cases, it might seem that {parameter} could be then
  • string.Empty
  • a value
  • a value filtered
  • only a filter
Meaning, your examples could use several times 'query' action depending on the module concretely called and filters provided :

=> Content Manager API
~/api/contents/metadata
to retrieve all fully content types definition

~/api/contents/metadata/book
to retrieve full content type definition of only the one named 'book'

~/api/contents/metadata/book?$filter='parts'
to retrieve only the parts of the content type definition named 'book'
Here is a specific need to check a custom {parameter} : value + filter

~/api/contents/query/books
to query all published content items of the type named 'book'

~/api/contents/query/books?$filter='123'
to query the content item of type 'book' wich identifier is '123'
Here is a specific need to check a custom {parameter} : value + filter

=> Projection API
~/api/projection/query/myBooksProjection
to return the results of 'myBooksProjection'

~/api/projection/query/myBooksProjection?$filter=<input>
to return the results of 'myBooksProjection ' wih provided filters
Here is a specific need to check a custom {parameter} : value + filter

=> Search API
~/api/search/query/?$filter=<input>
Here is a specific need to check a custom {parameter} : only filter (value is string.Empty)

Waiting to read responses if you find this an interesting approach :)
Coordinator
Aug 11, 2014 at 6:30 PM
The filtering is actually already implemented in my fork, and I agree with the idea. You need to be able to select which part of the object you want to load.

The default route is already defined in orchard, the examples where just examples, I agree it should follow these rules.

It seems the default action should always be query. Not sure if it should be implicit then. Not all services would also implement something else, like metadata, hence it would make sense to make only 'metadata' action explicit where it's used.
Coordinator
Aug 14, 2014 at 1:41 AM
Edited Aug 14, 2014 at 1:41 AM
In my fork I called your filter "expand". I have some questions about it.

Do you thing it is better to return everything by default and the filter would be used to optimized the data, or return nothing by default and filter works like an include parameter? Having everything could be done then with 'filter=*'
Obviously this would only apply to content items results.
Aug 14, 2014 at 3:09 PM
I also think 'query' would be the main action refering to a CQRS naming convention.
This scenario would be a GET action regards to REST standards for a syndication use (reading from mobile, embedding contents in other site, etc ...) .
Commands would have to be defined through POST actions per module, because they naturally imply business logic.

If this is set, default routing could indeed be applied.
$filter and $expand are then, to my point of view, two parameters of 'query' that can have two different behaviours

$filter should be used to get a subset of data from main action. It might refer to a Where<T> clause.
$expand should be used to ensure consistency of data retrieved and running over the middleware. It might refer to an Include<T> clause.
Both could be obviously composed.

To be discussed
Aug 17, 2014 at 11:47 AM
Here are my suggestions, ideas in no particular order etc...
  • REST API following as close to REST principles as pragmatically possible
  • Response to be XML and JSON through content negotiation and formatted correctly (i.e. camel case for JSON)
  • Response to be clean and simple - not verbose
  • URL's to be clean and force a version /api/v1/<resource>
  • HTTPS by default
  • CORS enabled
  • JSONP enabled
  • Caching
  • Authentication and permissions tie-in
  • API Keys, URL to be used on, usage limits and usage stats - tied into users
  • Would be nice to have it separated by tenant and the ability to share it across tenants
  • Localisation through Accept-Language header
  • HATEOAS where possible and pragmatic
  • I'm not mad about ODATA but that functionality in a standard REST way
  • Allowance for nested entities in response
  • HEAD for last modified - useful for mobile offline syncing
  • Meta data to be included with standard response - for example...:
 "metadata": {
        "totalCount": 1000,
        "returnCount": 10,
        "pageSize": 10,
        "numberOfPages": 100,
        "currentPage": 1,
        "timeStamp": "2014-08-17T10:37:09.022975+00:00"
    },
    "results": [
        {
          ....
         }]
Just a quick brain dump.

Steve
Aug 26, 2014 at 8:15 AM
Edited Aug 26, 2014 at 8:19 AM
Hello,

What do you think of accepting a DisplayType parammeter for Content Item queries? It will expect DisplayTypes defined within each module as Detail, Summary ...

It will be an alternative to use the expand parammeter you mentioned earlier.

From my point of view a big difference between Orchard as a Framework for developing and a classical DDD project is the lack of DTOs and mappers between DTO-Aggregates. It is not bad, because those add a tedious work to developer and extra proccesing to the app. The way you avoid it is by DisplayType, that is no other thing that a convention used by the consumer of the data to say the provider only return me just the data I need in this scenario.

Those data can be a content part, a combination of two, or one data from one content part and two from another, etc. What I mean is that using content part as the lowest grain of detail for getting data is not enough. In that way you are forcing developers to split its content parts thinking on how many different scenario the information will be served. But at domain level you shouldn't be worried about this because if you're driven by this you end splitting everything in too many small content parts.

An extra thing can be done if you use DisplayType is to stablish user access permissions per DisplayType. It helps the administrator to stablish permissions per scenario instead of doing it per content type that is less practical. Many times you could need to show some data from a content part but you cannot show all the data from other one content part despite you need to return data from both. An easy way of doing this is checking permissions of the user for this DisplayType asked and delegate the composition of the display type on the module which defined it instead of letting the consumer decide which data to get with the expand parammeter.

What do you think?

@sebastienros regarding the expand parrameter I like how you propose to do it "to return everything by default and the filter would be used to optimized the data, or return nothing by default and filter works like an include parameter? Having everything could be done then with 'filter=*"
Sep 3, 2014 at 5:12 PM
Edited Sep 3, 2014 at 5:19 PM
To go on the discussion, filtering and data consistency seem to be the two main goals of the external query reader feature.
($filter and $expand)

Parameters may suit to the OData protocol and I have tried few months ago to start a custom provider on Orchard: Orchard.OData

You may consult in details relative code here: orchardsyndication.codeplex.com
However relevant code may be shrinked to :
               // from DataService<T>.OnStartProcessingRequest(ProcessRequestArgs args)
                var queryArgs = HttpUtility.ParseQueryString(args.RequestUri.Query);
                this._expression = ... // from queryArgs["$filter"]; 
                this._expression = new DSPExpressionVisitor().Visit(this._expression)
                this._includes = ... // from queryArgs["$expand"]

                // from IDataServiceQueryProvider.GetQueryRootForResourceSet
                var contentTypeDefinition = ((ResourceType)resourceSet.CustomState).Name; 

                // dependency of Orchard.ContentManagement.IContentManager
                this._contentQuery = this._contentManager 
                    .Query()
                    .ForType(new string[] { contentTypeDefinition })
                    .ForVersion(VersionOptions.Published);

                if (this._includes.Any())
                {
                    // reflection code to get proper Expand<> method from this._includes
                    this._queryHints = ... 
                    this._contentQuery = this._contentQuery.WithQueryHints(this._queryHints);
                }

                // == TODO == apply WHERE condition through visitor onto expression 
                // before been applied laterly on CLR with all fetched data from repository
                
                // == Here apply Linq2Objects onto full set of already retrieved data 
                // expression has already been inspected with InterceptedQueryVisitor 
                this._queryResults = this._contentQuery.List();
                var lambda = Expression
                     .Lambda<Func<IQueryable<ContentItem>>>(this._expression)
                     .Compile();
                this._queryResults = lambda().AsEnumerable();
This maybe could provide the feature expected.
But $filter may have to be previously visited properly and routed into the nHibernate final query "before" full table is loaded.
Meaning a dedicated inspector for Linq2Sql should be reimplemented while using IContentQuery.Where<TContent, TRecord>() clause.
Based on DSPExpressionVisitor, I think this could be done while implementing another custom ExpressionVisitor and overloading at least two methods
                 // to navigate on tree until contentPart and/or field is found
                 protected override Expression VisitMethodCall(MethodCallExpression node)
                // To apply the Where() method onto IContentQuery 
                // left or right node as a ContentPart or ContentField subtree been visited
                 protected override Expression VisitBinary(BinaryExpression node) 
However, this strategy might only provide "intersection" complex queries. Meaning only $filter expression build with "and" could be applied properly while the Where<> would be applied sequentially to extract the subsequence of results. But, why this should not be a known constraint in feature requirements ?
Thanks for your comments.
Sep 4, 2014 at 5:54 AM
Related to OData maybe this sample of a non-IQueryable data source as NHibernate to be queried using the OData query syntax using AspNet Web API OData would help.

http://blogs.msdn.com/b/webdev/archive/2013/02/25/translating-odata-queries-to-hql.aspx
Sep 4, 2014 at 9:57 AM
As far as I think on this I see two REST APIs are needed for two different purposes:
One a kind of OData API that exposes data from the db for those who want to make its own SPA app fed it with Orchard data using any query they need.
The other REST API is one which should be provided with a js Library which should be a kind of Orchard shapes engine in the client. It would load templates and configuration like placement file in the client-side provide by the REST API. For rendering shapes it would need data so the Rest API should be a Shape based Api where the consumer asks for the data for certain shape or shapes and it returns its/their corresponding model. Problem is to have a clear view of what this API should offer the js client side has to be clear how it will be designed.

I'm a newbie (just finishing lombiq videos ;) ) for me a lot of aspects of Orchard are like a black box, however I hope my naive opinion can be a help for you.
Sep 4, 2014 at 1:24 PM
I think you are pointing a right question. What's the feature really expected with such an API ?

From a CQRS strategy, I was personnally focusing on syndication-reader scenarios while mobile apps and/or external websites consume orchard's published contents, for their purpose. My point was to provide a query provider, and then in a second step to specialize from dedicated modules the commands that imply business logic embedded into these modules. Sort of splitting GET requests from POST ones in a REST common behavior.

Waiting for your comments.
Sep 5, 2014 at 10:59 AM
Yes it looks that the first question to answer is what is expected to solve the new API.

What you expect is a generic Rest API useful for any content consumer. I agree it is needed.

However, I think is also needed an API that helps us to move to Client UI renderization concepts as Shapes, Widgets, Dynamic forms to use them within SPA apps (mobile or desktop). It will help to leveraging the server from this tasks when we think it is interesting for the sake of performance or for the sake of user experience.But also it will leverage programmers to rewrite in the SPA app logic for rendering those contents because it will use the Theme we have set for the Orchard app. The point is those UI concepts are a great attractive of Orchard because they are great tools for reusing and personalization UI. So,it is a pity that we cannot harness them when we write an SPA based on an Orchard app.

@sebastenros what is the purpose you follow with the new API?
Sep 17, 2014 at 4:13 PM
jersiovic wrote:
Related to OData maybe this sample of a non-IQueryable data source as NHibernate to be queried using the OData query syntax using AspNet Web API OData would help.

http://blogs.msdn.com/b/webdev/archive/2013/02/25/translating-odata-queries-to-hql.aspx
Thanks for pointing this resource.
Finally, standard DSPExpressionVisitor seems to be a very complex challenge to implement here, and not sure this should work on all OData protocol usages.

However a similar solution could be indeed to parse a syntactic tree with this other OData API while checking the request on starting the odata process.
Prefiltering on parts could be applied through something based on this System.Web.Http.OData.NHibernate.NHibernateFilterBinder

However not sure if "whereclause" on fields could be applied this way.
These other clauses should still need to be applied on Linq2Sql laterly.

As mentioned here, NHibernate.OData, the ideal goal shall be to parse queryString to fullfill a complete Where<T> on IContentQuery or IHqlQuery "before" a List() call

Would not be an easy work, but as this developer has done, transform odata expression on orchards' one may be an option while parsing our own DynamicQueryable to apply recursively on an IContentQuery handler.

What's your point of view ?
Sep 25, 2014 at 11:36 PM
Thanks Bertrand's new module,
I think injecting odata transofrmed query into nHibernate could really be done here through ISessionLocator ?

HqlPlayground
HqlPlaygroundAdminController
Oct 2, 2014 at 9:57 PM
Hi, tried to compile all these resources and embed them into a Orchard.OData module.
You may find current changes here.

The main idea is to change previous query behavior while not applying user expression onto full content items with Linq2Objects
            try
            {
                // == Here apply expression onto full set of already retrieved data while expression as been inspected with InterceptedQueryVisitor 
                //this._queryResults = this._contentQuery.List();
                //var lambda = Expression.Lambda<Func<IQueryable<ContentItem>>>(this._expression).Compile();
                //this._queryResults = lambda().AsEnumerable();

                // == Here prefilter query before been applied on repository
                var ids = this.FilterAndOrderContentQuery(this._sessionLocator.For(typeof(ContentItem)));
                this._queryResults = this._contentManager.GetMany<ContentItem>(ids, VersionOptions.Published, QueryHints.Empty);
                this._queryResults = this._queryResults.Select(CloneAndExpandContent); 
            }
            catch {
                this._queryResults = new List<ContentItem>() { };
            }
First step was to take a look to ODataLib and parse from query string a FilterClause as expressed into this aspnet sample
Second step was to navigate on OData semantic tree as explained in NHibernate.OData
Finally and maybe the most difficult task is to manage subexpression while trying to access field or part property.
Thanks to these excellent resources, I got a better view of internal orchard data model.
Join content parts
Join content fields

Just need to manage now $skip and $top tokens plus ordering criteria and that would do expected job :)
Oct 3, 2014 at 3:03 AM
Very interested in seeing where this Orchard RESTful Api is going. Have a project coming up which we want to implement an Api for, and I'd rather leverage as much of Orchard's core as possible rather than writing a completely bespoke implementation.

I think Orchard could be a powerful platform for SPAs. I'd also like to expose the same endpoints for mobile apps to consume.

Haven't got much to offer to the convo at this stage, but will be following it closely.
Oct 7, 2014 at 6:33 PM
Thanks to these resources
To complete discussion here, I agree with this comment on excellent Bertrand's blog.
The killer "app" for me would be to take the Content Type system, the Users/Roles system, etc... from Orchard, wrap a REST-based API around it and turn it as an Azure service (like Azure Mobile Services). Literally call it "Azure Orchard" or "Azure Orchard Services."

You may find then a new version of Orchard.OData implementation, however with kown limitations (regards to Orchard's lower-level fields storage)
Waiting for comments :)
Oct 28, 2014 at 7:00 PM
A post that describes how to use and advantages of this OData module
Jan 12, 2015 at 2:57 PM
Trying to up the topic

Some relative source from msdn that may be studied to define current needs :
Introduction to REST and Web .Net API
Feb 11, 2015 at 8:19 AM
Edited Feb 11, 2015 at 8:20 AM
At which point is this topic?

@Ostephan your OData module exposes data to everybody?, can be set per user access restricitions?
Feb 16, 2015 at 2:31 PM
You're right.
Current data exposed in my module is only published contents, as an anonymous viewer may consume them on the website through a browser.
I guess this would not be very difficult to catch current authenticated user to filter these data.
In addition, the module has been mentionned on odata.org, as other CMS built-on ASP.Net stack.
May 3, 2015 at 8:51 AM
Edited May 3, 2015 at 8:53 AM
Hello Sebastien

Being new to Orchard and very curious about the amazing work you all have done and without knowing the deeper inner workings in Orchard...

Developing a REST-api to Orchard seems like the same Microsoft did when developing MVC and WEBAPI separately, and now in the vNext combine those two together again.
From a CMS-perspective I'm a having a hard time trying to see any value in developing a REST-framework.

I would rather see Orchard leveraging on existing technologies, why not use NancyFX, imho the perfect REST and MVC. NancyFX can serve both views and raw responses, very dynamic, a bit like Orchard shapes :)

SteveTaylorUK mentioned a list, NancyFX support most of those as a minimum and has done so for many years.

In my opinion Orchard should focus on CMS-functionality that can take the gold from the old-timers:
Keep up the good work :)
Jun 23, 2015 at 4:31 PM
Based on RESTier concepts, I updated Orchard.OData module while parsing ContentTypes metadata through last ODataLib component.
Code is inspired from RESTier sample
You may find package from gallery, or codeplex.
Do not hesitate to comment.
Jul 8, 2015 at 7:30 PM
At work we've used orchard as a back end for apps serving up data via web api (these instances also serve as a website exposing much of the same data).

This got me thinking about how to do this without having to write any custom code. It felt that any webapi implementation needed to be consistent with the current mvc implementation which meant not cutting out content part drivers and field drivers as many of these contain logic which can't simply be cut out.

I have a working version that serves up any content using shape attribute bindings(I've implemented the core shapes) to create a flexible view model from the shape based model created via contentmanager.builddisplay, which is then serialised in the usual web api way. This allows for being able to map data however you want (ie flattened, nested etc) and using placement, alternates etc and in a very generic way. It is set up in the same way as the mvc implementation using contentmanager.builddisplay, the results of which it then passes through the display management via custom helper. This uses a small modification in the display management to set a flag on shape bindings (see the code) to indicate there purpose (ie html display or in this case model translation). As with the core controller it serves up a single item however this includes projections so you can quickly and easily serve up whatever queries you like(with paging as well).

I also have a very simple example(on a feature branch) of an admin controller taking a more rigidly structured json model and using the contentmanager.updateeditor method to model bind via the drivers and create content items, again this mirrors the current mvc implementation.

Please note that I haven't done anything with layoutpart as I did this 5 months ago before 1.9 was around so you'll need to use content items that don't use this (ie use body part).

It's more of a proof on concept that an actual implementation but I'd been really interested to see what people make of it. I have a fork containing the core alterations and then a module with the api:
Developer
Jul 10, 2015 at 6:47 PM
We could also supply responses with the HAL convention: http://stateless.co/hal_specification.html