Let's create a new Gallery!

Topics: Announcements
May 22, 2015 at 11:25 PM
We've discussed this before: we need a new and improved Gallery. I'm sure there were other discussions but I could find only this one: https://youtu.be/xOL1tk10f8c?t=20m36s

So what's the issue with the current Gallery? The most painful points seem to be that:
  • you have to create module/theme packages and upload them to the Gallery (and keep these up to date) to publish an extensions and
  • it's hard to tell whether an extension is compatible with a certain version of Orchard.
Then let's create a Gallery that (on top of what is there today) as a start has solutions for these two issues!

Instead of package uploads we could have source control integration: similar to what the Media Theme on DotNest does you'd give it a URL to a zip file containing the extension's code and there would be a webhook to call when you want the Gallery to fetch, package and publish your extension. This is a provider-agnostic way of setting up automatic publish from source control (works with anything if there's a zip to download). E.g. for a GitHub repo you'd do the following:
  1. Configure the URL of the source download of a branch (e.g. https://github.com/OrchardCMS/Orchard/archive/master.zip for Orchard's master branch) on the Gallery. This would be a third option besides uploading a package and specifying a package URL under https://gallery.orchardproject.net/Contribute/NewSubmission.
  2. You get a secure API endpoint unique to your extensions. You configure a GitHub webhook to call that API endpoint every time you push to your repository. Similarly webhooks are available on Bitbucket.
  3. When you push to the repo and update the branch used for Gallery publishing the Gallery downloads the zip, updates the displayed details from the manifest file, creates a nupkg and makes it available for download, just as today.
For version compatibility the task can be outsourced to users by having a vote feature similar to that of the Wordpress Plugin Directory (see e.g. here at the bottom right: https://wordpress.org/plugins/wp-super-cache/).

We at Lombiq can undertake to do the following tasks, in this order:
  1. Take the current code of the Gallery and update everything to be compatible with and make use of the new features of Orchard 1.9. That is, if the Gallery code can be re-used.
  2. Host a public test version.
  3. Implement source control integration.
  4. Implement version compatibility voting feature.
If nobody has anything against it we can start, we'd only need access to the current source of the Gallery. Let's stay focused now and don't care about e.g. the Gallery's design but let's try to make it up-to-date first and implement these two crucial new features. Then we can talk about any other improvements.
May 23, 2015 at 9:29 AM
A few thoughts.
  • Why link to a zip? Surely this leads to build outputs being stored in source control, which is not ideal. I can see that for those debs that do not have access to a CI environment like Team City that this might be a quick way. Alternatives include just pointing at a source location, but I guess there might be concerns that the gallery server might have more work to do then. As long as it is also possible to support direct package upload (via API) then you support both models
  • The closer that the format is to a basic nuget package the better.
  • We are probably in the minority of use cases, but a private gallery would potentially be far more useful to us than a public one, as our modules are all bespoke and commercially sensitive. Again if the gallery was a completely standard nuget feed, then we could make our own decision how to host, as long as the orchard admin tools ( UI and command line) understand a plain nuget server. presenting and filtering packages becomes only about understanding their meta data (e.g module.txt). Within orchard. You could even have an orchard module that then acts as a gallery server front to a nuget feed, with a site recipe to match.
May 23, 2015 at 11:01 AM
The zip file would be the full source zipped up, i.e. what source control providers (e.g. GitHub, Bitbucket) commonly automatically make available (again, on GitHub this is the "Download ZIP" button) and what version control systems commonly have implemented somehow (i.e. archiving). This wouldn't be an Orchard extension nupgkg, just a plain zip (the package would be created by the Gallery automatically). Because of this you don't need to store zips in source control and you don't need build support.

Ideally the Gallery would be an up to date NuGet feed, yes. It started as such but the NuGet library in Orchard being a very old one I guess it's now out of date. Updating this would be a later step, but for now we'd use Orchard's standard packaging services (that the packaging commands also do when you generate an extension package from the command line) which could then be updated in the future.

BTW privately I wouldn't use a Gallery server but rather share extensions via source control (think git submodules).

And it just came to my mind that the Gallery's source is on Codeplex, here: https://orchardgallery.codeplex.com/
May 23, 2015 at 1:09 PM
I guess my thought is that extending a nuget server source is not as good as writing a product that simply consumes a standard feed (not saying anything about difficulty of course).

We do use git sub modules for the few things that we can, but mostly our scenario concerns what we deploy to what environments, rather than what we build.

What does the current orchard packager cover that nuget pack does not?
May 23, 2015 at 1:26 PM
Markdown support for package descriptions would be nice thing to see!
May 23, 2015 at 3:58 PM
I don't think Orchard packaging does anything that NuGet doesn't, since it's an old version of NuGet.

Markdown is an interesting idea. We should keep in mind that descriptions should be possible to be auto-generated from extension manifest files; with Markdown this is possible.
May 23, 2015 at 4:38 PM
Maybe I was a bit quick there - for the markdown formatting I was referring to the package description page on the gallery, however I don't know how much this method of browsing packages is actually used. The problem with the gallery format at the moment is that it offers much more package information than is contained within the package manifest. For example, you can add multiple preview images and a standalone description. How will the new gallery address continuity in this area?

For the dashboard feed I think that it should absolutely display the description from the manifest file. At the moment it pulls the description from the package's gallery page and this looks somewhat noisy in a list view.

On a tangent to preview images, something which has bothered me is that they're not square. We should absolutely make all preview images square.

I'd like to put myself forward for developing a theme for the gallery and developing general usability stuff if this is going to be part of this project.
May 23, 2015 at 4:57 PM
I was also talking about Markdown for Gallery descriptions. It's just that these descriptions are currently automatically pre-filled from the manifest's data (which is a good thing) and whatever we add there it should be thus compatible with the format the manifests use (e.g. thus allowing HTML for example, what naturally is a bad idea, wouldn't work since manifests can't operate with HTML).

However the manifest'd content and what should be displayed in the Gallery shouldn't necessarily be the same (e.g. the manifest can contain some technical info irrelevant in the Gallery) so it's still good to keep the two separately editable IMO.

And if you can take part in improving the usability then by all means, this is great.
May 23, 2015 at 6:16 PM
Edited May 23, 2015 at 6:21 PM
Npm is a great example to look at: it has git integration, and gets a lot of things right. In particular, to publish a package, I just cd to the module's directory and npm publish. That's it. It already knows who I am, and can get all the info it needs from the package.json file. That is applicable to Orchard directly.

Npm also recently added private packages. It's super easy to use, just one additional parameter on the publish command-line. That reduces the need for private galleries (although that would certainly be an option), and reduces the cost of entry. Also, that brings us closer to the possibility of paid packages ;)

Images are a pain in the current gallery. Why not make it a convention that the images for the gallery (icon, and screenshots) are actually part of the package, with corresponding properties in the manifest to point at them?

As for Markdown description and documentation, let's just do what everybody else is doing: if there is a readme.md file at the root of the package, just use that as the description. Npm does it, Github does it. Let's do it.
May 23, 2015 at 6:18 PM
BertrandLeRoy wrote:
As for Markdown, let's just do what everybody else is doing: if there is a readme.md file at the root of the package, just use that as the description. Npm does it, Github does it. Let's do it.
Was going to suggest this - totally agree.
May 23, 2015 at 7:35 PM
Good idea with the Readme! I'd keep the option to manually edit the description appearing in the Gallery but optionally it could be updated from the Readme.

I think having a publishing command line goes sort of in another direction than source control integration and still it would need a manual publish (and to implement everything related for the command line).
May 24, 2015 at 2:09 AM
Not really: the publish command can do the initial publication, and then updates can happen through the source control that's been set-up from the manifest.
May 24, 2015 at 6:00 PM
Ah I see. This looks quite complicated (the command line should authenticate you and store credentials for re-use, it has to detect if you use source control and process that to set it up automatically...) so I think this can be a later step once the experience is full on the web UI.
May 25, 2015 at 6:42 PM
Localization features: upon uploading/updating a theme/module in the Gallery, the master .po file could be automatically generated. This would allow us to have all the modules and themes added to the Orchard CMS Gallery project in Crowdin and keep them up to date very easily (even automating the whole process is possible).