Docker is broken right now in many ways.
- Docker images are huge (typically for me in the range of 1 to 1.5 GB each)
- Pushing and pulling Docker images to/from the Docker registry is slow - even if you push/pull only new layers. Nobody on the Docker forums could explain the widely differing loading times of layers (from slow to fast)
- There is no trivial way to replicate Docker containers.
- The concept of data containers is as long broken as there are no reasonable way to replicate and backup data containers. There is no trivial way for moving data between containers.
- Backup methods specific to certain services do no longer work on the container level.
- Docker builds are slow (even on fast machines).
- Monitoring Docker containers is not trivial and hard to achieve. Our fat development server feels sluggish with a high load running only 3 Docker containers. I currently do not see obvious options for tracking down the real issue.
- Docker runs stable (or not )depending on the underlaying Linux distro and kernel. I can not find any officially supported recommendations for distros and kernels.
- The Docker registry is a huge bag of unsorted Docker image crap - it feels like the early PyPI years where every @@@@@ felt the power to upload every crap into the wild.
I am still using Docker for a few usecases like providing a demo instance for XML Director or for having containers with Base-X and eXist-db used for running unittests.
Docker has potential but it has many serious issues. Of course the Docker fan boys will crucify me for this post. Also looking into CoreOS Rocket over the next days however Docker feels much more mature than Rocket but Rocket also has to start from somewhere...the concepts of Rocket appear more well thought than Docker, the MongoDB of virtualization.
I have been working with Docker over the fews in order build a demo system for XML Director. Docker in general looks nice and promising but there are lot of things that cause some headaches:
- incremental builds with caching is a nice feature in theory. In real life you often have to make changes at the begin of the Dockerfile and the complete build dance starts from scratch
- Docker builds are slow
- Pushing an image is slow
- Pulling and updating images is slow and unreliable
- Various kernel panics on OpenSuse 13.1 (no crashes with CentOS 6)
- Docker "internal" corruptions that could be resolved only by removing the local /var/lib/docker folder
- It is not possible to start a build at a certain point within the Dockerfile without some tricks
- Fiddeling a buildout configuration takes a lot of time because you can not make use of buildout caches without tricks
- Docker images can be big. In my case the XML Director image with Plone and eXist-db ended up with a final size of 2.1 GB
- from reading/hearing: Docker had several severe security issues in the past
Docker is a new project and moving and evolving at fast lightspeed. That's why it is important keeping an eye on it.
XML-Director will be a new-generation XML content management system based on the Plone 5 CMS with either eXist-db or Base-X as backend. Additional components will provide DOCX to XML and XML/HTML to PDF/EPub conversion, support for desktop and web-based XML editors.
I am pleased to announce the start of a new big project in 2015: XML Director
XML-Director will be a new generation XML content-management-system focused on
- combining the best technologies on the market into a superb XML solution
Based on almost 20 years of experience in the electronic publishing and publisher business and based on our experience with Produce & Publish over the last years it is time to bring our tools to the next level.
What will XML-Director be?
- First, XML-Director will be a classical open-source project open to everyone interested in Plone, XML, CMS, PDF, EPub, conversions etc.
- A classical CMS solution for XML content with Plone as the basis. Plone's content-type framework Dexterity makes it very easy to create new XML-ish content-types either through code or through the web. The existing core feature of Plone like security, role-based user management, fine-grained security model will be full available within XML-Director.
- XML-Director will be a storage for XML content (from small to large). XML-Director will support out-of-the-box both XML databases eXist-db and Base-X.
- DOCX to XML conversion: XML-Director will provide a pluggable API for integrating external conversion services like C-REX or solutions build on top of open-source software like LibreOffice encapsulated through a webservice. Our goal is to provide out-of-the-box conversion support of DOCX to Docbook and DITA.
- PDF generation will be based on our Produce & Publish solution that exists now for many years with PDFreactor and PrinceXML as primary converters. However free solutions like WKHTMLTOPDF will also remain supported. In addition we are looking into further PDF conversion options like Antennahouse Formatter and Speedata Publisher.
- Configurable content-types
- Support for XML metadata either in Plone or XML or mixed
Usecases for XML-Director
- Technical documentation
- Authoring and production of guideline documents, SOPs etc.
- classical XML production and publishing workflows
- Plone CMS (version 4.3 and later 5.X)
- eXist-db or Base-X
- PDF conversion: PDFreactor, PrinceXML, WKTOHTMLPDF
- EPub conversion: Calibre, custom implementation
- DOCX conversion/import: C-REX, custom implementation
- All modules of XML-Director will be freely available as open-source (GPL2) on Github
- specific custom solutions build on top of XML-Director where confidentiality is an issue
- commercial converters like PrinceXML or PDFreactor
- Everyone is invited to participate the project. The code is on Github: fork it and improve it
Four new XML related Dexterity fields
I decided to integrate eXist-db much tigher with Plone as initially planned (for strategic reasons). After heaving fighting with z3c.form I finally got to the point where I have now 4 new different Dexterity fields that store their data in eXist-db instead of the ZODB.
- XMLText: this field holds an arbitrary XML document. XML validation is performed both client-side and server-side. The widget for XMLText is the ACE editor.
- XMLXPath: this field can be used to pull in parts of an content from an arbitrary XMLText field by specifying the name of the XMLText field and an arbitrary XPath expression
- XMLImage: as as the Dexerity image field but with eXist-db as storage
- XMLBinary: same as the Dexterity file field but with eXist-db as storage
Here is a screencast of the Produce & Publish XML Edition that features an integrated environment for
- conversion of DOCX to XML/HTML
- conversion XML/HTML to PDF
- publishing of PDF and HTML to the web or other services
The solution is powered by
- Plone CMS
- C-REX for DOCX to XML/HTML conversion
- PDFreactor for PDF generation
- eXist-db XML database
The non-commercial and generic parts of the project will be released as open-source software early next year.
The Plone integration with eXist-db is already available as zopyx.existdb package.
Supporting WebDAV services, Samba Shares or Dropbox in Plone? mr.mount might be the solution!
Over the last days I blogged several times (link, link) about the ongoing integration of Plone with XML databases (eXist-db in particular as part of an ongoing XML publishing project with Plone). The integration is based on the WebDAV API of eXist-db and my module zopyx.existdb that integrates the hierarchical structure of eXist-db directly into Plone (similar to the Plone Reflecto add-on by Wichert Ackermann). Because of the integration layer is WebDAV it was almost trivial providing the same integration support for the XML database BaseX. But wait...there are other services providing access through WebDAV. During the Plone Conference in Bristo l gave OwnCloud a quick try and it was easily possible to mount OwnCloud (or a subfolder) into Plone for reading and writing. But then Asko had the crazy idea to try to mount Plone within Plone through WebDAV and it actually worked. Here is a screencast where plone.org/events was mounted into my local Plone sandbox. Just saying: the integration with other services works best for binary stuff like images, videos, office documents etc. It is not very suitable to access Plone test-ish content because Plone provides such content either rendered as complete HTML pages through the HTTP port or as document source over the WebDAV port. There might be options to deal with this some way but this is actually not the first scope of my project. So my eXist-db integration got a more wider scope: providing a unique way for attaching external data sources into Plone.
After a glass of wine I had the idea to turn the existing code into a more generic module - codename "mr.mount". On the implementation level I am using the pyfilesystem module that provides a generic and unified API for hooking arbitrary filesystem services into Python (as long as there is a driver of the FS). Fortunately pyfilesystem provides out-of-the-box support for WebDAV, local filesystem, SFTP or even AWS S3. So supporting all these filesystem services in Plone should not be that hard. I evaluated Dropbox access some time ago with an older version of pyfilesystem and it worked nicely (apart from issues dealing with the OAuth token from Dropbox). After some research I found two drivers for SMB/Samba for pyfilesystem...so supporting Dropbox and Windows/Samba shares in Plone appears feasible.
To make it clear: the current approach is not a replacement for the ZODB and not another storage layer for Archetypes/Dexterity. It is all about providing a smooth integration of external data sources. The core implementation right now provides traversal support from Plone into those external services including basic UI support for managing folders/collections (add, rename, delete), for up- and downloading content, ZIP export/import and a basic integration with the ACE editor. I am sceptical about a more deeper integration into Plone as it was implemented in Reflecto. I do not think that external content should be treated in a way a primary Plone content. So I would hestitate providing some kind of indexing support because that did not work out well in Reflecto but I am open to suggestions.
These are my ideas about mr.mount and the integration of other services into Plone. The implementation of mr.mount is not of high priority since my basic goal of talking with Plone to eXist-db is perfectly working for my project. If you are interested bringing this idea forward (with suggestions and funding), please get in touch with me.
As part of the Plone Conference 2014 I finally ended up with the perversion to mount a Plone site through WebDAV into another Plone site. In this a quick and dirty screencast where you can see that it is possible to mount plone.org/events as a subfolder into a local Plone 4.3 installation with the option to navigate through the substructure and loading content like image, binary files etc. With some more further work it would be possible to render text-ish content from the remote site like news items or documents within the context of the local site.
As part on my work on zopyx.existdb I made some progres towards mounting arbitrary WebDAV services into Plone. With only a few changes to the current zopyx.existdb codebase I was able to connect Plone with BaseX (another open-source XML database) and OwnCloud. In both cases I was able to create/remove folders and create/remove contents through Plone directly within BaseX or OwnCloud. So this solution is the best solution right for Plone to integrate with arbitrary WebDAV services.
Here are the results of the Plone Developer Survey based on 103 submissions - thank you all.
While some of the charts deliver some interesting information and trends, the real valuable information can be found in the various comments left in the sections about likes and dislikes in Plone. I leave it to the community drawing their own conclusions from the numbers and comments. At least I have some more information and input for my "Why Plone will die" talk to be given at the Plone conference in Bristol in two weeks.