Overview over all blog items

Oct 05, 2006

There is always hope...

A Plone 2.0b1 -> 2.5.1 migration adventure

At some point you have to migrate a Plone site from time to time. So I helped a friend to migrate his institute website to Plone 2.5.1. Unfortunately the original site was something like Plone 2.0 beta 1 (at least the migration tool was telling that). The first tries using the default Plone migration failed in differently ways badly depending on the Zope and Plone version (we also tried to migrate to Plone 2.1 first but with no success). So no way out? No chance to get the data in a sane way back into Plone 2.5? Don't give up the hope, there is always hope!

Actually the following steps worked well to perform a half-way automated migration:
  • Set up a clean Plone 2.5.1 instance with Zope 2.9.5/Python 2.4.3
  • Try to export parts of old site using the ZMI export. You will get a bunch of .zexp files e.g. for the top-level folders
  • Move the .zexp file to your Plone 2.5.1 instance and try to import the .zexp files through the ZMI inside a scratch folder. It might be necessary to install some of the old 3rd-party products used within your old site in order to make the .zexp import work!
  • I wrote a small migration script that walks of the hierarchy of the old objects and creates new instances inside a shadow folder. The current implementation recreates folders, documents and files. Adding support for images and other types should be straight-forward. Basically after creating the new object you must call the mutator methods of the new object with the data of the old object (requires some sniffing in the old CMFDefault code)
  • The script does not care about ownership issues, workflow states or other esoteric metadata since this was not necessary in this particular case. It might be useful for other people where the standard Plone migration fails however if there is a  need to migrate your content without  engaging a bunch of cut&paste monkeys.

I found this brilliant tal:condition inside a Plone template today:

      <div class="row"
             tal:condition="python:test(content_type.startswith('text'), 1, 0)">
                <h2 i18n:translate="heading_file_contents">File contents</h2>
                <pre tal:content="here/get_data|here/data|nothing">

Some Plone users had a hard-time in the past with the configuration of TextIndexNG3. Since the out-of-the-box installation should be running on a bare Zope system, it has been impossible to ship TextIndexNG3 with a configuration that would work out-of-the-box with Plone. Userd has to edit a configure.zcml file in order to register adapters for Plone content-types properly. This caused unnecessary headaches for users and myself. The upcoming TextIndexNG V 3.2.0 release will auto-configure all adapters properly (at least for the default Plone content-types) and it will no longer be necessary to touch any configure.zcml file. This will hopefully help avoiding the need to explain how to edit XML files :-) This auto-configuration feature currently only works with Zope 2.10..have to check how to backport the functionality back to at least Zope 2.9.

Stefan Holek gave a great presentation at the Plone Conference 2006 in Seattle about mistakes and things to avoid when programming with Zope & Plone.

As a developer you should take your time and step through the slides:


I am sure that even advanced programmers will find violations of the 28 Lurker Laws in their code. This is a must-read.

Dec 16, 2006

Kudos to Philipp

Web Component Development with Zope 3, Second edition available

Yesterday I received (finally) the second edition of Web Component Development with Zope 3 written by young master Philipp von Weitershausen. Compared to the first edition the book is much better structured and more complete. It covers the most recent features of Zope 3 very well although the chapters can't go into very detail (otherwise the book would possibly have a thousand pages or more). Even as an advanced Zope 2 developer this book contains a lot of new stuff and new insights  The book is a must-have for all Zope 2 and Zope 3 developers and it is the best Zope book published ever....really great work...put the book on your x-mas wish list.

In Zope 2.10 we replaced the ZPT implementation with the one from Zope 3  (with some additional wrapper code). The Zope 3 ZPT implementation is pretty much clean and unicode-aware. So the next step is to make Zope more aware of unicode. For that purpose the Zope 2.11 implementation will use unicode as internal representation for the ZopePageTemplate file class (which is used by ZPTs stored within the ZODB). For output purposes (WebDAV/FTP) each ZopePageTemplate instance now has an output-encoding property that controls the conversion from unicode. While processing a ZPT everything happens on the basis of unicode. It is now up to the ZPublisher to convert the rendered output to some other encoding. This encoding is either given through the Zope default configuration or set by the application itself by using RESPONSE.setHeader('content-type', 'text/html; charset=xxxx').

However Products.PageTemplates also contains PageTemplate and PageTemplateFile classes. I think it does not make sense to enforce unicode directly inside the classes. The PageTemplate class for example is used by CMFCore.FSPageTemplate. It makes more sense to convert the data inside  these classes to unicode instead of messing up the base classes.

Near term goals:

  • backport  the latest ZPT changes to Zope 2.10.2 since Zope 2.10.0 and Zope 2.10.1 are partly broken concerning encoding issues and WebDAV/FTP support
  • making CMFCore.FSPageTemplate unicode-aware (you should be able to specifiy the encoding of a filesystem-based PT through its .metadata file)
  • checking how the ZPT unicode-awareness affects Five

Archetypes-based content-types are traditonally tied to the ZODB as backend storage. It has always been hard making the content available through a RDBMS - the only solution for a long time has been SQLStorage. However the implementation and the behavior was more than flaky. Lately projects like collective.tin (Laurence Rowe) and Contentmirroring by Kapil appeared on the scene.

Jul 01, 2010

Using ZODB demostorage for migration testing

A small hint when doing migrations

A common usecase when working with migration related issues in Zope world is:

  • writing migration code
  • testing the migration code until it work

Usually you test the migration code against a copy of some production ZODB storage. However migration code usually modifies your storage. Here comes demo storage into the game. Any ZODB storage can be wrapped through a demo storage proxy. All changes to the storage will be persistent in memory only and the wrapped storage remains untouched. By doing this you can test you migration code against the copy of the production over and over again without replacing it all the time with a fresh copy. The only thing you have to configure is to modify your ZODB storage configuration and insert <demostorage> around the configuration of your storage to be wrapped.

PDF document icon ProducePublishCloudEdition.pdf — PDF document, 151 kB (154,844 bytes)