Most power users mays already know this. But this is a real enlightenment for newbies who are stuck with the command line pdb utility. Even for experienced developper as I am… supposed to be, this is a real advantage to have on screen at the same time the source code, with breakpoints in the margin of it, the inspector for your global and local variables, the access to the call stack and execution frames.

Start reading the fine manual here and come back reading how to get this in Zope 2.

Of course I assume you have a recent Eclipse (Indigo) and Pydev (2.2.4)

Find in your Eclipse software directory the full path of the directory that contains “pydevd.py“. In other words :

$ cd /your/eclipse/root
$ find . -name 'pydevd.py'

In my MacBook the file is found in “/Developer/eclipse/plugins/org.python.pydev.debug_2.2.4.2011110216/pysrc“

(note that upgrades of PyDev may change this path to something else)

Edit your development buildout config file and add this in your plone.recipe.zope2instance part:

[instance]
# recipe = plone.recipe.zope2instance
...
extra-paths =
    /Developer/eclipse/plugins/org.python.pydev.debug_2.2.4.2011110216/pysrc

Of course re-run your buildout. Open the debug perspective in Eclipse and start the PyDev debugger server (see link above).

You can now add anywhere you want (need) the line “import pydevd;pydevd.set_trace()“, or use the shortcut “pydevd” to insert your first hard breakpoint.

Start your Zope instance and make the necessary clicks to execute the line that has the hard breakpoint. Wow, great : the next line of Python to be executed is hilited from there, you cand step, step over, go to the next return, and even add soft breakpoint double clicking in the left margin anywhere in your source code.

All other things you may need to know about debugging with PyDev are here and usable “as is” in a Zope server.

Another great feature for those who develop with Windows : when debugging with Eclipse/Pydev you can start your development instance as a service, when before you needed to run the sloooooow “instance fg” and benefit of Eclipse remote debugging.

Complex Python based solutions are now built with the great zc.buildout, like complicated ZEO clusters with heavily customized Plone site or others.

zc.buildout is a great solution that’s now essential to share among developers a custom project as well as for distributing to users or customers. As everybody know, zc.buildout grabs components from the Internet (mostly from the Pypi site) and makes the application assembling those components according to the integration directives in the various *.cfg files. But talking about zc.buildout in details in not the purpose of this blog post of course.

Installing the solution in the customer’s computers is supposed to be easy and straightforward. “Unfortunately”, running zc.buildout requires to bootstrap the project with the famous “bootstrap.py” file.

This script downloads and installs some essential resources that enable to run the “bin/buildout” command, like distribute and zc.buildout itself. But with some customers, there is no possible Internet connection on the installation target, due to a paranoïd strict security policy. And I don’t want to write a complicated installation manual that works around this issue. In addition, these customer have little or no IT required skills.

After asking to some experts in the Plone mailing lists, some answers provided some pointers from which I made my “offline_bootstrap.py” script that must be used in place of the classical “bootstrap.py” script in such situations.

The solution

Prepare your buildout structure

In particular, you must ensure that your “buildout.cfg” file has these options:

[buildout]
...
# All packages are downloaded locally.
download-cache = downloads
# All eggs are installed here
eggs-directory = eggs
# You need this if there is an "extends = http://..."
extends-cache = ext-cache
...
versions = versions
...
[versions]
...
# Freeze your zc.buildout preferred version!
zc.buildout = x.y.z

Add to the buildout directory the “offline_bootstrap.py” script you can grab here.

From the same place add a “bootstrap_resources/” directory. In this directory, add:

• “distribute_setup.py“
• “distribute-x.y.z.tar.gz“. Important: Use the same version as the one indicated by the DEFAULT_VERSION in above mentioned “distribute_setup.py“. Find it from here.
• “zc.buildout-x.y.z.tar.gz“. Important: Use exactly the same version as the one pinned in your “buildout.cfg” as above described. Find it from here

From now you can bootstrap your zc.buildout and carry on  developing/testing/documenting your Python project as usual.

Installing in an no-Internet platform

Your beautiful project is ready to be installed in a customer computer that has no connection to the Internet. First add a new file in your buildout: “production_install.cfg” with these few lines:

[buildout]
extends = production.cfg
install-from-cache = true
offline = true
newest = false

Of course, I assume you made a “production.cfg” buildout profile that smiles on your integration platform.

You may now remove the database (typically the “var/” folder) from the buildout directory and make a tarball.

Inflate this tarball in the place your customer wants it on his computer. From the newly created folder, you just need to say this to the console :

$ python offline_bootstrap.py -c production_install.cfg
$ bin/buildout -c production_install.cfg

That should be all unless other application specific operations.

Other things about “offline_bootstrap.py”

“offline_bootstrap.py” accepts any option and argument the usual “bootstrap.py” takes. Except “–setup-source” and “–download-base” that are set specifically by “offline_bootstrap.py“.

Questions or comments ?

… are welcome.

ZCML is a great feature in Zope 2 since the first Five years. Exploring a well designed Python package starts with its ZCML files and its main “interfaces.py” module.

But newbies as I have been and others have pain to remember the various ZCML directives available, as well as the detailed attributes of them.

Of course there are some documentations in printed books or in various blogs or docs about most directives. But there’s no central point (unless I didn’t search correctly) where you can read the documentation for all ZCML directives available in your Zope 2 instance with third party components that could add new ZCML vocabulary.

Of course, you may grep all “meta.zcml” files in your instance, search for the appropriate namespace, then the appropriate directive. And finally open the Python file that defines the schema of the mysterious ZCML, just as I explained in another ZCML related blog post.

Grepping the “zope.configuration”, I hopefully found a magic function that provides a structured tree of all the registered ZCML directives well suited to provide a live documentation.

After some hours of work and exploration, I’m proud to give to the Zope 2 community “aws.zope2zcmldoc” that provides a live documentation on all ZCML namespaces, directives and subdirectives installed in your Zope 2 instance.

Some screenshots

Available from the standard Zope control panel

All ZCML namespaces

All directives from the "browser" namespace

The "pages" directive in details

You can notice that the description text are translated in your favorite language when the translation is available.

aws.zope2zcmldoc is the new companion of Products.DocFinderTab, plone.reload, Products.Clouseau and zope2.zodbbrowser (and perhaps others I forgot, sorry) to your development instances.

Any feedback, is welcome.  Enjoy !

Ah yes, and many thanks to my actual employer Alter Way for sponsoring this piece of software.

Lots of you are still in process of maintaining Plone 3 sites for yourself or customers and didn’t go to Plone 4 for some good reason.

The Plone 3 bundle is powered and tested with the latest Zope 2.10. This Zope version itself includes ZODB3 3.7. That has lots of issues that are now gone with later versions.

But wait, it’s not possible to use ZODB3 3.10.x since it seems it requires Python 2.6. ZODB3 3.9.x too has too many API changes and it seems that we need to go to a full Zope 2.11 to support this lates version.

ZODB3 3.8 seems a reasonable choice to improve the persistence support of your Plone 3 apps.

• It compiles with Python 2.4- It has a lot of new features
• It has a lot of bugfixes
• It has lots of additional tests, thus should be more reliable

Have a look at http://pypi.python.org/pypi/ZODB3/3.8.6#whats-new-in-zodb-3-8-6-2010-09-21 to see all the details.

This sample buildout extends the standard “buildout.cfg” that comes with the ZopeSket template “plone3_buildout”, and replaces the ZODB3 that comes with Zope 2.1.0 by the ZODB3 3.8.6 egg, as well as some Zope 2 components that need an explicit upgrade to cope with this new ZODB3 version.

[buildout]
# buildout.cfg as built from plone3_buildout ZopeSkel template
extends = buildout.cfg
eggs +=
    zope.proxy
    ZODB3
    zodbcode
    tempstorage

[zope2]
skip-fake-eggs =
    ZODB3
    zope.proxy
    zodbcode
    tempstorage
[versions]
# Marked as additional fake egg in [zope2]
ZODB3 = 3.8.6
zope.proxy = 3.4
zodbcode = 3.4.0
tempstorage = 2.11.3

Some warnings and gotchas

• This configuration has been tested as simple instance deployment. I didn’t yet test a ZEO cluster in such situation.
• This configuration has been tested successfully with only some well known third party extension : LinguaPlone, Collage, PloneFormgen, Ploneboard. Some third party Plone extensions that may play with ZODB inners should be tested in depth (run at least unit tests) before going in production.

Another thing

If something is going wrong with this new ZODB, you can go back to the original ZODB3 that comes with Zope 2.10.x. Unless you have been playing with new storages and options that come with ZODB3 3.8.
Any feedback of others who tried this recipe or similar ones is welcome.

I have been investigated some months ago with Encolpe Degoute on how to set a workflow state on a content when the workflow is DCWorkflow powered (Plone of course). Things are not that simple since DCWorkflow does not provide a public API for this. Instead, we must execute the various transitions that are slow and may trigger unwanted events.
This small scripts uses deep inners private API of DCWorkflow (yeah, that’s evil, but…) and makes the expected job:

• Sets the workflow state of a content to whatever state without executing any transition

• Sets its security martix as expected

• Reindexes content security

Here is the baby… In the hope it will be useful to others and maybe get improvements…

It has been used in many Plone apps and migration utilities.

from Products.CMFCore.utils import getToolByName
from DateTime import DateTime

def sample(folderish):
    folderish.invokeFactory(type_name='Document', id='blah', title="Blah")
    blah = folderish['blah']
    changeWorkflowState(blah, 'published', comments="No comment")
    return

def changeWorkflowState(content, state_id, acquire_permissions=False,
                        portal_workflow=None, **kw):
    """Change the workflow state of an object
    @param content: Content obj which state will be changed
    @param state_id: name of the state to put on content
    @param acquire_permissions: True->All permissions unchecked and on riles and
                                acquired
                                False->Applies new state security map
    @param portal_workflow: Provide workflow tool (optimisation) if known
    @param kw: change the values of same name of the state mapping
    @return: None
    """

    if portal_workflow is None:
        portal_workflow = getToolByName(content, 'portal_workflow')

    # Might raise IndexError if no workflow is associated to this type
    wf_def = portal_workflow.getWorkflowsFor(content)[0]
    wf_id= wf_def.getId()

    wf_state = {
        'action': None,
        'actor': None,
        'comments': "Setting state to %s" % state_id,
        'review_state': state_id,
        'time': DateTime(),
        }

    # Updating wf_state from keyword args
    for k in kw.keys():
        # Remove unknown items
        if not wf_state.has_key(k):
            del kw[k]
    if kw.has_key('review_state'):
        del kw['review_state']
    wf_state.update(kw)

    portal_workflow.setStatusOf(wf_id, content, wf_state)

    if acquire_permissions:
        # Acquire all permissions
        for permission in content.possible_permissions():
            content.manage_permission(permission, acquire=1)
    else:
        # Setting new state permissions
        wf_def.updateRoleMappingsFor(content)

    # Map changes to the catalogs
    content.reindexObject(idxs=['allowedRolesAndUsers', 'review_state'])
    return

• It’s damn fast. Consider using changeWorkflowState if you need to set the workflow on a huge set of contents in one transaction. Read a content migration or a bulk content creation.

• You can set the workflow state you want, including a state that’s no in the workflow associated with the content ;o)

Cons:

• Doesn’t use legacy API. So this may or may not work with future versions of DCWorkflow.

• Doesn’t use the transitions. This is an intentional feature for speeding up all this. As a consequence, no transition script or event is triggered. But this is perhaps

Not recorded in workflow history. Honestly this is not an issue for my use case.
As a counterpart, there’s no control on the validity of the state value in the context (global workflow or placeful workflow). Buggy user code may issue content on which no workflow transition is possible.

Last week, I read a conversation with John Stahl and Mustapha Benali about the PLIP 185, and realized that this PLIP is almost iw.memberreplace (or the opposite).

So I spend a couple of hours on that component to add the last details on that component. Et voilà, the last release of iw.memberreplace (1.0.0-RC1) fulfills now that PLIP: the original member can now be removed – if defined in a mutable users source.

I swear that in the future, I will read the open PLIPs before creating a new component ;o)

Enjoy…

iw.memberreplace control panel

openxmllib is a pure Python package built on lxml that parses an ECMA373 office file (read *.docx, *.pptx, *.xlsx recorded from MS Office 2007). It can actually extract – very fast – the indexable words from an MS Office 2007 document. It may be used from any Python app, even from others (Java, PHP, C++ …)

Products.OpenXml is a component for Plone 2.5 -> 3.2 that registers the MIME types and icons for the known extensions for these office files and provides transform rules to indexable text, such MS Office 2007 documents in ATFile or any content type with a searchable FileField are indexed.

Both are avalaible at the cheeseshop.

Future directions

Yeah, those babies make satisfying job at the moment, I have not enough room to copy the testimonies of satisfied users. But users want more…

Have real plain text

openxmllib actually provides all words from a document. This is very fast because some XPath expressions make the job. But when the result is appropriate for indexing, it is not human friendly. The words are returned in any order and there’s no way to understand what’s written.

Have HTML preview

Yes, previews of office documents in a Plone site are great. ARFilePreview shows some of what I want to do: having a nice HTML document, visually as close as possible from the printed document.

Volunteers?

Unfortunately, I have not much time to spend for this, and the ones who asked these features didn’t want to participate or fund my work. You are interrested for adding these features, please let me know. Skills in XSLT and in the ECMA373 standard are required for this.

Many thanks by avance.

This component replies to a well known use cases and request from our customers, especially from intranet managers.

How do we do on our Plone intranet when Mr Foo dismisses our company and Ms Bar is entitled to do his job?

Ms Bruni just married and she’s now Mrs Sarkozy. Her login is now c.sarkozy and she lost the permissions she had on some contents she used to work on. How shall I handle this? Should I browse in the site finding all her stuffs and change the sharings, property (…) accordingly?

Some weeks ago, this was in a bunch of code I included in private site products. Some days ago, I released it as a public component.

Just install iw.memberreplace, open its control panel, select the desired options and let it do all the job – that otherwise would require hours of digging – for you.

Enjoy, and as always, feedback and contributions are welcome.

Anyway, there’s actually an annoying issue on upgrade steps: upgrade steps for component “foo” are exposed in sites where the “foo” component is not installed.

Wow, managers on instances where various different Plone sites are installed have to be careful on this. Running an upgrade step of a component that is not installed in a site may be harmful. Even if upgrade scripts are supposed to be defensively programmed.

Having several Plone extensions, I made a decorator based generic safety belt. Of course this does not prevent GS showing the upgrade steps but executing such upgrade steps raises an explicit error message and does not execute the upgrade step.

Just add these lines to your “utils.py”:

from zope.component import getUtility
from Products.CMFCore.interfaces import ISiteRoot
from config import PROJECTNAME

class NotInstalledComponent(LookupError):
    def __init__(self, cpt_name):
        self.cpt_name = cpt_name
        return

    def __str__(self):
        msg = ("Component '%s' is not installed in this site."
               " You can't run its upgrade steps."
               % self.cpt_name)
        return msg

class IfInstalled(object):
    """The decorator"""
    def __init__(self, prod_name=PROJECTNAME):
        """@param prod_name: as shown in quick installer"""
        self.prod_name = prod_name

    def __call__(self, func):
        """@param func: the decorated function"""
        def wrapper(setuptool):
            portal = getUtility(ISiteRoot)
            qi = portal.portal_quickinstaller
            installed_ids = [p['id'] for p in qi.listInstalledProducts()]
            if self.prod_name not in installed_ids:
                raise NotInstalledComponent(self.prod_name)
            return func(setuptool)
        wrapper.__name__ = func.__name__
        wrapper.__dict__.update(func.__dict__)
        wrapper.__doc__ = func.__doc__
        wrapper.__module__ = func.__module__
        return wrapper

Then your upgrade scripts should be decorated like this in your “upgrades.py”

...
from utils import IfInstalled
@IfInstalled()
def someUpgradeFunction(setuptool):
 # Stuff as usual...

You’re done…

As we’re supposed to kill the CMF skins layer in the Plone components, when tailoring the old style products to components, we put all static data (images, javascripts, stylesheets) in resources directories, python scripts to views and adpters, controller page templates to plone.app.form or plone.app.z3cform schemes, and so on…

But the new style resource directory doesn’t take care about DTML as it does for page templates. And for some kinds of stuffs like dynamic CSS/Javascript, or making a CSV file, DTML is yet the better suited than ZPT.

Yes you can publish DTML based views/pages/viewlets. This is not obvious but not that much complicated. This small example shows how to add a stylesheet using the standard Plone “base_properties” CSS data.

First the ZCML bunch at …/browser/configure.zcml:

<browser:page
  name="mystytles.css"
  for="Products.CMFPlone.interfaces.IPloneSiteRoot"
  class=".stylesheet.MyStylesheet"
  permission="zope2.Public"
/>

So we’ll have that sthylesheet published at http://<your-site>/mystyles.css.

Now the …/browser/stylesheet.py module:

...
import os
from Globals import DTMLFile

from Products.Five.browser import BrowserView
...
this_dir = os.path.dirname(os.path.abspath(__file__))

templates_dir = os.path.join(this_dir, 'templates')
# Don't add ".dtml" to the file name though the file is "mystyles.css.dtml"
mystylesheet_dtml = DTMLFile('mystyles.css', templates_dir)

...
class MyStylesheet(BrowserView):

    def __call__(self, *args, **kw):
        """This view is published"""

        # Wrap acquisition context to template

        template = mystylesheet_dtml.__of__(self.context)
        # Note that you can provide other named args below you might need in
        # your template
        return template(context=context)

And finally our DTML template at …/browser/templates/mystyles.css.dtml

/* <dtml-with "context.base_properties"> (do not remove this 
<dtml-let portal_url="context.absolute_url()"> (do not remove this 
*/

.silly {
  color: &dtml-discreetColor;;
}
.foo {
  background: &dtml-portal_url;/some-image.gif;
}

/* </dtml-let></dtml-with> (do not remove this   */

You just need to add “mystyles.css” to the Plone CSS registry but that’s another story. That’s all folks.

Note that you could prefer using z3c.zrtresource if you’re bleeding-edge oriented developer.