Back on iw.memberreplace

I blogged some months ago about iw.memberrreplace. In some words, iw.memberreplace provides a tool that clones the security features of an unser to another one (ownership, DC creator, sharings, group membership). No more hassle digging around huge Plone site and hundreds of clicks to do this.

Last week, I read a conversation with John Stahl and Mustapha Benali about the PLIP 185, and realized that this PLIP is almost iw.memberreplace (or the opposite).

So I spend a couple of hours on that component to add the last details on that component. Et voilà, the last release of iw.memberreplace (1.0.0-RC1) fulfills now that PLIP: the original member can now be removed – if defined in a mutable users source.

I swear that in the future, I will read the open PLIPs before creating a new component ;o)

Enjoy…

iw.memberreplace control panel

OpenXml and openxmllib – looking for volunteers

Abstract

openxmllib is a pure Python package built on lxml that parses an ECMA373 office file (read *.docx, *.pptx, *.xlsx recorded from MS Office 2007). It can actually extract – very fast – the indexable words from an MS Office 2007 document. It may be used from any Python app, even from others (Java, PHP, C++ …)

Products.OpenXml is a component for Plone 2.5 -> 3.2 that registers the MIME types and icons for the known extensions for these office files and provides transform rules to indexable text, such MS Office 2007 documents in ATFile or any content type with a searchable FileField are indexed.

Both are avalaible at the cheeseshop.

Future directions

Yeah, those babies make satisfying job at the moment, I have not enough room to copy the testimonies of satisfied users. But users want more…

Have real plain text

openxmllib actually provides all words from a document. This is very fast because some XPath expressions make the job. But when the result is appropriate for indexing, it is not human friendly. The words are returned in any order and there’s no way to understand what’s written.

Have HTML preview

Yes, previews of office documents in a Plone site are great. ARFilePreview shows some of what I want to do: having a nice HTML document, visually as close as possible from the printed document.

Volunteers?

Unfortunately, I have not much time to spend for this, and the ones who asked these features didn’t want to participate or fund my work. You are interrested for adding these features, please let me know. Skills in XSLT and in the ECMA373 standard are required for this.

Many thanks by avance.

Safely replace a Plone member

Hi,

This component replies to a well known use cases and request from our customers, especially from intranet managers.

How do we do on our Plone intranet when Mr Foo dismisses our company and Ms Bar is entitled to do his job?

Ms Bruni just married and she’s now Mrs Sarkozy. Her login is now c.sarkozy and she lost the permissions she had on some contents she used to work on. How shall I handle this? Should I browse in the site finding all her stuffs and change the sharings, property (…) accordingly?

Some weeks ago, this was in a bunch of code I included in private site products. Some days ago, I released it as a public component.

Just install iw.memberreplace, open its control panel, select the desired options and let it do all the job – that otherwise would require hours of digging – for you.

Enjoy, and as always, feedback and contributions are welcome.

Safer GenericSetup upgrade steps

Upgrading Plone extensions using GenericSetup upgrade steps. A great idea. Easy to develop, easy to use and document. A unified upgrade interface for every Plone extension.

Anyway, there’s actually an annoying issue on upgrade steps: upgrade steps for component “foo” are exposed in sites where the “foo” component is not installed.

Wow, managers on instances where various different Plone sites are installed have to be careful on this. Running an upgrade step of a component that is not installed in a site may be harmful. Even if upgrade scripts are supposed to be defensively programmed.

Having several Plone extensions, I made a decorator based generic safety belt. Of course this does not prevent GS showing the upgrade steps but executing such upgrade steps raises an explicit error message and does not execute the upgrade step.

Just add these lines to your “utils.py”:

from zope.component import getUtility
from Products.CMFCore.interfaces import ISiteRoot
from config import PROJECTNAME

class NotInstalledComponent(LookupError):
    def __init__(self, cpt_name):
        self.cpt_name = cpt_name
        return

    def __str__(self):
        msg = ("Component '%s' is not installed in this site."
               " You can't run its upgrade steps."
               % self.cpt_name)
        return msg

class IfInstalled(object):
    """The decorator"""
    def __init__(self, prod_name=PROJECTNAME):
        """@param prod_name: as shown in quick installer"""
        self.prod_name = prod_name

    def __call__(self, func):
        """@param func: the decorated function"""
        def wrapper(setuptool):
            portal = getUtility(ISiteRoot)
            qi = portal.portal_quickinstaller
            installed_ids = [p['id'] for p in qi.listInstalledProducts()]
            if self.prod_name not in installed_ids:
                raise NotInstalledComponent(self.prod_name)
            return func(setuptool)
        wrapper.__name__ = func.__name__
        wrapper.__dict__.update(func.__dict__)
        wrapper.__doc__ = func.__doc__
        wrapper.__module__ = func.__module__
        return wrapper

Then your upgrade scripts should be decorated like this in your “upgrades.py”

...
from utils import IfInstalled
@IfInstalled()
def someUpgradeFunction(setuptool):
 # Stuff as usual...

You’re done…

Have your views rendered with DTML

Have your views rendered with DTML

As we’re supposed to kill the CMF skins layer in the Plone components, when tailoring the old style products to components, we put all static data (images, javascripts, stylesheets) in resources directories, python scripts to views and adpters, controller page templates to plone.app.form or plone.app.z3cform schemes, and so on…

But the new style resource directory doesn’t take care about DTML as it does for page templates. And for some kinds of stuffs like dynamic CSS/Javascript, or making a CSV file, DTML is yet the better suited than ZPT.

Yes you can publish DTML based views/pages/viewlets. This is not obvious but not that much complicated. This small example shows how to add a stylesheet using the standard Plone “base_properties” CSS data.

First the ZCML bunch at …/browser/configure.zcml:

<browser:page
  name="mystytles.css"
  for="Products.CMFPlone.interfaces.IPloneSiteRoot"
  class=".stylesheet.MyStylesheet"
  permission="zope2.Public"
/>

So we’ll have that sthylesheet published at http://<your-site>/mystyles.css.

Now the …/browser/stylesheet.py module:

...
import os
from Globals import DTMLFile

from Products.Five.browser import BrowserView
...
this_dir = os.path.dirname(os.path.abspath(__file__))

templates_dir = os.path.join(this_dir, 'templates')
# Don't add ".dtml" to the file name though the file is "mystyles.css.dtml"
mystylesheet_dtml = DTMLFile('mystyles.css', templates_dir)

...
class MyStylesheet(BrowserView):

    def __call__(self, *args, **kw):
        """This view is published"""

        # Wrap acquisition context to template

        template = mystylesheet_dtml.__of__(self.context)
        # Note that you can provide other named args below you might need in
        # your template
        return template(context=context)

And finally our DTML template at …/browser/templates/mystyles.css.dtml

/* <dtml-with "context.base_properties"> (do not remove this 
<dtml-let portal_url="context.absolute_url()"> (do not remove this 
*/

.silly {
  color: &dtml-discreetColor;;
}
.foo {
  background: &dtml-portal_url;/some-image.gif;
}

/* </dtml-let></dtml-with> (do not remove this   */

You just need to add “mystyles.css” to the Plone CSS registry but that’s another story. That’s all folks.

Note that you could prefer using z3c.zrtresource if you’re bleeding-edge oriented developer.

Plone needs a webmaster role

Yes, we have the best open-source professional CMS of the galaxy – and beyond – out here. With the age and of course the help of the framework developers and lots of others, Plone gained lots of functionnalities. A huge set of options to tweak a Plone site such it is your Plone site are now available through the Plone content panel.

As Plone professional, I have to train site managers for customers. And most of the new site managers I have to train are not IT professional. Most are marketing or communication executives. And when most of them have good functional skills and know what they want to do with their Plone site and how to provide information and services to visitors, it is somehow dificult to teach them about HTML/CSS filtering, Javascript adds, SMTP connection, Plone extension (un, re)installation, ZMI access and other potentially harmful control panels.

As a consequence, I sometimes need to hide potentially harmful control panels to such non technical managers on these sites.

I’m sure lots of you, dear Plone service providers who read these lines, are often facing such situations. That’s why I think that there’s a place for a new OTB role in Plone. I’ll call it “webmaster” at the moment.

Such user could do most things a Manager is enabled to do through the Plone except accessing those harmful control panels.

To make short, a webmaster would have all Manager permissions granted except “View management screens”. Of course, from there it’s fairly easy to hide the links to ZMI, Extensions management and others from the control panels, but it does not solve lots of other things that requires some tweaks in the control panels.

Any though ?

Using decorated functions in ZCML

As python programmer, I love the elegant way to tweak function behaviours with decorators.

In order not to be executed when another profile is invoked, most setup handler functions must start with:

def setupSomeStuff(context):
    if context.readDataFile("mysite.txt") is None:
        return
    # Let's do the job baby
    ...
    return

As the site I’m working on needs a lot of handlers that are used in various conditions like this one. But there are lots of such…

<gs:importStep
   name="Products.MySite.setupSomeStuff"
   title="Stuff that site"
   description="D'ya know what stuffing means..."
   handler="Products.MySite.setuphandlers.setupSomeStuff">
   <depends name="Products.MySite.importSiteStructure" />
</gs:importStep>

…and of course as many setup handler functions.

In order to shorten a little bit the code, I made a simple decorator that executes the setup handler only in the context of the extension profile. Should be fine.

def thisProfileOnly(func):
    """Decorator that prevents the setup func to be used on other GS profiles.
    Usage:
    @thisProfileOnly
    def someFunc(context): ...
    """

    def wrapper(context):
        if context.readDataFile('mysite.txt') is None:
            logger.info("*NOT* Executing setuphandler function %s", func.__name__)
            return
        else:
            logger.info("Executing setuphandler function %s", func.__name__)
            return func(context)

@thisProfileOnly
def setupSomeStuff(context):
    # Let's do the job baby
    ...
    return

Let’s go baby… But wait… It does not work! Importing the profile doe not run the setup handlers.

Having a deeper look into all this, I found that the relevant GenericSetup registry does hold the decorated functions but the unbound wrapper itself. Bad news.

Is it a bug or a feature? Anyway, digging in the Python gurus blogs, I found how to work this around, augmenting the wrapper such it gets most of the decorated function signature. Follow the lines in red in the fixed decorator.

def thisProfileOnly(func):
    """Decorator that prevents the setup func to be used on other GS profiles.
    Usage:
    @thisProfileOnly
    def someFunc(context): ...
    """

    def wrapper(context):
        if context.readDataFile('modulo.txt') is None:
            logger.info("*NOT* Executing setuphandler function %s", func.__name__)
            return
        else:
            logger.info("Executing setuphandler function %s", func.__name__)
            return func(context)
    wrapper.__name__ = func.__name__
    wrapper.__dict__.update(func.__dict__)
    wrapper.__doc__ = func.__doc__
    wrapper.__module__ = func.__module__
    return wrapper

Yes, with such decorators, you may use decorated functions in your ZCML.

Note that this should be useless from Python 2.5, but I didn’t test in such situation (too lazy to try to run the Zope/Plone machinery with Python 2.5)

Other sources about advanced Python decorators:

• Charming Python: decorators make magic easy
• The decorator module

Eclipse/pydev and buildout

Pydev has a nice code completion feature when editing Python code. You start typing a name, and Eclipse/Pydev offers context specific completions with call tips that saves a lot of time when programming for a hugh framework like Plone.

This requires to stuff somehow the PYTHONPATH for your project. This is fairly easy with a Plone bundle install tarball, where you just need to provide the paths to the Products directory, the lib/python of your instance and the global Zope $SOFTWARE_HOME.

With an instance created with buildout, things are not as straightforward. Such an instance is made of tons of eggs that are not in the standard “site-packages”, in addition, Zope 2 style products may be located in various places. Adding all this manually in the Pydev project PYTHONPATH is a real nightmare.

Hopefully we have two recipe companions who can help us a lot configuring Eclipse + Pydev on your buildout instance:

• iw.recipe.cmd that builds some symlinks tree, with the help of some Python lines, for…
• pb.recipe.pydev that makes most of the job

From now we assume you already have a ready Plone 3.x instance.

With Eclipse, create a new Pydev project, say at the root of your buildout. Do not add anything to the Pydev – PYTHONPATH of that project. Now quit Eclipse. Your instance root should have a “.pydevproject” file.

Open your buildout.cfg and add that stuff:

[buildout]
...
zope-directory = /path/to/your/Zope-2.10.5
...
parts =
    ...
    make_pydev_init_files
    pydev
    ...
# Do not add parts that add eggs or products after the above parts
...
[make_pydev_init_files]
# we need this (a Products directory with symlinks to all plone products)
# to have completion of code in the Products namespace
recipe = iw.recipe.cmd:py
on_install = true
cmds =
      >>> import os
      >>> dirs = """${instance:products}""".split("\n")
      >>> prodlinks = os.path.join("""${buildout:directory}""".strip() , 'pydevlinks')
      >>> Products = os.path.join(prodlinks,'Products')
      >>> if not os.path.isdir(prodlinks): os.mkdir(prodlinks)
      >>> if not os.path.isdir(Products): os.mkdir(Products)
      >>> file(os.path.join(Products , '__init__.py'),'w').write('#')
      >>> for dir in dirs:
      >>>     if not dir: continue
      >>>     for product in [os.path.join(dir,a) for a in os.listdir(dir) if os.path.isdir(os.path.join(dir,a))]:
      >>>         linkname = os.path.join(Products, os.path.basename(product))
      >>>         if not os.path.islink(linkname): os.symlink(product,linkname)

[pydev]
recipe = pb.recipes.pydev
eggs = ${instance:eggs}

# See [make_pydev_init_files] below
extra_paths =
    ${buildout:directory}/pydevlinks
    ${buildout:zope-directory}/lib/python
pydevproject_path = ${buildout:directory}/.pydevproject

Re-run your buildout. That’s OK? Now open Eclipse and view your project properties. Et voilà…

As you can see we use “os.symlink” to fake a products hierarchy. Windows (NTFS) users should install a symlink tool like “junction” and tweak the script of the “make_pydev_init_files” part.

Caveat: do not change manually anything that’s in the /prodlinks directory of your buildout unless…

Many thanks to Tim Knapp and Sylvio for the hints in the products developers mailing list. Now this is in a blog.

A simple benchmark getting the site object

Hi,

Ho many of you complain about Plone performances? The latest Plone release are said to be faster than the former ones.

Her is a sample benchmark that compares the three ways to get the portal (Plone) object.

• The first one, I found it when wandering in the Zope 3 subset that ships with Zope 2.10.
• The second one is the recommanded practice since Plone 3, getting the "plone_portal_state" multi adapter.
• And the third one is the (somehow deprecated?) CMF style to get the portal object.

Here is the code as external method, just copy it in an appropriate place, and add the external method where you want in your Plone site.

from zope.app.component.hooks import getSite
from zope.component import getMultiAdapter
from Products.CMFCore.utils import getToolByName
from time import time as time_now
from StringIO import StringIO

COUNT = 1000

def testGetPlone(self):
    """Small benchmark"""

    out = StringIO()
    request = self.REQUEST
    t0 = time_now()

    for x in xrange(COUNT):
        plone = getSite()
    t1 = time_now()

    for x in xrange(COUNT):
        plone = getMultiAdapter(
            (self, request),
            name="plone_portal_state").portal()
    t2 = time_now()

    for x in xrange(COUNT):
        plone = getToolByName(self, 'portal_url').getPortalObject()
    t3 = time_now()

    print >> out, COUNT, "times getSite Zope 3 function:", t1 - t0, "seconds"
    print >> out, COUNT, "times plone_portal_state multi adapter:", t2 - t1, "seconds"
    print >> out, COUNT, "times old style getToolByName:", t3 - t2, "seconds"
    return out.getvalue()

Yeah and here are the results on my MacBook:

1000 times getSite Zope 3 function: 0.000494956970215 seconds
1000 times plone_portal_state multi adapter: 0.0551428794861 seconds
1000 times old style getToolByName: 0.0382490158081 seconds

What conclusions can we expose here?

1. The old style and deprecated "getToolByName" is faster than the new style way that uses the "plone_portal_state" multi adapter that is supposed to cache the result! I ran the test several times to confirm this is not a bad trip or beer abuse effect. No, this is true, just continue using the old CMF style way to get a tool or the portal.
2. The (poorly used in Plone) Zope 3 "getSite" does not need a context or request object to get the site object. Wow! Interresting in lots of design situations. And in addition, it is 100 times faster than the "official" APIs.
3. Zope 3 offers new services that deserve to be used more and more. Plone suffers of serious performances issues, more particularly for authenticated users, sorry to say this but it’s true. We shouldn’t need to use ZEO or Squid when 10 authenticated users manage content in a Plone site.

Okay, that benchmark does not match real applications. Who searches 1000 times the portal object in a Plone based app?

But there are certainly dozen of other tricky things in Plone, Archetypes or CMF that should be re designed to address the Plone performance issues.

Have you guys found other hints to have a faster Plone without the need to cache somewhere?

Archetypes sucks…

Eh, not my personal opinion but…

That’s what we can read in some posts and blogs. So, why does AT sucks according to these opinions?

• It’s old style python (read mostly plain 2.1 python) and doesn’t use the latest features that came up to Python 2.4 (decorators, misc new style classes features…).
• There’s no out of the box support for repetead fields or groups of fields (read “we can’t add n files or images”)
• AT is uselessly noisy, and it’s code is now as messy as obese.
• AT is slow, more specifically for authors, and is a major cause of Plone lazyness.
• AT APIs are sometimes complicated. We use “content.getField(’foo’).set(content, value)” when Python 2.4 could let us just type “content.fields.foo = value” or better “content.foo = value“.
• AT is not Zope3 “ish”, or does the minimal stuff to run within Plone 3. And yes, it uses always the old style Zope 2 interfaces, CMF skins layer, nested complex ZPT macros where viewlets could do better job.
• The set of metadata that ship with AT is not really extensible or replaceable though it’s named ExtensibleMetadata.

So… Let’s KILL Archetypes! Yes, and what else?

Okay there’s new kids on the block. There’s lot of buzz around good newcomers in the field of AT, like the new plone base contents (see the plone.app.content egg) you can use if you don’t need advanced features.

Or you could wait for the promising new framework Devilstick that will let us define the data model with XML.

Just don’t forget that if Plone ships with a nice collection of content types, has so many third party content types and rich services, if Plone is often referred as the best open-source CMS, if you can sell Plone based competitive solutions to your customers you just need to say “Thank you so much Archetypes developers“.

In addition, most (all?) of above listed issues of today’s Archetypes can be fixed in the future without breaking support for actual content types.

Due to so many skilled developers and rich solutions Archetypes is here to stay for years and years. Yeah!