be Groovie


Ben Bangert is a San Francisco Bay Area programmer, best known for his open-source work creating and contributing to Python libraries such as Pylons, Beaker, and Routes.
He currently works at Mozilla.


Advanced Caching with Django and Beaker

After seeing more than a few blog posts and packages attempt to provide more advanced caching capability for Django, it occurs to me I should actually just blog how to use Beaker in Django, rather than just keep mumbling about how “Beaker already does that”. So, if you’ve needed caching in Django that goes beyond using just one backend at a time, or maybe can actually cope with the Dog-Pile Effect, this is the blog entry for you (Until I flesh it out further into actual docs on the Beaker site).

Install Beaker

This is simple enough, if you have easy_install available, just:

easy_install -U Beaker

Or if you prefer to download tar files, grab the Beaker 1.4 tar.gz file

Configuring the Cache

Setting up Beaker’s cache for your Django app is pretty easy. Since only a single cache instance is needed for an app, we’ll set it up as a module global.

Create a file in your Django project with the following contents:

from beaker.cache import CacheManager
from beaker.util import parse_cache_config_options

cache_opts = {
    'cache.type': 'file',
    'cache.data_dir': 'cache/data',
    'cache.lock_dir': 'cache/lock'

cache = CacheManager(**parse_cache_config_options(cache_opts))

There’s a lot more options available, such as memcached, configuring multiple cache backends at once, etc. Now that you know how to provide the configuration options, further customization can be done as needed using the Beaker configuration docs. (Note the very handy cache region configurations which make it easy to toggle cache backend configurations on the fly!)

Using the Cache

Beaker provides a conveinent decorator API to make it easy to cache the results of functions. In this example we’ll just sleep and make a string including the time, add this to your

import time
from datetime import datetime

from django.http import HttpResponse

from YOURPROJECT.beakercache import cache

def hello(request):
    def fetch_data():
        return 'Hello world, its %s' %
    results = fetch_data()
    return HttpResponse(results)

In this case, the cached data is in a nested function with the decorator. It could of course be in the module elsewhere as well.

Hook the view function up in your, and hit the view. The first time it will wait a few seconds, then it will return the old time until the cache expires (10 seconds in this case).

The cached function can also accept positional (non-keyword) arguments, which will be used to key the cache. That is, different argument values will result in different copies of the cache that require those arguments to match.

That’s it, it’s really quite easy to use.

Update: It occurs to me this post does say ‘advanced’, and that example wasn’t very advanced, so here’s something a bit more interesting. Let’s configure cache regions to make it easy to toggle how long and where something is cached. Cache regions allow you to arbitrarily configure batches of settings, a ‘region’. Later you can then indicate you want to use that region, and it uses the settings you configured. This also make its easy to setup cache policies and change them in a single location.

In this case, we’ll have ‘long_term’ and ‘short_term’ cache settings, though you can of course come up with as many regions as desired, with the name of your choice. We’ll have the long_term settings use the filesystem, since we want to retain the results for quite awhile and not have them pushed out like memcached does. The short_term settings will use memcached, and be cached for only 2 minutes, long enough to help out on those random slashdog/digg hits.

In the file:

from beaker.cache import CacheManager
from beaker.util import parse_cache_config_options

cache_opts = {
    'cache.type': 'file',
    'cache.data_dir': 'cache/data',
    'cache.lock_dir': 'cache/lock',
    'cache.regions': 'short_term, long_term',
    'cache.short_term.type': 'ext:memcached',
    'cache.short_term.url': '',
    'cache.short_term.expire': '1200',
    'cache.long_term.type': 'file',
    'cache.long_term.expire': '86400',

cache = CacheManager(**parse_cache_config_options(cache_opts))

Now in our

import time
from datetime import datetime

from django.http import HttpResponse

from testdjango.beakercache import cache

def hello(request):
    def fetch_data():
        return 'Hello world, its %s' %
    results = fetch_data()
    return HttpResponse(results)

def goodbye(request):
    def fetch_data():
        return 'Bye world, its %s' %
    results = fetch_data()
    return HttpResponse(results)

Beaker 1.4 Released

Beaker 1.4 has now been released, and addresses several fairly important bugs. First, the full changelog:

  • Fix bug with hmac on Python 2.4. Patch from toshio, closes ticket #2133 from the TurboGears2 Trac.
  • Fix bug with occasional ValueError from FileNamespaceManager.do_open. Fixes #10.
  • Fixed bug with session files being saved despite being new and not saved.
  • Fixed bug with CacheMiddleware overwriting configuration with default arguments despite prior setting.
  • Fixed bug with SyntaxError not being caught properly in entry point discovery.
  • Changed to using BlobProperty for Google Datastore.
  • Added domain/path properties to the session. This allows one to dynamically set the cookie’s domain and/or path on the fly, which will then be set on the cookie for the session.
  • Added support for cookie-based sessions in Jython via the JCE (Java Cryptography Extensions). Patch from Alex Grönholm.
  • Update Beaker database extensions to work with SQLAlchemy 0.6 PostgreSQL, and Jython.

Note that the beaker database extension now works on Jython, and the cookies for sessions can be set dynamically during a request (for sites that operate across multiple domains/sub-domains).

Most importantly though, a bug in the import of the Google back-end has been fixed, which caused installation failures on Beaker 1.3.x.

Docs can be found on the Beaker site.

To upgrade your Beaker with easy_install:

easy_install -U Beaker

This release is also notable as the majority of the fixes were contributed by several web framework communities. Thanks for the patches!

Comments and Web Services

I’m trying out Disqus for comments on the blog, for one main reason, I just didn’t feel like implementing comments myself. I am somewhat wary of services that seem integral to a blog, like where the comments are, which is why I’ve been leery of using services like this for some time.

What happens if Disqus runs out of VC and goes belly-up? I’ve seen this with a few other web services, and no one using it is very happy when it occurs. On the other hand, I did find some irony in the fact that one of the features Disqus pitches users is, “Don’t lose your comments if the blog disappears”.

On the other hand, I really appreciate the capabilities a central service-based comment system brings. I’ve inadvertently used it on other blogs, and was very pleasantly surprised to get updates and actually be able to keep up on what was happening on the blogs I commented on.

Are other people worried about using services by new companies or am I just overly paranoid?

I’d almost feel better about it if I could pay five bucks a year or something, as I’d at least have some better reassurance that the company is actually making money like I do with Flickr. Or maybe it’d be nice if companies that are profitable to say so, though it occurs to me that even that isn’t a guarantee, as someone could come along and buy them up, then decide to terminate their functionality (like that one site that people used and were pissed about when Six Apart bought them and shut it down, but for some reason I can’t remember the name of it at all).

Blog upgraded! Now running on Pylons + MongoDB

I’ve now updated the blog software powering my blog, which is very long overdue. In the past, this blog was run off Typo, which apparently now hosts their home site off the github (its moved a dozen times in the past 3+ years).

Typo always worked moderately well for me, however, I found it sluggish (Prolly Rails there), and incredibly horrid on ram usage. It was not at all unusual to see it running past 700 megs of RAM after running for just a few weeks, which is a bit annoying as the machine only has 4GB total and is running quite a few things.

After last weeks SF Python Meetup on MongoDB I figured it was time to get a little actual MongoDB usage under my belt. I also inadvertently implemented enough of the MovableType XMLRPC API as I didn’t want my app to be too extensive, just enough for me to post to my blog.

So in the end, I had a small set of requirements for the replacement:

  1. Not be horribly slow
  2. Not take up huge amounts of RAM
  3. Retain all existing URL’s (It really annoys me when people break their old links)
  4. Compatible with my blog software (MovableType / MetaWeblog XMLRPC API)
  5. Not screw up the RSS/Atom feeds and cause Planet Python to show all my posts as if they were new (I’ve seen this happen to a few people on occasion)

I wanted to build it myself, because of course, the world definitely needs more blog apps, and I wanted one that used MongoDB. So for those curious, here’s the source code to the blog app.

It’s rather rough, as its fairly custom built just for my needs, nor do I have any plans to expand it into some general purpose blog engine, with themes, etc. The only other thing pending at the moment is to add the ability to comment again, as I haven’t quite gotten that feature in yet. For those trying it out, the README should help get started, but its very rough (thus the name of the package).

Strings, Unidecode, and Slugs

When copying some functionality I needed from the Rails app (to retain URL compatibility), I noticed two things it did which I thought was handy. To convert a title into slug for the article, it used a fairly sophisticated scheme relying on two other packages.

First, was the use of a Ruby port of a Perl package called Text::Unidecode which is pretty cool, and converts UTF-8 chars into their closest ASCII version. I figured someone must’ve ported it to Python as well, and sure enough, someone did! It wasn’t on the cheeseshop though, which was unbearable for me, so I’ve posted it to the Cheeseshop so others can easy_install it.

Next up, was a Ruby library called stringex, which add’s a few things to Rails, including a string method called ‘to_url’. That method does a variety of transformations to remove all those funny characters from a title, and do a bunch of other neat changes of common characters to human readable versions (source for those conversions).

I ported the key module of stringex to Python, and it resides in my blog app. If someone would like to extract it and make it into its own package, or even better, if I somehow missed the fact that someone else has ported it already, let me know (tweet me @benbangert!).

I’ll be writing up my thoughts on making a small app with MongoDB, and how it differs from my experience working with CouchDB for PylonsHQ in a later post for those curious.

Making a better TextArea

I’ve been working on some Javascript to enhance the TextArea elements on the PylonsHQ Snippets section, and have noticed that… well, TextArea’s suck.

The hack I’ve seen is to use one of the newer features of browsers, the editable or ‘design’ attribute’s for div elements I believe. This lets one build a very snazzy amount of features, such as syntax highlighting, code completion, etc., but I don’t think I needed to go that far.

I only have one main design goal, this TextArea is for the user to enter RestructuredText so it’d be awesome if the TextArea acted in a way that made rst a bit nicer. The obvious two things that came to mind:

  1. Tab key indents 4-spaces
  2. Hitting return on an indented line, will retain the indentation on the next line

I’ve actually gotten some Javascript, hobbled together from various parts of the net, along with an ‘enter’ key handler I wrote myself on bitbucket.

Course, it’d also be nice to have a button or key combo, that will indent/unindent a selection in the TextArea as well.

Anyone else have any Javascript they’ve hobbled together in the past to make TextArea a little nicer for restructured text?

Beaker 1.3 is juicy caching goodness

Beaker 1.3 is out, actually, its been out for awhile and I’m just not getting around to blogging the fact. It’s a shame I’ve been a bit too busy lately to blog this earlier because in addition to some bug fixes it has some nice new features that make it even easier to use in any Python script/application/framework.

First, to air my dirty laundry, the important bug fixes in Beaker 1.3:

  • Fixed bug with (non-cookie-only) sessions not being timed out
  • Fixed bug with cookie-only sessions sending cookies when they weren’t supposed to
  • Fixed bug with non-auto sessions not properly storing their last accessed time

The worst thing with the first two of these is that they were regressions that snuck in despite unit tests that exercised the code fairly decently. They’re fixed now along with more comprehensive tests to help prevent such regressions occurring again.

Beaker has always had session’s, and caching, but except for Pylons I’ve yet to see anyone actually use Beaker’s caching utility. I’ve seen the SessionMiddleware used in other WSGI based frameworks, but not the caching, which is kind of a shame since it:

  • Supports various backends: database, file-based, dbm file-based, memcached
  • Has locking code to ensure a single-writer, multiple reader model (This avoids the dreaded dog-pile effect that caching systems such as the one in Django experience!)

For clients that hit the cached function while its already being regenerated, Beaker serves the old copy until the new content is ready. This avoids the dog-pile effect, and keeps the site snappy for as many users as possible. Since the lock used is disk-based though, this does mean you only avoid the effect per machine (unless you’re locking against NFS or a SAN), so if you have 20 machines in a cluster, the worst the dog-pile effect can get is that you’ll have 20 new copies generated and stored.

Now, in Beaker 1.3, to try and encourage its use a bit more, I’ve added a few decorators to make it easier to cache function results. Also with Mike Bayer’s suggestion, there is now cache regions to make it easier to define various caching policy short-cuts.

Cache Regions

Cache regions are just pre-defined sets of cache instructions to make it easier to use with your code. For example many people have a few common types of cache parameters they want to use:

  • Long-term, likely to a database back-end (if used in a cluster)
  • Short-term, not cached as long, perhaps to memcached

To set these up, just tell Beaker that about the regions you’re going to define, and give them the normal Beaker cache parameters for each region. For example, in this Pylons app, I define 2 cache regions in the INI:

beaker.cache.regions = short_term, long_term
beaker.cache.short_term.type = ext:memcached
beaker.cache.short_term.url =
beaker.cache.short_term.expire = 3600

beaker.cache.long_term.type = file
beaker.cache.long_term.expire = 86400

Note: For those wondering about multiple memcached servers, just put them in as the url with a semi-colon separating them.

If you want to use the caching outside of Pylons without middleware (ie, as a plain library), that’s a bit easier now as well:

from beaker.cache import CacheManager
from beaker.util import parse_cache_config_options

cache_opts = {'cache.data_dir': './cache',
              'cache.type': 'file',
              'cache.regions': 'short_term', 'long_term',
              'cache.short_term.type': 'ext:memcached',
              'cache.short_term.url': '',
              'cache.short_term.expire': '3600',
              'cache.long_term.type': 'file',
              'cache.long_term.expire': '86400',

cache = CachManager(**parse_cache_config_options(cache_opts))

And your cache instance is now ready to use. Note that using this cache object is thread-safe already, so you just need to keep one around in your framework/app (Can someone using Django explain where you’d keep a reference to this object around so that you could get to it in a Django view?).

New Cache Decorators

To make it easier to use caching in your app, Beaker now includes decorators for use with the cache object. Given the above caching setup, lets assume you want to cache the output of an expensive operation:

# Get that cache object from wherever you put it, maybe its in environ or request?
# In Pylons, this will be: from pylons import cache
from wherever import cache

def regular_function():
    # Do some boring stuff

    # Cache something
    @cache.region('short_term', 'mysearch')
    def expensive_search(phrase):
        # Do lookup with the phrase variable
        return something
    return expensive_search('frogs')

The second argument to the region decorator, ‘mysearch’. That isn’t required unless you have two function’s of the same name in the same module, since Beaker records the namespace of the cache using the function name + module + extra args. For those wondering what a Beaker namespace is, its a single cache ‘block’. That is, lets say you wanted to cache 4 versions of the same thing, but change them differently depending on the input parameter. Beaker considers the thing to be a namespace, and the things that change the thing being cached are the cache keys.

Only un-named arguments are allowed on the function being cached. These act as the cache keys so that if the arguments change, a new copy is cached to those arguments. This way you can have multiple versions of the function output cached depending on the argument it was called with.

If you want to use arbitrary cache parameters, use the other decorator:

# Get that cache object from wherever you put it, maybe its in environ or request?
# In Pylons, this will be: from pylons import cache
from wherever import cache

def regular_function():
    # Do some boring stuff

    # Cache something
    @cache.cache('mysearch', type='file', expire=3600)
    def expensive_search(phrase):
        # Do lookup with the phrase variable
        return something
    return expensive_search('frogs')

This allows you to toggle the cache options per use as desired.

If there’s anything else I can do to make it easier to use Beaker in your application, be sure to let me know (Yes, I know more docs would help, this blog post was a first attempt to help out on that front, more docs on the way!).

Pylons 0.9.7 Released

I’m pleased to announce after a rather lengthy release candidate period, that Pylons 0.9.7 is finally out. Pylons 0.9.7 brings a good amount of changes to Pylons from 0.9.6 while still retaining a fairly hefty amount of backwards compatibility to ensure a mostly painless upgrade.

Some helpful documentation on the new release:

Major changes in 0.9.7:

  • Switched to using WebOb for the request/response object
  • Various performance improvements to object initialization
  • Beaker and Routes updates
  • Middleware improvements, and optimizations

This is a huge step forward for Pylons, and I’d like to thank all of the contributers who have helped make Pylons what it is today. We’ve knocked off more bugs for this release than any before, which shows just how far the Pylons community has come:

  • 0.9.5 tickets: 45
  • 0.9.6 tickets: 64
  • 0.9.7 tickets: 160

And we have finally made a huge dent in the historical “lack of docs” problem that Pylons previously suffered from with the new Sphinx generated docs and a comprehensive Pylons book.

The full changelog which describes the major changes (Look for the bits marked with WARNING that might affect backwards compatibility).

0.9.7 (February 23, 2009)

  • WARNING: A new option is available to determine whether or not an actions

arguments should be automatically attached to ‘c’. To turn off this implicit behavior in config[‘pylons.c_attach_args’] = False This is set to True by default.

  • WARNING: Fixed a minor security hole in the default Pylons error page that

could result in an XSS security hole.

  • WARNING: Fixed a security hole in the default project template to use the

StaticURLParser to ensure arbitrary files can’t be sent.

  • WARNING: Refactored PylonsApp to remove legacy PylonsApp, moved

session/cache and routes middleware into the project template. This will require projects to be updated to include those 3 middleware in the projects

  • Changed to using WebTest instead of paste.fixture for app testing.
  • Added render_mako_def to render def blocks within a mako template.
  • Changes to cache_decorator and cached_template to support Beaker API

changes in version 1.1. 1.0.3 is still supported.

  • Fix HEAD requests causing an Exception as if no content was returned

by the controller. Fixes #507. Thanks mvtellingen, Petr Kobalicek.

  • Fix a crash when returning the result of “etag_cache“ in a controller.

Fixes #508.

  • “response” flag has been removed from pylons.decorators.cache.beaker_cache,

as it sends all headers along unconditionally including cookies; additionally, the flag was taking effect in all cases previously so prior versions of beaker_cache are not secure. In its place, a new option “cache_headers” is provided, which is a tuple of specific header names to be cached. It defaults to (‘content-type’,’content-length’).

  • “invalidate_on_startup” flag added to beaker_cache, which provides a

“starttime” to the cache such that when the application is started or restarted, the cache entry is invalidated.

  • Updating host to use for development binding.
  • Added option to specify the controller name with a controller variable

in the controller’s module. This name will be used for the controller class rather than the default naming scheme.

  • egg_info now restores projects’ paster_plugins.txt,

allowing paster shell to work again after the egg-info directory was lost. fixes #282. Thanks sevkin.

  • The paste_deploy_config.ini_tmpl template is now located at

package/config/deployment.ini_tmpl for new projects.

  • Project’s default test fixtures no longer hardcode test.ini; the ini

file used can now be specified via the nosetests —with-pylons argument (defaults to test.ini in setup.cfg). fixes #400.

  • ``validate now defaults to translating FormEncode error messages via Pylons’ gettext catalog, then falls back to FormEncode’s. fixes #296. Thanks Max Ischenko. * Fixed SQLAlchemy logging not working in paster shell. Fixes #363. Thanks Christoph Haas. * Added optionally engine initialization, to prevent Buffet from loading if there’s no ‘buffet.template_engines’ in the config. * Updated minimal template to work with Tempita and other new templating changes. * Fixed websetup to parse location config file properly when the section isn’t ‘main’. Fixes #399. * Added default Mako filter of escape for all template rendering. * Fixed template for Session.remove inclusion when using SA. Fixed render_genshi to properly use fragment/format options. Thanks Antonin Enfrun. * Remove template engine from load_environment call. * Removing template controller from projects. Fixes #383. * Added signed_cookie method to WebOb Request/Response sub-classes. * Updated project template to setup appropriate template loader and controller template to doc how to import render. * Added documentation for render functions in pylons.templating. * Adding specific render functions that don’t require Buffet. * Added forward controller.util function for forwarding the request to WSGI apps. Fixes #355. * Added default input encoding for Mako to utf-8. Suggested in #348. * Fixed paster controller to raise an error if the controller for it already exists. Fixes #279. * Added to template dir in project template if the template engine is genshi or kid. Fixes #353. * Fixed jsonify to use application/json as its the proper mime-type and now used all over the net. * Fixed minimal template not replacing variables properly. Fixes #377. * Fixed ``validate decorator to no longer catch exceptions should they be

raised in the action that is supposed to display a form. Fixes #374.

  • Fixed paster shell command to no longer search for egg_info dir. Allows

usage of paster shell with installed packages. Suggested by Gavin Carothers.

  • Added mimetype function and MIMETypes class for registering mimetypes.
  • WARNING: Usage of pylons.Response is now deprecated. Please use

pylons.response instead.

  • Removed use of WSGIRequest/WSGIResponse and replaced with WebOb subclasses

that implement methods to make it backwards compatible with the Paste wsgiwrappers.

  • Fixed missing import in template controller.
  • Deprecated function uses string substitution to avoid Nonetype error when

Python optimization is on. Fixes #334.

  • E-tag cache no longer returns Content-Type in the headers. Fixes #323.
  • XMLRPCController now properly includes the Content-Length of the response.

Fixes #310, thanks Nicholas.

  • Added SQLAlchemy option to template, which adds SQLAlchemy setup to the

project template.

  • Switched project templating to use Tempita.
  • Updated abort/redirect_to to use appropriate Response object when WebOb is


  • Updated so that 404’s properly return as Response objects when WebOb is in

use instead of WSGIResponse.

  • Added beaker_cache option to avoid caching/restoring global Response values

that were present during the first cache operation.

  • Adding StatusCodeRedirect to handle internal redirects based on the status

code returned by the app. This replaces the use of ErrorDocuments in projects.

  • Refactored error exceptions to use WebError.
  • WSGIController now uses the environ references to response, request, and

the c object for higher performance.

  • Added optional use of WebOb instead of paste.wsgiwrapper objects.
  • Fixed bug with beaker_cache defaulting to dbm rather than the beaker

cache app-wide default.

  • The —with-pylons nose plugin no longer requires a project to have been

registered with setuptools to work.

  • The config object is now included in the template namespace.
  • StaticJavascripts now accepts keyword arguments for StaticURLParser.

Suggested by Marcin Kasperski.

  • Fix pylons.database.AutoConnectHub’s doInTransaction not automatically

connecting when necessary. Fixes #327.

New PylonsHQ Site Launches

The new PylonsHQ site has now launched!

The new site is running on the latest Pylons 0.9.7 code-base backed by the CouchDB database. New features that have been added:

Unfortunately, we were unable to integrate the Wiki’s auth, so that will still require a separate login for now.

Comments are through-out the site, to ensure that feedback isn’t missed and of course there’s many more features planned that are coming soon. The site isn’t quite 100% complete, as a few links here and there are likely broken yet (like the tutorial links on the front page). I’ll be putting out frequent updates to remedy this and any other little bits that need more polish.


Public launch of Stanford Intellectual Property Litigation Clearinghouse

At long last, we’ve finally launched the Stanford IPLC website that I’ve been working on for the past year. It’s quite nice to finally have something out there that I can show people, though I know its definitely more of a niche area of interest, as not everyone is probably as interested in intellectual property litigation as I am. :)

This site is running Pylons of course, with various other technologies I’m unable to disclose powering the back-end.

Note that signing up requires a valid e-mail address as e-mail confirmations are sent out to them. For those that are keen to keep up on what’s going on with patent litigation, hopefully our website can help out.

WSGI/Pylons Talk in San Francisco tonight (Dec 3rd)

I’ll be giving a talk tonight about WSGI, making apps using it, and Pylons tonight in San Francisco. If you live in the Bay Area and have been wanting to learn more about some of the packages utilizing WSGI, as well as Pylons, RSVP soon as they’d like to get some numbers for food.

Here’s the meetup event page for the talk. This is one of the longer talks I’ve given, so I should actually have enough time to cover the topics (WSGI, WSGI Middleware, making low-level WSGI apps with WebOb, making small WSGI stacks, and Pylons) in good detail, as well as some demonstrations.