Blog upgraded! Now running on Pylons + MongoDB

I’ve now updated the blog software powering my blog, which is very long overdue. In the past, this blog was run off Typo, which apparently now hosts their home site off the github (its moved a dozen times in the past 3+ years).

Typo always worked moderately well for me, however, I found it sluggish (Prolly Rails there), and incredibly horrid on ram usage. It was not at all unusual to see it running past 700 megs of RAM after running for just a few weeks, which is a bit annoying as the machine only has 4GB total and is running quite a few things.

After last weeks SF Python Meetup on MongoDB I figured it was time to get a little actual MongoDB usage under my belt. I also inadvertently implemented enough of the MovableType XMLRPC API as I didn’t want my app to be too extensive, just enough for me to post to my blog.

So in the end, I had a small set of requirements for the replacement:

  1. Not be horribly slow
  2. Not take up huge amounts of RAM
  3. Retain all existing URL’s (It really annoys me when people break their old links)
  4. Compatible with my blog software (MovableType / MetaWeblog XMLRPC API)
  5. Not screw up the RSS/Atom feeds and cause Planet Python to show all my posts as if they were new (I’ve seen this happen to a few people on occasion)

I wanted to build it myself, because of course, the world definitely needs more blog apps, and I wanted one that used MongoDB. So for those curious, here’s the source code to the blog app.

It’s rather rough, as its fairly custom built just for my needs, nor do I have any plans to expand it into some general purpose blog engine, with themes, etc. The only other thing pending at the moment is to add the ability to comment again, as I haven’t quite gotten that feature in yet. For those trying it out, the README should help get started, but its very rough (thus the name of the package).

Strings, Unidecode, and Slugs

When copying some functionality I needed from the Rails app (to retain URL compatibility), I noticed two things it did which I thought was handy. To convert a title into slug for the article, it used a fairly sophisticated scheme relying on two other packages.

First, was the use of a Ruby port of a Perl package called Text::Unidecode which is pretty cool, and converts UTF-8 chars into their closest ASCII version. I figured someone must’ve ported it to Python as well, and sure enough, someone did! It wasn’t on the cheeseshop though, which was unbearable for me, so I’ve posted it to the Cheeseshop so others can easy_install it.

Next up, was a Ruby library called stringex, which add’s a few things to Rails, including a string method called ‘to_url’. That method does a variety of transformations to remove all those funny characters from a title, and do a bunch of other neat changes of common characters to human readable versions (source for those conversions).

I ported the key module of stringex to Python, and it resides in my blog app. If someone would like to extract it and make it into its own package, or even better, if I somehow missed the fact that someone else has ported it already, let me know (tweet me @benbangert!).

I’ll be writing up my thoughts on making a small app with MongoDB, and how it differs from my experience working with CouchDB for PylonsHQ in a later post for those curious.

Making a better TextArea

I’ve been working on some Javascript to enhance the TextArea elements on the PylonsHQ Snippets section, and have noticed that… well, TextArea’s suck.

The hack I’ve seen is to use one of the newer features of browsers, the editable or ‘design’ attribute’s for div elements I believe. This lets one build a very snazzy amount of features, such as syntax highlighting, code completion, etc., but I don’t think I needed to go that far.

I only have one main design goal, this TextArea is for the user to enter RestructuredText so it’d be awesome if the TextArea acted in a way that made rst a bit nicer. The obvious two things that came to mind:

  1. Tab key indents 4-spaces
  2. Hitting return on an indented line, will retain the indentation on the next line

I’ve actually gotten some Javascript, hobbled together from various parts of the net, along with an ‘enter’ key handler I wrote myself on bitbucket.

Course, it’d also be nice to have a button or key combo, that will indent/unindent a selection in the TextArea as well.

Anyone else have any Javascript they’ve hobbled together in the past to make TextArea a little nicer for restructured text?

Beaker 1.3 is juicy caching goodness

Beaker 1.3 is out, actually, its been out for awhile and I’m just not getting around to blogging the fact. It’s a shame I’ve been a bit too busy lately to blog this earlier because in addition to some bug fixes it has some nice new features that make it even easier to use in any Python script/application/framework.

First, to air my dirty laundry, the important bug fixes in Beaker 1.3:

  • Fixed bug with (non-cookie-only) sessions not being timed out
  • Fixed bug with cookie-only sessions sending cookies when they weren’t supposed to
  • Fixed bug with non-auto sessions not properly storing their last accessed time

The worst thing with the first two of these is that they were regressions that snuck in despite unit tests that exercised the code fairly decently. They’re fixed now along with more comprehensive tests to help prevent such regressions occurring again.

Beaker has always had session’s, and caching, but except for Pylons I’ve yet to see anyone actually use Beaker’s caching utility. I’ve seen the SessionMiddleware used in other WSGI based frameworks, but not the caching, which is kind of a shame since it:

  • Supports various backends: database, file-based, dbm file-based, memcached
  • Has locking code to ensure a single-writer, multiple reader model (This avoids the dreaded dog-pile effect that caching systems such as the one in Django experience!)

For clients that hit the cached function while its already being regenerated, Beaker serves the old copy until the new content is ready. This avoids the dog-pile effect, and keeps the site snappy for as many users as possible. Since the lock used is disk-based though, this does mean you only avoid the effect per machine (unless you’re locking against NFS or a SAN), so if you have 20 machines in a cluster, the worst the dog-pile effect can get is that you’ll have 20 new copies generated and stored.

Now, in Beaker 1.3, to try and encourage its use a bit more, I’ve added a few decorators to make it easier to cache function results. Also with Mike Bayer’s suggestion, there is now cache regions to make it easier to define various caching policy short-cuts.

Cache Regions

Cache regions are just pre-defined sets of cache instructions to make it easier to use with your code. For example many people have a few common types of cache parameters they want to use:

  • Long-term, likely to a database back-end (if used in a cluster)
  • Short-term, not cached as long, perhaps to memcached

To set these up, just tell Beaker that about the regions you’re going to define, and give them the normal Beaker cache parameters for each region. For example, in this Pylons app, I define 2 cache regions in the INI:

beaker.cache.regions = short_term, long_term
beaker.cache.short_term.type = ext:memcached
beaker.cache.short_term.url = 127.0.0.1:11211
beaker.cache.short_term.expire = 3600

beaker.cache.long_term.type = file
beaker.cache.long_term.expire = 86400

Note: For those wondering about multiple memcached servers, just put them in as the url with a semi-colon separating them.

If you want to use the caching outside of Pylons without middleware (ie, as a plain library), that’s a bit easier now as well:

from beaker.cache import CacheManager
from beaker.util import parse_cache_config_options

cache_opts = {'cache.data_dir': './cache',
              'cache.type': 'file',
              'cache.regions': 'short_term', 'long_term',
              'cache.short_term.type': 'ext:memcached',
              'cache.short_term.url': '127.0.0.1:11211',
              'cache.short_term.expire': '3600',
              'cache.long_term.type': 'file',
              'cache.long_term.expire': '86400',
}

cache = CachManager(**parse_cache_config_options(cache_opts))

And your cache instance is now ready to use. Note that using this cache object is thread-safe already, so you just need to keep one around in your framework/app (Can someone using Django explain where you’d keep a reference to this object around so that you could get to it in a Django view?).

New Cache Decorators

To make it easier to use caching in your app, Beaker now includes decorators for use with the cache object. Given the above caching setup, lets assume you want to cache the output of an expensive operation:

# Get that cache object from wherever you put it, maybe its in environ or request?
# In Pylons, this will be: from pylons import cache
from wherever import cache

def regular_function():
    # Do some boring stuff

    # Cache something
    @cache.region('short_term', 'mysearch')
    def expensive_search(phrase):
        # Do lookup with the phrase variable
        return something
    return expensive_search('frogs')

The second argument to the region decorator, ‘mysearch’. That isn’t required unless you have two function’s of the same name in the same module, since Beaker records the namespace of the cache using the function name + module + extra args. For those wondering what a Beaker namespace is, its a single cache ‘block’. That is, lets say you wanted to cache 4 versions of the same thing, but change them differently depending on the input parameter. Beaker considers the thing to be a namespace, and the things that change the thing being cached are the cache keys.

Only un-named arguments are allowed on the function being cached. These act as the cache keys so that if the arguments change, a new copy is cached to those arguments. This way you can have multiple versions of the function output cached depending on the argument it was called with.

If you want to use arbitrary cache parameters, use the other decorator:

# Get that cache object from wherever you put it, maybe its in environ or request?
# In Pylons, this will be: from pylons import cache
from wherever import cache

def regular_function():
    # Do some boring stuff

    # Cache something
    @cache.cache('mysearch', type='file', expire=3600)
    def expensive_search(phrase):
        # Do lookup with the phrase variable
        return something
    return expensive_search('frogs')

This allows you to toggle the cache options per use as desired.

If there’s anything else I can do to make it easier to use Beaker in your application, be sure to let me know (Yes, I know more docs would help, this blog post was a first attempt to help out on that front, more docs on the way!).

Pylons 0.9.7 Released

I’m pleased to announce after a rather lengthy release candidate period, that Pylons 0.9.7 is finally out. Pylons 0.9.7 brings a good amount of changes to Pylons from 0.9.6 while still retaining a fairly hefty amount of backwards compatibility to ensure a mostly painless upgrade.

Some helpful documentation on the new release:

Major changes in 0.9.7:

  • Switched to using WebOb for the request/response object
  • Various performance improvements to object initialization
  • Beaker and Routes updates
  • Middleware improvements, and optimizations

This is a huge step forward for Pylons, and I’d like to thank all of the contributers who have helped make Pylons what it is today. We’ve knocked off more bugs for this release than any before, which shows just how far the Pylons community has come:

  • 0.9.5 tickets: 45
  • 0.9.6 tickets: 64
  • 0.9.7 tickets: 160

And we have finally made a huge dent in the historical “lack of docs” problem that Pylons previously suffered from with the new Sphinx generated docs and a comprehensive Pylons book.

The full changelog which describes the major changes (Look for the bits marked with WARNING that might affect backwards compatibility).

0.9.7 (February 23, 2009)

  • WARNING: A new option is available to determine whether or not an actions

arguments should be automatically attached to ‘c’. To turn off this implicit behavior in environment.py: config[‘pylons.c_attach_args’] = False This is set to True by default.

  • WARNING: Fixed a minor security hole in the default Pylons error page that

could result in an XSS security hole.

  • WARNING: Fixed a security hole in the default project template to use the

StaticURLParser to ensure arbitrary files can’t be sent.

  • WARNING: Refactored PylonsApp to remove legacy PylonsApp, moved

session/cache and routes middleware into the project template. This will require projects to be updated to include those 3 middleware in the projects middleware.py.

  • Changed to using WebTest instead of paste.fixture for app testing.
  • Added render_mako_def to render def blocks within a mako template.
  • Changes to cache_decorator and cached_template to support Beaker API

changes in version 1.1. 1.0.3 is still supported.

  • Fix HEAD requests causing an Exception as if no content was returned

by the controller. Fixes #507. Thanks mvtellingen, Petr Kobalicek.

  • Fix a crash when returning the result of “etag_cache“ in a controller.

Fixes #508.

  • “response” flag has been removed from pylons.decorators.cache.beaker_cache,

as it sends all headers along unconditionally including cookies; additionally, the flag was taking effect in all cases previously so prior versions of beaker_cache are not secure. In its place, a new option “cache_headers” is provided, which is a tuple of specific header names to be cached. It defaults to (‘content-type’,’content-length’).

  • “invalidate_on_startup” flag added to beaker_cache, which provides a

“starttime” to the cache such that when the application is started or restarted, the cache entry is invalidated.

  • Updating host to use 127.0.0.1 for development binding.
  • Added option to specify the controller name with a controller variable

in the controller’s module. This name will be used for the controller class rather than the default naming scheme.

  • setup.py egg_info now restores projects’ paster_plugins.txt,

allowing paster shell to work again after the egg-info directory was lost. fixes #282. Thanks sevkin.

  • The paste_deploy_config.ini_tmpl template is now located at

package/config/deployment.ini_tmpl for new projects.

  • Project’s default test fixtures no longer hardcode test.ini; the ini

file used can now be specified via the nosetests —with-pylons argument (defaults to test.ini in setup.cfg). fixes #400.

  • ``validate now defaults to translating FormEncode error messages via Pylons’ gettext catalog, then falls back to FormEncode’s. fixes #296. Thanks Max Ischenko. * Fixed SQLAlchemy logging not working in paster shell. Fixes #363. Thanks Christoph Haas. * Added optionally engine initialization, to prevent Buffet from loading if there’s no ‘buffet.template_engines’ in the config. * Updated minimal template to work with Tempita and other new templating changes. * Fixed websetup to parse location config file properly when the section isn’t ‘main’. Fixes #399. * Added default Mako filter of escape for all template rendering. * Fixed template for Session.remove inclusion when using SA. Fixed render_genshi to properly use fragment/format options. Thanks Antonin Enfrun. * Remove template engine from load_environment call. * Removing template controller from projects. Fixes #383. * Added signed_cookie method to WebOb Request/Response sub-classes. * Updated project template to setup appropriate template loader and controller template to doc how to import render. * Added documentation for render functions in pylons.templating. * Adding specific render functions that don’t require Buffet. * Added forward controller.util function for forwarding the request to WSGI apps. Fixes #355. * Added default input encoding for Mako to utf-8. Suggested in #348. * Fixed paster controller to raise an error if the controller for it already exists. Fixes #279. * Added __init__.py to template dir in project template if the template engine is genshi or kid. Fixes #353. * Fixed jsonify to use application/json as its the proper mime-type and now used all over the net. * Fixed minimal template not replacing variables properly. Fixes #377. * Fixed ``validate decorator to no longer catch exceptions should they be

raised in the action that is supposed to display a form. Fixes #374.

  • Fixed paster shell command to no longer search for egg_info dir. Allows

usage of paster shell with installed packages. Suggested by Gavin Carothers.

  • Added mimetype function and MIMETypes class for registering mimetypes.
  • WARNING: Usage of pylons.Response is now deprecated. Please use

pylons.response instead.

  • Removed use of WSGIRequest/WSGIResponse and replaced with WebOb subclasses

that implement methods to make it backwards compatible with the Paste wsgiwrappers.

  • Fixed missing import in template controller.
  • Deprecated function uses string substitution to avoid Nonetype error when

Python optimization is on. Fixes #334.

  • E-tag cache no longer returns Content-Type in the headers. Fixes #323.
  • XMLRPCController now properly includes the Content-Length of the response.

Fixes #310, thanks Nicholas.

  • Added SQLAlchemy option to template, which adds SQLAlchemy setup to the

project template.

  • Switched project templating to use Tempita.
  • Updated abort/redirect_to to use appropriate Response object when WebOb is

used.

  • Updated so that 404’s properly return as Response objects when WebOb is in

use instead of WSGIResponse.

  • Added beaker_cache option to avoid caching/restoring global Response values

that were present during the first cache operation.

  • Adding StatusCodeRedirect to handle internal redirects based on the status

code returned by the app. This replaces the use of ErrorDocuments in projects.

  • Refactored error exceptions to use WebError.
  • WSGIController now uses the environ references to response, request, and

the c object for higher performance.

  • Added optional use of WebOb instead of paste.wsgiwrapper objects.
  • Fixed bug with beaker_cache defaulting to dbm rather than the beaker

cache app-wide default.

  • The —with-pylons nose plugin no longer requires a project to have been

registered with setuptools to work.

  • The config object is now included in the template namespace.
  • StaticJavascripts now accepts keyword arguments for StaticURLParser.

Suggested by Marcin Kasperski.

  • Fix pylons.database.AutoConnectHub’s doInTransaction not automatically

connecting when necessary. Fixes #327.

New PylonsHQ Site Launches

The new PylonsHQ site has now launched!

The new site is running on the latest Pylons 0.9.7 code-base backed by the CouchDB database. New features that have been added:

Unfortunately, we were unable to integrate the Wiki’s auth, so that will still require a separate login for now.

Comments are through-out the site, to ensure that feedback isn’t missed and of course there’s many more features planned that are coming soon. The site isn’t quite 100% complete, as a few links here and there are likely broken yet (like the tutorial links on the front page). I’ll be putting out frequent updates to remedy this and any other little bits that need more polish.

Enjoy!

Public launch of Stanford Intellectual Property Litigation Clearinghouse

At long last, we’ve finally launched the Stanford IPLC website that I’ve been working on for the past year. It’s quite nice to finally have something out there that I can show people, though I know its definitely more of a niche area of interest, as not everyone is probably as interested in intellectual property litigation as I am. :)

This site is running Pylons of course, with various other technologies I’m unable to disclose powering the back-end.

Note that signing up requires a valid e-mail address as e-mail confirmations are sent out to them. For those that are keen to keep up on what’s going on with patent litigation, hopefully our website can help out.

WSGI/Pylons Talk in San Francisco tonight (Dec 3rd)

I’ll be giving a talk tonight about WSGI, making apps using it, and Pylons tonight in San Francisco. If you live in the Bay Area and have been wanting to learn more about some of the packages utilizing WSGI, as well as Pylons, RSVP soon as they’d like to get some numbers for food.

Here’s the meetup event page for the talk. This is one of the longer talks I’ve given, so I should actually have enough time to cover the topics (WSGI, WSGI Middleware, making low-level WSGI apps with WebOb, making small WSGI stacks, and Pylons) in good detail, as well as some demonstrations.

Book Meme

Via :

  • Grab the nearest book.
  • Open it to page 56.
  • Find the fifth sentence.
  • Post the text of the sentence in your journal along with these instructions.
  • Don’t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

From The Omnivore’s Deilemma by Michael Pollan:

And then of course there’s the corn itself, which if corn could form an opinion would surely marvel at the absurdity of it all — and at its great good fortune.

I don’t usually do the meme thing, but I happened to have a book on my desk so it took so little effort… only one chapter to go!

Python 2.6 release and Pylons buildbots

Python 2.6 came out yesterday, so I figured I might as well see if Pylons works on it. Pylons already has a set of buildbots that builds Pylons along with some of its dependencies, so it was fairly trivial to add another builder to verify things ran swimmingly on Python 2.6.

Unfortunately, as one can see looking at the build results, things weren’t so great. It appears that nose had a Python 2.6 incompatibility which is used to run all the various Pylons tests, meaning that they all failed so far mainly because the testing tool was Python 2.6 incompatible.

Making Buildbot nicer

While I wait for the new nose to be released, I did at least discover a little bug in my new webapp that provides a nicer view of the buildbot result set. I’ve been fairly displeased with the lack of conciseness of buildbot’s waterfall display for awhile, and noticed that if only buildbot had a few more xmlrpc methods then it’d be trivial to build my own more kind interface.

I should note that the waterfall display isn’t totally horrible, the Django folks spiced their builders up with some CSS work…. which reminds me, it isn’t looking very good for those running on trunk at the moment. ;)

So after making my own little buildbot fork to add some additional custom xmlrpc methods to, I’ve come up with my own buildbot status viewer. I’m sure a more talented designer could spice it up even more, but it gives me the pertinent data I’m interested in without all the boring “builder connected, builder took a vacation” messages that cloud up the waterfall. Also, rather than displaying the cryptic “shell_21 failed” messages, it actually uses the names I attached, and shows them quite clearly for the last build.

I’ll submit some patches for these xmlrpc additions to buildbot when I get the time, but right now I mainly needed the Mercurial 1.0 hook compatibility (that was broken for quite awhile in buildbot), and a fairly specific set of information from the xmlrpc methods that I wasn’t sure others would want.

I’m looking forward to trying out the new nose so that I can hopefully verify Pylons is good to go on Python 2.6 as Phil Jenvey’s been working tirelessly on patches to Beaker and other dependencies to make them 2.6 compatible. Any suggestions or thoughts on improving my buildbot viewer are welcome. :)

The Revised Bailout

Amazingly enough, despite all the hoop-la of how the bailout will pass, on Monday, it failed. It was quite entertaining to me to see a bill that everyone seemed so sure would pass, to just go and fail.

In the meantime I’ve found more than a few interesting links that only go to re-inforce my prior thoughts on the bailout.

The vagueness I noted in some sections of the bailout was quite intentional.

Even as policy makers worked on details of a $700 billion bailout of the financial industry, Wall Street began looking for ways to profit from it.

Ouch, well that sure is reassuring.

How’d they do? Pretty dang good. They made sure no pesky regulations would affect their golden parachutes, or reduce their paychecks, and they expanded the definition of what assets could be bought to extend beyond just mortgage related securities, to any troubled asset. If some bank has a laundry list of dumb investments that aren’t paying off well, they’re as eligible as anything else if the Treasury considers them at risk of failing (though even what kind of ‘risk of failing’ is required before the Treasury buys such assets seems vague).

Isn’t the point of the free market to let companies that do stupid stuff fail from it?

The NY Times also mentions that many of the financial institutions want to manage the assets that are bought as well, and get paid for it of course, which would generate them a very hefty profit as well.

I haven’t seen a lot in the mainstream media about the huge list of economists (over 165 and counting) who oppose the bailout, which is curious.

So far, this countries track record of hastily passing massive and far reaching pieces of legislation based on hysteria hasn’t worked out too well. It has resulted in a war and all sorts of legislation that is still having ill effects on basic liberties and freedoms. The revised bill which includes more bits to appeal to Republicans doesn’t seem any better, and is still co-written by the financial industry eager to get some more free money.

Ron Paul has some good thoughts on why the bailout is a bad idea, and Michael Moore has an amusing write-up of how to save the banks with plenty of good facts and of a few questionable ones (but overall a good read). I particularly like his #2 point for paying for the bailout.

I disagree with Michael Moore though, mainly because I think Ron Paul and the economists have the right point. Investors should not be subsidized. Investors making risky bets, should take their losses. They clearly want to keep the profit, let them keep the loss as well.

Regardless, I hope the new version doesn’t get passed as well, though given how disappointing the politicians have been for me and the fact that the general public got creeped out watching the investors having a fire-sale means that the financial industry will likely be able to get the new bill passed. Nothing like some hysteria in the stock market to help some legislation creep through.

Ugh.

Update: A rather interesting look at the distribution of wealth in America, figure 5 showing the difference increasing.

Update 2: I should note that the new bailout bill that has had tastier pork tossed in, increased in size by $110 billion, and in size from the 101 page version I read, to a new 451 page bill.

That pretty much negates the odds of most people having enough time to read it, and who knows what other tidbits got buried in it.