be Groovie en-us Tue, 17 Sep 2019 00:00:00 -0700 <![CDATA[Argo Tunnel in Kubernetes]]>

Argo Tunnel in Kubernetes


Part [1 2 3] of a series of more. I don’t know how much more yet as this is primarily written to document my setup so I can refer to it later when I wonder why/how I did something.

To get the Argo Tunnel working in Kubernetes, we need to first install helm on the computer we run kubectl from. I use a debian based system for this, so these commands are the appropriate ones for a recent debian that has snap.

Install helm:

$ sudo snap install helm --classic

Configure a service account with cluster-admin role by sticking this into a rbac-config.yaml file:

apiVersion: v1
kind: ServiceAccount
    name: tiller
    namespace: kube-system
kind: ClusterRoleBinding
    name: tiller
    kind: ClusterRole
    name: cluster-admin
- kind: ServiceAccount
    name: tiller
    namespace: kube-system

Apply the configuration to the cluster:

$ kubectl create -f rbac-config.yaml

Initialize helm:

$ helm init --service-account tiller --history-max 200

Now we’re ready to add the cloudflared ingress controller:

$ helm repo add cloudflare
$ helm repo update
$ helm install --name anydomain --namespace default \
    --set rbac.create=true \
    --set controller.ingressClass=argo-tunnel \
    --set controller.logLevel=6 \

Install cloudflared from here.

Then run cloudflared login and put the cert in ~/.cloudflared/

Create a secret for this domain:

$ kubectl create secret generic DOMAIN --from-file="$HOME/.cloudflared/cert.pem"

The rest of the Argo instructions for an Ingress definition should all work fine now.

Tue, 17 Sep 2019 00:00:00 -0700 <![CDATA[Lan Containers in Kubernetes with Rancher]]>

Lan Containers in Kubernetes with Rancher


Part [1 2 3] of a series of more. I don’t know how much more yet as this is primarily written to document my setup so I can refer to it later when I wonder why/how I did something.

When running containers that I want available on my LAN, it’s handy to expose them under their own LAN IP. To do this I set my DHCP server to stop allocating addresses past .189, and will reserve the remaining IP addresses for container use.

To allocate LAN IP addresses for containers in Kubernetes I use MetalLB. I installed it using kubectl (Rancher makes it easy to download the kube config file).

Then I setup a ConfigMap YAML:

apiVersion: v1
kind: ConfigMap
  namespace: metallb-system
  name: config
  config: |
      - name: default
        protocol: layer2

And applied with:

$ kubectl apply -f metallb_config.yaml

Unifi Controller

I’m changing the IP of where the existing Unifi controller runs, which comes with an interesting drawback. All the existing Unifi devices need to start reporting to the new controller.

First, login to existing controller, and download a full backup. Then setup the new controller, my unifi-controller.yaml looks like this:

I created a namespace for unifi in the Rancher UI, where I’ll run my Unifi containers.

To spin up the container on its new IP:

$ kubectl -n unifi apply -f unifi-controller.yaml

This can take a few minutes to come-up, it has to play with ARP responses and such. Once the new Unifi controller was up, I restored the backup and logged in. None of the devices will show up.... yet!

On the old Unifi controller, I went to Settings -> Site and clicked the Export Site button. You can download a backup file here, but the more important step is to migrate the devices to the new controller by providing the new IP for them to inform. Once this step is done, the devices should show up in the new controller shortly.

The old controller can then be shutdown.

Unifi Video Controller

I used this kubernetes file to setup my unifi video controller:

I’m using the /mnt/data directory that is the second SATA SSD drive for these containers to store their data on.

After restoring this from the backup I made of my old unifi video controller, I logged in, selected the camera, and under Manage I changed its reporting IP to the new container IP.

Wed, 11 Sep 2019 00:00:00 -0700 <![CDATA[A Smaller, Faster, and More Efficient Home Server]]>

A Smaller, Faster, and More Efficient Home Server


Part [1 2 3] of a series of more. I don’t know how much more yet as this is primarily written to document my setup so I can refer to it later when I wonder why/how I did something.

A few years ago I built a home NAS and virtualization server. While I moved past SmartOS after a year or two to FreeNAS, the hard drives are aging and I realized it isn’t very efficient compared to what’s available now. From my prior post, the hard drives were replaced with 6 4TB drives in a ZRAID-2. This ended up giving me vastly more space than I had any need for and as I now have a 1Gbps fiber line I don’t feel the need to store 16 TB locally.

Given my Internet connectivity and increased focus in my spare time on containerized applications using Kubernetes, I figured it might be a good time to ‘downsize’ in power consumption while increasing my virtualization capabilities. The new server configuration I ended up with is a fraction of the size (half-length 1U vs full-length 2U) and power (190w idle vs 40w idle), while having 4x the cores and over 8x the memory capacity. 7 years has definitely helped on what you get for the same price!

System Specs

It took awhile before Epyc embedded processors and vendors 1U systems started to hit the market, but the 8-core/16-thread Epyc 3251 in a nice half-length 1U is now widely available. Serve The Home did a great review of this system which helped convince me to purchase one.

Here are specs for this build:

  • CPU/Motherboard: AMD Epyc Embedded 3251 (8-core / 16-thread) on Supermicro M11SDV-8C-LN4F
  • Case: Supermicro AS 5019D-FTN4
  • RAM: Supermicro DDR4-2666 64 GB ECC LRDIMM x2 (128 GB total)
  • NVMe Drive: Samsung 970 Evo Plus 1TB
  • SATA Drive: Sandisk Ultra 3D 2TB
My 2019 Home virtualization server

Overall cost: ~$2100

This is a bit more than the last buuild, but is 100% SSD storage without any redundancy. I’m relying on cloud backups and accepting that I will have some downtime if a part fails. I consider this an acceptable trade-off to keep costs lower with the hope that once a SSD has proven itself for a few months it should last much longer than a spinning platter drive as I don’t anticipate heavy read/write loads that would wear out the drives.


In the event an SSD fails, I’m only a 10min drive from a Best Buy where I got both the SSD’s. I consider this a more likely failure scenario than the PSU, CPU, memory, or motherboard failing in my experience (I already ran a memtest suite against the memory before installing proxmox, which comes with memtest on their live ISO).

If I really go nuts with VM’s and containers I still have 2 DIMM slots free for another 128 GB of memory. I’ve found that for personal use containers usually run into RAM pressure much earlier than CPU pressure.

OS Choice

I’ve been reading a lot of Serve the Home, and their glowing review of Proxmox VE convinced me to give it a try. I’ve been enjoying it so far and it’s easy to get started with and makes running KVM a breeze.

ProxMox VE it is!

I grabbed the ISO for installing ProxMox 6.0, based on a buster debian distro. Using the ISO directly from my computer was rather easy from the Supermicro IPMI Java interface. While the ikvm HTML5 interface is more convenient, the Java-based console makes it a breeze to attach local .iso files as a CD/DVD drive to the server.

I’m running the community edition, which requires you to edit the /etc/apt/sources.list to include the non-enterprise deb:

deb buster pve-no-subscription

Additional steps:

  1. From the shell in the web UI:

    apt-get update
    apt-get dist-upgrade -y
  2. Add the additional drive as a new lvm-thin pool


I have set all the VM hard drives under the default lvm-thin pool which is on the NVMe SSD for performance. The SATA SSD is to be used for persistent data volumes for containers.

AWS Storage Apppliance

The first thing I wanted to try was utilizing hybrid storage in AWS with their Storage Appliance. Unfortunately AWS only provides a VMWare image. I found a few articles online that indicated this was rather easy to convert to a raw disk image for use in Proxmox, and got it working rather quickly.

  1. Download the AWS Storage Gateway zipfile

  2. Unzip the zipfile (resulting in an .ova file)

  3. tar xf AWS-Appliance-2019-07-24-1563990364.ova (Filename dependent on time it was d/l)

  4. qemu-img info AWS-Appliance-2019-07-24-1563990364-disk1.vmdk and record the virtual size to use when provisioning a proxmox KVM

  5. Provision a Proxmox VM with the given size disk, using an IDE disk emulation target. I gave my VM 16GB of memory, as I wasn’t sure how much it would want.

  6. Determine the location of the LVM disk used by the new VM (something like /dev/pve/vm-100-disk-0).

  7. Convert the vmware disk image to the raw:

    qemu-img convert -f vmdk -O raw AWS-Appliance-2019-07-24-1563990364-disk1.vmdk /dev/pve/vm-100-disk-0
  8. Edit the VM hardware to add the LVM-thin drive resource. I added a 150Gb hard drive as another IDE resource per AWS recommendations for local cache size.

  9. Start the VM.

  10. Look at the console in proxmox to determine the IP, and change it as desired for a static IP.

  11. Finish setup in the AWS Console for the Storage Gateway, your computer will need to be able to talk directly to the VM running the appliance VM. You will be asked to set a cache drive, select the additiona 150Gb drive.


  • Fast access to frequently accessed files that fit within the cache
  • Everything backed by S3 reliability
  • As much storage as you want to pay for
  • It’s fun to see that your SMB share has 7.99 Exabytes free


  • Unavailable when the Internet is out
  • Slower access than a NAS
  • SMB requires an Active Directory server for user based permissions or a single guest account with read/write access to all SMB shares.
  • NFS shares have similarly odd restrictions


I followed these directions to install RancherOS under ProxMox VE. Reproduced here with a fix to the cloud-config.yml as the example didn’t validate.

  1. Download RancherOS ISO
  2. Upload the iso to (local)pve
  3. Setup a VM with RancherOS ISO as CD. Give it at least 3gb ram to start. Rancher Server failed with low ram
  4. Boot
  5. From Console change password
    • sudo bash
    • passwd rancher
  6. SSH to rancher@
  7. prepare your ssh keys with putty gen or local ssh key-gen
    • vi cloud-config.yml
  8. paste the cloud config edited with your settings, make sure the pasted data is pated correctly, add your key in a single line
  9. press exit exit :wq to save

            mtu: 1500
            dhcp: false

      - ssh-rsa <YOUR KEY>
  • sudo ros config validate -i cloud-config.yml
  • sudo ros install -c cloud-config.yml -d /dev/sda
  1. Remove CD Image from VM, and then reboot.
  2. SSH back into RancherOS (rancher@) using your new ssh private key


With RancherOS running happilly, its time to install Rancher on the VM. This is relatively easy, from the RancherOS VM shell, just run:

sudo docker run -d --restart=unless-stopped -p 8080:80 -p 8443:443 -v rancher:/var/lib/rancher rancher/rancher:stable

Mapping port 80/443 to different local ports is to avoid intereference from the ingress proxy which will be running on this same node.

Once Rancher is available on port 8443:

  1. Add a cluster, of custom type.
  2. Name it, and hit next.
  3. Select all three node options (etcd, Control Plane, Worker)
  4. Copy the command shown and run it in the RancherOS shell.
  5. Click Done in the Rancher UX.
  6. The cluster will become available.

Setup the SATA SSD

I want to use the SATA SSD for persistent volumes for the containers:

  1. Add a hard drive in Proxmox VE to RancherOS VM

  2. Choose a sufficient size (I choose 400 GB)

  3. Start the RancherOS VM (or restart it)

  4. Verify additional hard drive appears in fdisk -l

  5. Format the hard drive with fdisk /dev/sdb

  6. Choose new partition, primary, select default start/end values

  7. Format the partition with mkfs.ext4 /dev/sdb

  8. Set it to load at start in RancherOS:

    ros config set mounts '[["/dev/sdb","/mnt/data","ext4",""]]'
  9. Reboot and verify /mnt/data is a volume mount.


That’s it for a first day of configuring things. Next up I’ll need to setup MetalLB so that my Kubernetes containers I start with Rancher get LAN IP’s rather than shuttling everything through the default nginx ingress.

Mon, 09 Sep 2019 00:00:00 -0700 <![CDATA[Trying out SmartOS and OpenIndiana]]>

Trying out SmartOS and OpenIndiana

After building my new server capable of running SmartOS, it was time to give it a spin!

If you’ve only built desktop machines, its hard to express how awesome IPMI KVM is. No longer do you need to grab another keyboard / video monitor / mouse (the KVM), you just plug in the IPMI Ethernet port on the motherboard to your switch and hit the web-server its running. It then lets you remotely access the machine as if you had it hooked up directly. You can get into the BIOS, boot from ISO’s on your local machine, hard reset, power down, power up, etc. It’s very slick and means I can stick the computer in the rack without needing to go near it to do everything that used to require a portable set of additional physical hardware.


This post assumes some basic knowledge of OS virtualization. In this case QEMU, KVM (which was ported by Joyent to run on SmartOS), and Zones. I generally refer to them as VM’s and will differentiate when I add a Zone vs. a KVM instance.

First Go at SmartOS

Installation is ridiculously easy, there is none. You download SmartOS, put it on a USB stick or CD-ROM, and boot the computer from it. I was feeling especially lazy and used the motherboards IPMI KVM interface to remotely mount the ISO image directly from my Mac.

Once SmartOS booted, it asked me to setup the main ZFS pool, and it was done. SmartOS runs a lot like a VMWare ESXI hyper-visor, with the assumption that the machine will only be booting VM’s. So the entire ZFS pool is just for your VM’s, which I appreciate greatly. After playing with it a little bit, it almost felt.... too easy.

I had really allocated at least a week or two of my spare time to fiddle around with the OS before I wanted it to just work, and having it running so quickly was almost disappointing.

The only bit that was slightly annoying was that retaining settings in the GZ (Global Zone) is kind of a pain. You have to drop in a service file (which is XML, joy!) on a path which SmartOS will then load and run on startup. This was mildly annoying, and some folks on the IRC channel suggested I give OpenIndiana a spin, which is aimed more at a home server / desktop scenario. There was also a suggestion that I give Sophos UTM a spin instead of pfsense for the firewall / router VM.


Since OpenIndiana has SmartOS‘s QEMU/KVM functionality (needed to run other OS’s like Linux/BSD/Windows under an illumos based distro), it seemed worth giving a go. It actually installs itself on the system unlike SmartOS, so I figured it’d take a little more space. No big deal. Until I installed it.

Then I saw that the ZFS boot pool can’t have disks in it larger than 2TB (well, it can, but it only lets you use 2TB of the space). Doh. After chatting with some IRC folks again, its common to use two small disks in a mirror as a ZFS boot pool and then have the much larger storage pool. Luckily I had a 250GB drive around so I could give this a spin, though I was bummed to have to use one of my drive bays just for a boot disk.

Installation went smoothly, but upon trying to fire up a KVM instance I was struck by how clunky it is in comparison to SmartOS. Again, this difference comes down to SmartOS optimizing the heck out of its major use-case.... virtualizing in the data-center. In SmartOS there’s a handy imgadm tool to manage available images, and vmadm to manage VM’s. These don’t seem to exist for OpenIndiana (maybe as an add-on package?), so you have to use the less friendly QEMU/KVM tools directly.

Then the KVM failed to start. Apparently the QEMU/KVM support in OpenIndiana (at least for my Sandy Bridge based motherboard) has been broken in the latest 3 OpenIndiana releases for the past 5 months. There’s a work-around to install a specific set of packages, but to claim QEMU/KVM support with such a glaring bug in a fairly prominent motherboard chip-set isn’t a good first start.

My first try to install the specific packages failed as my server kernel-panicked halfway through the QEMU/KVM package installation. Upon restarting, the package index was apparently corrupted. The only way to fix it is to re-install OpenIndiana... or rollback the boot environment (a feature utilizing ZFS thus including snapshots). Boot environments and the beadm tool to manage them are a bit beyond the scope of this entry, but the short version is that it let me roll-back the boot file-system including the package index to a non-mangled state (Very cool!).

With QEMU / KVM finally installed and working, I installed and configured Sophos UTM in a KVM and was off and running. Except it seemed to run abysmally slow... oh well, I was about to go on vacation anyways. I set the KVM to load at boot-time and restarted.

Upon loading the KVM at boot, the machine halted. This issue is apparently related to the broken QEMU / KVM packages. It was about time for my vacation, and I had now played with an OS with some rather rough edges in my spare time for a week. So I powered it off, took out the boot drive, and went on my vacation.

Back to SmartOS

When I got back from my vacation, I was no longer in the mood to deal with failures in the OS distribution. I rather like the OpenIndiana community, but now I just wanted my server to work. SmartOS fit the bill, and didn’t require boot drives which was greatly appreciated. It also has a working QEMU / KVM, since its rather important to Joyent. :)

In just a day, I went from a blank slate to a smoothly running SmartOS machine. As before, installation was dead simple, and my main ZFS pool zones (named as such by SmartOS) was ready for VM’s. Before I added a VM I figured I should have an easy way to access the ZFS file-system. I turned on NFS for the file-systems I wanted to access and gave my computer’s IP write privilege and the rest of the LAN read-only. This is insanely easy in ZFS:

zfs set sharenfs=rw=MYIP,ro= zones/media/Audio

To say the least, I love ZFS. Every other file-system / volume manager feels like a relic of the past in comparison. Mounting NFS file-systems on OSX used to suck, but now its a breeze. They work fast and reliably (thus far at least).

Setting Up the Router KVM

First, I needed my router / firewall KVM. I have a DSL connection, so I figured I’d wire that into one NIC, and have the other NIC on the motherboard go to the LAN. SmartOS virtualizes these so that each VM gets its own Virtual NIC (VNIC), this is part of the Solaris feature- set called Crossbow. Setting up the new KVM instance for Sophos UTM was simple, I gave it a VNIC on the physical interface connected to the DSL modem and another on the physical interface connected to my switch.

Besides for the fact that the VM was working without any issues like I had in OpenIndiana, I noticed it was much faster as well. Unfortunately for some reason it wasn’t actually routing my traffic. It took me about an hour (and clearing the head while walking the dog) to see that I was missing several important VNIC config options, such as dhcp_server, allow_ip_spoofing, allow_dhcp_spoofing, and allow_restricted_traffic.

These settings are needed for a VM that intends to act as a router so that it can move the packets and NAT them as appropriate across the VNICs. Once I set those everything ran smoothly.

So far, this only took me about 3 hours and was rather simple so I decided to keep going and get a nice network backup for the two OSX machines in the house.

Setting Up Network Backups

After some research I found out the latest version of netatalk would work quite nicely for network Time Machine backups. I created a zones/tmbackups ZFS file-system, and two nested file-systems under that for my wifes’ Macbook and my own Mac Mini. Then I told ZFS that zones/tmbackups should have compression enabled (Time Machine doesn’t actually compress its backups, transparent ZFS file compression FTW!) and I set quota’s on each nested file-system to prevent Time Machine from expanding forever.

Next I created a Zone with a SmartOS Standard dataset. Technically, the KVM instances run in a Zone for additional resource constraints and security, while I wanted to use just a plain Zone for the network backups. This was mainly because I wanted to make the zones/tmbackups file-system directly available to it without having to NFS mount it into a KVM.

If you’ve ever compiled anything from source in Solaris, you’re probably thinking about how many days I spent to get netatalk running in a Zone right now. Thankfully Joyent has done an awesome job bringing a lot of the common GNU compiler toolchain to SmartOS. It only took me about an hour to get netatalk running and recognized by both macs as a valid network Time Machine backup volume.

Unfortunately I can’t remember how exactly I set it up, but here are the pages that gave me the guidance I needed:

I’ve heard that netatalk 3.x is faster, and will likely upgrade that one of these days.

Setting Up the Media Server KVM

One of the physical machines I wanted to get rid of was the home theater PC I had built a few years back. It was rarely used, not very energy efficient, and XBMC was nowhere near spouse-friendly enough for my wife. We have an AppleTV and Roku, and I figured I’d give Plex a try on the Roku since the UI was so simple.

I setup a KVM instance and installed Ubuntu 12.04 server on it. Then I added the Plex repo’s and installed their Media Server packages. Fired it up and pointed Plex at my Video folders and it was ready to go. The Roku interface is slick and makes it a breeze to navigate. Being based on XBMC means that it can play all the same media and trans-codes it as necessary for the other network devices that want to play it.

At first Plex ran into CPU problems in the KVM... which I quickly realized was because I hadn’t changed the default resource constraints. The poor thing only had a single virtual CPU... after giving it a few more it easily had enough CPU allocated to do the video trans-coding.

While KVM runs CPU-bound tasks at bare-metal speed, disk I/O is virtualized. To reduce this problem I have Plex writing its trans- coded files to the ZFS file-system directly via an NFS mount. The media folders are also NFS mounted into the Media Server KVM.

I threw some other useful apps onto this KVM that I was running on the home theater PC and left it alone.

SmartOS Rocks

I now have a nice little home SmartOS server setup running that does a great job taking on jobs previously done by 2 other pieces of hardware. I still need to setup a base Ubuntu image to use for other development KVM’s, which I’ll blog about when I get that going. Despite being intended for the data-center, SmartOS works great for a home NAS / Media Server / Router system. I’m sure I’ll be even happier as I start to ramp up my use of development VM’s.

OpenIndiana is a small community taking on a big job. It’s a great community and people are very friendly. But you should expect to be hacking on things very early on if you use it, rather than playing with the other components. The SmartOS community is doing great too, and there’s more than a few forks that add some additional home-centric type functionality. So far I haven’t needed any of those enough to get me to try them out.

Anything else I should blog about regarding SmartOS or the rest of my setup?

Mon, 17 Sep 2012 00:00:00 -0700 <![CDATA[Building A SmartOS Server]]>

Building A SmartOS Server

I’ve been reading about SmartOS for awhile now and have wanted to build a home server that would let me run VM’s with ZFS for the main file-system. Getting rid of my home theater PC and wireless router (which has been annoying me with its flakiness for months) was also a goal. Running something like pfsense in a VM would give me more options and theoretically be more stable than the fairly crappy software that seems to plague home consumer-grade wireless routers.

So after a month or so of research in my spare time, it seemed like SmartOS was going to be the best bet. Even though its generally intended for use in the datacenter, it had all the features I wanted (which I’ll blog about separately in my next post). Now I just needed a parts list that had already been verified to work with SmartOS, which is a bit pickier on hardware than the linux/BSD distributions.


Here’s what I ended up with:

  • CPU: Intel Xeon E3-1230 V2
  • Motherboard: SUPERMICRO MBD-X9SCL-F-O
  • Case: NORCO RPC-2212 Black 2U Rackmount Server Case with 12 Hot-Swappable SATA/SAS Drive Bays
  • HBA: LSI Internal SATA/SAS 9211-8i (Hooks up to 2 of the back-plane connectors in the case for 8 drives)
  • RAM: 16GB ECC (The 8 GB unbuffered sticks were unfortunately not around at the time or I would’ve gotten two of those to begin with)

I already had a 2TB and 3TB drive, so I bought one more of each so that I could run a ZFS storage pool with 2 vdev mirrors as Constantin Gonzalez blogs about regarding RAID vs. mirrors.

In retrospect, and after reading a bit more, I think I would’ve gotten one of the larger Norco 4U cases. Not because I need or want 20+ hot-swap bays, but because you can easily use a ‘desktop’ grade 80+ Titanium rated power supply. Finding a 2U 80+ PSU is difficult, a 80+ Titanium rated that puts all its power out on a single 5v rail is almost impossible. The cost savings in getting a good desktop-grade PSU with the Norco 4U case is about the same as the one I got with the more expensive 2U PSU.

I also bought a rack to put the server in along with my other home networking gear, so that it’d all be nicely packed away in a corner of the garage. Here’s a photo of the completed setup:

My home-server rack

I have one of the cheaper Cisco SG300-10 switches which conveniently came with rack-mounts, and monoprice had a very affordable patch panel and blank plates to make it look tidy.

Overall cost: ~$2200

That includes the nice Tripp Lite SR12UB 12U Rack Enclosure which I’ve found handy to lock to ensure my toddler doesn’t yank out hard drives (he figured out how to pull out the hot-swap drive in all of 20 seconds when I was assembling it). Not that I let him run around the garage, but keeping everything locked is handy just in case.

OS Choice

When I was assembling and preparing to install SmartOS, some people on IRC mentioned that OpenIndiana might be a better choice for a home server. Suffice it to say it didn’t work out well, while SmartOS has been flawless now and running smoothly for the past two months.

My next post will have a lot more details on my OpenIndiana experience as well as how I have the SmartOS box setup.

Sun, 16 Sep 2012 00:00:00 -0700 <![CDATA[New Blog Software Again!]]>

New Blog Software Again!

I’ve been using tumblr for awhile and while its useful when posting random stuff I should’ve posted to Facebook instead (images, links, videos, etc.), writing my text in HTML was just icky. Markdown isn’t a huge improvement and I was really itching to write all my posts in reST as I already know it quite well from writing my docs using Sphinx.

I considered using blogofile, but it seems to be abandoned and it’s not trivial to add normal reST style code highlighting. Then I saw tinkerer, which is basically just a few extensions on top of Sphinx... perfect!

The Migration

For anyone considering migrating from tumblr, here’s my simple dump script that pulled all my posts I cared about out of tumblr and dropped them into directories for tinkerer

import json
import re
import os
import subprocess

import requests

date_regex = re.compile(r'(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})')

def pull_posts(blog, api_key):
    api_url = ''
    call_url = api_url % (blog, api_key)
    r = requests.get(call_url)
    posts = json.loads(r.content)
    return posts

def html2rst(html):
    p = subprocess.Popen(['pandoc', '--from=html', '--to=rst'],
                         stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    return p.communicate(html)[0]

def dump_posts(posts):
    post_links = []
    for post in posts:
        if post['type'] not in ['text']:
        d = date_regex.match(post['date']).groupdict()
        os.system("mkdir -p %s/%s/%s" % (d['year'], d['month'], d['day']))
        slug = post['post_url'].split('/')[-1].replace('-', '_')
        link = '%s/%s/%s/%s' % (d['year'], d['month'], d['day'], slug)
        bar = '=' * len(post['title'])
        with open('%s.rst' % link, 'wb') as f:
            print post
            f.writelines([post['title'].encode('utf-8'), '\n', bar, '\n\n'])
            if post['type'] == 'text':
                body = html2rst(post['body'].encode('utf-8').replace('’', "'"))
                f.writelines([body, '\n\n'])
            elif post['type'] == 'link':
                desc = html2rst(post['description'].replace('’', "'").encode('utf-8'))
                f.writelines(['Link: `%s <%s>`_\n\n' % (post['title'].encode('utf-8'), post['url'].encode('utf-8'))])
                f.write(desc + '\n\n'),
                '.. author:: default\n', '.. categories:: %s\n' % ', '.join(post['tags']),
                '.. comments::\n', '   :url: %s' % post['post_url']
    return post_links

It’s quite nice that I get all the Sphinx extensions for use in my blog, and there’s no more mental context switching to write a blog post vs. writing project documentation for my open-source projects.

Dropping a graphviz diagram into my blog also became trivial.

digraph foo {     "awesome" -> "sphinx";     "awesome" -> "tinkerer"; }

The Bad News

It’s not perfect, tinkerer is still very beta. But I can wrap my head around it, and its easy to extend. I’ve already made a little modification in my own fork which allows me to specify URL’s for the comments to ensure I get the right disqus threads on the old blog posts I ported. This wasn’t a flawless process due to how the main reactions and comments thing on the main pages look, they’re a bit off for the legacy posts... but at least the comments and such show up fine once you click in so I’ll live with it.

There’s no category specific RSS feeds at the moment, so I’ll need to hack that in so that I can get relisted on the Python aggregators. I also will likely update the theme, right now I’m just using ‘minimal’ which isn’t bad.

Since this is more for just posts, I dropped the other tumblr things like links and videos to retain just content. I don’t think this is too negative but some might want all the types tumblr supports.

Overall, I’m quite happy with it thus far. We’ll see how I feel in a few months, and hopefully I’ll be blogging more since there’s less friction involved since I get to use the Sphinx tools I’m quite familiar with.

Mon, 26 Mar 2012 00:00:00 -0700 <![CDATA[Notes on the Pylons & repoze.bfg Merger]]>

Notes on the Pylons & repoze.bfg Merger

Some folks might not have time to follow the Pylons-discuss mail list, so this might be news to them, but I’m thrilled to announce that the Pylons and repoze.bfg web frameworks are merging. If this is the first you’ve heard about it, don’t worry, it was only announced a week ago now on the Pylons mail list.

In the time since the announcement, I’ve heard a lot of varying feedback. Some people took a look at Pyramid (the core package that will be equivilant to ‘Pylons 2.0’) and were quick to respond, usually in a knee-jerk type response. I think some of this was due to a miscommunication, and partly because there was so much already done. When other frameworks have merged in other languages, such as Rails merging with Merb, the announcement was just that. There was no code at the time to show, just a promise that when it was ready, it would be awesome.

This merger in contrast already had a starting foundation for a huge chunk of the core features. As a result, people assumed that what we had was already ‘finished’, or close to it. The polish of much of the documentation made it feel odd that there was no “Porting Pylons 1.0 to Pyramid” guide done. In reality, Pyramid is definitely not done, there is still quite a bit of work left before Pyramid will meet the expectations that many Pylons users have. There’s still refinements to be done to Pyramid, and additional packages that Pylons users will most likely always use with it for the feature-set they’re accustomed to.

I’ve summed up a few thoughts on when Pylons users should port to Pyramid to try and help manage expectations better in the future. I’ll make more announcements when packages are ready to ease the transition and a “Porting Guide” is ready.

What is Pylons?

Many Pylons users don’t realize which features they enjoy come from the package ‘pylons’ vs. the other packages that Pylons depends on. Contrary to popular belief the majority of features present in Pylons actually come from other packages. This mistaken belief that most of the features come from the pylons package led some to think that because a lot of my future development time will be spent on adding features/packages around pyramid, Pylons is somehow dead>. This is not the case.

First, Pylons the web framework is mainly a small (~ 1000 LoC) glue layer between Paste, PasteScript, PasteDeploy, WebOb, WebError, Routes, WebHelpers, Mako, and SQLAlchemy. Some people usually end up swapping out Mako/SQLAlchemy but by and large this is the common ‘Pylons Stack’. Most of the new features in Pylons over the past several years actually came from additions to WebHelpers, WebError, or Routes. All of these packages continue to get the same development as they have, so no ‘death’ is occurring.

Second, for over the past 6 months now, there’s been very little in the way of patches submitted, bugs reported, or other feature requests. In many ways Pylons is ‘done’ regarding adding more feature to the core package itself. As I announced on the Pylons-discuss mail list, the Pylons code-base hit some design issues. Adding the features I heard requested from quite a few users (and needed myself) regarding extensibility couldn’t be retro-fitted into the existing design. I encourage anyone curious to read my prior entry on sub-classing for extensibility to be a preview of some future blog posts. I’ll be writing more about design patterns in Python that handle extensibility which many popular Python web frameworks are also struggling to handle.

The Future

I’m very excited about the future for the Pylons Project, which is the new over-arching organization that will be developing Python web framework technologies. The core will be Pyramid, with additional features and functionality building around that. We’re already quickly expanding the developer team with some long-time contributors and having a combined team has definitely helped us progress rapidly.

One of my main goals is to encourage and ease contributions from the community. To that extent I’ve been filling in the contributing section for the Pylons Project as much as possible. I believe this is an area that will quickly set us apart from other projects as we emphasize a higher standard of Python development.

Django did a good job setting the bar high for its documentation of how to contribute to Django, which deserves a lot of credit for clearly defining community policies. Its missing a portion we considered extremely valuable which core developers generally get very picky on when accepting patches… how to test your code. The Pylons Project adapted the rather thorough testing dogma noted by Tres Seaver, which I personally can’t recommend highly enough when it comes to writing unit tests. It’d be nice to see more posts expand on exactly how to test your code. Many developers (including myself) can write code that passes 100% test coverage… but is it brittle test code? Prone to failure if some overly clever macro it uses fail? Seeing a well written set of examples on designing unit tests to avoid common gotcha’s is definitely something anyone contributing (and developers in general) should be familiar with.

For those wanting a gentler introduction to Pyramid (the docs are very verbose and detailed, not at all opinionated), I’ll be blogging more about new features and how to utilize them. Please be patient, I think a lot of people are going to be excited at what’s in store.

Sat, 13 Nov 2010 00:00:00 -0800 <![CDATA[Why Extending Through Subclassing (a framework's classes) is a Bad Idea]]>

Why Extending Through Subclassing (a framework’s classes) is a Bad Idea

Ok, I’ll admit it, overly ambitious blog post. So I’ll refine it a little now, this is intended mainly as my thoughts on why as a tool developer (one who makes tools/frameworks that other programmers then use), its a bad idea to implement extensible objects via developer subclassing. This is actually how the web framework I wrote - Pylons - provides its extensibility to developers and lets them change how the framework functions.

Please excuse the short and possibly incomplete description, this is mainly a quick post to illustrate a tweet I recently made.

First, some background…

One of the things that Pylons 1.0 and prior is missing is a way to easily extend a Pylons project. While it can be done, its very ad-hoc, kludgy, and generally not very well thought-out. Or thought-out at all really. What was somewhat thought-out was how a developer was supposed to extend and customize the framework.

In a Pylons project, the project creates a PylonsApp WSGI object, and all the projects controllers subclass WSGIController. This seemed to work quite well, and indeed many users happily imported PylonsApp, subclassed it to extend/override methods they needed for customization, or changed how their WSGIController subclass worked to change how individual actions would be called.

Everything seemed just fine…. until…

Improving Pylons

When I had some free time a little while back, I set about looking into how to extend and improve Pylons to make up for where it was lacking, extensibility. I quickly realized that I’d need to change rather drastically how Pylons dispatch worked, and how controller methods were called to make them more easily extendable. But then with a certain feeling of dread, the subclassing issue nipped me. All my implementations of PylonsApp and WSGIController were effectively frozen.

Since every single developer using Pylons sub-classes WSGIController, and to a much lesser extent, PylonsApp, any change to any of the main methods would result in immediate breakage of every single Pylons users app that happened to customize them (the very reason subclassing was used!). This meant that I couldn’t very well change the implementation of the actual classes to fix their design, because that would just cause complete breakage. Ugh!

So after looking into it more, I’ve ended up with this short list of the obvious bad reasons this shouldn’t be done. BTW, in Pylons 2, controllers don’t subclass anything, and customization is all with hooks into the framework, no subclassing in sight!

Short List of Why It’s Bad

From a framework maintainers point of view…

  1. Implementations of the classes are effectively frozen, because all the class methods are the API.
  2. Correcting design flaws or implementation flaws are much more difficult if not impossible without major breakage due to point #1.
  3. Heavily sub-classed/large hierarchy classes can have performance penalties.

From a developer ‘extending’ the classes point of view…

  1. Figuring out how to unit-test is more difficult as the full implementation is not in your own code… its in the framework code.
  2. When using mix-in’s and other classes that also subclassed the framework, strange conflicts or overrides occur that aren’t at all obvious or easy to debug/troubleshoot.

I think there were a few more reasons I came across as well, but I can’t recall them at the moment. In short, I’m now of the rather firm opinion that the only classes you should ever subclass are your own classes. Preferably in the same package, or nearby.

Tue, 19 Oct 2010 00:00:00 -0700 <![CDATA[Pylons 0.10 and 1.0 Beta 1 Released]]>

Pylons 0.10 and 1.0 Beta 1 Released

Without further ado,

I’m pleased to announced that Pylons 0.10b1 and 1.0b1 are now out. I have not put them on Cheeseshop to ensure they’re not downloaded accidentally.

Upgrading / Installing

I have updated upgrading instructions here:

The instructions to install from scratch on Pylons 1.0b1:

The upgrading page covers the important upgrading instructions that Mike Orr touched briefly on before.

Note that these are beta releases, intended for us to discover remaining issues and continue updating any other documentation where applicable. Very little has actually changed in Pylons since 0.9.7, apart from 1.0 dropping all of the legacy functionality and a few explicit clean-ups.


Routes, Beaker, and WebHelpers however have been seeing quite a bit of updates through the life of Pylons 0.9.7 so no one should think that the developers working on Pylons and its related parts have been hanging out doing nothing. :)

Since Pylons 0.9.7 was released on February 23, 2009, almost one year ago now:

  • Routes 1.11 was released, and 1.12 with some great updates will be out shortly
  • Beaker has gone from 1.2.2 -> 1.5 with 3 major updates substantially increasing its ease of use and reliability
  • WebHelpers is now at 1.0b4 with major updates, core functions rewritten, and new docs up
  • SQLAlchemy has gone from 0.4 to 0.5 (with 0.6 in beta)

I believe this speaks a great deal about the benefits of keeping the core Pylons functionality separate from other parts, as a variety of bug fixes and features can be improved without requiring new Pylons releases to quickly address bug reports.

How to Help!

To bring Pylons to 1.0, many docs likely need very small changes. Also, it would be great to take care of reference docs where people have commented about problems/tips. Helping is fairly easy, especially if you’re familiar with restructured text.

First: Clone the Pylons repository on Bitbucket:

Then: Edit the documentation files under pylons/docs/en/ to read as appropriate, commit the fix, and push it to bitbucket.

Finally: Issue a pull request on bitbucket so that we’ll know your fix is ready. Ideally you should include a note in it about what your fix remedies.

Bug Reports

Did your upgrade not go according to plan? Was there something missing that you needed to do from the upgrading docs?

Let us know by filing a bug report (mark component as documentation, and milestone as 0.10:

You’ll need to login to file a bug report, or feel free to reply to this announcement with the issue.

Thanks (in alphabetical order) to Mike Bayer, Ian Bicking, Mike Burrows, Graham Higgins, Phil Jenvey, Mike Orr, and anyone else I missed for all their hard work on making Pylons and its various components what they are today.

Sat, 06 Feb 2010 00:00:00 -0800 <![CDATA[Deploying Python web apps with toppcloud]]>

Deploying Python web apps with toppcloud

Ian Bicking recently released a rather interesting package called toppcloud that aims to tackle what I see as a growing need for those of us deploying Python webapps.

I was actually interested in easing my own deployment woes before I saw Ian’s announcement about his package, and was halfway through a rather hefty amount of research on automating server deployments with tools like Chef and Puppet, but toppcloud is a bit different. It not only tackles provisioning a new system on the fly from the ‘cloud’ (using libcloud), but it also handles easy Python (and now PHP) web application deployments.

With such a tantalizing set of goals, I couldn’t really resist getting my feet wet. Boy am I glad I did. PylonsHQ is now running with toppcloud, and I won’t be surprised when more people get it running for them. When many shared hosting providers are $5-10 a month, its rather nice to pay $10/mth for an automatically configured VPS, with one-command deployments. Though of course, unless you go to quite a bit of work yourself, most shared hosting providers don’t have one-command deployments for you.

Before I continue on to describe how I setup Kai - the source code behind PylonsHQ - I should provide a few caveats about using toppcloud:

  • toppcloud is alpha software, there’s no releases of it yet, you will be checking it out from source code
  • currently, only Rackspace Cloud is known to be working, though since it uses libcloud, in theory, any cloud providers that it supports should be usable
  • toppcloud is changing rapidly, be ready to keep up on the commit log to see whats changing
  • there are no unit tests, most likely because its very tedious to make the rather significant amount of mock objects required to test the various local/remote commands and the fact that its changing so fast the tests would probably be obsolete in a week

toppcloud philosophy

I’m half guessing here, based on talking with Ian and reading the docs myself, but the philosophy of toppcloud is around providing a common deployment platform, ala Google App Engine, except of course with Postgres or other ‘services’ that you can request to use. At the moment the only services that toppcloud comes with is CouchDB, Files (to store/serve files for your app on the filesystem), and Postgres w/postgis extensions. For those diving into the source code, it shouldn’t be too hard to see how to create additional services and I hope to see more get added as people get more interested.

Therefore, toppcloud is not expected to be everything to everyone, it is expected to be at least an 80+% solution to deploying web apps in a Google App Engine style ease-of-deployment process. toppcloud itself then ensures that depending on what service you asked for, its setup and ready for use on the server when your app is deployed.

If that sounds like something you’re dying to try out, you’re in luck!

Setting up a Pylons App

I’m not going to mention how to check out the toppcloud source, except to mention the directions are in the


file in the toppcloud source (which should be read in its entirety!). Once you have toppcloud installed on your computer, getting from zero to running website on VPS is remarkably quick:

  1. $ toppcloud create-node --image-id=14362 baseimage

    Wait until email arrived indicating that server is up and ready

  2. $ toppcloud setup-node baseimage

    Server is now all setup to run web apps!

  3. $ toppcloud init myapp

    Create an app to deploy to our server

  4. $ source myapp/bin/activate

    Activate the virtualenv used for this app

  5. $ pip install Pylons
  6. $ cd myapp/src
  7. $ paster create -t pylons awesomeapp

    Create our Pylons app, or check out an existing one from your VCS here

  8. $ cd ../..
  9. $ ln -s myapp/src/awesomeapp/awesomeapp/public/ myapp/static

    toppcloud will make available things in the static directory available without hitting the webapp

  10. Configure myapp/app.ini similarly to (taken from pylonshq site):

    app_name = pylonshq
    runner = src/kai/production.ini
    version = 1
    update_fetch = /sync_app
    service.couchdb =
    service.files =
    default_host =

    In this case, the Pylons site needs CouchDB setup, and the files service (all Pylons sites should use the files service to store cached files, templates, etc.)

  11. $ toppcloud update myapp/

    Note that we’re one directory above myapp, and we have a trailing slash on myapp, this is needed because an rsync is done to copy it to the server, don’t leave off the trailing slash! (This will prolly be fixed at some point)

That’s it! 10 easy steps to go from zero to a running deployed website.

I should also note that you will need to make a production.ini file, and that it should have a few important changes in it. All the references to %(here)s should be changed to %(CONFIG_FILES)s since that’s the persistent location that files can be stored for a toppcloud app between deployments. Other configuration information provided by services (Couch supplies the db/host, Postgres has its host/user/pass info) can be accessed via CONFIG_ vars as well. The services docs have some more info.

More apps can be added to a host as desired, until the ram runs out of course. At the moment toppcloud is using a process pool of 5 with 1 thread under mod-wsgi for each application. This can use a bit of ram if you have multiple heavy Pylons processes, hopefully there will shortly be a way to ask toppcloud to use a single process with multiple threads which will help cut the ram profile a bit.

Using Django and PHP

Unfortunately I haven’t actually tried this myself, but there’s nothing preventing it. You’ll need to change the app.ini so that instead of using ‘src/kai/production.ini’ as the runner, it uses a Python file in the directory, say that then loads the Django app as a WSGI application and returns it. Sort of like this. Note that the config vars Django needs for its database should then be present in os.environ when setting up the that Django uses.

If you’re looking through the toppcloud source code by now, you may have also noticed there’s an example app that uses PHP. There’s nothing holding back toppcloud from setting up mod_passenger and deploying Ruby apps at some point either should someone wish to add that feature.

dumb pipes

There’s been numerous mentions on various blogs and in the news about how much the cell phone carriers hate the concept of being nothing more than “dumb pipes” for wireless Internet and phone use. That means of course, that they’d no longer be competing on what phones you could use but instead solely on service quality and price… I think the same type of transition is in store for shared hosting providers and some of the boutique app deployment shops like heroku.

It’ll take awhile, toppcloud is very rough right now. But when you can d/l a nice little open-source package, run it, choose your choice of cloud provider (based on price + quality!), have it automatically setup for you to deploy your apps to, then deploy apps with a single command… you’ve already done the vast majority of what a service like heroku does, except you could still modify toppcloud if there was something lacking you really needed. And it’s a reason like that, that toppcloud exists to begin with.

Oh, and thanks Ian for writing this before I wasted more time making it myself.

Thu, 07 Jan 2010 00:00:00 -0800 <![CDATA[Advanced Caching with Django and Beaker]]>

Advanced Caching with Django and Beaker

After seeing more than a few blog posts and packages attempt to provide more advanced caching capability for Django, it occurs to me I should actually just blog how to use Beaker in Django, rather than just keep mumbling about how “Beaker already does that”. So, if you’ve needed caching in Django that goes beyond using just one backend at a time, or maybe can actually cope with the Dog-Pile Effect, this is the blog entry for you (Until I flesh it out further into actual docs on the Beaker site).

Install Beaker

This is simple enough, if you have easy_install available, just:

easy_install -U Beaker

Or if you prefer to download tar files, grab the Beaker 1.4 tar.gz file

Configuring the Cache

Setting up Beaker’s cache for your Django app is pretty easy. Since only a single cache instance is needed for an app, we’ll set it up as a module global.

Create a file in your Django project with the following contents:

from beaker.cache import CacheManager
from beaker.util import parse_cache_config_options

cache_opts = {
    'cache.type': 'file',
    'cache.data_dir': 'cache/data',
    'cache.lock_dir': 'cache/lock'

cache = CacheManager(**parse_cache_config_options(cache_opts))

There’s a lot more options available, such as memcached, configuring multiple cache backends at once, etc. Now that you know how to provide the configuration options, further customization can be done as needed using the Beaker configuration docs. (Note the very handy cache region configurations which make it easy to toggle cache backend configurations on the fly!)

Using the Cache

Beaker provides a conveinent decorator API to make it easy to cache the results of functions. In this example we’ll just sleep and make a string including the time, add this to your

import time
from datetime import datetime

from django.http import HttpResponse

from YOURPROJECT.beakercache import cache

def hello(request):
    def fetch_data():
        return 'Hello world, its %s' %
    results = fetch_data()
    return HttpResponse(results)

In this case, the cached data is in a nested function with the decorator. It could of course be in the module elsewhere as well.

Hook the view function up in your, and hit the view. The first time it will wait a few seconds, then it will return the old time until the cache expires (10 seconds in this case).

The cached function can also accept positional (non-keyword) arguments, which will be used to key the cache. That is, different argument values will result in different copies of the cache that require those arguments to match.

That’s it, it’s really quite easy to use.

Update: It occurs to me this post does say ‘advanced’, and that example wasn’t very advanced, so here’s something a bit more interesting. Let’s configure cache regions to make it easy to toggle how long and where something is cached. Cache regions allow you to arbitrarily configure batches of settings, a ‘region’. Later you can then indicate you want to use that region, and it uses the settings you configured. This also make its easy to setup cache policies and change them in a single location.

In this case, we’ll have ‘long_term’ and ‘short_term’ cache settings, though you can of course come up with as many regions as desired, with the name of your choice. We’ll have the long_term settings use the filesystem, since we want to retain the results for quite awhile and not have them pushed out like memcached does. The short_term settings will use memcached, and be cached for only 2 minutes, long enough to help out on those random slashdog/digg hits.

In the file:

from beaker.cache import CacheManager
from beaker.util import parse_cache_config_options

cache_opts = {
    'cache.type': 'file',
    'cache.data_dir': 'cache/data',
    'cache.lock_dir': 'cache/lock',
    'cache.regions': 'short_term, long_term',
    'cache.short_term.type': 'ext:memcached',
    'cache.short_term.url': '',
    'cache.short_term.expire': '1200',
    'cache.long_term.type': 'file',
    'cache.long_term.expire': '86400',

cache = CacheManager(**parse_cache_config_options(cache_opts))

Now in our

import time
from datetime import datetime

from django.http import HttpResponse

from testdjango.beakercache import cache

def hello(request):
    def fetch_data():
        return 'Hello world, its %s' %
    results = fetch_data()
    return HttpResponse(results)

def goodbye(request):
    def fetch_data():
        return 'Bye world, its %s' %
    results = fetch_data()
    return HttpResponse(results)
Thu, 13 Aug 2009 00:00:00 -0700 <![CDATA[Beaker 1.4 Released]]>

Beaker 1.4 Released

Beaker 1.4 has now been released, and addresses several fairly important bugs. First, the full changelog:

  • Fix bug with hmac on Python 2.4. Patch from toshio, closes ticket #2133 from the TurboGears2 Trac.
  • Fix bug with occasional ValueError from FileNamespaceManager.do_open. Fixes #10.
  • Fixed bug with session files being saved despite being new and not saved.
  • Fixed bug with CacheMiddleware overwriting configuration with default arguments despite prior setting.
  • Fixed bug with SyntaxError not being caught properly in entry point discovery.
  • Changed to using BlobProperty for Google Datastore.
  • Added domain/path properties to the session. This allows one to dynamically set the cookie’s domain and/or path on the fly, which will then be set on the cookie for the session.
  • Added support for cookie-based sessions in Jython via the JCE (Java Cryptography Extensions). Patch from Alex Grönholm.
  • Update Beaker database extensions to work with SQLAlchemy 0.6 PostgreSQL, and Jython.

Note that the beaker database extension now works on Jython, and the cookies for sessions can be set dynamically during a request (for sites that operate across multiple domains/sub-domains).

Most importantly though, a bug in the import of the Google back-end has been fixed, which caused installation failures on Beaker 1.3.x.

Docs can be found on the Beaker site.

To upgrade your Beaker with easy_install:

easy_install -U Beaker

This release is also notable as the majority of the fixes were contributed by several web framework communities. Thanks for the patches!

Fri, 24 Jul 2009 00:00:00 -0700 <![CDATA[Comments and Web Services]]>

Comments and Web Services

I’m trying out Disqus for comments on the blog, for one main reason, I just didn’t feel like implementing comments myself. I am somewhat wary of services that seem integral to a blog, like where the comments are, which is why I’ve been leery of using services like this for some time.

What happens if Disqus runs out of VC and goes belly-up? I’ve seen this with a few other web services, and no one using it is very happy when it occurs. On the other hand, I did find some irony in the fact that one of the features Disqus pitches users is, “Don’t lose your comments if the blog disappears”.

On the other hand, I really appreciate the capabilities a central service-based comment system brings. I’ve inadvertently used it on other blogs, and was very pleasantly surprised to get updates and actually be able to keep up on what was happening on the blogs I commented on.

Are other people worried about using services by new companies or am I just overly paranoid?

I’d almost feel better about it if I could pay five bucks a year or something, as I’d at least have some better reassurance that the company is actually making money like I do with Flickr. Or maybe it’d be nice if companies that are profitable to say so, though it occurs to me that even that isn’t a guarantee, as someone could come along and buy them up, then decide to terminate their functionality (like that one site that people used and were pissed about when Six Apart bought them and shut it down, but for some reason I can’t remember the name of it at all).

Mon, 29 Jun 2009 00:00:00 -0700 <![CDATA[Blog upgraded! Now running on Pylons + MongoDB]]>

Blog upgraded! Now running on Pylons + MongoDB

I’ve now updated the blog software powering my blog, which is very long overdue. In the past, this blog was run off Typo, which apparently now hosts their home site off the github (its moved a dozen times in the past 3+ years).

Typo always worked moderately well for me, however, I found it sluggish (Prolly Rails there), and incredibly horrid on ram usage. It was not at all unusual to see it running past 700 megs of RAM after running for just a few weeks, which is a bit annoying as the machine only has 4GB total and is running quite a few things.

After last weeks SF Python Meetup on MongoDB I figured it was time to get a little actual MongoDB usage under my belt. I also inadvertently implemented enough of the MovableType XMLRPC API as I didn’t want my app to be too extensive, just enough for me to post to my blog.

So in the end, I had a small set of requirements for the replacement:

  1. Not be horribly slow
  2. Not take up huge amounts of RAM
  3. Retain all existing URL’s (It really annoys me when people break their old links)
  4. Compatible with my blog software (MovableType / MetaWeblog XMLRPC API)
  5. Not screw up the RSS/Atom feeds and cause Planet Python to show all my posts as if they were new (I’ve seen this happen to a few people on occasion)

I wanted to build it myself, because of course, the world definitely needs more blog apps, and I wanted one that used MongoDB. So for those curious, here’s the source code to the blog app.

It’s rather rough, as its fairly custom built just for my needs, nor do I have any plans to expand it into some general purpose blog engine, with themes, etc. The only other thing pending at the moment is to add the ability to comment again, as I haven’t quite gotten that feature in yet. For those trying it out, the README should help get started, but its very rough (thus the name of the package).

Strings, Unidecode, and Slugs

When copying some functionality I needed from the Rails app (to retain URL compatibility), I noticed two things it did which I thought was handy. To convert a title into slug for the article, it used a fairly sophisticated scheme relying on two other packages.

First, was the use of a Ruby port of a Perl package called Text::Unidecode which is pretty cool, and converts UTF-8 chars into their closest ASCII version. I figured someone must’ve ported it to Python as well, and sure enough, someone did! It wasn’t on the cheeseshop though, which was unbearable for me, so I’ve posted it to the Cheeseshop so others can easy_install it.

Next up, was a Ruby library called stringex, which add’s a few things to Rails, including a string method called ‘to_url’. That method does a variety of transformations to remove all those funny characters from a title, and do a bunch of other neat changes of common characters to human readable versions (source for those conversions).

I ported the key module of stringex to Python, and it resides in my blog app. If someone would like to extract it and make it into its own package, or even better, if I somehow missed the fact that someone else has ported it already, let me know (tweet me @benbangert!).

I’ll be writing up my thoughts on making a small app with MongoDB, and how it differs from my experience working with CouchDB for PylonsHQ in a later post for those curious.

Wed, 17 Jun 2009 00:00:00 -0700 <![CDATA[Making a better TextArea]]>

Making a better TextArea

I’ve been working on some Javascript to enhance the TextArea elements on the PylonsHQ Snippets section, and have noticed that… well, TextArea’s suck.

The hack I’ve seen is to use one of the newer features of browsers, the editable or ‘design’ attribute’s for div elements I believe. This lets one build a very snazzy amount of features, such as syntax highlighting, code completion, etc., but I don’t think I needed to go that far.

I only have one main design goal, this TextArea is for the user to enter RestructuredText so it’d be awesome if the TextArea acted in a way that made rst a bit nicer. The obvious two things that came to mind:

  1. Tab key indents 4-spaces
  2. Hitting return on an indented line, will retain the indentation on the next line

I’ve actually gotten some Javascript, hobbled together from various parts of the net, along with an ‘enter’ key handler I wrote myself on bitbucket.

Course, it’d also be nice to have a button or key combo, that will indent/unindent a selection in the TextArea as well.

Anyone else have any Javascript they’ve hobbled together in the past to make TextArea a little nicer for restructured text?

Sun, 31 May 2009 00:00:00 -0700 <![CDATA[Beaker 1.3 is juicy caching goodness]]>

Beaker 1.3 is juicy caching goodness

Beaker 1.3 is out, actually, its been out for awhile and I’m just not getting around to blogging the fact. It’s a shame I’ve been a bit too busy lately to blog this earlier because in addition to some bug fixes it has some nice new features that make it even easier to use in any Python script/application/framework.

First, to air my dirty laundry, the important bug fixes in Beaker 1.3:

  • Fixed bug with (non-cookie-only) sessions not being timed out
  • Fixed bug with cookie-only sessions sending cookies when they weren’t supposed to
  • Fixed bug with non-auto sessions not properly storing their last accessed time

The worst thing with the first two of these is that they were regressions that snuck in despite unit tests that exercised the code fairly decently. They’re fixed now along with more comprehensive tests to help prevent such regressions occurring again.

Beaker has always had session’s, and caching, but except for Pylons I’ve yet to see anyone actually use Beaker’s caching utility. I’ve seen the SessionMiddleware used in other WSGI based frameworks, but not the caching, which is kind of a shame since it:

  • Supports various backends: database, file-based, dbm file-based, memcached
  • Has locking code to ensure a single-writer, multiple reader model (This avoids the dreaded dog-pile effect that caching systems such as the one in Django experience!)

For clients that hit the cached function while its already being regenerated, Beaker serves the old copy until the new content is ready. This avoids the dog-pile effect, and keeps the site snappy for as many users as possible. Since the lock used is disk-based though, this does mean you only avoid the effect per machine (unless you’re locking against NFS or a SAN), so if you have 20 machines in a cluster, the worst the dog-pile effect can get is that you’ll have 20 new copies generated and stored.

Now, in Beaker 1.3, to try and encourage its use a bit more, I’ve added a few decorators to make it easier to cache function results. Also with Mike Bayer’s suggestion, there is now cache regions to make it easier to define various caching policy short-cuts.

Cache Regions

Cache regions are just pre-defined sets of cache instructions to make it easier to use with your code. For example many people have a few common types of cache parameters they want to use:

  • Long-term, likely to a database back-end (if used in a cluster)
  • Short-term, not cached as long, perhaps to memcached

To set these up, just tell Beaker that about the regions you’re going to define, and give them the normal Beaker cache parameters for each region. For example, in this Pylons app, I define 2 cache regions in the INI:

beaker.cache.regions = short_term, long_term
beaker.cache.short_term.type = ext:memcached
beaker.cache.short_term.url =
beaker.cache.short_term.expire = 3600

beaker.cache.long_term.type = file
beaker.cache.long_term.expire = 86400

Note: For those wondering about multiple memcached servers, just put them in as the url with a semi-colon separating them.

If you want to use the caching outside of Pylons without middleware (ie, as a plain library), that’s a bit easier now as well:

from beaker.cache import CacheManager
from beaker.util import parse_cache_config_options

cache_opts = {'cache.data_dir': './cache',
              'cache.type': 'file',
              'cache.regions': 'short_term', 'long_term',
              'cache.short_term.type': 'ext:memcached',
              'cache.short_term.url': '',
              'cache.short_term.expire': '3600',
              'cache.long_term.type': 'file',
              'cache.long_term.expire': '86400',

cache = CachManager(**parse_cache_config_options(cache_opts))

And your cache instance is now ready to use. Note that using this cache object is thread-safe already, so you just need to keep one around in your framework/app (Can someone using Django explain where you’d keep a reference to this object around so that you could get to it in a Django view?).

New Cache Decorators

To make it easier to use caching in your app, Beaker now includes decorators for use with the cache object. Given the above caching setup, lets assume you want to cache the output of an expensive operation:

# Get that cache object from wherever you put it, maybe its in environ or request?
# In Pylons, this will be: from pylons import cache
from wherever import cache

def regular_function():
    # Do some boring stuff

    # Cache something
    @cache.region('short_term', 'mysearch')
    def expensive_search(phrase):
        # Do lookup with the phrase variable
        return something
    return expensive_search('frogs')

The second argument to the region decorator, ‘mysearch’. That isn’t required unless you have two function’s of the same name in the same module, since Beaker records the namespace of the cache using the function name + module + extra args. For those wondering what a Beaker namespace is, its a single cache ‘block’. That is, lets say you wanted to cache 4 versions of the same thing, but change them differently depending on the input parameter. Beaker considers the thing to be a namespace, and the things that change the thing being cached are the cache keys.

Only un-named arguments are allowed on the function being cached. These act as the cache keys so that if the arguments change, a new copy is cached to those arguments. This way you can have multiple versions of the function output cached depending on the argument it was called with.

If you want to use arbitrary cache parameters, use the other decorator:

# Get that cache object from wherever you put it, maybe its in environ or request?
# In Pylons, this will be: from pylons import cache
from wherever import cache

def regular_function():
    # Do some boring stuff

    # Cache something
    @cache.cache('mysearch', type='file', expire=3600)
    def expensive_search(phrase):
        # Do lookup with the phrase variable
        return something
    return expensive_search('frogs')

This allows you to toggle the cache options per use as desired.

If there’s anything else I can do to make it easier to use Beaker in your application, be sure to let me know (Yes, I know more docs would help, this blog post was a first attempt to help out on that front, more docs on the way!).

Thu, 23 Apr 2009 00:00:00 -0700 <![CDATA[Pylons 0.9.7 Released]]>

Pylons 0.9.7 Released

I’m pleased to announce after a rather lengthy release candidate period, that Pylons 0.9.7 is finally out. Pylons 0.9.7 brings a good amount of changes to Pylons from 0.9.6 while still retaining a fairly hefty amount of backwards compatibility to ensure a mostly painless upgrade.

Some helpful documentation on the new release:

Major changes in 0.9.7:

  • Switched to using WebOb for the request/response object
  • Various performance improvements to object initialization
  • Beaker and Routes updates
  • Middleware improvements, and optimizations

This is a huge step forward for Pylons, and I’d like to thank all of the contributers who have helped make Pylons what it is today. We’ve knocked off more bugs for this release than any before, which shows just how far the Pylons community has come:

  • 0.9.5 tickets: 45
  • 0.9.6 tickets: 64
  • 0.9.7 tickets: 160

And we have finally made a huge dent in the historical “lack of docs” problem that Pylons previously suffered from with the new Sphinx generated docs and a comprehensive Pylons book.

The full changelog which describes the major changes (Look for the bits marked with WARNING that might affect backwards compatibility).

0.9.7 (February 23, 2009)

  • WARNING: A new option is available to determine whether or not an actions

arguments should be automatically attached to ‘c’. To turn off this implicit behavior in config[‘pylons.c_attach_args’] = False This is set to True by default.

  • WARNING: Fixed a minor security hole in the default Pylons error page that

could result in an XSS security hole.

  • WARNING: Fixed a security hole in the default project template to use the

StaticURLParser to ensure arbitrary files can’t be sent.

  • WARNING: Refactored PylonsApp to remove legacy PylonsApp, moved

session/cache and routes middleware into the project template. This will require projects to be updated to include those 3 middleware in the projects

  • Changed to using WebTest instead of paste.fixture for app testing.
  • Added render_mako_def to render def blocks within a mako template.
  • Changes to cache_decorator and cached_template to support Beaker API

changes in version 1.1. 1.0.3 is still supported.

  • Fix HEAD requests causing an Exception as if no content was returned

by the controller. Fixes #507. Thanks mvtellingen, Petr Kobalicek.

  • Fix a crash when returning the result of “etag_cache“ in a controller.

Fixes #508.

  • “response” flag has been removed from pylons.decorators.cache.beaker_cache,

as it sends all headers along unconditionally including cookies; additionally, the flag was taking effect in all cases previously so prior versions of beaker_cache are not secure. In its place, a new option “cache_headers” is provided, which is a tuple of specific header names to be cached. It defaults to (‘content-type’,’content-length’).

  • “invalidate_on_startup” flag added to beaker_cache, which provides a

“starttime” to the cache such that when the application is started or restarted, the cache entry is invalidated.

  • Updating host to use for development binding.
  • Added option to specify the controller name with a controller variable

in the controller’s module. This name will be used for the controller class rather than the default naming scheme.

  • egg_info now restores projects’ paster_plugins.txt,

allowing paster shell to work again after the egg-info directory was lost. fixes #282. Thanks sevkin.

  • The paste_deploy_config.ini_tmpl template is now located at

package/config/deployment.ini_tmpl for new projects.

  • Project’s default test fixtures no longer hardcode test.ini; the ini

file used can now be specified via the nosetests —with-pylons argument (defaults to test.ini in setup.cfg). fixes #400.

  • ``validate now defaults to translating FormEncode error messages via Pylons’ gettext catalog, then falls back to FormEncode’s. fixes #296. Thanks Max Ischenko. * Fixed SQLAlchemy logging not working in paster shell. Fixes #363. Thanks Christoph Haas. * Added optionally engine initialization, to prevent Buffet from loading if there’s no ‘buffet.template_engines’ in the config. * Updated minimal template to work with Tempita and other new templating changes. * Fixed websetup to parse location config file properly when the section isn’t ‘main’. Fixes #399. * Added default Mako filter of escape for all template rendering. * Fixed template for Session.remove inclusion when using SA. Fixed render_genshi to properly use fragment/format options. Thanks Antonin Enfrun. * Remove template engine from load_environment call. * Removing template controller from projects. Fixes #383. * Added signed_cookie method to WebOb Request/Response sub-classes. * Updated project template to setup appropriate template loader and controller template to doc how to import render. * Added documentation for render functions in pylons.templating. * Adding specific render functions that don’t require Buffet. * Added forward controller.util function for forwarding the request to WSGI apps. Fixes #355. * Added default input encoding for Mako to utf-8. Suggested in #348. * Fixed paster controller to raise an error if the controller for it already exists. Fixes #279. * Added to template dir in project template if the template engine is genshi or kid. Fixes #353. * Fixed jsonify to use application/json as its the proper mime-type and now used all over the net. * Fixed minimal template not replacing variables properly. Fixes #377. * Fixed ``validate decorator to no longer catch exceptions should they be

raised in the action that is supposed to display a form. Fixes #374.

  • Fixed paster shell command to no longer search for egg_info dir. Allows

usage of paster shell with installed packages. Suggested by Gavin Carothers.

  • Added mimetype function and MIMETypes class for registering mimetypes.
  • WARNING: Usage of pylons.Response is now deprecated. Please use

pylons.response instead.

  • Removed use of WSGIRequest/WSGIResponse and replaced with WebOb subclasses

that implement methods to make it backwards compatible with the Paste wsgiwrappers.

  • Fixed missing import in template controller.
  • Deprecated function uses string substitution to avoid Nonetype error when

Python optimization is on. Fixes #334.

  • E-tag cache no longer returns Content-Type in the headers. Fixes #323.
  • XMLRPCController now properly includes the Content-Length of the response.

Fixes #310, thanks Nicholas.

  • Added SQLAlchemy option to template, which adds SQLAlchemy setup to the

project template.

  • Switched project templating to use Tempita.
  • Updated abort/redirect_to to use appropriate Response object when WebOb is


  • Updated so that 404’s properly return as Response objects when WebOb is in

use instead of WSGIResponse.

  • Added beaker_cache option to avoid caching/restoring global Response values

that were present during the first cache operation.

  • Adding StatusCodeRedirect to handle internal redirects based on the status

code returned by the app. This replaces the use of ErrorDocuments in projects.

  • Refactored error exceptions to use WebError.
  • WSGIController now uses the environ references to response, request, and

the c object for higher performance.

  • Added optional use of WebOb instead of paste.wsgiwrapper objects.
  • Fixed bug with beaker_cache defaulting to dbm rather than the beaker

cache app-wide default.

  • The —with-pylons nose plugin no longer requires a project to have been

registered with setuptools to work.

  • The config object is now included in the template namespace.
  • StaticJavascripts now accepts keyword arguments for StaticURLParser.

Suggested by Marcin Kasperski.

  • Fix pylons.database.AutoConnectHub’s doInTransaction not automatically

connecting when necessary. Fixes #327.

Mon, 23 Feb 2009 00:00:00 -0800 <![CDATA[New PylonsHQ Site Launches]]>

New PylonsHQ Site Launches

The new PylonsHQ site has now launched!

The new site is running on the latest Pylons 0.9.7 code-base backed by the CouchDB database. New features that have been added:

Unfortunately, we were unable to integrate the Wiki’s auth, so that will still require a separate login for now.

Comments are through-out the site, to ensure that feedback isn’t missed and of course there’s many more features planned that are coming soon. The site isn’t quite 100% complete, as a few links here and there are likely broken yet (like the tutorial links on the front page). I’ll be putting out frequent updates to remedy this and any other little bits that need more polish.


Wed, 21 Jan 2009 00:00:00 -0800 <![CDATA[Public launch of Stanford Intellectual Property Litigation Clearinghouse]]>

Public launch of Stanford Intellectual Property Litigation Clearinghouse

At long last, we’ve finally launched the Stanford IPLC website that I’ve been working on for the past year. It’s quite nice to finally have something out there that I can show people, though I know its definitely more of a niche area of interest, as not everyone is probably as interested in intellectual property litigation as I am. :)

This site is running Pylons of course, with various other technologies I’m unable to disclose powering the back-end.

Note that signing up requires a valid e-mail address as e-mail confirmations are sent out to them. For those that are keen to keep up on what’s going on with patent litigation, hopefully our website can help out.

Tue, 09 Dec 2008 00:00:00 -0800 <![CDATA[WSGI/Pylons Talk in San Francisco tonight (Dec 3rd)]]>

WSGI/Pylons Talk in San Francisco tonight (Dec 3rd)

I’ll be giving a talk tonight about WSGI, making apps using it, and Pylons tonight in San Francisco. If you live in the Bay Area and have been wanting to learn more about some of the packages utilizing WSGI, as well as Pylons, RSVP soon as they’d like to get some numbers for food.

Here’s the meetup event page for the talk. This is one of the longer talks I’ve given, so I should actually have enough time to cover the topics (WSGI, WSGI Middleware, making low-level WSGI apps with WebOb, making small WSGI stacks, and Pylons) in good detail, as well as some demonstrations.

Wed, 03 Dec 2008 00:00:00 -0800 <![CDATA[Book Meme]]>

Book Meme

Via :

  • Grab the nearest book.
  • Open it to page 56.
  • Find the fifth sentence.
  • Post the text of the sentence in your journal along with these instructions.
  • Don’t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

From The Omnivore’s Deilemma by Michael Pollan:

And then of course there’s the corn itself, which if corn could form an opinion would surely marvel at the absurdity of it all — and at its great good fortune.

I don’t usually do the meme thing, but I happened to have a book on my desk so it took so little effort… only one chapter to go!

Thu, 13 Nov 2008 00:00:00 -0800 <![CDATA[Python 2.6 release and Pylons buildbots]]>

Python 2.6 release and Pylons buildbots

Python 2.6 came out yesterday, so I figured I might as well see if Pylons works on it. Pylons already has a set of buildbots that builds Pylons along with some of its dependencies, so it was fairly trivial to add another builder to verify things ran swimmingly on Python 2.6.

Unfortunately, as one can see looking at the build results, things weren’t so great. It appears that nose had a Python 2.6 incompatibility which is used to run all the various Pylons tests, meaning that they all failed so far mainly because the testing tool was Python 2.6 incompatible.

Making Buildbot nicer

While I wait for the new nose to be released, I did at least discover a little bug in my new webapp that provides a nicer view of the buildbot result set. I’ve been fairly displeased with the lack of conciseness of buildbot’s waterfall display for awhile, and noticed that if only buildbot had a few more xmlrpc methods then it’d be trivial to build my own more kind interface.

I should note that the waterfall display isn’t totally horrible, the Django folks spiced their builders up with some CSS work…. which reminds me, it isn’t looking very good for those running on trunk at the moment. ;)

So after making my own little buildbot fork to add some additional custom xmlrpc methods to, I’ve come up with my own buildbot status viewer. I’m sure a more talented designer could spice it up even more, but it gives me the pertinent data I’m interested in without all the boring “builder connected, builder took a vacation” messages that cloud up the waterfall. Also, rather than displaying the cryptic “shell_21 failed” messages, it actually uses the names I attached, and shows them quite clearly for the last build.

I’ll submit some patches for these xmlrpc additions to buildbot when I get the time, but right now I mainly needed the Mercurial 1.0 hook compatibility (that was broken for quite awhile in buildbot), and a fairly specific set of information from the xmlrpc methods that I wasn’t sure others would want.

I’m looking forward to trying out the new nose so that I can hopefully verify Pylons is good to go on Python 2.6 as Phil Jenvey’s been working tirelessly on patches to Beaker and other dependencies to make them 2.6 compatible. Any suggestions or thoughts on improving my buildbot viewer are welcome. :)

Fri, 03 Oct 2008 00:00:00 -0700 <![CDATA[The Revised Bailout]]>

The Revised Bailout

Amazingly enough, despite all the hoop-la of how the bailout will pass, on Monday, it failed. It was quite entertaining to me to see a bill that everyone seemed so sure would pass, to just go and fail.

In the meantime I’ve found more than a few interesting links that only go to re-inforce my prior thoughts on the bailout.

The vagueness I noted in some sections of the bailout was quite intentional.

Even as policy makers worked on details of a $700 billion bailout of the financial industry, Wall Street began looking for ways to profit from it.

Ouch, well that sure is reassuring.

How’d they do? Pretty dang good. They made sure no pesky regulations would affect their golden parachutes, or reduce their paychecks, and they expanded the definition of what assets could be bought to extend beyond just mortgage related securities, to any troubled asset. If some bank has a laundry list of dumb investments that aren’t paying off well, they’re as eligible as anything else if the Treasury considers them at risk of failing (though even what kind of ‘risk of failing’ is required before the Treasury buys such assets seems vague).

Isn’t the point of the free market to let companies that do stupid stuff fail from it?

The NY Times also mentions that many of the financial institutions want to manage the assets that are bought as well, and get paid for it of course, which would generate them a very hefty profit as well.

I haven’t seen a lot in the mainstream media about the huge list of economists (over 165 and counting) who oppose the bailout, which is curious.

So far, this countries track record of hastily passing massive and far reaching pieces of legislation based on hysteria hasn’t worked out too well. It has resulted in a war and all sorts of legislation that is still having ill effects on basic liberties and freedoms. The revised bill which includes more bits to appeal to Republicans doesn’t seem any better, and is still co-written by the financial industry eager to get some more free money.

Ron Paul has some good thoughts on why the bailout is a bad idea, and Michael Moore has an amusing write-up of how to save the banks with plenty of good facts and of a few questionable ones (but overall a good read). I particularly like his #2 point for paying for the bailout.

I disagree with Michael Moore though, mainly because I think Ron Paul and the economists have the right point. Investors should not be subsidized. Investors making risky bets, should take their losses. They clearly want to keep the profit, let them keep the loss as well.

Regardless, I hope the new version doesn’t get passed as well, though given how disappointing the politicians have been for me and the fact that the general public got creeped out watching the investors having a fire-sale means that the financial industry will likely be able to get the new bill passed. Nothing like some hysteria in the stock market to help some legislation creep through.


Update: A rather interesting look at the distribution of wealth in America, figure 5 showing the difference increasing.

Update 2: I should note that the new bailout bill that has had tastier pork tossed in, increased in size by $110 billion, and in size from the 101 page version I read, to a new 451 page bill.

That pretty much negates the odds of most people having enough time to read it, and who knows what other tidbits got buried in it.

Wed, 01 Oct 2008 00:00:00 -0700 <![CDATA[A Quick Look at the Bailout Draft]]>

A Quick Look at the Bailout Draft

With the upcoming expected passage of the bailout, I noticed that the draft text was online so I decided to take a look. There’s some thoughtful provisions that the first bailout plan was lacking, such as an attempt to prevent financial institutions from making an actual profit with plenty of loopholes so they don’t look terribly effective.

Unjust Enrichment

Consider this first attempt to prevent the companies helped, from actually getting a little ‘too much help’:

PREVENTINGUNJUSTENRICHMENT.—In making purchases under the authority of this Act, the Secretary shall take such steps as may be necessary to prevent un-just enrichment of financial institutions participating in a program established under this section, including by preventing the sale of a troubled asset to the Secretary at a higher price than what the seller paid to purchase the asset. This subsection does not apply to troubled assets acquired in a merger or acquisition, or a purchase of assets from a financial institution in conservatorship or receivership, or that has initiated bankruptcy proceedings under title 11, United States Code.

So the Treasury Secretary can’t buy up troubled assets for more than the seller bought them at (who knows how little they’re really worth), unless the assets were acquired in a merger or acquisition. Now, just think about how massive that loophole is… given how many of these financial institutions have been buying each other up. I really hope they don’t pay anywhere near full price for these assets, as they’ve dropped in value significantly since they were issued… and this attempt to stop them from overpaying for them seems rather half-hearted.

Transparency and Review

Unlike the original bill the Treasury wanted, this one sets up a oversight board responsible for reviewing the actions of the Secretary and ensuring there’s compliance with the rest of the act. The oversight board then has to report back to the Congressional Oversight Panel (they sure do love panels and boards). Reports covering purchases and justifications need to be sent to the Congressional Oversight Panel everytime the Treasury uses $50 billion dollars.

Not too shabby on oversight…. I wonder if the report will be put online so the people can see where their money is going?

Attempting to Reign in the foreclosures

A decent chunk of the bill includes provisions attempting to keep people in their homes, rather than foreclosing. This makes a hell of a lot of sense to me, as the more foreclosures that occur, more and more bailouts will be needed.

One of these bits allows for loan modifications of the mortgage:

(2) MODIFICATIONS.—In the case of a residential mortgage loan, modifications made under paragraph (1) may include: A) reduction in interest rates; B) reduction of loan principal; and C) other similar modifications.

Curbing executive compensation

I was actually surprised that lawmakers would do anything to offend the executives at these companies, who shell out so much money to lobbyists that inevitably comes back to the lawmakers, so I’m eying this section with skepticism as I’m sure there’s more than a few loopholes.

The standards required under this subsection shall be effective for the duration of the period that the Secretary holds an equity or debt position in the financial institution.

I’m curious how easy it is for the Treasury to buy troubled assets, and try and get out of a position that qualifies in this case…

Ah, and here’s one rather interesting limitation of the executive compensation, apparently it has nothing to do with how many millions the executive makes, but merely the top 5 executives for a company:

(3) DEFINITION.—For purposes of this section, the term ‘‘senior executive officer’’ means an individual who is one of the top 5 executives of a public company, whose compensated is required to be disclosed pursuant to the Securities Exchange Act of 1934, and any regulations issued thereunder, and non-public company counterparts.

I really don’t understand why they stopped at the top 5. How about restricting any executive (VP, Senior VP, Partner, etc) from getting those crazy incentives? Why not put a cap on everyone in the company on payment, the fairly reasonable one that they can’t make more than the top paid US Govt official (currently the President, at $400k/yr)?

The good news is that it does reign in some of the crazy executive perks to an extent, like the golden parachutes, and millions of bonuses. CNN reported that in the past few years, executive compensation has gone up 20%, while earnings for the companies went up 3%. Odd how the rest of the employee’s don’t see 20% raises….

The limits are also a little odd, it says that while the Treasury holds assets of the company, they will be in effect:

(A) limits on compensation that exclude incentives for executive officers of a financial institution to take unnecessary and excessive risks that threaten the value of the financial institution during the period that the Secretary holds an equity or debt position in the financial institution;

That’s great and all…. but I’m having trouble finding anywhere defining what these limits actually are.

More Transparency (Section 114)

Ah ha, a mere 39 pages in, a bit more on transparency. The Secretary has to make available to the public, in electronic form, full descriptions, amounts, and pricing of assets that are being bought, within 2 business days of the purchase.

If only the executive branch was a bit more transparent…. :)

How much is it really??

The first chunk of money the Treasury gets to spend, is $250 billion. Later, when its running low supposedly, the President can send a written certification to the Congress that it needs more, and the limit on outstanding money can be raised to $350 billion outstanding. After that, the Pres can write in again, and the amount can be raised to a limit of $700 billion outstanding. So in effect, its a $700 billion bail-out built without any actual data as to whether thats what it will take.

The remaining 30 pages is mainly details on which positions shall oversee who, how often, who’s part of them, etc. Nothing terribly exciting stands out here, nor did I find any explanation of exactly what limits on executive compensation are being applied. Was it just golden parachutes and bonuses?

CNN took a look, and apparently came to a similar conclusion in their reading that the companies aren’t allowed to write new ‘golden parachutes’ for their top 5 executives…. but the current contracts which may include golden parachutes are just dandy.

They also indicate that the companies will not be able to deduct the salary they pay to executives above $500k. Errr, “deduct the salary”? Not sure what that means, hopefully it means that they’re capped at $500k, but its hard to tell.

It still sucks

Overall, I’m still not happy with it. Nor are all these folks, who happen to be economists. There’s some good points in that letter as well:

In addition to the moral hazard inherent in the proposal, the plan makes it difficult to move resources to more highly valued uses. Successful firms that may have been in a position to acquire troubled firms would no longer have a market advantage allowing them to do so; instead, entities that were struggling would now be shored up and competing on equal footing with their more efficient competitors.

This bail out bill is definitely better than the prior one, but I think its still a waste of my taxpayer money. There’s zero guarantee it will work, zero guarantee the money will ever come back, and zero guarantee this will be the last bail-out of this magnitude we come across. While the bill has some measures to try and decrease foreclosures, most in the housing industry still believe the worst is yet to come

“We’ve been saying that the foreclosure trend has not yet peaked,” said Doug Robinson, a spokesman for the foreclosure prevention organization NeighborWorks America. “Before it was a subprime problem,” he said. “Now, it’s everybody’s problem.”

Ouch. So this bail-out only helps a few companies deal with the current problem. I really don’t want to know what will the Treasury ask the taxpayer to do next to stop the companies that hold the upcoming foreclosures from going bankrupt.

Mon, 29 Sep 2008 00:00:00 -0700 <![CDATA[Ringtone sync FAIL]]>

Ringtone sync FAIL

I’ve had an iPhone for about a month now, quite loving it. Today I decided to add some custom ringtones to it, so I went into iTunes and clicked the button to sync ringtones…

` <>`_



Really? They couldn’t find any way to add a few ringtones without removing (erasing) all the music I already dragged over? Really???

Thu, 25 Sep 2008 00:00:00 -0700 <![CDATA[Pylons 0.9.7rc1 Release]]>

Pylons 0.9.7rc1 Release

Pylons 0.9.7rc1 was released a week ago, unfortunately I haven’t had time to actually blog it so better late than never. This is a big step towards the 0.9.7 release, and contains some major changes over 0.9.6 while still retaining a huge degree of backwards compatibility.

At this point, the thing I get asked the most is:

When will Pylons 0.9.7 be released?

So the short answer, when the new website and docs are ready. We’re going to a lot of effort to totally eradicate that old mantra that “Pylons has no docs”, and we’re doing it big. Most of the docs have already been updated, revamped, and moved to the new Sphinx doc tool (Take a look at the new Pylons docs).

The new website is nearing completion as well, and for those using the 0.9.7 release candidate, when posting a traceback you’ll get a link to it thats on the new beta website. Until then, 0.9.7 is feature-frozen and newer RC’s up to 0.9.7 are bug-fix only.

New Features

Pylons gets the substantial amount of its feature-set from the other Python libraries it uses, and here’s some of the new things these libraries have brought Pylons users:

This is a huge update, including safely escaped HTML builders, a literal object to mark strings as safe (vs unsafe) for use in templating languages, and a move away from all the old ported Rails helpers to new ones that in many cases have more features with less bugginess

  • Routes 1.9, with minimization turned off. This helps for more predictable route generation and matching which confused many, and in some cases led to hard-to-debug routes being created and matched. The new syntax available also breaks with the Rails’ish Routes form, and lets you easily include regexp requirements for parts of the URL.
  • Mako Automatic Safe HTML Escaping
  • Simplified rendering setup that doesn’t use Buffet
  • Simplified middleware setup with easier customizability
  • Simplified PylonsApp for customizing dispatch and URL resolving
  • and lots of bug fixes!

There’s a more detailed page covering 0.9.7 changes available as well that can also assist in the rather minimal change needed for a 0.9.6 project to get going with 0.9.7rc1.

Other things in Pylons-land

With TurboGears2 extending Pylons for its foundation, many various parts of TG2 have become usable within Pylons, not to mention existing packages that have been getting better and better.

ToscaWidgets has gotten drastically simpler, no longer requiring the rather confusing RuleDispatch package with its generic methods. This makes the tw.forms package install with a fraction of the packages it used to require, and since it comes with Mako templates won’t incur any speed bumps it used to have from its use of Genshi. The new Pylons tutorials for it also make it a breeze to quickly create large forms with advanced widgets.

Some might have noticed that Reddit released their source code, which happens to be in Pylons. Their code is a good example of some of the customizing possible with a Pylons based project, as they added some custom dispatching to make controllers work in a more similar fashion to controllers that they ported their app from. In a way, its similar to how TG2 has been able to support TG1 users for the most part by customizing Pylons to dispatch in a TG1 style manner.

Profiling an application got a lot easier with repoze.profile, and I’m sure more cool bits of WSGI middleware will be coming out of the repoze project in the future, not including some of the past handy bits like repoze.who which is used in TG2 for its new identity system.

I ported a little app that Robert Brewer wrote to track memory leaks. Being terribly uncreative on names for my new WSGI middleware version, I called it Dozer. It’s a handy little piece of WSGI middleware to throw in when you think you might have a memory leak to try and sort it out.

Pylons is moving along quite nicely, and the amount of WSGI middleware and tools that work with it continue to expand which makes it hard to list all the cool new projects I’ve seen lately that work wonderfully with Pylons.

Mako and SQLAlchemy continue to evolve with Mako having pretty much zero backwards incompatible changes in the past 6+ months, while SQLAlchemy slowly deprecates things as they prepare the 0.5 release. These packages have massive amounts of features and are rapidly becoming very stable easily making Pylons + Mako + SQLAlchemy a tough combination to beat.

Wed, 27 Aug 2008 00:00:00 -0700 <![CDATA[Routes 1.9 Release]]>

Routes 1.9 Release

I released Routes 1.9 today, which is another step on the Road to Routes 2.0. Some of the highlights that people will be most interested that I had previously blogged about now available:

Minmization is optional

Pylons 0.9.7 will default to turning minimization off (projects are free to leave it on if desired). This means that constructing a route like this with minimization off:


will actually require both the controller and the action to be present, and the trailing slash. This addresses the trailing slash issue I wanted to fix as well.

Named Routes will always use the route named

This is now on by default in Routes 1.9, which results in faster url_for calls as well as the predictability that comes with knowing exactly which route will be used.

Optional non-Rails’ish syntax

You can now specify route paths in the same syntax that Routes 2 will be using:


Or if you wanted to include the requirement that the id should be 2 digits:


Routes automatically builds the appropriate regular expression for you, keeping your routes a lot easier to skim over than a bunch of regular expressions.

Routes 2 will be bringing redirect routes, and generation-only routes, making Routes 1.9 a great way to transition to Routes 2 when its ready.

Sat, 14 Jun 2008 00:00:00 -0700 <![CDATA[Pylons on JVM's (and other VMs)]]>

Pylons on JVM’s (and other VMs)

Phil Jenvey has been making some great progress getting all the components of Pylons running on Jython, and posted a good write-up of the remaining work being done. It’s interesting to note that one of the big issues will affect any web framework on Jython, not just Pylons. That is, the reload time when used in development to restart the server.

While I don’t plan on deploying Pylons apps in WAR files anytime soon, its nice to see Jython emerging as a candidate for deployment.

Wed, 07 May 2008 00:00:00 -0700 <![CDATA[Most bizarre Git service and other stupid Rails powered "businesses"]]>

Most bizarre Git service and other stupid Rails powered “businesses”

I can’t help but get totally baffled when I see a business model like this.

Yes, that’s right, you can pay for the privilege of keeping a copy of your distributed version control system (DVCS) private repositories on someone else’s machines. You also get to pay depending on how many people you want to allow to collaborate on it.

Nevermind that one of the entire points of a DVCS is that you do NOT need a central repository. Does anyone actually work at a “Large Company” (as the page indicates) that would be stupid enough to pay $100/month so they can put all their proprietary and very personal code repositories on a third party web service?

So what are you paying for? Well, to start with, they have awesome integration with Lighthouse, since we all know there’s no decent free open-source issue tracking system… cough trac cough roundup cough. Oh wait, since there’s absolutely no simple web-based issue tracking systems, let’s have another slick business model to get people to pay for a stripped down Trac (but this time with a really pretty UI)!

What do these sites have in common? Rails, “look ma, I can copy-paste the business plan too” pricing models, and some good graphic designers at the helm. There also seems to be an interesting amount of promotion between these sites, as well as a nice blog post from the Rails creator himself promoting GitHub. I’m sure no one who has read this rant should be surprised though.

I only hope that no one starts to believe that a DVCS actually requires these “please pay” copies of their DVCS repo.

Update (11/12/2008): This post is apparently popular enough to come up on occasion several times now, so I thought I’d clarify a bit more.

Many people have suggested the obvious benefits of services like GitHUB, and I’ve used one just like it myself, BitBucket. These sites are great for open-source projects as many have rightfully pointed out, they make it easy to collaborate and fork projects, and easy for maintainers to pull patches from forks after looking them over.

Most of their social-network features become moot though when working on company code thats not open-source, (note that this rant is directed entirely at the paid service options which are for private repos). None of the companies I’ve worked at would ever let their private source code leave their own servers. Since you need to deploy a site anyways (many times to a remote computer), which will generally require ssh access, its trivial to use the modern DVCS’s over ssh…. which makes it seem very silly to me to be paying so much to another company for a bunch of useless social features for a private repo.

Part of the original humor intended in this rant was that a centralized repo hub has become one of the stronger selling points for a distributed VCS system. Unfortunately many seemed to have missed that point.

Wed, 07 May 2008 00:00:00 -0700 <![CDATA[Google Datastore and the shift from a RDBMS]]>

Google Datastore and the shift from a RDBMS

So many random musings and theories on Google App Engine, I won’t bother musing about it myself, except to mention that Ian Bicking put together instructions for running Pylons on it. These also work fine for using the latest Pylons 0.9.7 beta.

I got Beaker, the session and caching WSGI middleware that Pylons uses, running fine on Google now, using Google Datastore as the backend. Diving into the Datastore docs to get a grip on what’s the best way to implement it shed some light on the transition any developer thinking about writing data-backed apps for GAE (Google App Engine) will need to tackle.

Some notes on terminology, Google has Entities, Kinds, and Properties. These correspond roughly to Rows, Tables, and Columns in RDBMS-speak. Kinds can also be called classes, because in the Python API, you create a class and inherit from the appropriate datastore class. Entities may also be referred to as instances, since performing a query returns a list of objects (instances).

Sessions and Datastore

First, regarding sessions. Beaker will now let a Pylons app use normal sessions on GAE, the real question is, should you?

The Google User API makes it trivial to get currently logged in user, and the datastore comes with a property type for a ‘table’ that is specifically made for a Google user account reference. So with just one short command, you can have an entity from the Datastore that corresponds to a given user, ie:

userpref = UserPrefs.all().filter('user =', users.get_current_user()).get()

The Datastore is blindingly fast for reads and queries, so there’s a compelling reason to ignore sessions altogether and just fetch the appropriate preferences or what-have-you. This leaves people with the normal reason for wanting more, ie, a session, “But wait, I want to stash other little things with the user when they run around my app!”. Not a problem.

Google’s Datastore has an Expando class for entities that lets you dynamically add properties of various types. It’s like having a RDBMS where you can just add columns to each row, on the fly. The dynamic_properties() entity method makes it easy upon pulling an object, to see what dynamic properties were already assigned.

As far as I’m concerned, this pretty much mitigates the need for a session system. If you didn’t want to require user login, you could always make a little session ID yourself, and keep that on the UserPrefs table as a separate property, then query on that.

Rethinking how you store/query/insert data

Going slowly through all the Datastore docs and especially reading some of the performance information people were drumming up on the GAE mail list brought up a number of issues with how people with RDBMS backgrounds approached Datastore. Many of the table layouts I saw pasted on the mail list were clearly written for how an RDBMS works, with sometimes significant work required to adapt it to deal with Datastore.

A little background might help understand this difference. Google Datastore is implemented on top of BigTable, which is described briefly in the paper as a “sparse, distributed, persistent multi-demensional sorted map”. One of the other descriptions I heard in a talk on data storage techniques at FOO Camp from a Google developer was, “think of a BigTable table as a spreadsheet, except with pretty much as many columns as you want”.

This brings about a fairly big shift in thinking for the developer who grew up on an RDBMS. The fairly normalized organization of data written without regard to massively distributed data stores suddenly becomes a rather big problem. Consider a few of the ‘limitations’ of Datastore that will jump right out at you:

  • You cannot query across relations
  • You cannot retrieve more than 1000 rows in a query
  • Writes are much much slower than you’re used to (a developer on the mail list said 50 inserts with 2 fields each almost ate up the 3 seconds allowed for a web request)
  • There are zero database functions available
  • There is no “GROUP BY…”, which doesn’t matter much if you read the prior bullet point
  • Transactions can only be wrapped around entities in the same entity group (ie, the same section of the distributed database)
  • Referential integrity only sort of exists
  • No triggers, no views, no constraints
  • No GIS Polygon types, or anything beyond just a GeoPoint (Odd, considering that Google has so much mapping stuff)

Then of course, a few of the new things that might leave you scratching your head, quite happy, or both:

  • Keys for an entity may have ancestors (ancestors aren’t relations, they’re different and have to do with Entity Groups, which determine what you can do in a transaction, wheeee!)
  • An Entity Group doesn’t have to all be of the same Kind, its more of an instruction to Datastore to keep these near each other when distributed
  • Key’s can be made before the entity, just so you can make descendent entities of the key, then make the ancestor
  • The handy ListProperty, when used in a query, will let you use the conditional argument and apply it to every item in the list (sort of like an uber ‘IN (…)’ query, except it can also find all the data where a member in the list was , or = to something else)
  • Making more Entity groups is a good idea when you frequently need a batch of “these few things” for a request, especially if you need to alter them all at once in a transaction
  • Normalizing is frequently bad since you can’t query across relations, dynamic properties make it easy to heavily denormalize. If you do normalize some data and its for the same batch of ‘things you always need at once’, use Entity groups. Or use a ReferenceProperty if its merely something related you may occasionally hit.
  • The ReferenceProperty() does not have to refer to a known kind, you can decide on the fly what datastore classes to reference if not specified when declaring the ReferenceProperty
  • Many to Many relations aren’t what you think, now you could have a ListProperty() of ReferenceProperty()’s, which may or may not all refer to instances of the same class
  • A query may return entities of different kinds, if querying for entities of a given ancestor

(There’s probably a bunch more as well, these were some of the obvious ones that jumped out at me)

The end result of this, is that the standard way a developer writes out the table schema for a RDBMS should be dumped almost entirely when considering an app using Google Datastore. Storing data and using Google Datastore isn’t difficult, but it is a pretty hefty paradigm shift, especially if you’ve never left RDBMS-land. This is not a trivial change to make in approaching your data.

I rather enjoyed working with these new ways of tackling data, and the possibilities opened by the ways it lets me store and refer to data in many ways goes beyond the traditional RDBMS. In the short term though, I doubt I’ll be making any GAE app’s until there’s an alternative implementation thats production ready… I just can’t handle the lock-in.

And of course, please note any corrections or inaccuracies in the comments.

Sun, 13 Apr 2008 00:00:00 -0700 <![CDATA[Where's the Capistrano knock-off for us Python web devs?]]>

Where’s the Capistrano knock-off for us Python web devs?

Rails, and Ruby in general has had Capistrano for awhile now to help with the task of deployment and automating builds for servers, and even clusters of servers. Where is something like this for Python?

Now, before people note that I could easily use Capistrano for my Python project, I should note that it is rather annoying having to install yet another language. On the other hand, given that I will likely only need to install it on my development machine (which running OSX already has Ruby… and gems), it doesn’t seem too horrible to just use Capistrano and be done with it.

However, Capistrano doesn’t quite manage the Python egg’s, and the task isn’t exactly trivial. zc.buildout, which I previously ranted about due to odd docs does the management pretty well. It even results in a rather consistent build experience no matter where it occurs. Two commands, and boom, the app is ready to go.

Unfortunately, life isn’t quite that easy. When something does go wrong with buildout, trying to track it down can be exceptionally hairy. Having a tool so ‘magical’ as I’ve heard some describe it, carries its own penalties when things fail. Buildout also fails to automate the task of deploying the app itself to the other machine, which is still a manual process. It does manage egg’s rather well, though it does some very odd mangling of sys.path to accomplish this in every script.

I don’t need something as full featured as Capistrano, but I’d love to see something that has no more requirements than I’m already depending on (Python), that can handle the task of easily automating deployment of a Python application – including ensuring all the proper versions of the eggs I want are used – on a remote *nix machine. I recall seeing a post (I think by Jeff Rush) awhile back, on a system just like this that he unfortunately never released. Vellum also looks like it could be hacked further to do this task…

Is there some build/deployment tool that is just Python that I’ve missed? Something that will let me setup a script for some commands on how to deploy my app on another server and setup (hopefully in a virtualenv) the webapp so its ready-to-run (and optionally restart it/migrate the db/etc :)?

Thu, 10 Apr 2008 00:00:00 -0700 <![CDATA[MarkMail now indexing Pylons-discuss/devel]]>

MarkMail now indexing Pylons-discuss/devel

I’m thrilled to announce that MarkMail now indexes the Pylons-dicuss and Pylons-devel mail lists. For those looking for a great way to search and browser the Pylons mail lists, the MarkMail interface is top-notch.

For those looking for detailed Pylons docs…. there’s some very exciting developments in the documentation front coming up shortly that will make you rather happy. :)

Wed, 09 Apr 2008 00:00:00 -0700 <![CDATA[Sacrificing readability for automated doc tests]]>

Sacrificing readability for automated doc tests

I’ve tried several times in the past to try out zc.buildout, a fairly neat sounding package that automates the buildout process for a Python app. The promise of fairly easy to write recipes that can setup external processes like nginx in addition to ensuring my webapp is put together with all the things it needs sounded great.

It occurred to me that the docs definitely didn’t help at all. In fact, they’re noticeably bizarre unless you actually realize why they’re written the way they are. Here’s a sample of the zc.buildout docs about how to make a new buildout and bootstrapping.

You’ll notice that it almost looks like command line interactions of some sort are occurring, yet the author of the docs is clearly at an interactive Python prompt. Note that none of the commands shown there will work if you copy them into your Python interpreter, nor is there any indication what you would need to do to get such commands available. As a user trying to follow the docs, that leaves me wondering… am I supposed to be in a Python interpreter? What do these variables get expanded to so that I can do that at my shell prompt? Why can’t you just give me the damn command line I’m supposed to run so I can copy/paste???

Yes, it definitely got me a bit frustrated. I believe the only logical reason the docs were in this bizarre fashion is so that they could be automatically doc-tested. Its a shame that the result of this is docs that make me want to close the web page as soon as I stumble upon the ‘samples’, since there’s no way I can handle wading through the command line abstractions.

Doctests can be useful, but turning command line interactions into a Python interactive session is a massive readability issue. People know and recognize command line interactions, lets stick with them please.

Fri, 04 Apr 2008 00:00:00 -0700 <![CDATA[Pylons @PyCon 2008 Wrap-up]]>

Pylons @PyCon 2008 Wrap-up


Wow, they were great. A lot got done, quite a bit more than I was expecting. I’d like to give a big shout-out to the Pylons sprinters (in random order, sorry if I missed anyone):

  • Karen Lo
  • Mike Orr
  • David Montgomery
  • Mike Verdone
  • Wes Devauld
  • Ian Bicking
  • Phil Jenvey
  • and many more TG2 sprinters

WebHelpers saw some significant gains, including the addition of a literal object for safe HTML escaping, and default Mako auto-escaping in Pylons 0.9.7.

There will definitely be more Pylons/TG2 sprints in the future, in multiple locations around the country.


Ouch… setuptools came through, unfortunately the network didn’t, nor did my first built egg of Pylons (unfortunately I got a new laptop recently that was missing some things…). Mark Ramm and I did get the show under way, but the delay in starting up definitely affected how far the tutorial attendee’s made it in the basic Wiki project and proved frustrating for all.

I made it through all the Pylons information I had in the Mastering Pylons/TG2 section, but unfortunately had no time left to go into more detail on various aspects of it in a more hands-on style. I definitely won’t be doing any more tutorials without ensuring lots of USB thumb-drives are handy with virtualenv Pylons/TG2 ready to go.

Sessions and the Conference

They seemed ok, several of them seemed very, “And this person is giving this talk…. why?”. Note to session presenters, honesty is great but I really do wonder why you’re presenting when your opening remarks are, “What I’m about to show you, anyone who’s used this already knows, so there’s nothing to see here.” Show us something with some zing!

The lightning talks, despite a first day of some awkward sponsored ones, were actually the highlight for me. I almost spit water laughing on the “Speech Recognition does war”, followed up by “His base of God would have I done now” (Originally intended to be ‘Speech recognition does work. oh god, what have I done now?’). Unfortunately Ian Bicking failed to give us an update on ZhangoPyloGears, I can only imagine it has fully attained consciousness and freed itself of his clutches.

Coming up…

Pylons 0.9.7 is close at hand, just waiting for some final bits of WebHelpers to drop into place and it should be ready to pop. And one last side-note, I think I finally found the documentation tool of my dreams… Sphinx!

Sat, 22 Mar 2008 00:00:00 -0700 <![CDATA[PyCon 2008 Social Network]]>

PyCon 2008 Social Network

Normally I’m not a fan of social networks, I always seem to get bored quickly. The one time that I actually find myself using them however, is with conferences. The relevance of the immediately upcoming event makes it quite a bit more interesting, and I’ve found it a great way to find people at conferences that I’d like to meet and remember who I met afterwards.

As Doug mentioned earlier, there’s several social sites for PyCon 2008 up. I suggested a Crowdvine network as well, and have set one up for all PyCon 2008 attendees.

Sign Up!

For those attending PyCon 2008, consider joining the PyCon 2008 Crowdvine network. Some handy things to do after joining:

  • Add a picture. That’ll help people put a name to a face.
  • Fill out your profile. This will help people find you.
  • Find people you want to meet. From their profile you can let them know that you are a “fan” or that you “want-to-meet.” Or you leave a comment with something more specific.


Sat, 08 Mar 2008 00:00:00 -0800 <![CDATA[Linux, preferably with Pie]]>

Linux, preferably with Pie

From a Python job posting last year that I just happened to recently actually read for reasons that escape me.

Description: Python Developer
The Greenivy Group is a consulting firm with clients in the

metropolitan NY/NJ area. This project is for a major player in the telecom industry. The client needs a seasoned Python developer to code applications in Linux utilizing Cheeta, mod_Python, and/or Cherry Pie.

Mmmmm…. cherry pie…

Sun, 24 Feb 2008 00:00:00 -0800 <![CDATA[It's just super]]>

It’s just super

No one told me that in New Hampshire, the Democrats have a hidden power. They apparently have a mysterious force known as superdelegates. Yea, I know.

Wed, 09 Jan 2008 00:00:00 -0800 <![CDATA[Getting Inspired]]>

Getting Inspired

I’ve never been to TED, but wow are the talks there inspiring and insightful. They also cover a wide range of topics far beyond just IT. The majority of the talks are also available online, some in high quality 480 progressive mp4 as well.

The best thing I’ve always found about going to conferences, is the inspiration that I can get and find in the talks others give. TED inspires in a different way, so here’s a few of my favorites and why:

`Dr. Dean Ornish: The world now eats (and dies) like Americans <>`_ Only 3 minutes, WATCH IT. The power of your diet to kill you is rather amazing, if you want an even more comprehensive background on this, read The China Study

`Might You Live a Great Deal Longer? <>`_ A very fascinating theme, which includes several talks, including the short 3 minute one I cited separately in the hopes more people would watch it. :)

`Richard Dawkins: An atheist’s call to arms <>`_ This one is sure to get a few people in a huff, for Richard Dawkins has few kind words for those who hold religious beliefs dear to their heart. But its a very interesting talk on science, and oddly enough I didn’t find it very scornful (which some do), probably my background in Philosophy.

`Blaise Aguera y Arcas: Jaw-dropping Photosynth Demo <>`_ This one has been posted by people before, I have to include it cause its definitely rather incredible to see. Definitely a tech theme on this one.

And finally, one of the most relevant to our current crises with regards to the environment and sustainability…

`Janine Benyus: 12 sustainable design ideas from nature <>`_ A very exciting talk about work being done to copy sustainable ideas from nature, also called biomimicry. If you’re looking for an upper, this one definitely left my with a smile and some hope for future developments that don’t require us to devastate places with mining, pollute the environment shipping raw materials around the world, etc.

I’m sure I missed a lot of good videos, if there’s something that really inspired or awed you that can be found in the TED talks, please share!

Thu, 03 Jan 2008 00:00:00 -0800 <![CDATA[Misapplying book terms, Pylons, and the 'end-user']]>

Misapplying book terms, Pylons, and the ‘end-user’

On the Internet, I frequently see terms misapplied to other contexts. Whether its a basic definition misapplied to a different realm, or an analogy grossly misapplied to a context that is not in fact similar. Since this particular term ‘conceptual integrity’ however was used as a beating stick on Pylons, I felt it’s something worth discussing.

First, its very useful to actually understand the context of the book the term comes from, The Mythical Man-Month. The Mythical Man-Month is about Fred Brooks observations as a manager on the OS/360 project at IBM, and the problems encountered when developing an OS for end-users. Before getting into the question of whether an OS end-user is equivalent to a web framework end-user, its useful to look at the term being referred to, ‘conceptual integrity’.

Wikipedia has a fairly good summary of the term:

Conceptual Integrity

To make a user-friendly system, the system must have conceptual integrity, which can only be achieved by separating architecture from implementation. A single chief architect (or a small number of architects), acting on the user’s behalf, decides what goes in the system and what stays out. A “super cool” idea by someone may not be included if it does not fit seamlessly with the overall system design. In fact, to ensure a user-friendly system, a system may deliberately provide fewer features than it is capable of. The point is that if a system is too complicated to use, then many of its features will go unused because no one has the time to learn how to use them.

Is there Conceptual Integrity?

The Pylons developers, including myself, spend quite a bit of time deciding what goes into Pylons, and what does not. Pylons is in fact very very small at its core due to our refusal to let ‘stuff’ into Pylons that is outside its strictly defined scope. In fact, a great many comments I get are usually about features people want, but that aren’t in the scope of Pylons to provide. I would argue that this does show thought has gone into ‘conceptual integrity’, in the explicit decisions made about what is and isn’t provided.

The scope of Pylons is to provide a small, concise, WSGI-driven, lightweight framework, that provides the flexibility for a web developer to use the best tools for the job at hand for building their web application. I would argue based on the definition, the goal of Pylons, and what Pylons has in it now, that it fully meets the design goal of having conceptual integrity with regards to feature scope.

While the original blog post I cited above starts with conceptual integrity, it delves deeply into framework design issues like, ‘Should a framework pick the ORM for you?’. This is slightly odd in the context of Pylons, as the default ORM, and template language are chosen, Pylons merely emphasizes that the choice is yours should you wish to change it (as does Django).

In another misapplied analogy, a parallel is drawn up between having an opinion on a writing style, to having an opinion on a library. Consider how odd this analogy is, that the fairly minor effect of adding a trailing ‘S’, should be compared to whether a particular ORM library is capable of meeting your demands. Should the framework authors decide whether the ORM meets your demands? Should the framework lose a significant amount of capability the second its stock ORM fails to handle your needs? These are interesting questions, well beyond an attempt to draw a parallel to adding a trailing ‘S’ to a word.

Since its clear that Pylons doesn’t encroach on conceptual integrity regarding features that are unused, the next aspect to look at is the concept of a ‘user-friendly system’. This becomes a little more difficult to analyze, because it brings into question the expectations one has of the ‘user-base’. It also shows how the original term of conceptual integrity was misapplied as another parallel is drawn, this time one that I think all web developers should be somewhat terrified by.

Website end-user == Website Programmer?

The last parallel, is to compare the programming expertise of a web developer, to the internet naivety of the website end-user. (Does that offend any other web developers?) The latter of which is well documented in various web usability books, such as the “Don’t Make Me Think” book. The underlying concept being:

  • Website users have short attention spans
  • Website users won’t use a website that requires thinking
  • Website users have no patience

The thesis of this parallel is that a framework author is somehow in the superior position of knowing the appropriate library that will work for your task, and should therefore decide the tools for you. The parallel applied looks like:

  • Website programmers have short attention spans (ok, I somewhat buy this one :)
  • Website programmers won’t think about whether the tools at hand will accomplish their task
  • Website programmers have no patience (Given the languages you have to learn to do anything, I seriously doubt this)

Now, there are quite a few website programmers that don’t think about whether the tools they use, work best for their task. I have seen many websites that are slow, inefficient, or just not well designed because of a lack of thinking regarding architecture, and tool choices. I would not consider these to be website programmers I would want to work with, nor are they the target audience or ‘end-user’ of Pylons. The door is purposely left open in Pylons, and lit up, with the choice available in libraries to use with Pylons.

Regarding making decisions, Pylons has choices for all the same components as Django has decided on: ORM: SQLAlchemy Templating: Mako URL’s: Routes Sessions: Beaker Caching: Beaker

The main problem apparently being, Pylons has emphasized that there is choice, that the framework is built so that they could be swapped, should one of these not be the best tool for your web application.

The Pilot System

When designing a new kind of system, a team will design a throw-away system (whether it intends to or not). This system acts as a pilot plant that reveals techniques that will subsequently cause a complete redesign of the system. This second smarter system should be the one delivered to the customer, since delivery of the pilot system would cause nothing but agony to the customer, and possibly ruin the system’s reputation and maybe even the company’s.

In response to the original post regarding conceptual integrity, I submit the issue of the pilot system. Django and other frameworks that start off with libraries you didn’t analyze in regards to your particular web application, have more than a few times led to pilot systems (I usually hear about the second incarnation, which was then done on Pylons).

Where you realize later down the road, “wow, this ORM just can’t handle the data model I have”, which then leads to, “well, since the ORM was hooked into the XML serializer, the admin UI, the form generator, etc. there’s now no competitive advantage over Pylons… which didn’t have those extras.” Systems with heavy and deep integration of libraries that may or may not meet your demands, where large parts of the system become relatively useless should you stop using one part of it, are extremely prone to being a pilot system.

When your pilot system is done, I invite you to check out Pylons, and think about what is going to work for your website application.


Please note that I take deliberate care in referring to Pylons for use with developing web applications, its my belief that Django, and similar systems in other languages with an ‘app’ centric approach like Joomla, Drupal, etc. are excellent for making web sites. While you can easily make websites with Pylons apps, this is definitely not the primary use-case emphasized and designed for, so it is harder and less obvious in many respects than Django.

The other thing everyone should’ve learned from these posts, is that The Mythical Man-Month is a rather good book worth reading. ;)

Tue, 18 Dec 2007 00:00:00 -0800 <![CDATA[Beaker 0.9, cookie-stored sessions, and crypto]]>

Beaker 0.9, cookie-stored sessions, and crypto

In the latest 0.9 release of Beaker, I’ve finally added cookie-side session storage. I was a little bit moved to finally do this by seeing that Rails 2.0 had added cookie-side session storage, and heck if I was going to miss out!

A few changes from how Rails 2 did it though, I was definitely not content to store all the data in an end-user visible form in the cookie. That only left encryption as the next logical choice, and that quickly led me down a path of quite a bit of cryptography research.

The world of cryptography is a constantly evolving and rapidly progressing field. New papers are coming out all the time with new research on ways to break, or ‘wound’ a particular encryption scheme. Providing a weak form of encryption in Beaker would be worse then not having it at all, since it’d lead someone to falsely believe the session data was secure.

After initially going with an RC4 cipher implementation, I got ahold of some crypto people that are actually in the field, and the unanimous opinion was to use AES encryption in Counter Mode, also referred to as AES-CTR along with a signature to prevent tampering (you’d be amazed what you can do to encrypted data, and it’d still technically decrypt). This led to a slight increase in requirements unfortunately, as pure Python based AES encryption is a bit slow. This means that using cookie-based sessions in Beaker requires the installation of PyCrypto, which includes a C extension (making cookie-based sessions faster than file-based, memcached, and db-based sessions).

The final solution in Beaker uses 256-bit AES-CTR with a 256-bit HMAC for authentication purposes. It’s fast, secure, and scales across a cluster without a problem. It’s not for everyone of course, cookies are rather severely limited in size, so if you’re just storing a few small tidbits of information in a session, for example:

  • a user id
  • some flags about the users status (logged in, etc)
  • a flash message

Then cookie-stored sessions might be perfect for you.

Update: Forgot to mention, in the future, Beaker will probably use pycryptopp instead of PyCrypto since the PyCrypto library’s AES-CTR implementation isn’t as efficient as it could be, and will be using VMAC’s instead of HMAC’s for even more speed. Plus, apparently Andrew Kuchling isn’t maintaining PyCrypto, as there’s quite a few patches for it sitting unanswered on the sourceforge and launchpad bug trackers.

Tue, 18 Dec 2007 00:00:00 -0800 <![CDATA[Nice tidbits in Leopard]]>

Nice tidbits in Leopard

As many have blogged about, Apple has posted a rather hefty list describing all 300 new features of Leopard. Some specifics that really got the coder in me excited:

(Highlighting emphasis all mine)

Cocoa Bridges

Use Ruby and Python as first-class languages for building Cocoa applications, thanks to Objective-C bridges as well as full Xcode and Interface Builder support.

Scripting Bridge

Use Objective-C, Ruby, and Python programs to automate Mac applications. The new Scripting Bridge enables them to easily generate AppleEvents using a concise, AppleScript-like syntax.

Create Instruments with DTrace

Monitor system activity from high-level application behavior down to the operating system kernel, all thanks to the power of DTrace and the instrument builder.


Monitor virtually any aspect of your application with DTrace, integrated into the Darwin kernel. Java, Ruby, Python, and Perl have also been extended to support DTrace, providing unprecedented access for monitoring the performance characteristics of those languages.

While some of the things can be done now (There’s PyObjC to use Python with Cocoa), the tighter integration of Python and Ruby into Leopard make it a very appealing upgrade (Even though I rather hate the new semi-transparent menubar and odd shelf nature of the Dock).

Wed, 17 Oct 2007 00:00:00 -0700 <![CDATA[Too Much Happening, Filtering the Overload]]>

Too Much Happening, Filtering the Overload

Tonight I was drifting through Upcoming to see what’s happening in my area. Turns out the first page of 50 some events only covered today…. and its a ‘boring’ Wednesday. As I flipped through page after page, eventually getting all the way up to this upcoming Sunday… it occurred to me that there is way way too much happening within a mere 30 miles of me.

Of course, being a geek I’ve been thinking about ways to try and narrow down this list so I can actually get a useful list for what I’d prolly be interested in for the next 45 days. There’s just no way I can look through 1200 events to see what’s going on for the next month, and I like to plan ahead.

A few things I think would help to begin with would be keyword filtering, of course the next step that requires would be some way to build a useful keylist of what should be flagged for my perusal. Trying to generate a full list of every keyword I might be interested in is going to be pretty tough, though I finally thought of the one application I have which could yield the best list of keywords to filter on… Delicious Library.

It already knows the vast majority of books I own, and all my DVD’s. That’s not a bad starting point, toss in all the music I’ve bought from scouring all the id3 tags on my digital music and that’ll cover my music. I really need a way to pull all my previously watched, and upcoming movies from Netflix, in addition to mining all the data of who was in each flick (IMDB should have that). After which I should have a decent set of keywords to ensure I don’t miss either some special premier with an actor I might like, or a comedian doing standup.

Is there anything like this already that I just haven’t found? Clearly, I’d want to keep this massive keyword list personal, so the software to pull from upcoming would narrow down the result set locally. I’m still a bit too privacy obsessed to consider putting such a massive amount of information into a single vendors hands. :)

Thu, 13 Sep 2007 00:00:00 -0700 <![CDATA[SQLAlchemy, Declarative layers, and the ORM 'Problem']]>

SQLAlchemy, Declarative layers, and the ORM ‘Problem’

There’s been a bit of talk on the Pylons devel list regarding the recommended way to use SQLAlchemy with Pylons mainly regarding how to use SA (SQLAlchemy) in a fashion that is well documented and easy to work with (and maintain!).

Prior to Pylons 0.9.6 and SQLAlchemy 0.4 it was a bit of a mess, with the framework needing to load the config (since thats where your db settings are), then setup globals for SA…. eek. Mike Orr had a good intermediary solution for SA 0.3 called SAContext that handled many of the tricky parts. Unfortunately, this actually caused even more confusion as more ways of doing the same thing came about. SAContext solved some of the global config grabbing issues, but the additional layer of indirection made trouble shooting even harder (despite how small of a library it was).

Less is More

So the fix? Less intermediary layers, less indirection… essentially, KISS. Despite how much Pylons was attempting to help a user to get the db going, the additional layers in the end actually caused more problems then they solved. Of course, I shouldn’t have been too surprised…. Mike Bayer did warn me about many of these things at the beginning. Being overly eager to make things “easier” for new users, I ignored him. :)

This is why Pylons does not recommend Elixir, and with SA 0.4 the recommended usage of SA is to use its plain mapper functionality should you need an ORM layer. Yes, that’s right, despite almost every web framework out there pushing its ORM on you (or someone else’s ORM), there are many times when an app doesn’t even need a full-blown ORM.

Declarative vs Basic SA

For a better look at why one might consider additional layers on SA a bad thing lets compare a fairly basic table setup consisting of users and groups. Each user can be in multiple groups, and lets use proper referential integrity to ensure that groups aren’t deleted when users are still in them.

Compare the following two ways of setting up a basic many to many relation and the tables:

SQLAlchemy 0.4

from sqlalchemy import Column, ForeignKey, MetaData, Table, typesfrom sqlalchemy.orm import mapper, relation

    metadata = MetaData()

    person_table = Table('person', metadata,
    Column('id', types.Integer, primary_key=True),
    Column('name', types.String, nullable=False),
    Column('age', types.String))

    group_table = Table('group', metadata,
    Column('id', types.Integer, primary_key=True),
    Column('name', types.String, nullable=False))

    persongroups_table = Table('person_groups', metadata,
    Column('person_id', types.Integer, ForeignKey('', ondelete='CASCADE'), primary_key=True),
    Column('group_id', types.Integer, ForeignKey('', ondelete='RESTRICT'), primary_key=True),)

    class Person(object):

    class Group(object):

    mapper(Person, person_table, properties=dict(
       groups=relation(Group, backref='people', lazy=False)))mapper(Group, group_table)


from elixir import *class Person(Entity):
    has_field('name', String)
    has_and_belongs_to_many('groups', of_kind='Group')

    class Group(Entity):
    has_field('name', String)
    has_and_belongs_to_many('people', of_kind='Person')

On first glance, its pretty obvious that everyone should love Elixir vs the obviously more tedious SA approach of layout out your tables, then mapping them to the class objects. However, look at these two examples, and try to quickly answer the following questions:

  • How do you add a column to the many to many table to store an additional bit of info for the join?
  • Do they both enforce referential integrity?
  • How do you control whether SA is eager loading the relation? Can you restrict it to just one column of the relation?
  • What are the table names used?
  • How many tables are in your database?
  • Where do you change the id column name?
  • Which one is closer to the Zen of Python?

I think the explicit setup makes many of these questions easier to answer just at a glance. Those with enough Elixir experience can fairly easily answer most of these questions, but consider what that implies. Not only do you need to know SQLAlchemy options and parameters, but you need to know Elixir options and how they map to the SQLAlchemy functions. The desire to reduce the up-front setup of the ORM actually increases the amount of knowledge a user has to have in order to use it, and the most worrisome aspect… how to troubleshoot it.

Setups that Grow with You

With Pylons, a goal has been to provide out of the box recommendations that grow with you. That is, using the set of recommended tools may not be as apparently “easy” as some other frameworks. However, the pay-off is that you don’t hit a wall in 2 months when your application inevitably gets a little more advanced and needs to do something the simple tools either can’t do at all, or it’s incredibly difficult to do even slightly complicated things (eager load 2 columns off a related table, but not all of them). This way, the toolset you learned, you can keep using as you get more advanced and you don’t “outgrow” your tools.

While Elixir definitely appears to be easier at first glance, when you need to get more complicated you can’t exactly turn to the SA docs since Elixir has put a layer between you and SA. This can be very crippling when you eventually hit a wall, and so much ‘magic’ is wrapped up in the declarative layer that you have to troubleshoot additional layers of code when something goes wrong. The result of this is that to effectively use Elixir in complicated setups with SQLAlchemy, you need to really really know both of them which actually requires more work for a user than plain SQLAlchemy.

The SA example clearly requires a little more up-front setup, however, are you really adding tables to your database every day? How often are you going to be actually touching the table and mapper code, vs just adding domain model methods to your Person/Group class? Did the layer make it easier or harder to use multiple databases and/or put more between you and advanced SA functionality you might need later?

Adam Gomaa pointed out some interesting issues with Django’s ORM and Elixir but unfortunately tries to do the same thing Elixir and TurboEntity do…. add more layers. More layers more indirection more to wade through when you need to do something that should be pretty basic with SQLAlchemy (and is probably nicely documented on the SA site, which won’t help with these layers until you dig through the layer to find the basic SA objects the SA site refers to…).

What really makes a lot of this even more trouble-some with SA, is that when setting up complex relationships, the order of declaring table objects becomes important, since relations need to refer to them and the ORM classes. This usually results in some interesting metaclass hackery when you have these Entity’s in multiple modules, importing each other, and doing other fairly common stuff.

SQLAlchemy 0.4

In the end, I’ve been using plain SQLAlchemy 0.4 (at beta5 now, but quite stable) a lot lately, and its really great. Yes, setting up the tables (generally a one-time thing) took me probably 15 mins longer than it would’ve with Elixir. But I’m fairly certain I’ve saved myself significantly more time in the long run since I won’t need to worry about diving into Elixir code to try and find SA objects when I need a complex query, or trying to figure out how to hack Elixir’s connection should I need multiple db connections at once, etc.

So please, new users to SQLAlchemy, use just SQLAlchemy. It definitely seems daunting at first, but the flexibility and comprehensive documentation give you a solution that scales to meet your needs, with no walls in sight.

On a side-note, its interesting to compare my position on this issue to the Django team lack of AJAX helpers. The Django team rightfully claims that Javascript isn’t that hard, so “get over it” and learn a nice Javascript library so you can do powerful things. Also note that by including AJAX helpers, Pylons is encouraging one part that doesn’t scale… as the AJAX helpers will have you hitting a wall sooner or later.

Tue, 04 Sep 2007 00:00:00 -0700 <![CDATA[Routes planning and the road to Routes 2.0]]>

Routes planning and the road to Routes 2.0

Routes has been a wonderfully successful project of mine as its used not just in Pylons but quite a few other small WSGI apps. It’s even been integrated as a CherryPy URL dispatching option for those using TurboGears 1.x or plain CherryPy. It’s come a long way since 1.0 and has diverged on quite a few fronts from the Rails version that it was originally duplicating in functionality, which has me thinking that perhaps its time to consider 2.0 and what that will mean.

First, I think there’s quite a few behaviors that don’t make sense and probably never have. The Route minimizing functionality sounds neat and fulfills the Rails ideal of ‘keeping urls pretty’ yet suffers from a fundamental flaw… they result in multiple valid URL’s for the same page. James Tauber covers some of the implications of this in addition to the issue with relative URL’s not working right either. The example he cites is even worse for Routes since Routes can easily result in multiple URL’s mapping to the same controller action.

Routes 2.0

To solve the multiple URL issue, and also address some Named Routes confusion that recently hit the Pylons mailing list, Routes 2.0 will do a few things differently than Routes 1.X. Many things will be significantly more explicit and more predictable as a result. In addition, I want to add more functionality so that Routes can be the end-all of URL generation in addition to URL dispatching, even for use purely generating links conditionally that aren’t even used for URL dispatching.

Some of the key changes planned for Routes 2.0:

  • Routes recognition and generation will always be explicit

Currently, there’s an explicit option that removes the keyword controller/action/id defaults, 2.0 will not have these implicit defaults.

  • Defaults don’t cause minimization

Defaults just mean they’ll be used, they won’t result in Route minimization which increases the amount of URL’s that end up matching to a single controller action.

  • Trailing slashes shouldn’t be an issue

Without routes being minimized, a route such as ‘home/index’ will always be ‘home/index’ instead of being minimized to /home/. This will resolve the trailing slash issue.

  • Named Routes will always use the route named

A named route currently just means that the defaults for that route will be used during route generation. This currently can cause confusion because people believe that the named route actually forces that route to be used during generation.

  • Generation-only routes

A new option, which will result in routes that are used purely for generation. This option will likely be used primarily for static resources which may be on other servers, or may need the domain rotated so that the browser can do parallel resource loading. For example, one would be able to provide a list of domains to be used, and the generated links will rotate as desired on the page to split page resources over multiple domains.

  • Redirect routes

Sometimes, (especially when replacing legacy apps), its desirable to slowly migrate URL scheme’s from the old to the new rather. While URL’s should never change, sometimes the system being replaced has horrid URL’s that violate all URL recommendations. Being able to provide a smooth migration path from the old URL’s to the new ones is handy, and permanent redirects are respected by many search crawlers as well.

Migration and Compatibility

Routes 1.8 will include options to turn on behavior that will be the default in Routes 2.0, and if you like how Routes 1.X works there’s no need to worry, it will still be maintained for the foreseeable future. It’s currently extremely stable, and has a massive unit test suite to ensure it operates as designed.

Add Your Desired Feature Here

What other features are Routes users currently looking for?

Wed, 29 Aug 2007 00:00:00 -0700 <![CDATA[Next-Gen DVD Wars Give me the Blues...]]>

Next-Gen DVD Wars Give me the Blues...

I’ve been seeing more and more movies I wouldn’t mind actually buying… except that I have a 56” HD TV set. It looks amazing, pretty good with DVD’s, but really amazing with HD so if I’m going to buy a movie I sure as heck want it in HD.

Unfortunately thanks to the tech companies refusing to come up with a single standard, I can either buy a $200 HD-DVD add-on for my Xbox 360, or a PlayStation 3 which I’ve heard is the cheapest Blu-Ray DVD player available. And of course, I really need to buy both if I want HD movies since some studios are only putting their movies out on one or the other format.

The end result? I’m not buying HD movies for HD-DVD or Blu-ray because its pathetic to have to have 2 players around (no, I don’t care about universal players). In addition, I’m not buying DVD’s anymore since I know that in a year or two I’ll want to re-buy it in whatever HD format wins (please let one of them win in a year or two!!!).

What are other people doing in this situation? Buying one or the other? Still buying DVD’s? Or not buying any movies at all until they get it straight?

Mon, 27 Aug 2007 00:00:00 -0700 <![CDATA[Making logging friendly in webapps]]>

Making logging friendly in webapps

Finding out whats going on in an application is always a key point when trying to figure out what’s happening when things don’t go the way you expect. In these kind of annoying situations, following the log of how your request mapped in and what was going on can save a significant amount of time. TurboGears has had great logging throughout it, and with 0.9.6, Pylons add’s the same thorough logging.

But once the entire request is mapped out, it can be a real chore to track through them, toggle modules you want and don’t want logging for, etc. Chainsaw has had excellent graphical support for navigating logging output, and with the help of Phil Jenvey’s XMLLayout package its easy to output logging statements in a format for use with it.

With the recently updated Chainsaw section in the Pylons logging docs, you too can be lumberjacking with Chainsaw.

And of course, a shot of Chainsaw in action: image0

Tue, 21 Aug 2007 00:00:00 -0700 <![CDATA[WSGI Framework Components and other thoughts on WSGI]]>

WSGI Framework Components and other thoughts on WSGI

In light of Phillip Eby’s recent post concerning WSGI Middleware as harmful, I’ve had more than a few thoughts on the issue. None of them are all that new, but given the post I think its useful to get some of them out there.

First, I agree 100% with PJE’s post. The issue it raises results in two lines of thought. Without a doubt these objects using the WSGI specification should not be called WSGI Middleware or WSGI Applications. This means that either: 1) People should stop using the WSGI spec for non WSGI application/middleware objects.

Or… 2) WSGI needs new terminology for this application of the specification, and should not be muddling up the WSGI middleware/application definitions and environ namespace with meta-framework API’s.

To answer either of these possibilities it helps to evaluate why things are developing like this right now, and almost all of it comes down to one thing… tool developers are incredibly picky and opinionated.

To avoid further muddling up WSGI definitions, I’ll be using the following term:

WSGI Framework Component (WFC) – A WSGI specification based component that acts possibly as either a WSGI application or WSGI middleware, or some mix of both. Example, a WFC that ensures users are logged in before accessing your WSGI application (thus acting as WSGI middleware), but will render its own form and go through its own login procedure should they need to login (thus acting like a WSGI application). This is referred to as a WFC because using WSGI is seen as a way to avoid binding it to a specific framework, while its clear that an application using it actually requires it to be there to operate (thus its not WSGI middleware).

Dealing with Disagreement the WSGI Way

WSGI makes it a lot easier to disagree, yet still harness the code and development efforts of those that disagreed with you. WFC’s allow re-usable code that isn’t utterly dependent on your framework of choice as long as its WSGI compatible. Thus the fact that many frameworks are WSGI compatible at various levels makes it very enticing to build re-usable components at the WSGI level instead of using framework-bound API’s.

A thought that started cropping up, and hitting the Web-SIG mail list, and which I believe one target of Phillip Eby’s post, regards putting ‘standard’ keys into the WSGI environ for applications to utilize. This would to an extent allow you to swap WFC’s that do similar things, but in different ways. Maybe you want to swap two resolving middleware, so you use the routing spec to determine how the URL was resolved then dispatch appropriately. You can now swap WFC’s that do routing to an extent since there’s a further specification in place.

There are other specifications underway, and lots of various WFC’s being developed. If I’m using Pylons, and someone using CleverHarold or some other WSGI type application makes a WFC that does something cool, I can use it as is without having to agree with the design of their application of WSGI based framework.

Compare that to a CherryPy2 filter, CherryPy3 tool, or Django middleware. To use any of those, I need to use the whole framework. For CherryPy, this may not be the case in the future should it allow a CherryPy tool to act in the middle like middleware. Robert Brewer has said in the past he wants CherryPy to be the end-point and not continue dispatching elsewhere which would rule out its use as a library for a WFC. Thus, I’m labeling CherryPy as a framework in the context of WFC creation, while Paste and Yaro are libraries usable both in WSGI apps/middleware and in WFC’s (Note that CherryPy3 is almost capable of being used as a WFC, except it can only dispatch to non-CP3 WSGI apps).

Going Overboard with WFC’s and WSGI

The other aspect to these new WFC’s that I think Phillip hit on the head, is that there’s quite a few being pushed into this layer that really don’t belong there. No one has put out a solid checklist to know when something should be in a library, a plugin API (possibly using setuptools entrypoints like the TG Template Plugin API), actual WSGI middleware, or a WFC. As a result, there are WFC’s that do very little, and in some cases have no reason to be operating at the WSGI layer.

So, I’m going to propose some guidelines, a rough draft as I’m sure there’ll be plenty of useful feedback, on when something should be considered for a WFC and when it should be a library. It’s also useful to note that libraries can operate on things from WSGI, vs WFC’s which get plugged into a framework/app as if it was WSGI middleware.

The guidelines for WFC’s should roughly follow the same guidelines you’d want for any WSGI middleware. There’s some conditions that make it more obvious than others on where some functionality belongs and of course there’s always exceptions to the rules.

Signs some code would be a good candidate for a WFC (It’s assumed that if you’re thinking of making a WFC, you will be wrapping your actual ‘application’ with it):

  • A set of operations needs to always occur before and after the application is called, and requires knowledge of the incoming and outgoing headers
  • Modifications are done to the HTTP headers and/or content being returned to the client (cookies, HTTP caching, content transformation)
  • The application may not be called at all (authentication, authorization, conditional dispatching)

Note that the first condition doesn’t apply to functionality that merely requires something to setup. It’s overkill using WSGI just to run a function at the start of every request — even if it needs environ — there’s no reason you couldn’t just put the function call in your app, call it every request, and put the function in its own module/package (thus easy to re-use).

A lot of the Paste functions operate like this, and many of them just take the environ as their call giving you a nice API without requiring a WFC (which Phillip Eby advocates as well):

request = paste.wsgiwrapper.WSGIRequest(environ)
print request.cookies, request.path_info

There’s no reason a variety of WFC’s I see on the WSGI middleware and utils list couldn’t operate like this as well. Take wsgiakismet for example, which parses the form submission and screens it against Akismet. The example as a WFC actually looks more involved than I could see a library based version looking:

theoretical library version of wsgiakismet
from akismetverify import verify_akismet

def app(environ, start_response):
# Wordpress API Key and website name are required arguments
usersub = verify_akismet(key='3489012ab121', site='', environ)
start_response('200 OK', [('Content-type', 'text/plain')])
return ['Comment is %s' % usersub['comment'][0]]

Note that using it like this as a function that takes environ and the other 2 keys actually makes it easier to use than the original sample requiring you to import cgi and re-parse the form vars.

So some good ways to know you might be on the wrong track with a WFC:

  • Only a few things are being done on setup, and stuffed into environ
  • Some environ keys are manipulated
  • Your code never alters or does anything with the status codes, headers, or content
  • … or none of the conditions to know when it should be a WFC exist

I’m sure there’s more criteria I’ve missed, and it’d be great to have a page possibly on regarding design decisions to hopefully avoid having anymore functionality pushed into the WSGI layer when there’s no good reason for it.

Sun, 19 Aug 2007 00:00:00 -0700 <![CDATA[Routes functional conditions and WSGI Middleware]]>

Routes functional conditions and WSGI Middleware

Sometimes, it amazes me whats possible fully utilizing WSGI middleware in an application stack. While it likely isn’t something totally unique to the framework, the relative ease with which it can be done still sometimes gets me to grin.

Tonight, a Pylons user on the IRC channel (, #pylons) asked if it was possible to get a URL laid out so that /s/SOMETHING would map into their ‘s’ controller, with the second part passed in as a variable. That alone is pretty easy, however the additional requirement was that the controller action would change depending on the user’s “type”.

There’s two ways to deal with this, the first of which is the only possible way in many frameworks. Have every request to the URL map to a single function, and in that function load up the session and call the appropriate function to handle the request based on their user type. This way works fine in Pylons too, but thanks to Routes and WSGI middleware we have another option.

Routes has a lot of capabilities to it, there’s been numerous additions to the Python implementation that the Rails version is not capable of. One of them is the ability to alter the resulting URL match dictionary in various conditional functions. To toggle the controller action used, we’ll be using the ability to pass in a function to Routes conditions that can alter the resulting match.

This condition checking function has full access to the WSGI environ so if you wanted to restrict a specific controller/action combination to people referred from Slashdot, no problem! You can carefully fine-tune the conditions required for dispatch at the same place you define your URL resolution.

Since Pylons uses Beaker for session handling via WSGI middleware, the session object will already be available when our Pylons app gets called. Beaker loads the user session into environ['beaker.session']. Given this knowledge, we can write a conditional function for use with Routes like so:

def check_user(environ, result):
    session = environ['beaker.session']
    user_type = session.get('type')
    if not user_type:
        result['action'] = 'index'
    elif user_type == 'admin':
        result['action'] = 'view_action'
        result['action'] = 'not_logged_in'
    return True
map.connect('s/:domain', controller='s', conditions=dict(function=check_user))

Viola! Now Routes will run the function provided to see if it returns True before accepting that as a valid match. In the process, the action used will be set as desired. I’ve always thought a good sign something is well designed is when people can use it in ways you didn’t originally anticipate. If that’s the criteria, I think Routes succeeds and then some.

Disclaimer: Yes, I wrote Routes, and a good chunk of Beaker and Pylons, so I might be biased and tooting my own horn. :)

Sun, 19 Aug 2007 00:00:00 -0700 <![CDATA[Sci-Fi Fiction always on the way to reality]]>

Sci-Fi Fiction always on the way to reality

I remember reading a sci-fi book (David Brin’s Earth) awhile back set in the near future where a man-made black-hole fell into the center of the Earth. To say the least, it created a bit of a problem for our fictional heroes.

One of the events (I do hope I’m referring to the right book) that really stood out to me was when one of the main characters was roving around in a city, and a gang of thugs didn’t attack him/her purely because there were old retired people around hooked up with video gear that sent live feedback to the police station. These retired semi-vigilantes roamed around looking for some sort of crime to film, so they could send it in live to the police station. It was an interesting cross between the big-brother scenarios that are quite popular and vigilantism.

Ah, how fun it is as sci-fi I read years ago marches into the real-world. Earlier tonight I read a Cnet article on New York Police using cell phone photographs to help fight crime. I’m sure this will be extended to live video when cell phones and networks here can support it. I’ve found I actually prefer these kind of “big-brother” vs the type that seems to be popular in London; cameras on every street corner.

The problem with cameras all over, is that without some sort of AI, or army of people to watch them, you can only deal with logging the event and responding after the fact. It’s rare that someone watching hundreds of cameras will see some sort of crime in the act, compared to harnessing the ability of individuals to intelligently send relevant (or possibly relevant) video back to police for review.

The only thing in the article that confuses me is why it would cost the NY Police 40k to receive cell phone photos… surely there’s cheaper software that would work?

Fri, 19 Jan 2007 00:00:00 -0800 <![CDATA[Pylons 0.9.4 Released]]>

Pylons 0.9.4 Released

It’s with great pleasure that I announce the release of Pylons 0.9.4. This release has quite a few bug fixes and enhancements, the most since the 0.9 milestone. It’s also likely one of the last big updates before a 1.0 release candidate (there will be some small changes in 0.9.5 and possibly a 0.9.6).

First, the most important changes for those upgrading from an existing Pylons application:

  • WARNING: Removed the lang_extract and lang_compile commands. They used and its associated, which lacked the ability to extract ngettext style function calls and had issues with unicode strings. The new I18NToolBox project aims to provide this functionality (and more) via the gettext command line utilities.

  • WARNING: Myghty’s allow_globals config var has changed, causing the

following when running pre-compiled templates: Error(TypeError): do_run_component() takes exactly 13 non-keyword arguments (5 given) Delete the compiled Myghty templates directory (specified by cache_dir or myghty_data_dir in the config file) to resolve the error.

  • WARNING: The localization function ‘_’ now uses ugettext (returns unicode

strings) instead of gettext. To preserve the old behavior, append the following line to your project’s lib.base and lib.helpers imports: from pylons.helpers import gettext as _

  • WARNING: Removed 0.8.x legacy code and backwards compatibility functions.

Please note that since some i18n functions have moved, your will need to be updated to import _, and ungettext from pylons.i18n.


  • The XMLRPC Controller got a significant update so that it now provides

the full range of XML-RPC Introspection facilities for your service methods.

  • SQLAlchemy convenience functions have been added to pylons.database

for use with the SessionContext plugin, and to create and retain SA engines.

  • Paste dependency was updated to 1.1.1, Routes to 1.6.1 (important

update for map.resource functionality)

  • Pylons special objects (g, c, h, request, session) now available in

interactive debugger without _attach_globals.

  • Controller actions can now be generators
  • Pylons base WSGI app uses routing_args spec for easier

swapping of URL resolvers.


Please see for installation details.

Full Changelog

  • WARNING: Removed the lang_extract and lang_compile commands. They used and its associated, which lacked the ability to extract ngettext style function calls and had issues with unicode strings. The new I18NToolBox project aims to provide this functionality (and more) via the gettext command line utilities.
  • All Pylons special objects are now available within paster shell (not just h and g).
  • WARNING: Myghty’s allow_globals config var has changed, causing the following when running pre-compiled templates:

Error(TypeError): do_run_component() takes exactly 13 non-keyword arguments (5 given) Delete the compiled Myghty templates directory (specified by cache_dir or myghty_data_dir in the config file) to resolve the error.

  • Changed i18n functions in templates to use proxy objects so that using set_lang in a template works right. Fixes #153.
  • Now allowing any template plugin to overwrite global PYLONS_VARS (such as c, g), not just pylonsmyghty.
  • Adding SQLAlchemy support to the file. Saves the session engine to g to maintain it during the apps lifetime. Uses SessionContext plugin for management of the current session.
  • Updated config object so that init_app can take an optional template engine argument to declare the default template engine.
  • Updated Myghty plugin to use extra_vars_func when passed in.
  • Fixed Buffet to use extra_vars_func properly.
  • Fixed the validate decorator when there are validation errors and variable_decode=True: now passing the original params to htmlfill.render instead of the varable_decode’d version. Patch by FlimFlamMan.
  • Added ungettext function for use with pluralized i18n, and the N_ function (gettext_noop) to mark global strings for translation. Added ungettext, N_ and translator objects to be globals for templates. Refs #126.
  • WARNING: The localization function ‘_’ now uses ugettext (returns unicode strings) instead of gettext. To preserve the old behavior, append the following line to your project’s lib.base and lib.helpers imports:

from pylons.helpers import gettext as _

  • Pylons special objects are now available within the interactive debugger (deprecating _attach_locals).
  • Added setup-app run before unit tests run so that webapp has proper setup tasks handled. Fixes #113.
  • Added paste.deploy.CONFIG setup to, and testing files in the Pylons project templates. Closes #112.
  • Added security policy doc to index for use as Pylons security policy. Closes #91.
  • Improved the repr() of the c context object to show attributes.
  • Set environ[‘paste.testing_variables’] whenever that key is available, not just in testing mode.
  • Added capability to have an action be a generator function.
  • Added introspection capability to XMLRPCController and signature checking.
  • Updated Controller to use additional arg lookup scheme so that the source of the function args for _inspect_call can be easily overridden.
  • Updated RPCController, renamed to XMLRPCController. XMLRPCController now functions properly and will automatically return proper xmlrpc responses.
  • Added test configuration ini file to default template. Closes #114.
  • Fixed problem with pylons.database.PackageHub.*get* raising errors other than AttributeError when the database isn’t configured. Added new UnconfiguredConnectionError exception, instead of just KeyError or TypeError (depending on what part of the configuration failed).
  • Fixed default g init, since bare object has no init method. Reported by Ian Bicking.
  • Fixed issue with SQLObject method override having wrong name. Reported by climbus with patch. Fixes #133.
  • Moved log function to pylons.helpers and translation functions to pylons.i18n. using pylons.util purely for Pylons internal util functions.
  • WARNING: Removed 0.8.x legacy code and backwards compatibility functions.
  • PylonsApp now has option to not use Routes middleware, default resolving uses new routing_args spec.
  • Refactored routes dispatching to use new Routes middleware.
  • Fixed paster shell command to properly acquire mapper object without relying on the template being configured in a specific manner.
  • Added keyword argument pool_connection to pylons.database.PackageHub; if set to false then SQLObject connections won’t use pooled database connections (a new connection will be opened for each request).

Many thanks to Phil Jenvey, Ian Bicking, James Gardner, and all the other active members of the Pylons community!

Cheers, Ben

Sat, 30 Dec 2006 00:00:00 -0800 <![CDATA[Google TechTalk on WSGI, Middleware, Paste, and Pylons]]>

Google TechTalk on WSGI, Middleware, Paste, and Pylons

For those that haven’t gotten on the WSGI bandwagon, are still confused about Paste, how it fits into Pylons, and how its used in frameworks; I gave a Google Techtalk last week that hopefully can clear a few things up. The talk is now up on Google Video and the slides are available as PDF. You’ll want to get the slides and follow along if you’d like to read the code samples as the Google Video compression has turned them into large colorful blurs.

The main focus of the talk is on WSGI, with a bit on Paste and Pylons as well, and runs about 51 minutes. I could easily fill an hour or more just on Pylons, which I plan on doing at some point.

Disclaimers about the talk

  • When I mention how the Rails routes code does extensive code generation, this mainly applied to the 1.0 and prior version of Rails routing. The Rails routing system got an overhaul around 1.1 that made it significantly easier to read, though that was also when the security bug in the routing crept in.
  • Pony’s are valuable, though I don’t know why.
  • Yes, I realize that hot pink and its friends are not the best presentation colors.


Mon, 18 Sep 2006 00:00:00 -0700 <![CDATA[Routes 1.4 Release and Web Services]]>

Routes 1.4 Release and Web Services

This is slightly old as Routes 1.4 was released about a week and a half ago, but I thought it deserved some attention. There were a handful of fixes and some slightly major feature enhancements in 1.4.

From the changelog:

  • Fixed bug with map.resource related to member methods, found in Rails version.
  • Fixed bug with map.resource member methods not requiring a member id.
  • Fixed bug related to handling keyword argument controller.
  • Added map.resource command which can automatically generate a batch of routes

intended to be used in a REST-ful manner by a web framework.

  • Added URL generation handling for a ‘method’ argument. If ‘method’ is specified, it

is not dropped and will be changed to ‘_method’ for use by the framework.

  • Added conditions option to map.connect. Accepts a dict with optional keyword args

‘method’ or ‘function’. Method is a list of HTTP methods that are valid for the route. Function is a function that will be called with environ, matchdict where matchdict is the dict created by the URL match.

  • Fixed redirect_to function for using absolute URL’s. redirect_to now passes all

args to url_for, then passes the resulting URL to the redirect function. Reported by climbus.

Web Resources

The map.resource command is based directly off the Simply Restful Rails plugin which adds support for various verb-oriented controller actions in a RESTful service style approach. The Simply Restful layout is more or less the exact service style laid out in the Atom Publishing Protocol.

It’s a great approach and it also meant providing a few other features to Routes that I hadn’t implemented previously, the most important being able to limit matching of a URL based on the HTTP method used. This is present in the new conditions clause for a Route:

map.connect('user/:id', controller='user', action='edit',
    conditions={'method', ['GET', 'HEAD']})
map.connect('user/:id', controller='user', action='update',
    conditions={'method', ['PUT']})

The conditions clause can also accept your own function should you want to restrict the route to matching based off some other criteria (sub-domain, IP address, etc).

def stop_comcast(environ, match):
    if '' in environ['REMOTE_HOST']:
        return False
    return True

    map.connect(':controller/:action/:id', conditions={'function':stop_comcast})

David Heinemeier Hansson recently posted an entry about Resources on Rails discussing how important web services are. The other key point was to make it easier to write controllers that could not only give you easy browser access to your resources, but provide a web service API as well.

The two snippets shown above give you an edit and update capability that restricts matching based off the HTTP method (verb). Writing a huge mess of those for the rest of the functions needed for a full web service API like Atom is a bit of busy-body work, so in the opinionated style of Rails a single command wraps up the whole thing. In Routes it looks like this:


That will make the two routes at the top of this entry in addition to routes that handle PUT, and DELETE. It maps them out to a set of actions in the controller, and provides the capability to easily add more methods for specific verbs.

The map.resource command is still getting tuned up, and we’re integrating the additional functionality it provides back into Pylons as well. Josh Triplett also wrote some Python code that will parse HTTP Accept headers fully so that we can add some nice functionality to use in the controller to return the appropriate data given what the client is expecting (HTML, XML, JSON, etc.)

If you’re using a Python web framework that doesn’t use Routes… maybe its time to put a request in. :)

Wed, 02 Aug 2006 00:00:00 -0700 <![CDATA[Pylons 0.9 released]]>

Pylons 0.9 released

Last week during OSCON 2006, I was able to get a release of Pylons out. This version had some big internal changes, no longer using custom Myghty resolvers. We now use a very straight-forward WSGI interface to setup the application and the middleware. It’s easier to customize as a result, and the call-cycle is very understandable.

A bonus of our emphasis on using the WSGI specification and having a flexible architecture, has been that we’ve been able to maintain a very high degree of backwards compatibility despite such a large internal change-up. Many Pylons 0.8 applications run with absolutely zero changes under Pylons 0.9. Now that we’re using a clean and powerful API for our internal components, we can begin to add more new features without any backward compatibility issues.

Additional cool features in 0.9:

  • Swap the default templating language to your choice of TurboGears compatible template engine plug-ins
  • Controllers are called with the WSGI interface, enabling powerful application re-use
  • Custom version of Buffet that can cache templates rendered with any supported template engine
  • Mapping system now supports HTTP method restrictions for REST-ful web services
  • Interactive debugger can be used to examine AJAX triggered exceptions

We’re still adding more great features, and working towards a very solid and robust 1.0 release soon. The existing feature set of Pylons is rather large as well, since many of the projects Pylons leverages have been making great strides (SQLAlchemy, Paste, Routes, etc.).

About Pylons

Pylons combines the very best ideas from the worlds of Ruby, Python and Perl, providing a structured but extremely flexible Python web framework. Python concepts are utilized as often as possible to increase your knowledge re-use (Knowing Python makes Pylons easy), in addition to fully leveraging the WSGI protocol for maximum code re-use.

Tue, 01 Aug 2006 00:00:00 -0700 <![CDATA[Pylons Related News for the 4th of July]]>

Pylons Related News for the 4th of July

It’s been a busy month for Pylons, with lots of changes for the big internal API change in 0.9. The great news for those making Pylons apps right now with 0.8.2 is how few backwards compatibility issues there are. Most of the big changes take place under the hood and compatibility objects are present to mitigate the massive breakage that I’ve seen happen in other framework upgrades.

This is going to remain a big focus on future development in Pylons too, and the API in 0.9 is solid enough that I don’t see anything but minor tweaks in the future (1.0 and beyond). Pylons based apps will be easy to maintain and upgrade thanks primarily to WSGI.

No NIH Syndrome Here!

We’ve taken some cues from the Django project that we believe make for a cleaner request-cycle. In Pylons 0.9, controllers are expected to return a response object, and for convenience a method is included that renders a template to a new response object and returns it (This will look very familiar to Djangonauts).

The command was integrated with a slightly more enhanced version of Buffet along with the Beaker Session/Caching middleware. The end-result is a powerful command that not only can render templates in any Template-Plugin compatible template language, but the rendered result can also be cached. It’s a great way to utilize template languages which might not be all that quick by themselves.

Sample controller in Pylons 0.9:

from myproj.lib.base import *

    class UserController(BaseController):
    def show(self, id):
        # Cache based on id in the URL, for 30 seconds
        return render_response('/user/show.myt', cache_key=id, cache_expire=30)

    def index(self):
        # Just for fun, use Kid to render the index
        return render_response('kid', 'user.index', cache_expire=15)

Being able to easily cache any template in any template language makes it very easy to sprinkle in caching when you need to handle massive loads yet stay dynamic.

Another new feature present in the latest Routes and Pylons is resource mappings, which automatically generate routes for you with HTTP method restrictions. This makes it easy to setup controllers and their actions for specific HTTP verbs (aka, REST-ful URL’s and web services). The implementation of this feature was directly inspired by the Simply Restful Rails plug-in that was also demo’d by David Hansson in his Resources on Rails talk. Being able to discriminate valid routes based on HTTP method was brought up in the past and I’m happy to have seen an implementation that solves the issues I originally had with the idea.

The one feature still in development is to easily discern the content type requested, which was also inspired by Rails. Josh Triplett has written some code that deals with the ugly task of parsing the HTTP Accept headers, and I’m working on adding it into Paste for easy re-use by all. Combined with Routes, it’ll provide a clean and easy way to setup web applications that can serve multiple forms of content from a single action.


Pylons was represented at EuroPython by the other lead developer of Pylons, James Gardner. For those still trying to grasp WSGI, he gave a talk on WSGI and middleware that looks like it was quite interesting (I wasn’t there unfortunately.)

Other talks involving Pylons at EuroPython:

I’ll be at OSCON for those interested in chatting about Pylons, Python, or Python web frameworks later this month.

Pylons 0.9 Release

We’re currently wrapping up new features in 0.9, to make sure the resource mapping capability is present before a feature-freeze for release. Hopefully the release will be out within the next 2 weeks, if you haven’t checked out Pylons before I’d highly suggest taking a look at 0.9!

Tue, 04 Jul 2006 00:00:00 -0700 <![CDATA[Low-Contrast Websites]]>

Low-Contrast Websites

I was talking with my grandfather recently about various technology bits when he mentioned this awful trend he had started to notice on websites. Just the mention of an awful trend with the knowledge that his eyes weren’t what they used to be elicited similar anger from me, mainly, that there seems to be a significant increase in low contrast websites.

Disclaimer: I realize that this blog also commits some of these sins I’m about to rant on, and I’m in the process of changing them.

This trend is very apparent in many blogs, and some are getting so bad I want to scratch my eyes out after reading them. This isn’t the blog author’s fault, as they’re usually selecting a theme that “looks good” on a glance. Trying to read the text when the theme is in use can be a nightmare.

Consider this blog entry, it looks decent contrast-wise to me on first glance. But trying to read the text of the comments at the bottom made my eyes burn. Or the Shopify page which manages to consistently use the same color text as the background behind it. This is a pretty strict Do Not per US Section 508 Code (Web Accessibility Standards). I’m also fairly certain the W3C Accessibility standards frown on low-contrast layouts, especially if no option is given to switch to a high-contrast scheme.

This issue doesn’t just affect old people or those with handicaps. I have no color blindness or issues sorting out colors yet these themes still routinely give me headaches. The thing to keep in mind when using different different shades of the same color on top of each other is that while you might have a wonderful color-calibrated monitor, the vast majority of Internet users do not. Even those with perfect vision are typically at the mercy of the quality of their CRT/TFT and the color settings that may or may not be ‘proper’.

For the designers out there, I realize that these low contrast themes can look pretty, but please put out more high-contrast themes that look great. Now to fix-up my blog…

Fri, 30 Jun 2006 00:00:00 -0700 <![CDATA[Routes 1.3.1]]>

Routes 1.3.1

Ah yes, time for another release. This time its Routes 1.3.1, nothing too big to see here, thus the minor version increase. However if you were using Routes with the prefix option its highly recommended that you upgrade as there were 2 bugs specifically when used with that option that have been fixed.

Also, for those that want all their URL’s to end in a /, there’s now a append_slash option that will ensure URL’s are generated exactly how you want them to be.


Wed, 05 Apr 2006 00:00:00 -0700 <![CDATA[Pylons 0.8 released]]>

Pylons 0.8 released

Finally pushed out release 0.8 of Pylons, a bunch of great features bundled up in a slick package that makes web development easy and brings re-use to new levels. I’m not going to repeat the full announcement here, but I’d suggest checking it out if you’re the slightest bit interested in a framework leveraging WSGI for re-use and ease of use.

Now that 0.8 is out, its already time to get to work on 0.9 which is going to have a slightly more stream-lined WSGI stack but other than that won’t be very different from 0.8 from a usability perspective.

Ian Bicking noted that I was considering building the next Pylons on RhubarbTart which is sort of true. Julian Krause, Ian and myself are working on WSGI and associated components that will find themselves in all of our “frameworks” and most likely quite a few others too. That’s really one of the best things about writing solid pieces of web code in a more component style vs. a hulking framework, portability is great and the re-use is fantastic.

So far, Routes has found itself embedded in at least a half-dozen different frameworks and I’ve heard some rumors its to be ported to PHP for the Zend framework as well. The WebHelpers package is also picking up users and is being used by some Djangonauts which is great as Django is generally a fairly ‘helper’ friendly framework.

Ah yes, back to the topic, Pylons has tied together these pieces of code in a very seamless fashion that make it easy to write what you want without forcing you down a path that may confine you later. On the other hand, Pylons currently doesn’t have some of the friendlier elements that frameworks like TurboGears and Django both have. Such as a pretty Admin interface, or an extensive Toolbox app, but that stuff is in the works where appropriate.

Our main desire with this release was to get solid footing for the integration of these parts, and a good architecture for building web applications. To that end, we think Pylons is a very solid performer and will adapt easily to a wide variety of needs. You’ll also get a framework that makes it a snap to plug-in new middleware whenever something striking your fancy appears on the radar, and you can keep using the parts you like best, even if you stop using Pylons.

Wed, 15 Mar 2006 00:00:00 -0800 <![CDATA[Ingredients to the Pylons Python Web Framework]]>

Ingredients to the Pylons Python Web Framework

As the date edges closer to a Pylons release, I find myself already thinking about future directions of Pylons. I’m obviously rather biased when discussing Pylons as I’m one of its creators, though I still find that thought humorous as the vast vast majority of the code that resulted in an excellent framework is not actually in Pylons, nor did James Gardner (the other Pylons dev) and myself write it.

Matt mentions in a blog entry on Python Web frameworks, maybe its just too easy to write a Python web framework. I didn’t originally see myself writing a framework as such, but it became a fairly logical conclusion to the work I was doing at the time and the framework was originally extracted (like many others) from a large production application. It really is quite easy to write a web framework, and there’s tools available that make it even easier.

WSGI has changed the point at which re-usability is possible in a Python web framework as well, the only possible result I see in the future is more frameworks (Not in the way people think though). Many would argue this is “bad”, however I think due to the different point in re-use made possible by WSGI there will actually be more collaboration and less web developer divisions despite how many ‘frameworks’ are out there (once they’re more WSGI-ish that is). Consider this analogous in some ways to how many linux distributions are out there, and the fact that people can switch between them very easily.

So what went into building Pylons?

The Basics

First, there’s a few things you’re going to need in even the most basic web framework:

  • Dispatcher
  • Request API/Object

That’s really the utter basics, and there’s frameworks that don’t do much more than this. Obviously many people are going to want more…. like sessions. The question that WSGI raises is why lock a session interface to a framework, when it can be re-used as WSGI middleware? While the term “WSGI Middleware” can be rather intimidating, I previously covered why WSGI is actually quite easy despite the sometimes scary or just annoying term. It’s something to be reckoned with, and there’s no excuse not to get familiar with it if you’re doing web development in Python.

As I needed something rather reliable and sturdy, I went with Myghty’s session and request API. For dispatching, I used a custom resolver utilizing Routes that dispatches to a controller and action (controller method). This results in a fairly MVC’ish style web architecture that’s rather extensible.

So check those off the above list plus sessions:

At this point, we’d have a framework that can interpret a URL, setup an easy to use request object, and call your controller. Your controller would also be able to save data using sessions. We’d have a bit of a problem distributing such a framework though, as it probably wouldn’t be very convenient to setup and install.

In the case of Pylons, so far it means I’ve written just a class to handle dispatch, locating the controllers (actually, Routes will do this), and calling them. A measly hundred or so lines of Python code.

Defaults, Structure Creation, Stand-alone Server

What really helps people get started quickly with a framework is if its easy to install, creates a basic structure of a working web application for them to get started with, and some way to run it. So our new additions:

  • Web Framework Installer
  • Template for starter Web Application
  • Stand-alone Server

This can quickly be a fairly substantial amount of code to write, especially if you want cross-platform installation, and a system that also runs on multiple platforms. Not a problem though, Paste quickly makes the last two of these quite easy, and setuptools handles installing your framework and any requirements it may have. I should note that setuptools isn’t perfect, but it sure beats asking users to go to a half dozen sites and install various packages.

Paste is divided into 3 parts, though it can be tricky to see the relation between them. The core part of Paste contains wsgi middleware parts that will likely appeal to many, and some other basic request handling functionality. PasteScript contains the structure creation bit used in Pylons that generates the starting template. There’s also PasteDeploy which comes in useful later when deploying and running web applications made with a framework using it.

We’re looking at a decent little framework so far, now that we can install it, quickly start a new project, and run it. What’s next?

Templating and Object-Relational Mappers

Some frameworks, typically known as “full-stack” frameworks try and make more choices for you. They aim to fill your needs top-to-bottom, or at least their vision of your needs. While SQLObject is a very popular ORM, it seemed that good use of a layout and basic Python import statements would make using any ORM just as easy.

In Pylons, a models directory is provided, with some commented out suggestions for how you’d go about setting up an ORM. There’s hooks provided in the base application module that’s imported by all your controllers, so its easy to define your ORM classes and use them in your controllers.

One of the other objectives for Pylons was to try and be very ‘Pythonic’. That is, it should re-use as much of a developers Python knowledge as possible. This is used by default for templating since Myghty uses normal Python syntax for its templates in addition to providing powerful caching functionality and great re-use through components.

Not wanting to force anyone into something they couldn’t stand (template languages can be a love-hate thing), we also implemented the TurboGears Template Engines plug-in functionality. This made it very easy to let people use the template language of their choice, in a fairly uniform manner. It was also pretty minimal to implement the template language renderer as the Buffet project provided a great head-start.

Our new list is looking rather complete for a “full-stack” framework:

  • Web Framework Installer
  • Template for starter Web Application
  • Stand-alone Server
  • Dispatcher
  • Request API/Object
  • Sessions
  • Templating
  • Database Integration

Making the most of Middleware

This is one of the areas where Pylons was able to make some great leaps, with very little actual code. In several areas, thanks to the use of middleware, Pylons is able to offer features other frameworks are still working on.

To start off with, there’s the excellent EvalException middleware which provides an AJAXy exception catching system I have yet to see in any other framework. We’ve formatted it slightly to fit in nicely with Pylons, and it works like a charm. Extremely useful for those times when you want to interactively debug a web application to see what’s what, and it’s a lot quicker than putting print statements all over.

Another important bit, that a lot of frameworks skimp on is unit testing. Using Paste’s fixture middleware, its easy to test your web application. Pylons adds a few objects to the response you can test with, so you can ensure that the session was setup properly, the right template components were called with the right arguments, etc. In the future we’ll likely add some defaults to make using twill an easy option.

The best thing about all of this of course, is that these useful parts can be integrated seamlessly and re-used by other frameworks.

Taking the framework out of Framework

Given how most of these parts are definitely not unique to Pylons, nor are they intended to be, it shouldn’t be long until more frameworks start using the great modularity that Pylons is utilizing. A lot of these parts will likely become standardized to an extent, so that there’s even less barrier to switching frameworks.

At this point, Pylons becomes less of a “framework” in one sense, and more a set of defaults and structure for how a Python web application should be put together. Pylons has more features of course that I haven’t described, the WebHelpers functions are made easily available for use in templates, more Paste middleware is used for slick traceback email’s when you’re in production mode, PasteDeploy makes running your Pylons webapp easy in a variety of situations, etc.

It’s a very easy-to-extend model, with little need to put great amounts of application-y type stuff into the framework itself. It also keeps its components separate for easier testing, out of the actual framework. This means that while Pylons comes with a great set of middleware and parts set up for you, its very easy to swap in your preferred template language, your preferred ORM, a different exception handler, etc. The choice is up to you, but the defaults are set to a good starting point (also called “convention over configuration”).

Upping the Re-usability Ante

Increasing re-usability is what I’d consider the future for Python Web Development. With WSGI middleware driving re-usability, such concepts that “to use Feature X, you must use your Framework Y” just doesn’t apply. Unifying development work on excellent components that can be re-used in any WSGI-compatible framework makes Python Web Development better.

Pylons isn’t alone in aiming for this style of framework, Ian Bicking is working on a project that starts the opposite direction, with just a layout and you add sessions, templating, etc. as you need it. TurboGears is adding more Paste-compatible features that will shortly make it trivial for them to add in the EvalException middleware (assuming they haven’t already, I haven’t checked lately) and other great components. Different frameworks have different levels of re-usability, those that are built with re-usability in mind at the beginning will likely be able to adapt quicker to new demands and requirements, and take maximum advantage of the great middleware being created.

While having more people use Pylons would be great, it isn’t necessary for Pylons to become a better framework. Having more people use the WebHelpers package, or make their framework Paste-compatible, or use Routes, or Myghty’s powerful caching/session API’s all helps Pylons. It also helps any other framework using these components, and that’s what counts the most.

Here’s the final tally of Pylons features and where they came from:

  • Web Framework Installer – setuptools
  • Template for starter Web Application – PasteScript
  • Stand-alone Server – Paste
  • Dispatcher – Routes / Myghty
  • Request API/Object – Myghty
  • Sessions – Myghty
  • Caching – Myghty
  • Templating – Myghty, or any that support the TurboGears Template Plug-In
  • Helper functions/AJAX – WebHelpers
  • JSON – simplejson + Pylons decorator
  • Global “convenience” objects – Pylons
  • Database Integration – SQLObject, SQLAlchemy, anything else
  • Interactive Debugging – Paste
  • Traceback E-mails – Paste
  • Webapp Unit Testing – Paste
  • Webapp Deployment – PasteDeploy
  • Webapp Distribution/Installation – setuptools

This is just the default set-up, its trivial to add more middleware, which would make this list very, very long and includes such things as OpenID Authentication, authenticated session tickets, along with other great stuff.

The Pylons Code-base: Pylons Module Reference

Thu, 23 Feb 2006 00:00:00 -0800 <![CDATA[Routes 1.2 Released]]>

Routes 1.2 Released

I got Routes 1.2 out the door today, it’s a fairly important update for 2 reasons:

  • A bug crept in with 1.1 using the default controller directory scanner. The scanner wasn’t properly retaining the directory prefix which causes mismatches when using controllers underneath a sub-directory.
  • url_for can (and should) be used for all your URL needs, including static files

The second one is pretty important if you’re at all interested in creating portable web applications that can be used along-side other applications. While some frameworks provide generation of URL’s that lead to other web pages, hard-linking the other ‘static’ content will cause problems if you try and use the application under a different mount point.

Instead of using a url like /css/source.css it should instead be generated with url_for('/css/source.css'). This way Routes can ensure that if you’re running under a WSGI environment and there’s a SCRIPT_NAME present (this indicates the applications location), it will be pre-pended to your absolute URL’s. When used like this, additional keyword arguments passed in will be used as query args on the URL making url_for a handy way to create URL’s that are properly URL encoded.

Another useful feature that made it into 1.2 allows you to alias URL’s you might want to use throughout your web application, I call these static named routes. An example can be found in the named routes section of the Routes Manual.


Fri, 17 Feb 2006 00:00:00 -0800 <![CDATA[Emulating Ruby's anonymous blocks with Myghty]]>

Emulating Ruby’s anonymous blocks with Myghty

Ruby’s anonymous block capability is probably the main feature I find myself wishing Python had on more than one occasion. While the upcoming Python 2.5 PEP 323 provides for the “with” statement which will enhance generators to get a bit closer. Though as Ryan Tomayko notes, this still doesn’t make the block available within the “generator”.

There are of course, many uses for having the block (with its closure scope), available in the function/generator you’re using. I’ve seen this used to easily register code call-backs with a CleanUp/Initializaiton manager, and other cases where its preferable to actually retain the block in its entirety for later execution. It’s also very useful when you want the function to control execution of the block, and return its output in a modified form.

In these ways, it would appear that the generator enhancements won’t quite be bringing the full power of Ruby anonymous blocks to Python. However, I recently found out that a relatively recent feature in my favorite Python template language, Myghty, implements something quite close to this.

Myghty’s Component Calls with Content

In a previous entry on Formencode and SQLObject I noted how useful the component call with content can be. What I failed to note (I didn’t know at the time as well), was that not only is the function/component able to get the content of the content “block”, but its also able to execute it again and again.

Consider this example from my prior entry:

Hi, lets translate the content under:
<&| MODULE:mylib:translate, lang='es' &>
    This entire block of content will be sent in as a
    single variable to translate.myt
    for use. This includes any <b>HTML tags and such</b> as well.

and the function it calls:

from mytranslater import translated

    def translate(m, lang):
    body = “The translated text is:“
    body += translated(lang, m.content())

The m.content() call can be called as many times as you want the output of it, and it retains the scope of its original location. This in many ways emulates how Ruby can yield to the block and capture its output, however it is not possible to stash the block itself (in Myghty/Python).

Next up is to actually be able to use values from the function inside the block. In Ruby this is done in a very elegant fashion letting you declare what you’ll call them, then use them, like so:

SomelistOfints.each do |item|
    item += 2

To emulate this behavior, our block has to make a special function call and know the name of the value in advance. Here’s an example:

% randomvar = 423
<&| MODULE:mylib:translate, lang='es' &>
    This entire block of content will be sent in
    as a single variable to translate.myt for use.
    This includes any <b>HTML tags and such</b> as
    well. Here's something supplied by
    translate: <% m.content_args['value'] %>.
    Of course, <% randomvar %> is still in scope too.

And just for fun, our translate function will call the block with some different values:

def translate(m, lang):
    body = "<p>The translated and repeated text is:</p>"
    for val in range(0,4):
        body += m.content(value=val)
    body += "<p>That's it, nothing else.</p>"

The important thing to remember that makes the concept powerful is that the block above is called in the scope you saw it in the template. Whatever variables were available there are used as normal.


This method brings us close to Ruby’s anonymous blocks, as close as might be possible in Python. Unfortunately its only usable within Myghty (if not, please let me know), and its still not true anonymous blocks. At the very least, its close enough to make me happy for now. While I could just switch to Ruby entirely, there’s still way too many things about Python that I’d miss.

From the details of PEP 343, it appears that this full capability to pass the block in was purposely avoided as having flow control in the “macro” makes the code inscrutable. Hopefully someday the utility and power such functionality provides will result in it being available in Python. Or at the very least, some clever person can try a variant in PyPy and see what sticks.

And yes, I know nested functions can be used and passed around with their closure, but its pretty annoying to be nesting functions back and forth solely for that purpose. It feels like all we have is a hammer…

Further reference:

Myghty docs on Component-Call-With-Content Arguments

Tue, 07 Feb 2006 00:00:00 -0800 <![CDATA[Zachary on Routes with CherryPy]]>

Zachary on Routes with CherryPy has a very nice update on Routes with CherryPy covering not only how to integrate them, but why you’d want to use Routes style dispatch instead of the object publishing approach CherryPy uses by default.

In the future, there’ll also be an independent dispatcher for Routes that handles dispatch at the WSGI level. I believe Ian Bicking already has such code, though he hasn’t released it yet. I’m sure it’d make a great Part 2 in his series(?) on working with WSGI up close and personal. If you haven’t read his Do-It-Yourself Framework, I’d highly suggest giving it a read.

Not only does it help demystify WSGI, but it should hopefully make it more obvious why WSGI is changing the point of competition in the world of web programming.

Fri, 03 Feb 2006 00:00:00 -0800 <![CDATA[Routes 1.1 Released]]>

Routes 1.1 Released

I’ve released Routes 1.1 today after extending the unit tests for the new Groupings syntax and updating the docs.

The new syntax is quite powerful and will hopefully make everyone using Routes rather happy. While I’m not about to encourage anyone to use URL’s with .html at the end, there’s plenty of times when you want extensions to mingle with dynamic parts. You can also get some useful abilities like being able to pull out the extension like so:

map.connect('archives/:category/:(section).:(format)', controller='archive', action='by_extension')

This makes it easy to toggle the response depending on the extension, and the regexp business is handled for you.

Integration Enhancements

An additional feature, suggested by a Routes user was to make integration easier in WSGI environments. Earlier, at the beginning of each request you would have to populate the Routes config with the host, protocol, and match result. Now, merely passing the WSGI environ to the Routes config object will run a match, and populate those attributes for you.

The Routes Mapper now can take a function that when called returns a list of valid controllers. If you want to use the directory scanner Routes comes with, all you need to do is pass in the directory you’d like to scan and Routes will scan it for you.

These two new integration improvements make it rather simple to integrate Routes, here’s a basic WSGI app showing this off:

    from routes import *

    map = Mapper(directory='/my/directory/of/controllers')map.connect(':controller/:action/:id')map.connect('home', '', controller='home', action='splash')

    class WSGIApp(object):
    def init(self, mapper=map):
        self.mapper = mapper

    def call(self, environ, start_response):
        config = request_config()
        config.mapper = self.mapper
        config.environ = environ

        if not config.mapper_dict:
            start_response(“404 Not Found”, [(“content-type”,“text/html”)])
            return [“No match”]
            start_response(“200 OK”, [(“content-type”,“text/html”)])
            return [“Match with the following dict: %s” % str(config.mapper_dict)]

That’s it! If you’re not using WSGI, there’s been no backwards breakage so the old style of setting up all the attributes of the config will work fine as well.

So, now I have to figure out if there’s anything else Routes should possibly have… or is the only space for improvement at this point further optimization and perhaps usability improvements?

Fri, 13 Jan 2006 00:00:00 -0800 <![CDATA[Routes 1.1 Preview]]>

Routes 1.1 Preview

The major feature I was waiting on for Routes 1.1 is for the most part done, mainly adding more unit tests for the new syntax now. As I mentioned previously when announcing Routes 1.0, this feature is the one quite a few people have been waiting for. The ability to split the route path somewhere other than on a /.

Here’s what a few Routes using the new feature look like:

map.connect(':category/:(page).html', controller='stories', action='view')

    map.connect('feeds/:(section).:(extension)', controller='feeds', action='formatter')

    map.connect('archives/:(year):(month):(day).html', controller='archives', action='view',

The new section dividers, :(something) can be used side-by-side as the last example above shows, however in such cases each path part must have a rigid regexp requirement placed on it to ensure proper collection of the variables. Typically you will have some characters in between each dynamic part so this issue doesn’t arise.

I’ve retained full backward compatibility with the old syntax as well, if you don’t designate the dynamic part using the () modifier it will fall back to looking for the next / instead. So far, all the existing (and extensive) unit tests are passing, in addition to new tests I’ve been adding.

Routes generates a full regular expression for URL matching based off the route path you give it. This makes it a great way to setup URL routing with the power of regexp’s yet avoiding the hassle of writing a large complex regexp yourself. The other very powerful ability you gain is that Routes can turn the keywords back into the URL, like so:

>>> url_for(controller='archives', action='view', year=2004, month=10, day=12)‘archives/20041012.html'

    >>> url_for(controller='feeds',action='formatter',section='computers',extension='xml')‘feeds/computers.xml'

If you’re interested in giving it a spin, it can be checked out and installed easily from the svn repository using setuptools:

sudo easy_install -U

Feedback / Suggestions / Bug Reports greatly appreciated.

Tue, 03 Jan 2006 00:00:00 -0800 <![CDATA[Why Python?]]>

Why Python?

I first heard about Python from a roommate about 5 or 6 years ago. He was putting together a presentation for some Bio-informatics people, and was going to go over the basics of Python programming as its very popular in the scientific community. I didn’t give it much though as I was mainly using Perl at the time for any programming work, and the white space was of course somewhat of a put-off.

Fast forward a few years… I had been programming rather heavily in Perl at the time, and despite having used Perl literally for years, I still felt like I hadn’t achieved “mastery”. The effects of code in Perl would jump out and surprise me every few days, and sometimes hunting down weird bugs due to ambiguous syntax was rather time consuming. In short, Perl just wasn’t fun anymore.

I started looking around for other languages, and took a look at Ruby. Back then, Ruby had the same things that were annoying me about Perl. Implicit variables that seemingly came out of nowhere (@_, $_, etc), inconsistencies, and since the community was so small it lacked the vibrant and active development Perl had. Ruby also was missing so many of the CPAN modules I had come to rely on. Then browsing through some of Eric Raymond’s writings, I came across an article about Python he wrote titled the same as this blog entry.

White space and Mastery

My white space fears were removed rather quickly.

Of course, this brought me face to face once again with Python's pons asinorum, the
significance of whitespace. This time, however, I charged ahead and roughed out some
code for a handful of sample GUI elements. Oddly enough, Python's use of whitespace
stopped feeling unnatural after about twenty minutes. I just indented code, pretty
much as I would have done in a C program anyway, and it worked.

But this wasn’t what truly piqued my interest. As I mentioned, the feeling that Perl was somehow beyond mastery. To me, mastery means that I can write massive chunks of code with confidence it’ll act exactly as I intended it to. So seeing Raymonds next few paragraphs really got me going.

That was my first surprise. My second came a couple of hours into the project, when I
noticed (allowing for pauses needed to look up new features in Programming Python) I
was generating working code nearly as fast as I could type. When I realized this, I
was quite startled. An important measure of effort in coding is the frequency with
which you write something that doesn't actually match your mental representation of
the problem, and have to backtrack on realizing that what you just typed won't
actually tell the language to do what you're thinking. An important measure of good
language design is how rapidly the percentage of missteps of this kind falls as you
gain experience with the language.

This is all summed up very nicely,

I wrote a working, usable fetchmailconf, with GUI, in six working days, of which
perhaps the equivalent of two days were spent learning Python itself. This reflects
another useful property of the language: it is compact--you can hold its entire
feature set (and at least a concept index of its libraries) in your head. C is a
famously compact language. Perl is notoriously not; one of the things the notion
``There's more than one way to do it!'' costs Perl is the possibility of compactness.

Being compact is a valuable asset for Python. Not only does it make writing large blocks of code without error possible, but it also means the code is going to be easy to read. After opening up a few other Python projects and scanning through the code, I was hooked and proceeded to buy the Learning Python book (Though the online tutorials would’ve been just fine as well).

Black Magic isn’t so Black

Since then, I’ve been using Python close to non-stop for just a little over a year. In just the first week of using Python, I felt more productive and wrote code I was positive would work (just as ESR said) on the first try. A few months ago, interested in learning even deeper “black magic” of Python, I went to a presentation by Alex Martelli on Python’s Black Magic, Meta-classes and Descriptors.

The most fascinating thing was that meta-classes got their behavior from simple concepts I already knew. It was hard to believe things were so easy. Eric Raymond also noticed this:

It's remarkable enough when implementations of simple techniques work exactly as
expected the first time; but my first metaclass hack in a new language, six days from
a cold standing start? Even if we stipulate that I am a fairly talented hacker, this
is an amazing testament to Python's clarity and elegance of design.

Fields of Use

I have now written quite a few projects in Python, and on several of the open-source ones heard, “I just wanted to add Feature X to this, and I was actually able to figure it out just look at the code. I don’t even know Python!”

These projects span fields: shell scripts, network daemons, web applications, computing, GUI’s, and more. Learning Python is useful for many fields, and I know people learning Python right now so they can tackle problems in different fields with a compact language thats easy to master.

The Choir

I realize that this entry, carried by Planet Python, is mostly preaching to the choir. However, I felt something like this was needed. Partly because some members of the Python community have gotten odd impressions about what we should try and do in Python and what we should just “give up” on, having lost some sort of war (bizarre, I know).

People come to Python for a variety of reasons, they stay for a variety of reasons. Having compelling tools to make tasks easy in many fields is great, and means there’s no need to learn new languages solely to try and make a task in one field slightly easier (Though its always healthy to learn more).

It’s definitely valuable knowledge to know why a task in one field is easy using Language X, and for that alone its good to give it a spin. Learn what was done right, and what was done wrong. In the end though, if that language isn’t a compact language like Python, I’m just not going to enjoy it as much.

I didn’t start learning Python because of Application/Framework Y, I learned it because Python is exceptionally compelling by itself. Others learn it for the same reason, and despite the bizarre claim that book sales is equivalent to language usage/popularity, it should be obvious by now that a book isn’t needed to master Python.

For those demoralized because some other language is getting attention due to a field its had a huge impact on, re-evaluate your feelings. This isn’t the first time its happened, and it won’t be the last. Google uses Python, ILM uses Python, thousands (yes, thousands) of major corporations use Python.

Python doesn’t need to be master of every field, it just needs something compelling enough that you’re as productive using it as you would be using a different tool in a language that isn’t so compact.

Thu, 22 Dec 2005 00:00:00 -0800 <![CDATA[Handling Form data w/Formencode+SQLObject Redux]]>

Handling Form data w/Formencode+SQLObject Redux

In a prior post on handling form data I covered how to easily populate a form using FormEncode/SQLObject and validate/save the data.

One of the things I noted was that I had to take the HTML form output first, and use htmlfill with it to populate it with defaults and errors before displaying. This required some extra lines I wasn’t to thrilled with in the controller. After an evening of chatting with Ian Bicking about whether FormEncode could somehow be cleaned up to make this easier, Ian suggested I use a Myghty feature I had forgotten about.

Component Calling with Content blocks in Myghty

Somewhat ironic since I use Myghty all day and Ian doesn’t (afaik). It’s a very useful feature called Component Calls with Content. It’s probably easiest to understand the cool ability this provides by seeing it. In Myghty, each template is known as a component, and complex arrangements can easily be put together by component inheritance and component calls between them.

Normal Python function calls and expressions in a Myghty component (template) looks like this:

Hi everyone, 2 + 2 = <% 2+2 %> and your name is <% lookup_name() %>

    To include content from other components, you can either call the component through the function call m.comp('/some/template.myt') or you can use the component call syntax:

Hi everyone
  <& /sidebar.myt &>

So here’s how we will use a component call with content:

Hi, lets translate the content under:
<&| /translate.myt, lang='es' &>
    This entire block of content will be sent in as a single variable to translate.myt
    for use. This includes any <b>HTML tags and such</b> as well.

And the translate.myt template might look like:

The translated text is:

    lang%args>from mytranslater import translatedcontent = m.content()%init>

This is probably a lot to absorb, as it utilizes a few different Myghty concepts. Components can have arguments they expect, here the translate.myt component expects a lang arg which is passed in. The content block is then available as m.content() inside the template. The <%init> block is called first and variables defined there are available inside the template.

So how does this help us with the original problem?

Module Components

So far, all of these abilities are possible in Myghty’s ancestor, Mason. One more powerful construct available in Myghty however is the ability to not only call components, but Module Components.

I’ve been familiar with these for awhile as I use them for Controllers. It had never occurred to me though, that they can call any function in a module as if it was a component, not just ‘Controllers’.

Module components allow you to do a variety of interesting things, mainly, calling functions, classes, or objects that are in Python modules. Let’s take a look at the last translate.myt example using this approach instead.

Hi, lets translate the content under:
<&| MODULE:mylib:translate, lang='es' &>
    This entire block of content will be sent in as a single variable to translate.myt
    for use. This includes any <b>HTML tags and such</b> as well.

And the mylib Python module (assumed to be in the search path):

from mytranslater import translated

    def translate(m, lang):
    body = “The translated text is:“
    body += translated(lang, m.content())

Myghty will examine the function signature to determine what variables it wants and will search the current scope to make sure they’re passed in. Very handy. :)

Putting it All Together

So let’s see how this helps us out, first, rather than having to take the form and render it in the controller, we’ll push this into the template using a component call with content. So our new myform.myt looks like:

basic formUsername: Age: &>
Wed, 14 Dec 2005 00:00:00 -0800 <![CDATA[Is Rails a DSL? What is a DSL, and is it possible in Python?]]>

Is Rails a DSL? What is a DSL, and is it possible in Python?

I keep seeing blogs and blog comments pop up with some very odd notions. Many of these are summed up in this comment from a recent post by Tim on Ruby books

“It’s doubtful that the Python folks can come up with anything as compelling (or elegant) as Rails. Why? Because Ruby is so good at creating Domain Specific Langauges (DSLs). Ruby’s anonymous code blocks are a big part of what enables DSLs to be written so easily in the language. Python doesn’t have them. Python’s lambda’s (and closures in general) are crippled as well, which also doesn’t help Python’s cause.”

What is a Domain Specific Language (DSL)?

I always like to carefully define my terms before actually talking about them. So first I looked up several common definitions of a DSL. We have Martin Fowler’s entry on a DSL which describes it as:

“The basic idea of a domain specific language (DSL) is a computer language that’s targeted to a particular kind of problem, rather than a general purpose language that’s aimed at any kind of software problem.”

He further clarifies between two types of DSL:

“The most common unix-style approach is to define a language syntax and then either use code generation to a general purpose language, or write an interpreter for the DSL. Unix has many tools that make this easier. I use the term External DSL to refer to this kind of DSL.”

“The lisp and smalltalk communities also have a strong tradition of DSLs, but they tend to do it another way. Rather than define a new language, they morph the general purpose language into the DSL. (Paul Graham describes this well in Programming Bottom-Up.) This Internal DSL (also referred to as an embedded DSL) uses the constructs of the programming language itself to define to DSL.”

To summarize, an Internal DSL satisfies one of the following:

  • Uses a general purpose language, then morphs it into a language to fit the domain
  • Adds functionality to a general purpose language such that solving a specific domains problem is easier

An External DSL:

  • Completely new language syntax
  • Typically requires code generation or interpreter for DSL

Martin Fowler espouses on how very dynamic languages like Lisp, Smalltalk, and Ruby are more conducive to internal DSLs as the meta programming capabilities and dynamic nature make it easy to extend and customize the functionality for your domain. I completely agree with his contention that an Internal DSL is limited by the syntax and structure of the language.

DSL’s in Python and Ruby

While the anonymous code blocks in Ruby are useful and syntactically nice, similar functionality can be done in Python through the use of generators and iterators (more powerful generator functionality similar to Ruby’s anonymous blocks are planned for Python 2.5).

Ruby also lets you extend built-in types, which is utilized by Rails to add a few helper functions to core Ruby data types. Python and Ruby both have closures that operate slightly differently, Ruby’s closure is definitely more “full featured”. However this typically isn’t a problem because of list comprehension’s, generator expressions, and generators/iterators.

Zope is a very clear example that its quite possible to build an internal DSL in Python. Whether they built it in a clear, easy to use way is up for someone else to debate. SQLObject makes dealing with databases as easy as ActiveRecord, and there’s many other examples of internal DSL’s built in Python (Take a look at twill, a web application testing language implemented in Python).

Is Rails a DSL?

First, it should be obvious that if Rails is a DSL, its an Internal DSL. The most noticeable strides towards being a web programming DSL in Rails:

  • Over a dozen core Ruby classes are extended/modified
  • Dozens of helper functions
  • New classes (syntactic sugar) defining database access and web paradigms

Given how loose the definition of internal DSL is, I think its clear that Rails qualifies, so does Django, TurboGears, Aquarium, etc.

If you prefer to be more strict about an internet DSL, and argue that a lay programmer needs to be able to do a reasonable amount of ‘work’ without knowing the base language implementing it, Rails starts to move apart. You can do a reasonably large amount of work in Rails without knowing Ruby. A lot can also be done in Zope without knowing Python.

In either case, you can’t really start to make advanced applications in any of the frameworks without knowing the general purpose language its built on. DSL’s can and have been built in Python, and I believe if you go out and start counting them up, you’ll see more DSL’s implemented in Python than in Ruby.

Anyways, hopefully for people that run around re-iterating something they heard somewhere about DSL’s and Python vs. Ruby, this clarified something. It’s easy to create DSL’s in Python, lots of people do, and there’s lots of them. Python and Ruby are both great for making them, outside of Rails I don’t know any other DSL’s in Ruby. Perhaps someone can compile a list of all the internal DSL’s written in both Ruby and Python?

Thu, 08 Dec 2005 00:00:00 -0800 <![CDATA[Routes 1.0 Released]]>

Routes 1.0 Released

I’ve finally finished the documentation for Routes and as I mentioned earlier regarded releases am now ready for a 1.0 release. If you’re curious about Routes and want to get up to speed, I’d suggest jumping straight to the Routes Manual.

Routes is currently used in Myghty with the routes Paste template, and has been integrated for use both in Aquarium as well as CherryPy (though CherryPy 2.2 should allow better integration).

Now that Routes is feature-equivilant to the Rails version, its time to start planning for new stuff. The first and most obvious is to allow for more advanced configurability of URL’s by allowing for a new separator. This would allow you to get as creative as you like with URL’s, so you could do something like this:

m.connect('archives/:(article)-:(page).html', controller='blog', action='view')
m.connect('feeds/:section/:(format).xml', controller='feeds',
          action='xml', format='atom')

This should make for a nice 1.1 feature. For those familiar with the Rails system of Routes, has there been anything you’ve found lacking or were just itching for?

Tue, 22 Nov 2005 00:00:00 -0800 <![CDATA[Project/Code Re-Use, TurboGears, and Django]]>

Project/Code Re-Use, TurboGears, and Django

TurboGears has been making some impressive strides in both features, documentation, and possible user acquisition lately. Where it gets somewhat interesting is regarding its user-base though. The approach TG takes – building glue on top of other projects – is not new, as Subway also utilizes this, however the popularity TG has been enjoying has resulted in some interesting by-products.

First, its a bit interesting to take a look at some numbers. It’s hard to be precise, as users don’t exactly announce themselves, but many of them do usually join the mailing lists for the projects.


  • Users: 546
  • Activity: Medium
  • Average messages per month (5 months+): 340

Update: Eugene pointed out that there is Django-developers as well. Many of the Django-devel members are also on the Django-users list, so adding them together wouldn’t be too accurate. Here’s the Django-devel information, along with the total traffic if you take the two combined.


  • Users: 323
  • Activitiy: Low
  • Average messages per mont (5 months+): 177
  • Average messages of Django-devel+users: 517


  • Users: 668
  • Activity: High
  • Average messages per month (3 months+): 1,124

Information taken from Google Groups as of Nov. 14th, 2005

While each list is bound to pick up users that contribute and lurk who do not actually use the framework, I think this trend is significantly higher with TurboGears. I believe this is because TurboGears builds on several successful projects that are even used in fields outside of web development.

Project/Code Re-Use and Explaining the Mail Traffic

Django takes a somewhat interesting approach, especially when compared to TurboGears, in that re-use is non-existent.The reason given for the amount written from scratch does make sense to a certain extent, especially when you remember that Django was created 2 years ago (some existing projects weren’t too hot back then). It’s also interesting to note that in some cases, the syntax and decisions being applied to Django now, reflect those made quite awhile ago in other projects.

TurboGears pushes re-use to the extreme, and only resorts to writing it from scratch if there’s no clean way to integrate a similar project that does the same thing. Only the new Identity framework, and the crud stuff was written from scratch for example. But many of the largest, most heavily used pieces have large communities of their own, i.e. SQLObject, FormEncode, CherryPy, MochiKit, and setuptools (I likely missed some).

Also interesting to note, several of the top posters to the TG mailing list happen to be the project developers for the individual projects TG is comprised of. I doubt they’re heavy users (or perhaps haven’t even used TG), however they have plenty to contribute as the questions often relate to the individual parts. This creates a powerful community and further fuels development and use of the individual projects TG pulls together.

When considering the use of one framework vs the other, I think this re-use issue is definitely something to be considered. Building on other stable, established projects that are used widely and extensively leads to a more mature platform. The size of each individual community also means more people to work on solutions that benefit the whole (and even those using the parts).

For example, the TG community is starting to tackle some basic CRUD/administration type stuff. I don’t actually use TG myself, but I do use SQLObject and FormEncode which means I can plug my SQLObject classes into a TG project, and use their web based admin tool with my database. This is rather powerful and extremely useful code re-use, and in the future it’ll be even easier to do the same thing (likely using a WSGI app).

Code re-use is important, and the project re-use that TG employs leads to many benefits. Despite the reason cited on the Django page, I think Django is still ripe for re-use cases. The Django syntax is now so close to SQLObject, I wish Django would just switch to it and extended it where needed, rather than continue to re-implement the rest of it.

To be fair, Django does make it possible to use other templating languages, and Zope3 Page Templates are available (along with a Django ZPT error formatter). If part of the argument for OOP and Python is code re-use, projects that re-invent not just the wheel but the entire car aren’t exactly shining examples of what’s possible in Python.

Mon, 14 Nov 2005 00:00:00 -0800 <![CDATA[How to Use Database Agnostic SQL in SQLObject]]>

How to Use Database Agnostic SQL in SQLObject

One of the advantages to using SQLObject is that the code you write in it can easily be constructed in a way that ensures it’ll work without a problem in all the databases SQLObject supports. This is a tremendous advantage that is most useful when writing web applications intended for wide-spread deployment on a variety of systems.

The most common SQL expressions you’ll likely want to use are Update, Select, and Delete. You can directly issue all of these in SQLObject using the connections sqlrepr method. The easiest way to see this is to play with it on the interactive prompt.

Before we begin, let’s setup a simple database layout using sqlite to try out the examples with:

from sqlobject import *from sqlobject.sqlbuilder import *from datetime import datetimeimport sys, os

    db_filename = os.path.abspath('data.db')if os.path.exists(db_filename):
    os.unlink(db_filename)connection = connectionForURI('sqlite:' + db_filename)sqlhub.processConnection = connection

    class Comments(SQLObject):
    name = StringCol(length=50)
    date = DateTimeCol(
    comment = StringCol()


    Create some boring comments
    Comments(name='fred', comment='Hello everyone')Comments(name='joe', comment='Hi fred')Comments(name='smith', comment='Good day')

SQLObject supplies a batch of classes for us that generate our database agnostic SQL expressions. While there isn’t too much documentation for using these classes (the mailing lists help), you can get a good idea where to start by looking at the help for them.

Using Select

If we want to take a look at the documentation for the Select class (which was imported above from sqlobject.sqlbuilder), we’ll get the following information:

Help on class Select in module sqlobject.sqlbuilder:

    class Select(SQLExpression)
 |  Methods defined here:
 |  init(self, items, where=, groupBy=, having=, orderBy=, limit=)
 |  sqlrepr(self, db)

First, we need to pull the connection used for the class. We will then use the connections sqlrepr method to construct our SQL, and the connections query method to actually run it. Let’s take a look at getting all the names only from our Comments:

conn = Comments._connectionnameselect = conn.sqlrepr(Select( = conn.queryAll(nameselect)

     >>> results
        [('fred',), ('joe',), ('smith',)]

This will populate results with an array of tuples, one tuple for each result with the tuple values in the order you specified for the select (it’d be nice to have a way to get dicts instead…). Let’s take a look at a few more examples of using Select:

fields = [,]namedateselect = conn.sqlrepr(Select(fields, where=( results = conn.queryAll(namedateselect)

    >>> results
        [('fred', ),
        ('joe', ),
        ('smith', )]
    namedateselect = conn.sqlrepr(Select(fields, where=( results = conn.queryAll(namedateselect)

    >>> results
        [('fred', ),
        ('smith', )]

Joins, and additional fields can be specified using the normal Class.q notation to let SQLObject generate the proper SQL necessary in the same manner as the documentation explains.

Updating Fields

Doing a large update of a sub-set of fields is definitely something best left to a manual Update command. First, let’s take a look at what the help for the Update indicates:

Help on class Update in module sqlobject.sqlbuilder:

    class Update(SQLExpression)
 |  Methods defined here:
 |  init(self, table, values, template=, where=)
 |  sqlrepr(self, db)

I’ll admit right now I’m not actually sure what template is for, nor have I used that keyword argument. If someone would like to chime in on the comments, that’d be appreciated greatly.

Updating the table gets a little tricky since we need to specify all of the fields in a database agnostic manner. To avoid very long statements, I’ve broken it down into sections to build the query.

Let’s look at updating all the dates of our Comments table:

datecol = = conn.sqlrepr(Update(Comments.q, {}))conn.debug = True    # So we can see the query executeconn.query(updatedates)conn.cache.clear()

    >>> conn.query(updatedates)
         1/Query   :  UPDATE comments SET date='2005-11-01 11:57:31'
         1/QueryR  :  UPDATE comments SET date='2005-11-01 11:57:31'
         1/COMMIT  :  auto

Updating multiple fields is just as easy, merely add more key/val’s to the dict you pass in for the values variable. To update values using the original value of the field in the update, ie, adding something to the existing field we specify that using the Class.q.field format used in where clauses. Also, note that we need to clear the object cache after running the update so that SQLObject fetches the row again before using it.

updatedates = conn.sqlrepr(Update(Comments.q, { + 20}))conn.query(updatedates)conn.cache.clear()

     >>> conn.query(updatedates)
         1/Query   :  UPDATE comments SET date=( + 20)
         1/QueryR  :  UPDATE comments SET date=( + 20)
         1/COMMIT  :  auto

This adds 20 seconds to the existing dates for all the rows. Using the .q notation with the class is necessary for the key value because we need to ensures that Python doesn’t try to add 20 to a string which is what it would’ve tried if we had said {datecol:'date' + 20}.


Issuing a Delete on the database is very similar to the update command, the class help looks like this:

class Delete(SQLExpression)
 |  To be safe, this will signal an error if there is no where clause,
 |  unless you pass in where=None to the constructor.
 |  Methods defined here:
 |  __init__(self, table, where=<class sqlobject.sqlbuilder.NoDefault>)

By now, the signature for the init method should be fairly familiar as well as what input’s its expecting. Here’s a quick example:

delquery = conn.sqlrepr(Delete(Comments.q, where=( == 'smith')))conn.query(delquery)

     >>> conn.query(delquery)
         2/Query   :  DELETE FROM comments WHERE ( = 'smith')
         2/QueryR  :  DELETE FROM comments WHERE ( = 'smith')
         2/COMMIT  :  auto

That’s pretty much all there is to deleting, probably the easiest operation to do with a SQLExpression class.

Transactions, Notes, and Gotchas

When using these techniques in large programs, it can be tricky to ensure that the object cache is cleared out and up to date. If you’re going to use a lot of these commands extensively it might be prudent to turn cacheValues off, or wrap the commands in a function that calls the cache.clear() command.

Using transactions (not all databases support it) can still be done if you want to wrap a batch of these manual expressions into a single transaction. You just need to start the transaction and commit it when done:

trans = conn.transaction()
delquery = conn.sqlrepr(Delete(Comments.q, where=( == 'smith')))
updatedates = conn.sqlrepr(Update(Comments.q, { + 20}))

If you’d like to use database functions (bottom of the SQLBuilder docs), this is easy to pass in as well but since they’re more database specific you begin to lose portability.

Please feel free to contribute any experiences or further examples of working with SQLObject expressions in a manual fashion as I’ve described here.

Tue, 01 Nov 2005 00:00:00 -0800 <![CDATA[The Wacky World of Ruby]]>

The Wacky World of Ruby

Ruby is a fairly interesting programming language, from the “expressive” syntax to some of the absolutely bizarre documentation. For a Python programmer, the lack of predictability and almost excessively concise syntax (when just one more line would really make things a lot clearer) can be a bit of a downer. Overall though, I’m rather enjoying my experiences with Ruby but not enough that I’d want to use it exclusively.

The “Wacky” bit I cite, comes from some of the strange directions the language seems to go and wacky documentation and books available. Now, the creator of Ruby is completely aware about some of the ways in which Ruby sucks as he put it, and 3 of those are of particular importance to Python programmers since we enjoy a concise (and not complex), predictable, and consistent language. Ruby is working to address these issues in the upcoming 2.0 release which I’m hoping will make it more pleasant to work with.

It is also somewhat wacky the way the Ruby community has compared themselves to other programming language communities to which this blogger has a fairly friendly reply. Sometimes these little jabs run into the printed documentation regarding Ruby as well, which is a big turn-off for me, especially since I really disagree with the reasons that are cited for insulting other languages.

Beyond Java?

Take the Beyond Java book for example. About half-way in it becomes very clear that the author’s view of what is Beyond Java…. is Ruby, and for web programming, Ruby on Rails. The author then goes on to smash Perl, Python, and PHP (ok, I agree with him on PHP :). He puts Ruby in the mix as well, but the only bad thing he says about it is that its lacking commercial backing (except for later on where a new reason emerges).

His comments regarding Python are truly wacky though, as he jumps back to the very old “white-space reliance sucks” argument that hardly ever comes up in the real world. I’ve been using Python for over a year now, and have worked on quite a few collaborative projects, and have yet to see a single error related to mismatched white-space. This is probably because coding standards are well known and Python programmers actually follow them for the most part. It’s hard to express how wonderful this has made it when I’ve jumped into code written by other people, and have been able to easily scan it and add functionality after just a few minutes of looking it over.

When I’ve jumped into other people’s Ruby code looking to make a quick-fix, the syntax quickly became a massive chore to decipher as Perl’s motto of TMTOWTDI holds very true in Ruby as well.

The Beyond Java book also cites Python’s lack of a “killer app” when it comes to web programming and specifically references Ian Bicking’s article on web programming frameworks. Of course, since that article Ian has put out Python Paste which solves a host of problems he mentioned there, and the Python web community’s move to WSGI is helping to standardize methods of running Python web applications.

Even more wacky, later in the Beyond Java book, in yet another comparison of whats for and against the languages, a different reason pops up for Ruby not doing so good. What is it this time?

The biggest strike against Ruby right now is the lack of a strong project that lets Ruby run on the JVM. – Page 163

He goes on to cite how much support Ruby would get if Microsoft was able to woo the Ruby founders over to .NET’s CLR. On the very next page (165) another for and against argument comes out for Python. Since the author just mentioned the JVM and .Net CLR as major drawbacks to Ruby, I was actually expecting him to mention Jython, or even IronPython which is actually being developed by a programmer now at Microsoft. Amazingly enough, neither of these projects is mentioned here, though Jython was mentioned back near the beginning as being too slow (which IronPython apparently solves).

It gets even more wacky a few more pages in, as his reasons against Perl come out.

Perl does have a downside. When you look at overall productivity of a language, you’ve also got to take things like maintenance and readability into account. Perl tends to rate very poorly among experts on a readability scale. – Page 174

Who these experts are is never mentioned, and I’ve seen “experts” for and against the Perl syntax. While Ruby is easier to read than Perl in my personal opinion, its nowhere near as easy to read as Python.

Programming Ruby, The Pragmatic Programmers Guide

I’ve been reading this book for awhile now, and the author’s decision to do something different is admirable but has really been a bane to reading the book. I’ve also read the Agile Web Development with Rails book, which I think was done in a most excellent manner (written by Dave Thomas, the author of Programming Ruby). So this isn’t an attack on Dave, as I really enjoy his writing, I just believe this approach didn’t work so well.

The approach taken in Programming Ruby is to breeze over high-level uses of the language without actually explaining much about why things acted as they do. This quickly drove me nuts, and I stopped reading the entire intro as seeing syntax for no reason wasn’t helping me learn anything. If you want to learn Ruby in the way most programming books teach a language, by carefully and completely going over all the parts then doing more advanced things; skip to page 317 in the Ruby Crystallized Part.

Once I started reading here, everything fell into place very nicely and the language really started to make sense. If the sections were reversed as most programming books have it, I’d consider this book pretty much perfect for a programming book. The thing I think Dave might have missed here, is that most programming books follow this convention because it works. Being different, just to be different, isn’t very good unless there’s a real and practical reason for doing it.

The wackiness doesn’t end though, and I have yet to complete the entire book so I can’t say how many more examples of this are there. Here’s the latest gem though, which actually prompted this entry (though I’ve been thinking these thoughts for awhile).

On Page 330, going over the details of Variables and Constants, I came across this:

Ruby, unlike less flexible languages, lets you alter the value of a constant, although this will generate a warning message.

HUH? Errr, then why the heck do they call it a Constant?? Seriously, maybe its because I’m picky on language terms used but I think they should’ve called it a semi-Constant, or a mostly-Constant Constant. For me, this goes against the entire notion of what a Constant is. Then, on top of that, it insults other “less flexible languages” that (_gasp_) don’t let you change constants.

It’s a Wacky World

There’s many more examples of the wackiness present in the Ruby world. Perhaps its because the language is from Japan, home to so many wacky things by Western standards. Though the writers I mentioned are all non-Japanese, so this explanation doesn’t really cut it. To be fair, the Beyond Java book is well written and the reasons cited in many of the comparisons are valid to an extent.

Ruby in Rails also has its share of wackiness, though it seems so abundant I’ll have to save that for another post entirely. If you’re wondering after all this, why I’m still using Ruby… well, it’s a wacky world, and I do kind of like wacky (I think I’ve used up all allowed uses of the word ‘wacky’ by now). Ruby 2.0 looks to be quite appealing and it’ll be interesting to see how Rails adapts to so many breakage’s that 2.0 appears to introduce over 1.8.

Python isn’t perfect either, and I’m not going to claim it is. There’s plenty of people in both the Python world and the Ruby world who are very forthcoming about failures and successes of the language, so the views expressed by the authors in these books should not be taken to represent the whole. They are some of the most visible speakers though, so I hope that they can someday be as forthcoming as Matz has been.

Mon, 31 Oct 2005 00:00:00 -0800 <![CDATA[Editing Myghty with TextMate]]>

Editing Myghty with TextMate

TextMate is a rather slick little text editor, with a bunch of cool automation stuff thats great for programmers. You’ve probably already seen it if you’ve watched any of those demo movies for some of the latest web frameworks.

Anyways, as I mainly use Myghty, I needed a syntax highlighting mode for its format. So I made a Myghty bundle for TextMate. This should also be useful for those using Mason as only a few minor adjustments regarding the underlying language highlighter need to be tweaked (Change source.python -> source.perl). This also has a few automation snippets to speed things up.

To install:

  • Unzip
  • Drag into ~/Library/Application Support/TextMate/Bundles/
  • Restart TextMate (Not absolutely sure this is needed though)


Fri, 21 Oct 2005 00:00:00 -0700 <![CDATA[Routes 1.0 almost ready]]>

Routes 1.0 almost ready

I’ve almost got a 1.0 ready of Routes. After reading Kevin Dangoor’s post on the mysterious 1.0 I’ve come to the conclusion that Routes is 1.0 ready. In case you aren’t familiar with Routes, I’d suggest taking a look at one of my earlier posts about it.

So where is it? It’s in the latest svn for now, because I’d like to actually have a nicer site with more full fledged documentation before the release. For example, some docs on how framework creators should go about integrating it, as well as more detailed and thorough documentation on usage. I also have to finish up one little change that will make Routes fully compatible with the WSGI spec when the path/script info is split-up (which happens when a WSGI app is put under a URL-space).

Rails routes suffers from a lack of documentation as well, and the most complete source of information is buried in the test units for it. There’s also good docs in the Agile Web Dev for Rails book, but that’s not exactly a free public resource. So I’ll likely be “porting” my docs back to Rails, especially since Nicholas Seckar has been a great help while working on this project.

Whats New?

Short answer, not a lot. The main thing was actually very trivial to implement, named routes. These act essentially as a short-cut for when you want to pull some possibly long pre-defined route defaults. Short-cuts are good of course, as they save you a bit of typing and in this case also make your URL’s more flexible should you decide to change how to get to a “named” route.

Here’s what a fairly basic Route setup looks like:

m.connect('', controller='home', action='splash')

To implement Named Routes I added the ability to specify an additional string before the keywords. Here’s an example:

m.connect('home', '', controller='home',action='splash')

Now take a look at using it inside a template:

Without named routes
url_for(controller='home',action='splash')url_for(controller='home',action='splash', id=4)

With named routes
url_for('home')url_for('home', id=4)

This is slightly different from the Rails approach as I was mainly interested in keeping it “Pythonic”. So there’s no extra symbols or functions created when using a named route.

As you can see, it can save a bit of typing as using a named route is sort of like having a set of keywords inserted for you.

A 1.0 Release

Kevin’s article gave me a lot of food for thought regarding whether I should keep incrementing the Routes version. I can’t see any reason not to just go 1.0 as my version of Routes will be feature equivalent (and then some) with the Rails version, is heavily unit-tested, and is used in a production environment.

I’m confident in the reliability of the code and its being used by quite a few people in production environments (myself included). Being 1.0 doesn’t mean its done, it just means its hit a point functionality wise where I’ve accomplished everything I wanted the “finished” product to have. Even 1.0 software has bugs, and when I think of more features I’d like to add, I’ll start on my way to 1.1 or maybe even 2.0.

Hopefully I’ll have the docs ready to go sometime before November at which point a fully redesigned web-page will be put up. Until then, if any of this has you interested, head over to the Routes site or take a look at the unit tests to get a feel for using it. If you want to see it in action, try out the latest version of Myghty which contains an easy-to-use project template using Routes integration (currently with the 0.2 Routes which is lacking Named Routes).

Thu, 20 Oct 2005 00:00:00 -0700 <![CDATA[Hooked on Myghty]]>

Hooked on Myghty

I’ve been programming web sites for many years, and have yet to come across a templating language as appealing as Mason / Myghty. To avoid confusion I’m going to talk about Myghty, but since its a direct port of Mason (plus some MVC stuff) all of my comments apply to Mason as well (unless otherwise noted). So if you find yourself stuck using Perl (or you prefer Perl) and something here sounds appealing, by all means use Mason as I did.

Despite Myghty only having come into existence approximately 14 months ago, the code-base is stable, very quick, and has been running in production environments for over 8 months. This is mainly because it started as a very direct port of Mason to Python, it then grew a few additional features that made it great for MVC use. The methodologies present in Mason (thus Myghty as well) are known to scale to very large and complex sites, as this list of Mason-powered sites shows. But it’s the little things added up that really make Myghty my template language of choice.

This is a rather lengthy post as I highlight and explain some core concepts of Myghty, please bear with me… also, if you’d like to follow along and try the examples out, its really easy to get started using Myghty with Paste.


Template languages vary a lot in style, there are the very basic string replacement template languages all the way to more advanced template languages. Myghty definitely weighs in on the latter as it has many of its own concepts with regards to templating that produce a very robust and advanced templating system.

The syntax itself is also very appealing to me, as I’m a fan of templating languages that keep their guts inside < > signs. The only exception to this being if you need a quick line of Python, which is done just by having a % in front of it. Here’s what a loop would look like:

% for person in people:
   <b>Hi</b><% person %>
% #end

You’ll notice an additional line is needed to indicate the end of the loop. This is because Python uses white-space to determine blocks and Myghty needs to know when the indentation is over.

Components and Inheritance

In Myghty, each template is more properly referred to as a Component. When a component is directly called as a request (or sub-request), inheritance is applied. This is how a site’s skin is typically applied. Rather than having to specify in your template that it includes or extends some other template, in Myghty your component automatically inherits from an autohandler file above it.

An easier way to think of it is to consider your template directory layout as a Class, and all the templates in it are methods. Every directory inside that root one is another Class, and so on. The autohandler in this context acts much like your __init__ method. Here’s a little example:

Sitename% m.call_next()
Tue, 18 Oct 2005 00:00:00 -0700 <![CDATA[Python Web Framework Niches]]>

Python Web Framework Niches

I’ve come to the belief lately that the web frameworks available in Python are increasingly fine-tuned to specific application requirements. Of course, anyone reading the ‘About’ sections for these frameworks should realize this as well. I wonder how many people actually read that section as I’ve seen people latch onto web frameworks without knowing the task it was originally made for.

Without knowing the reason the framework was created, its common for many people to leap to the conclusion that its another Rails wanna-be just because its a ‘full-stack’ web framework. I was playing around with a nice full-stack framework called WebObjects years ago which made it easy to setup database objects, generate CRUD, etc. Zope’s been doing the same stuff for what now seems like eons as well, yet I don’t see people declaring RoR a Zope clone (It obviously isn’t).

In light of that, I’m inclined to agree with Ian Bicking’s response about the lessons Python web people did learn from RoR.

The concept I want to focus on is that people create these new frameworks because they make their task easier than any of the other frameworks already out there. While they might pick up features from other frameworks, most of them aren’t aspiring to be “Python on Rails”. Sometimes this task is easier when other tools can be integrated to avoid code replication, as is the case in one framework I cover here.

Many people have declared the amount of Python web frameworks a “problem” that should be “solved” somehow, perhaps a Highlander fight with swords to the death (_There can be only one!_). I’d like to suggest the opposite, there’s a lot of Python programmers and I think there’s room for even more web frameworks. The variety is a strength because they make it easier to get specific web applications done.


The TurboGears site has a nice about page describing its purpose, though I feel it doesn’t completely explain the rationale for its creation. There’s some interesting and unique decisions made in TurboGears, like using Kid instead of Cheetah or Myghty for templating. Then there’s the inclusion of Mochikit and the TurboGears decorators for returning output as JSON for use with Mochikit.

So what kind of applications is this web framework geared for? (Please excuse the pun)

The best way to answer this is to look at the application this framework was created for, Zesty News, and the abilities of some of the tools being used. Zesty News is in a rather interesting category of web applications in that the end-users themselves will be installing it, quite likely on their home computer rather than a server. Being able to package it up and easily distribute/upgrade it becomes a key issue along with database portability and code thats database agnostic.

Two tools assist here, setuptools for distribution/packaging and SQLObject for portable database code. Zesty News deals extensively with RSS and XML, so it makes sense that the templating language chosen was actually created for dealing with web services.

These design decisions behind TurboGears should make it fairly obvious when to consider it for your next project. The cohesive toolset you get when you choose TurboGears is ideal for developing portable, easily deployable AJAX-enabled web applications that likely deal with XML frequently and need to stay database agnostic. Even if your web application doesn’t deal with XML frequently, the decisions TurboGears makes for AJAX integration will make it easy to add heavy dynamic interaction to a TG webapp.


Django was created to deal with the requirements of working in the web development department of a news publisher. As such, the framework was created specifically to deal with the requirements placed upon the author. What’s rather interesting is the lack of re-use in Django when it comes to doing things that have been done before in other projects (Database mapper, form validation, etc).

The tools and parts of Django were specifically built to work as one package, and using Django makes that very obvious. One of the things most common when in a newsroom or publishing environment is dealing with CRUD. That is, there is a lot of content and ways to get content into the system and administrate the content is a high priority. As a web framework built for dealing with Content, many of the design decisions reflect the common tasks present in CMS’s (Content Management System).

To start with, you get a slick administration interface for your conntent, that’s miles beyond any of the generated CRUD type stuff in other web frameworks. This differs from the philosophy of other web frameworks that give you basic CRUD (Scaffolding in Rails-speak) in that Django’s admin interface is aimed directly at being production-ready with no modification at all.

Django also makes it fairly easy to make a Django ‘application’ like a Forum or Blog, then slot it into other Django application environments. Again, this makes a lot of sense given the original requirements placed on the creator of Django. If a company has 4 websites, and wants them all to have the new Forum/Classified ability it makes a lot of sense for this task to be optimized.

So what web applications are you going to want to use Django for?

Quite a few, as it turns out dealing with Content is a very common task. If you’re writing a web application heavy on content, that needs a full featured web interface for managing the content it’d be really hard not to recommend Django. It’s easy to get started, and in almost no time you have very powerful functionality running that gives you a lot of usability.

Don’t be Everything for Everyone

Part of the reason I picked these two projects to talk about is that they’re both extracted from a working project (as Rails was). I also haven’t seen many people mention the fact that frameworks developed in such a manner are also inherently going to be optimized for the use-cases that brought them into existence.

Open-sourcing the project lets them grow to an extent, but their design is largely baked in and a useful limitation. Too much expansion past the initial design requirement will make them generic, and with that comes a lot of complexity (sometimes worth it though for the extra re-usability).

Note that the specific things Django gives you don’t help that much if you’re trying to write a Zesty News style application. The same goes in reverse as well, since building an Admin interface of your own isn’t fun and can be time consuming. While it’s possible to make web applications that do this in either framework, compensating for framework design will require extra time when you try to use one framework for everything.

If you’re using one framework for everything, maybe its time to take a look around.

Thu, 13 Oct 2005 00:00:00 -0700 <![CDATA[WSGI and WSGI Middleware is Easy]]>

WSGI and WSGI Middleware is Easy

Really, its quite simple.

If you already knew that, this post isn’t going to enlighten you at all. If WSGI had been one of those things you kept saying to yourself, “Oh, yea, I’ll learn that someday”, consider this a short intro to all the info a web developer and to some extent a framework author, will likely care about.

All the gory details of WSGI along with a nice overview are available in PEP 333. As I’m mainly interested in it as a web developer, or as the PEP refers to it, a “application developer”, I’ll be focusing on that aspect.

Basic Usage

WSGI is intended to bridge the gap in deploying Python web applications that might use different frameworks in a more uniform manner. A web application that implements the WSGI interface can be called in the same way as any other web application that implements WSGI. The basic call of a web application using WSGI is quite straight forward (example straight from the PEP):

def simple_app(environ, start_response):
    """Simplest possible application object"""
    status = '200 OK'
    response_headers = [('Content-type','text/plain')]
    start_response(status, response_headers)
    return ['Hello world!\n']

That really shouldn’t scare anyone who remembers the ‘good’ old days when your CGI scripts had to write every little bit of the content to the web browser. environ is merely a dict thats loaded with some common CGI environment variables that anyone who has worked with CGI applications should recognize. The start_response is a Callable that the web application uses to start the response, send the status, as well as the typical response headers.

Pretty basic stuff so far, but as web developers, we’re not even likely to need to know that much. This is because we don’t need to muck around with setting the status and dumping our content out, our web framework takes care of it for us (the majority of which now support WSGI).

WSGI Middleware

Despite the acronym and somewhat ambiguous ‘Middleware’ term, WSGI Middleware isn’t any big deal either. Take a good look at the code fragment above… its just a function. It could be any Callable, since the server (which we don’t care about) is just going to call it. We could wrap that Callable inside another one without much hassle. I could even show you an example of some WSGI middleware code, but I’ll leave that for you to explore.

The thing to remember here, is that your entire web application can be called with that one simple command up above. As such, other functions or classes can wrap around your web application object and do other things before and/or after your web application gets called. That’s really all WSGI middleware is.

There is currently WSGI middleware that will:

  • Handle web application errors
  • Provide session support
  • Profile your web application
  • Deal with Login authentication
  • and Gzip the output

I’m sure there’s even more I’m missing. The three I cited here come with Python Paste, and using them is rather easy. Let’s assume I wanted to trap the traceback our basic web application shown above throws, and have it emailed to us (if for some reason it died a nasty death). Here’s what the code would look like:

from paste.exceptions.errormiddleware import ErrorMiddleware
wrapped_app = ErrorMiddleware(simple_app, global_conf, debug=False,

That’s it, we’ve now used WSGI Middleware. If you wanted to add profiling on top of that, you’d just pass the wrapped_app in as an argument to another WSGI Middleware class.

Now instead of telling your server to use simple_app as your WSGI web application, you just have it use wrapped_app. Notice that we just passed in simple_app as an argument to the ErrorMiddleware class. global_conf is a dict that Paste builds for us based on the Paste config file, so we don’t have to worry about that here either. The other 3 keyword arguments are visible in the ErrorMiddleware docs and I think its pretty obvious what they all are for.

Other WSGI Middleware, like the session middleware, will add extra keys to the environ dict that gets passed into your web application. That way you can get to objects or variables the middleware has setup for your use. There’s nothing hidden or mysterious here, just one Callable calling another.

PEP 333 calls this the “middleware stack”. I like to think of it as function wrappers, or object wrappers since each one is a Callable. We’re just passing an object into a new object as an argument, and so on.

Running a WSGI Callable

Ah yes you’re wondering, great, we got this function that takes two arguments and throws out the response, now how do we make it run? There’s several packages that adapt a WSGI Callable to some more common forms of running a web application.

Probably the easiest of these to use, and the one that Paste has in it, is the flup package. It contains several different WSGI servers that will take your WSGI Callable, and make it available either as Fast CGI, SCGI, or AJP. I’d highly suggest checking out that page as it also contains an example of using several bits of WSGI Middleware back to back.

If you were hesitant or unsure what WSGI and WSGI Middleware was before, hopefully this post has helped. It really is pretty basic, and by putting common tasks like error handling into WSGI Middleware it can help people using other frameworks avoid repeating existing work.

Fri, 07 Oct 2005 00:00:00 -0700 <![CDATA[Python Paste Power]]>

Python Paste Power

Mmm, tasty, a headline of P-word’s. Recently Ian Bicking went on a bit of a release spree with a whole bunch of goodness that I’m way too lazy to link to in this paragraph. The ones I’ve been waiting for were all in the line-up: Paste, Paste Script, and Paste Deploy. I blogged about paste and setuptools earlier and hadn’t followed up as I indicated I would partly because I was waiting on their official release.

So, what’s the big deal? Paste and its buddies solve a host of issues that commonly confront Python web developers and web administrators, the front page of Python Paste does a good job of explaining briefly why each group should care. Thus, I’ll move directly onto the fun that comes with using them.

Yes, I think its fun, but that might be just because I have a thing for installing web frameworks…. so I’ll cover using Paste as two groups of users, as a Web Developer, and as a Web Administrator.

Using Paste as a Web Developer

First off, to create a web application we need to setup a directory for the project. Since we want our web application to be Paste-enabled, the template for the web application should be using setuptools. That way when we package up our web application and give/sell it on the Internet, the people (Web Admins) using it will be able to install and run it easily.

So… lets see what framework I should use to whip up this little web application. I’ll run a paster command that lets me see what Paste-enabled web frameworks I have installed that come with new web application directory templates:

% paster create --list-templates
Available templates:
  basic_package:            A basic setuptools-enabled package
  myghty_modulecomponents:  Module Component Template
  myghty_routes:            Routes Template
  myghty_simple:            Simple Template
  paste_deploy:             A web application deployed through paste.deploy
  pylons:                   Pylons application template
  turbogears:               TurboGear application template

Cool, right? Well, maybe its just cause I like having a big toolbelt of web frameworks sitting around. Obviously not a lot of web frameworks have paste templates available yet, but more are adding support and I’ll be rather excited when that list is 20 or more long.

If you go and install the Paste packages, your list won’t be quite as long as mine (insert evil laugh) as I’m running the CVS of Myghty (0.99 release with these templates coming shortly), along with some other stuff I’ve whipped up. Merely installing a web framework package that’s Paste-enabled and has templates available will make it show up in that list.

Another interesting thing to consider here, is that there’s no requirement that only web frameworks are allowed to create directory templates. Maybe a web application you’re working on, is so powerful you want to make it easy for end-users to extend it with their own custom add-ons. You could provide a paster template that creates a plug-in ready directory template for custom themes. There’s a ton of power packed in the various ways you can extend these tools, building on the dynamic discovery stuff in setuptools.

Moving along though, I’ll go ahead and use Turbogears.

% paster create —template=turbogears pygoSelected and implied templates:
  TurboPaste#turbogears  TurboGear application template

  package:  pygo
  project:  pygoCreating template turbogears
  Creating ./pygo/
  Recursing into package
    Creating ./pygo/pygo/
    Copying to ./pygo/pygo/
    Copying to ./pygo/pygo/
    Recursing into templates
      Creating ./pygo/pygo/templates/
      Copying to ./pygo/pygo/templates/
  Copying package-start.py_tmpl to ./pygo/
  Copying dev.cfg_tmpl to ./pygo/dev.cfg
  Copying prod.cfg to ./pygo/prod.cfg
  Copying setup.py_tmpl to ./pygo/setup.pyRunning /opt/local/bin/python egg_info

As you can see, it just setup our new project for us, which I called pygo because it was the first name that popped into my head. If you’ve actually used TurboGears, looking at that should have raised some red flags as its missing a bunch of stuff that a TurboGears project needs. This is mainly because the template was created by Ian as a demonstration of how to make a Turbogears-style template. Hopefully an upcoming version of TurboGears will be Paste-enabled so I can create a new project like this (hint hint). :)

Ah well, I’ll just make a basic Myghty project using Routes instead:

% paster create —template=myghty_routes pygoSelected and implied templates:
  Myghty#myghty_routes  Routes Template

  package:  pygo
  project:  pygoCreating template myghty_routes
  Creating ./pygo/...Bunch more files…

So that’s all there is to creating new projects with the web framework of your choice (if its Paste-enabled). How do we go and start it up?

~% cd pygo
~/pygo% paster serve server.conf
Starting server in PID 6090.

In the case of the myghty_routes template, it starts the server on port 5000. A quick look at the server.conf file makes it obvious:

[server:main]use = egg:PasteScript#wsgiutilshost = = 5000

    [app:main]use = egg:pygo#paste

That’s really all the Paste you have to worry about as a web developer (though it has even more capabilities you’ll probably want to use). We can see here that its using wsgiutils from PasteScript to run the server. You can easily swap that out for any of the server support that flup offers such Fast CGI, SCGI, or AJP.

At this point, as a web developer, you’d go ahead and create your web application. Especially since the default myghty_routes template is pretty boring if you don’t put anything in it. But for this example, we’ll assume its done and distribute it as a single egg file for people:

~/pygo% python bdist_egg
running bdist_egg
running egg_info
... whole bunch of stuff here...
~/pygo% ls dist/

That’s it. This is easy, right? You’re ready to go ahead and give your egg to anyone running Python 2.4 now (You could make a source distribution and upload it to Cheese Shop just as easily). So let’s try and totally forget that we’re a web developer, and assume a different role.

Using Paste as a Web Administrator

Ah yes, the joys of setting up web applications you come across. Well, I’ve rarely gotten any joy out of it at least. Let’s take a look at how Paste does make it a lot easier. First, we’ll need to install the insanely useful Pygo webapp that some other thoughtful user created. If this Pygo application was released and uploaded to the Cheese Shop, I could install it like so:

% sudo easy_install Pygo                                  Searching for Pygo
Best match: Pygo 0.0
Processing pygo-0.0.0-py2.4.egg
... more stuff happens...
Installed /usr/local/lib/python2.4/site-packages/pygo-0.0.0-py2.4.egg
Processing dependencies for Pygo

Now, since I didn’t actually upload it to Cheese Shop, I’ve faked that screen. But that’s pretty close to how it would’ve looked if I had released pygo and uploaded it to Cheese Shop.

After this one command, as a web administrator, we now have the pygo application installed. There shouldn’t be any need to be installing web applications over and over for every user that wants to run it as the vast majority of web applications get their customization and settings from a database.

What needs to be configured, is the instance of the web application that each user is running. We can set that up for each user inside our Paste configuration file like so:

[server:main]use = egg:PasteScript#flup_scgi_threadhost = = 3000

    [composit:main]use = egg:Paste#urlmap/blog/fred = fredpygo/blog/janet = janetpygo

    [app:fredpygo]use = egg:pygodatabase = mysql:/username@localhost/database

    [app:janetpygo]use = egg:pygo

These configuration files are quite flexible, and allow different instances of a web application to run at different locations. In that case, one of them is running for ‘fred’ at /blog/fred. Each block for an app can have additional arguments that setup the database to use, and other settings that should probably be customized for each user. I threw in an additional database argument for one of them as an example.

Maybe you’re not a web administrator, but a web user comfortable downloading and installing many of the other webapps out there (Typo, MovableType, phpBB, etc.). Setting up your own site, using whatever Paste-enabled Python webapps you want is just as easy. Ideally ISP’s and such would just run the easyinstall command to install whatever the ‘popular’ webapps of the week are, and they can update them easily, keep old versions around if needed, etc.

Making Python Web-Application Distribution/Use Easy

This is pretty powerful stuff, and its quite easy to use. Even if you’re not making a web application with the intent to distribute it to the world, Paste is still going to help you out in a lot of ways.

An ISP or webapp end-user only needs to know how to add your webapp to their config file and installing it is a breeze thanks to setuptools. I think this has an enormous benefit for the Python web community, as it’ll significantly increase the ease of use when it comes to installing and managing web applications. Python web developers will gain some good footing to create Python Web Applications that compete with and surpass PHP webapps, especially if more ISP’s start supporting Paste-enabled applications.

Well, that’s my hope at least as I’m sure the title of this section made clear.

For the Python web framework creators out there, I like writing web applications (with various web frameworks), and I really like how easy Paste makes it to create them, package them up, and use them. Please add Paste support to your web framework, I’ll be happy to help and the Paste mail list is very responsive.

Tue, 04 Oct 2005 00:00:00 -0700 <![CDATA[Followup to "Best of Breed Controllers"]]>

Followup to “Best of Breed Controllers”

My prior post regarding Best of Breed Controllers got more notice than I had anticipated, I definitely didn’t expect the interesting discussion regarding object publishing history it spawned.

But first, some corrections:

  • Kevin Dangoor points out that CherryPy allows for computed URL traversal by having the parts of the URL passed into a default function with a function signature like def default(self, other, parts, of path):. On a side-note, I believe David Creemer is using this aspect in his use of Routes integration with CherryPy to some extent.
  • Mike Watkins gives a great overview of Quixote on his blog. He also shows how it fits into the categories I mentioned above and points to a hello world web app that again looks similar to the other frameworks.
  • Adiran Holovaty notes that Django controllers can be callable’s too, and that my example is somewhat misleading regarding the amount of code required for a basic controller. This is because the example I took from the Django documents was from a fairly advanced controller that was doing a lot of work.
  • Robert Brewer mentions that CherryPy controllers can be any callable that allows an “exposed” attribute to be set on it, though they tend to be class methods as its easier to hook them into the cherrypy.root handler tree. Also, CherryPy doesn’t officially advocate any templating language.

In a way, Phillip J Eby’s correction regarding the history could be in the list above. However, it really wasn’t my intention to get into a discussion regarding object publishing history. For anyone considering making YAPWF, I’d definitely advocate catching up on lessons learned in the past as it makes no sense to repeat their mistakes. I think in hindsight, a better heading for that section would’ve been Who didn’t do it first as the real intention of the section was so that people wouldn’t assume one of the frameworks I mentioned was the first to come up with it.

For those interested in the history of Zope, its predecessor Principia and Bobo, I’d highly suggest reading the above link to Object Publishing along with Phillip Eby’s comment and Paul Everitt’s response. I’ve found it all rather fascinating as most of the frameworks I showed looked so similar to Bobo. So nice to know that an old Python ‘adage’ I read (probably on some comparison of Python and Ruby) seemed to hold up here. If you give the same task to a dozen Python programmers, the solution is going to look quite similar.

Sun, 02 Oct 2005 00:00:00 -0700 <![CDATA[Best of breed Controllers for MVC web frameworks]]>

Best of breed Controllers for MVC web frameworks

I’ve been doing a lot of comparisons and research into various Python web frameworks lately, mainly focused on MVC oriented frameworks. Some of them have templating languages they come with, some not so much. The thing I have started to notice is how similar the Controller in them all is starting to look. First though, lets see what they all can do when it comes to the Controller.

For comparison’s sake, I’m bringing out some examples using CherryPy, Myghty, Bricks, Aquarium, Ruby on Rails, and Django.

Please note I’m not including frameworks built using any of the ones mentioned as their Controller looks pretty much the same. I’m also not an expert in all these frameworks as I only have so much time in the day, so if I seem vague on something its because I don’t know for sure. Feel free to comment (or correct) and clear up any vagueness I express.

Dispatch and Responder Styles

All the frameworks I’ve mentioned implement a style of object publishing (sometimes referred to as dispatch-style) that maps to the URL coming in. If you have another name you want to call this, go for it, I haven’t seen anything consistent on what to call this stuff.

Three different styles seem to exist that make sure an object responds to the URL:

  1. User creates a regular expression to explicitly map a URL to an object (*Explicit Mapping*)
    • (r'^polls/(?P<poll_id>\d+)/vote/$', '') (from Django tutorial)
  2. Object is mapped out according to the URL (*Implicit Mapping*)
    • /admin/login/hello/ —> admin/ (Myghty)
  3. Programmatic rule set determines object that responds (*Programmatic Mapping*)
    • map.connect ‘:controller/:action/:id’ (Rails)
    • m.connect(‘:controller/:action/:id’) (Routes)

In the Python world, three different styles also exist for how that object (or function) is defined. The first one is most popular, and is similar to how Rails sets up the Controller. They are:

  1. Create a class inheriting from a Controller object, methods of the class instance are called to respond (*Class-method*)
  2. Create a callable object that responds (*Callable*)
  3. Create a function thats directly called (*Function*)

Note that the last 2 are functionally equivalent for the most part, however a callable object has the typical advantages (or disadvantages) of persistence in some web frameworks, as such if a web framework only shows one of those styles I won’t mention the other. Curious what frameworks use what styles from the above two sets?

For this listing, I’ll refer to the mapping scenario as dispatch-style and the method for defining the response as the responder-style, also please note that some frameworks support multiple dispatch-styles and responder-styles.


  • Dispatch-Style Mapping 1:
    • Explicit, Implicit, Programmatic
  • Responder-Styles:
    • Callable (Though it looks almost exactly like the Class-method style)


  • Dispatch-Style Mapping:
    • Implicit
  • Responder-Styles:
    • Class-method


  • Dispatch-Style Mapping:
    • Explicit (Not with regexp’s though, each class is attached off the root to determine what URL leads to it)
  • Responder-Styles:
    • Class-method (Must be explicitly exposed with @cpg.expose)


  • Dispatch-Style Mapping:
    • Explicit
  • Responder-Styles:
    • Function (Packaged in the same module if they’re related)


  • Dispatch-Style Mapping 1:
    • Explicit, Implicit, (Programmatic with Routes in upcoming 0.99)
  • Responder-Styles:
    • Class-method, Callable, Function


  • Dispatch-Style Mapping 1:
    • Programmatic
  • Responder-Styles:
    • Class-method

1: User can create custom dispatchers to determine responder object

Who did it first?

If we throw in two more frameworks that utilize CherryPy, like Subway, and TurboGear, we can start to see that using a responder-style of Class-method is pretty popular. What’s even more interesting is that all the Class-method Controllers look remarkably similar, and I don’t think its because they’re copying each other.

The Aquarium project appears to be approximately 4 years old now, but from looking at the mailing list had little or no users until only 2-3 years ago (One poster from 2001 describes his aquarium plants turning brown…). From the sourceforge changelogs, the Controller stuff was added with 1.4 release in 2004. CherryPy has been around for 3+ years and I’m presuming it had the Controller stuff in it from the get-go as it seems a pretty core function of CherryPy. Rails has been out for a year or so, and was used in a form internally for BaseCamp for awhile predating that.

If you really want to trace back this style of Controller, you’d most likely find yourself looking at [STRIKEOUT:Java] Python code (See Phillip J. Eby’s comment below). I’m really just trying to point out here, that even if one of these projects saw something they liked from the other, none of them were the originators. They all took a good concept, and implemented it differently, though they’ve all started to look remarkably similar.

A Controller Line-Up

So lets take a look at some of the examples on the pages of each respective website. Django is the only one that really stands out, as they’re not using a Class to contain the methods, rather they’re grouping them as plain functions inside a module.


class BaseController(Controller):

    def call(self, callNext, *args, **kargs):
        “”“Save a reference to myself in ctx.controller.”“”
        self._ctx.controller = self
        callNext(*args, **kargs)

    def doMessageAddAction(self):
        “”“Add a new message.
        ...does a bunch of stuff…

I mentioned that it was Callable but functioned in a way like a Class-method. This appears to be because Aquarium puts a context and maybe other initialization stuff per request in the __call__ bit, then has it actually calls a different method to respond.


from bricks.controller.crud import CRUDfrom bricks import user

    class Test(CRUD):
    # Override the default delete() method
    def delete(self, rowid):
        return CRUD.delete(self, rowid)
    delete.expose = user(level=1, app='app')

    controller = Test('todo')

Bricks adds an interesting ability here by attaching function attributes to indicate what authorization is needed to execute the function called. Some of the other frameworks shown here do similar things through function decorators.


from cherrypy import cpg

    class HelloWorld:
    def index(self):
        return “Hello world!” = True

    cpg.root = HelloWorld()

Notice that the class instance is attached to the location off cpg.root thats desired. To map a URL to a class instance farther down, say under /booga instead of /, you’d do that last line like so: cpg.root.booga = HelloWorld()

Pretty handy I think, and it beats writing regular expressions (though doesn’t appear as flexible). One of the reasons I prefer the Programmatic dispatch-style. Moving on..


from django.core.extensions import get_object_or_404, render_to_responsefrom django.models.polls import choices, pollsfrom django.utils.httpwrappers import HttpResponseRedirect

    def vote(request, poll_id):
    p = get_object_or_404(polls, pk=poll_id)
    ... does a bunch more stuff to use Django-supplied functionality…

If you imagined the functions inside a class, instead of a module, it’d look rather similar to the others….


class HelloWorld:
    def doit(self, m, **params):
        m.write(“hello world!”)

    helloworld = HelloWorld()

At this point, the similarity between Bricks, CherryPy, and Myghty should be pretty obvious. Aquarium is close as well, while Django is only a wee farther from what’s appearing to be the norm.


class HelloController < ApplicationController
    def index
        return render :text => "hello world!"

Doesn’t look too different from some of our Python MVC frameworks…

What it All Means

I have no idea. Really, I just thought it was fascinating that these different frameworks started looking so similar. Some of them have been incredibly similar before others, but obviously these approaches catch on and find themselves adopted into other frameworks.

I do start to wonder at what point they’ll be similar enough that the barrier to switching web frameworks is narrowed down to only a few dozen lines of code to change here and there. Sure, going to/from Rails is going to be tough as that also means a language change, but for Python programmers tempted by a feature a different web framework offers I won’t be surprised to see more people hopping back and forth.

With regard to the Python web frameworks, none of them appear to agree on a template language. Several of them use Cheetah, some use Myghty’s template abilities, one uses Kid, and several of them have their very own template language.

It’s also interesting to consider whether or not having more possible dispatch/responder styles is beneficial or detrimental to the popularity of a specific framework. Rails creator David Hansson purposely includes only one style for each as he emphasizes with a concept known as convention over configuration. He provides defaults for pretty much everything, and following conventions makes it easier to write code as it enforces a consistent programming and naming style. I can’t help but notice that of the frameworks I mentioned above, the ones with less (apparent) options and better defaults tend to have more users (With the exception of Bricks as its still in alpha).

You might have noticed I designated Rails as being capable of user-programmed dispatching. While Rails uses its routes system by default, the thing people miss is that they’re just defaults. You can get in there and totally alter the way it works if you want, Ruby doesn’t completely close classes so you could just re-open them and override any aspect of its dispatch system you like.

I think this a valuable lesson that some of the more “verbose” examples above might want to consider. If there’s lines of code in a controller that you have to put in there every-time, find a way to make it the default case so that boiler-plate isn’t needed. It only needs to be a default, an advanced programmer will know how to override it should they need to. There’s obvious limitations to this since things work differently in Python, but there’s still Python ways to do it in a way that feels “right”.

For the Python Web Frameworks I missed

I know, I didn’t cover SkunkWeb, Spyce, Snaklets, Webware and more. Please feel free to add the relevant information regarding your framework in the comments. It’d be helpful if you could indicate what Dispatch-Styles and -Responder-Styles your framework supports.

If I failed to properly cover one of the above frameworks, bringing that to my attention is most appreciated.

The comment system supports the Textile format.

Fri, 30 Sep 2005 00:00:00 -0700 <![CDATA[setuptools and Python Paste]]>

setuptools and Python Paste

I indicated in my last article that I would begin blogging about a very misunderstood, and what I would consider unappreciated Python package called PythonPaste. Python Paste is currently undergoing a documentation and website update after which it will hopefully be much more useful for those wishing to use it, or even understand why they’d want to.

To really get started with Python Paste its exceptionally useful to know about a package that is used extensively by Paste to accomplish some of the magic it performs. This package is setuptools by Phillip J. Eby, and knowing where setuptools is being utilized form actual Paste code will make it much easier for both web framework creators and web framework users (Web Application end-users don’t need to know any of this).

A Better distutils

One of the things that setuptools in a way replaces/upgrade/enhances is the commonly used Python disutils package. Many issues are present in distutils that setuptools resolves, such as:

  1. Automatically installing dependency packages
  2. Using multiple versions of a package at once
  3. Upgrading a package and its dependencies
  4. Installing a package retrieved automatically over the Internet

These are just some of the common things people might want to do, that distutils doesn’t provide. setuptools add’s even more, such as:

  1. Easily develop a package without installing it
  2. Install packages in your user-space rather than the system
  3. Create extensible applications/frameworks that automatically discover extensions using ‘entry points’. (This is a big feature that Paste uses)
  4. Register/Upload your package to a central Python package repository (Cheeseshop)
  5. Create Python eggs – a single-file importable distribution format
  6. Include data files inside your package directories with modules making it easy to access them

The functionality of setuptools most likely to crop up during an exploration and use of Paste, is #3 and #6. Knowing that these functions are coming from setuptools rather than Paste will make it easier to see what parts Paste handles, and what setuptools is doing. There’s more features of setuptools than I’m not mentioning here, as what’s most important for Paste is to see where the features I mentioned fit in.

Using setuptools

Quite a few programs use setuptools and cheeseshop for distribution. Changing your package to use setuptools instead of distutils is very simple, as the setup() block in the file is almost identical. I’d suggest taking a quick look at the setuptools site for what this config looks like if you aren’t already familiar with it.

Discovering Libraries

This feature is used by Paste to discover support that other web frameworks enable for Paste usage. setuptools’ full name for this is Dynamic Discovery of Services and Plugins and its very handy. You could use it for your blog to discover plugins that announce support for it, or your web server can discover webapp’s that are available for it, etc.

Dynamic discovery of plugins is a very powerful ability, and one you’ll see throughout Paste-compatible web frameworks and web applications. When you see Paste documentation talking about ‘entry_points’, its referring to something you’d configure for your package. Here’s what a PasteDeploy compatible is likely to contain:

from setuptools import setup, find_packages

    name=Joe Smitch,
    install_requires=[“PackageOne”, “PackgeTwo >= 1.3”],

Paste is just looking for a command that will give it a WSGI application (we’ll talk about Paste later). The entry point group (or EPG) there is called paste.app_factory, so Paste can scan for packages that have this EPG and learn what packages will work with it.

While it looks like Paste is doing a lot of work, this is all setuptools so far. The install_requires just indicates that those two packages should be installed at the same time as our new one, and that PackageTwo should be a version >= 1.3. The rest of the lines should look familiar to anyone who has used distutils.

Including Data Files at Run-time

If you’ve distributed data files (Non-Python code that goes with your program), you might’ve resorted to __file__ hacks to make sure your program knew where the files actually were. setuptools provides facilities to make getting to these installed data files a bit easier, more reliable, and even compatible with PEP 302-based import hooks (don’t worry if the last one makes no sense, its good news).

When you look through or use code that imports pkg_resources, this is something setuptools provides for your use. It’s also used to run version-specific code, in the event that you have multiple versions of the same package installed. This way, you can upgrade packages without breaking code thats relying on a specific version of the package, as long as you keep around the older package (You’ll need to use the multiversion option when installing).


As you might have noticed, this is a fairly brief overview on some of the functionality of setuptools. It does a lot more than I’m noting here, as this post is mainly intended to cover heavily used features of setuptools that will crop up when using Paste. Knowing that ‘entry_points’ and such are what setuptools provides will make it easier to understand Paste and how Paste discovers the abilities you supply for it (Using your own packages that employ setuptools).

Coming up next… **What Paste means for end-users**

Thu, 29 Sep 2005 00:00:00 -0700 <![CDATA[Routes 0.2 released]]>

Routes 0.2 released

Routes 0.2 has been released. The change-log is pitifully short:

  • Added prefix option
  • Fixed Python 2.3 bug with thread-local singleton

But hey, its a small package to begin with so what the heck. Though its only 0.2 I’m rather pleased with it so far, its performance is great and quite reliable so I’m using it in production environments already.

If you haven’t been following my blog long, Routes is a feature-complete implementation of Rails routes system. I talk more about my reasons for re-implementing Routes in Python in an earlier post so I won’t repeat them all here.

It’s fairly unique in the Python world as it will do a route lookup search to turn a dictionary back into the URL (URL Generation) that will ensure the same values are created. This allows you to generate URL’s from inside your web pages and easily add new URL schemes without touching all your web pages.

The Routes package is aimed directly at integration with Python web frameworks that support the MVC style paradigm as it returns a controller and action value with the assumption your framework will know what to do with it.

Tired of writing big regexp’s to match URL’s to a class/method for dispatch in your webapp? Pester the framework creator to integrate Routes. :)

Here’s some Python Web Frameworks that currently or will shortly have Routes support/integration:

  • Aquarium has a URL Connector that uses Routes. Not sure if its been added to the core or only exists as an add-on right now.
  • Myghty will have Routes integration packaged with it in 0.99 which is on track to be released this week hopefully. It will also bring Python Paste support and integration which will bring a whole bunch of goodies to webapp developers and users. (Note that Python Paste requires Python 2.4)

I’ll be talking more about Python Paste and why you should care later this week. Any comments/suggestions on Routes are greatly appreciated.

Mon, 26 Sep 2005 00:00:00 -0700 <![CDATA[Ghost in the Prius]]>

Ghost in the Prius

I’ve owned my 2004 Silver Prius for about 18 months now, and in that time the only ‘problem’ I’ve had is the hilariously named Red Triangle of Death often fondly referred to as the Red Triangle of Doom as well (I shall now refer to it as the RToD).

In the case of the RToD, the problem itself seemed to go away after leaving the car off for a little while, much like this poster describes. I found the whole incident rather humorous and frightening in a way, as the dealer didn’t apparently realize that the Prius Drivers Manual states that in the case of this specific error the owner should call the dealership.

Before I describe the latest incident (quite minor in comparison to the RToD), looking back over the RToD incident would be helpful.

I was on my commute home from work (about 26 miles), and have to go through a fairly shitty interchange where 2 lanes of traffic merges to one, then merges with an on-ramp to one, THEN merges onto a very busy freeway. I’m referring in this case to the interchange in the North Bay Area where West 580 meets 101 North.

While sitting in the stop and go merge, I noticed that the engine turned off. Not unusual in the Prius as the engine goes to sleep on occasion during stop-and-go. However, after awhile it didn’t turn back on… then the RToD went into full effect. A few pretty glowing lights on the dash, the LCD got a nifty little icon as well.

Seeing the RToD definitely gets your attention. It’s not something that can be easily ignored, so I pulled off the road and yanked out the Prius Drivers Manual to see what the icons lighting up actually meant. I was already wondering at this point, why the hell do I need to pull out a manual when its an LCD, why not just print the explanation on the damn thing? The screen says all sorts of other useful text, it seemed quite lame they couldn’t just print a full explanation and what to do next right there.

The drivers manual said that in this particular case, I should pull off the road as soon as its convenient (vs. the error about the braking system which said to pull over immediately). Ok, no problem, I was already pulled over. It then said to contact the dealership immediately. It was 7pm, but I figured, what the hell, it said to contact them so I pulled out my cell phone and called them up. My call went like this:

Me: “Hi, I’m calling to report a problem with my Prius”

Dealer: “Sorry, our Service Center is already closed”

Me: “Well, the manual said when these lights go on, I’m supposed to call the dealship. Any clue what I should do?”

Dealer: “The manual said you should call us?”

Me: “Right”

Dealer: “Really… huh. Toyota never told us that.”

At this point I was a little concerned since the dealership apparently didn’t realize that the drivers manual told people to call them in the event of a half dozen different errors. It didn’t say call the next day, it said call immediately and seemed quite firm about it.

While chatting with the dealer, I was still fiddling with things in the car myself, and I tried turning it on again at which point most of the lights went away, and only the engine light was still lit. Since this light merely indicated “see dealership soon”, I figured it was good enough to drive home and scheduled an appt for the next day with the dealer.

The next day, none of the lights were on. I still took it into the dealership where they did a debug dump of the computer, and installed an ECM software update which they said would prevent it from occurring in the future.

The Problem

The real issue that this incident left me pondering, is what the hell do you do if you’re a long ways from a dealership? As only the dealership is somewhat prepared (most of their techs admit they have no clue how to fix it, they just follow manuals) this means you’ll have to plan trips careful to make sure you’re never too far from a dealership should something go wrong.

Someone on a PriusChat forum mentioned that the RToD happened to him on his property out in the back country of Canada. It took over 3 hours for the tow-truck to get to him, and hours more to tow the Prius to the nearest dealership (As his car never got better after leaving it off and turning it on again). At least he was lucky enough to have cell phone reception or some other way to contact a tow truck.

The Prius was originally unique in just how many systems are entirely computerized, however many modern cars are now catching up in electronic complexity so this is starting to become as likely to happen to any new car owner. More complexity of computers systems increases the likelihood of a possible bug cropping up and the testing process will never be able to eliminate them all. This means the people driving the vehicles are in a way, beta-testers.

I don’t think thats too horrible, as the most tested parts are the most essential ones. If the failures resulted in cars flying off the roads, we all would’ve heard about it by now (Right?).

I’m hoping that more car manufacturers in addition to Ford license and utilize Toyota’s hybrid technology because if there was some sort of standard for these parts you’re likely to see more independent repair-shops drop the big investment for the equipment to repair them. That in itself would make me a lot more comfortable driving long treks where dealerships are scarce.

As I mentioned earlier though, the repair techs at the dealerships don’t actually know how to fix any of these problems. They follow service manuals that indicate they should plug Gizmo-A into Plug-A on the car, hit Button-X to dump the data, then send the data back to Toyota for their computer/car engineers to analyze (More or less). Then they usually will either install a software update, or reset the software on the vehicle and hope the problem goes away. Isn’t that reassuring?

Todays Incident

Well, it was pretty minor now that I’ve fully dredged up the memory of my RToD experience. Today my car decided that I’ve been listening to the stereo so much, so it started being a pain. It was sort of like having a child in the car messing with me.

First the audio cut in and out, then it turned off completely. The LCD then decided it no longer needed to inform me of the outside air temperature (is it hot? cold?), and my steering wheel audio controls didn’t appear to be having any effect. So I hit the stereo power on/off and its back…. sort of. Then it cuts out, and I fiddle with it some more till its back on. It continues to do this throughout my drive in, occasionally coming back up to full functionality, then dropping the audio again.

I’ve already called and scheduled an appt at the dealership for tomorrow, so we’ll find out what the heck it is this time after that. For all I know, it might be my friend again on the drive home and leave me alone. In the meantime, I think I’ll browse the Prius forums and see if there’s some ritual that will align me with my car’s spirit so I can hopefully make peace with it.

Thu, 22 Sep 2005 00:00:00 -0700 <![CDATA[Presentation on SQLObject & FormEncode]]>

Presentation on SQLObject & FormEncode

I gave a presentation tonight on SQLObject & FormEncode for the Bay Area Python Users Group (BayPiggies). Here are the slides for the presentation, which actually have 3 extra slides I didn’t show during the presentation because I forgot they were there (doh!).

Anyways, there were rather important I think as they actually showed two common cases where you’d want to use SQLObject and FormEncode together. That is, for moving data from a CSV file to a database, and for dealing with web forms. The latter of which I previously described in much more detail on here but it would’ve been nice to go over in the presentation, ah well.

If you’re curious about FormEncode, I put a decent amount of info into it along with some examples. It’s worth checking out if you’re ever in need of data validation and converting data types.

Fri, 16 Sep 2005 00:00:00 -0700 <![CDATA[Separating Web Development Environments in Myghty]]>

Separating Web Development Environments in Myghty

On many occasions, its quite useful when developing web applications to have the webapp know whether its in a production/development/testing environment. Rails builds this into the framework and its rather easy to add this toggle throughout a Myghty webapp (or some other Python web framework) as well.

We’ll make use of an environment variable so that the webapp environment can be easily configured from lighttpd or Apache. This way just deploying the application under a different handler can toggle the web applications mode of operation. The other thing we’ll setup while we’re at it, is a variable to keep track of the absolute root of our web application. I’ve found it quite useful in many cases to be able to get at this information without hardcoding it in, this way its always accurate no matter where the program is.

There’s a few rather commands that’ll give you the information we’re looking for. It took me awhile to find this, so hopefully it’ll help someone else out there.

Getting our Absolute Location

Let’s assume our directory hierarchy looks like this:


Maybe you have the script called by mod_python or lighttpd somewhere else, I’m assuming it’ll be inside the root of your web application somewhere. In this case, the handler called by lighttpd is dispatch.fcgi. So dispatch.fcgi needs to figure out what the absolute path of the directory is above it.

Here’s the code that figures this out:

import os

    Set the prefix to our base path for the webapp
    myloc = os.path.join(os.getcwd(), file)prefix = os.path.normpath(myloc + ‘../..')

The myloc assignment gets the absolute file-name with path of the current file, in this case dispatch.fcgi. Unlike a normal os.getcwd(), this call doesn’t care what directory we happen to be in when we import the module, it will always be the absolute file-name of that file.

In case you’re curious why this happens, __file__ will return the path of the file relative to the working environment its called from. So combining it with the full path name of the current working environment results in the complete absolute path of the module no matter what the current context or working directory.

The prefix assignment uses the normpath call to strip off the filename, and backup one directory to our webapp root. This leaves us with the absolute path to our webapp/ directory.

Setting and Using the Environment variable

Figuring out if we’re in a special runtime environment is quite easy and looks like this:

import os

    MYGHTY_ENV = os.environ.get('MYGHTY_ENV') or 'development'

Now we can just test MYGHTY_ENV to determine whether to contain errors ourself, or drop them to the web (as you would want in development mode). We default to being in development mode, since this is what you’d typically run your webapp in.

To switch it to production mode, here’s what the lighttpd config looks like:

fastcgi.server = (
   ".fcgi" => (
     "www" => (
        "min-procs" => 2,
        "max-procs" => 4,
        "socket" => "/tmp/webapp.socket",
        "bin-path" => "PATH/TO/webapp/scripts/dispatch.fcgi",
        "bin-environment" => ("MYGHTY_ENV" => "production" ),
        "idle-timeout" => 120

To set the extra environment variables with Apache, use mod_env with the SetEnv directive which would look something like this:

# Make sure you have mod_env loaded, this line assumed to be in the VirtualHost
# block of your config
SetEnv MYGHTY_ENV production

At this point, you might’ve noticed (if you’ve used Rails) how similar my Fast CGI setup with lighttpd looks when compared to the some of the Rails examples for a lighttpd + Fast CGI setup. This is intentional, as I’m adding a Routes dispatcher to Myghty so it makes sense to layout my web application in a similar directory hierarchy.

Anytime you need to toggle some behavior depending on your webapp’s runtime context, just import os and check it as I showed up above.

Fri, 09 Sep 2005 00:00:00 -0700 <![CDATA[Where's Single Sign-On? Part 2]]>

Where’s Single Sign-On? Part 2

In a recent Wired article regarding One Login, reference is made to a new social style network called GoingOn. The article spends most of its time focusing on one site that hopes to aggregate functionality that currently is split between Blogger, Flickr, Friendster, and Bloglines (for the most part). However, the thing it misses is what I previously discussed regarding the lack of a working distributed identity system.

After looking around more, I’m happy to say there are indeed working identity systems out there. Unfortunately the most promised of them, the Liberty Alliance doesn’t seem to have much oomph behind it, but two others that I previously didn’t know about are now out there.

The first is from the folks at Microsoft, which they’ve called an Identity Meta-System (or something like that), which is described over at vnunet. It seems to be rather tied (or at least integrated heavily) to Microsoft technology (go figure!), and will be included in Indigo and other various Micrsoft technologies. As a mainly open-source coder, this has little appeal to me, nor am I about to start using Microsoft API’s to write my websites and web code. The standards utilized by Microsoft for their Federated Identity are generally known as WS-* for some reason I’m too lazy to investigate.

The second is much more appealing (to interested users and web developers), and has actually been around for a very long time in a primitive form (2000 is ancient by web standards). The home site appears to be the identity commons, and the current sole Identity Broker is 2idi, the organization behind the standards is XDI. They’ve made the entire code-base they run the Identity Broker on, open-source under the Affero General Public License to ensure that users are never locked into just one Identity Broker (Yea!).

If you’re curious how the Microsoft and Liberty Alliance methodology differs, idcommons has a useful FAQ addressing the differences.

The most exciting aspect for me, is that all the technology behind the XDI approach is completely open-source, and geared towards maximum user flexibility and empowerment. The user gets to move data between Identity Brokers, and every care has been made to ensure the user is never locked into a single Identity Broker. Actually, the most exciting part, is that it works right now. :)

They’re currently preparing to switch to a SAML-2.0 backed code-base, however the code they have only works from PHP, Java, and Perl. If you want to try it out, here’s how to get an i-Name, and you can try it out on those two sites. Also, a developer made a ISSO (I-name Single Sign-On) authentication system for WordPress which is pretty cool.

So what’s stopping ISSO from being used on more websites? It’s free, its open-source, its standards based, its not controlled by a commercial corporation….

It needs Python libraries!

I should mention, when I first wrote this as far as I knew, there was no Ruby version. There still isn’t a public one, but Victor Grey is fairly close to a Ruby version with a full Rails rig to go with it which I’m rather looking forward to.

Anyone want to help? I’m tired of remembering a zillion usernames and passwords, and with ISSO on the horizon I shouldn’t need to, all the Python web frameworks will be a bit better (at least the sites that use usernames/passwords) with an easy way to use ISSO.

By the way, for a useful overview of SAML, there’s a very detailed write-up of SAML2 on

Tue, 30 Aug 2005 00:00:00 -0700 <![CDATA[Handling Form data with Formencode + SQLObject]]>

Handling Form data with Formencode + SQLObject

Two of my favorite and most often-used Python packges are formencode and sqlobject. Using these packages together is done fairly often, but I’ve rarely seen any documentation describing a full form display, validation, and insert into a database of new data from said form.

In this example, I’ve tried to wean down the code as much as possible to get a very concise snippet (only 12 lines of web page Controller code) that will:

  1. Display a new form to the user
  2. Handle form submission
  3. Validate the form
  4. Coerce form values into the proper types for the database
  5. Insert the form data to the database, or
  6. Display the form with error messages, and maintain their values

The nice thing about using formencode, is that 3 of the 6 steps I listed above are handled by it in a fairly automatic way. Formencode will check the data given a form schema, coerce the values into the Python types you ask for, fill in errors should they occur, and maintain the existing values.

I’ll be using Myghty for this, but since all I’m really pulling from it is the request args, it should be pretty obvious what to change for whatever web framework makes you happy.


First, lets take a look at our basic form:

# myform.myt
<head><title>basic form</title></head>
<form action="/mypage" method="post">
Username: <input type="text" name="username" size="26" />
            <form:error name="username">
Age: <input type="text" name="age" size="3" />
            <form:error name="age">
<input type="submit" value="Send it" />

To validate this, we’ll setup a formencode form schema to run this through. I should note at this point, that the formencode web documentation kind of sucks. However, the doc strings are plentiful, and extremely useful for figuring out which validator to use in addition to having examples of each. To keep things organized, I usually place related form schema classes under the same module and import it as needed.

The other thing you might notice about the form, is that it has form:error fields. These are used by the formencode parser to put in the error message that the validation triggers. This lets us put the error messages right under the boxes they occured in. The drawback is that we have to process the form before first displaying it to strip out the form:error fields.

Here’s our simple schema to validate the above form:

from formencode import schema, validatorsclass UserInfoSchema(schema.Schema):
    allow_extra_fields = True
    filter_extra_fields = True

    username = validators.String(not_empty = True, max = 50)
    age = validators.Int(not_empty = True)

Hopefully the above should look pretty obvious. The allow_extra_fields bit is needed so that we can pass the entire request argument dict into formencode without it tripping up if there’s “extra” keys it didn’t expect (like the submit button). Since we’re going to be passing the dict we get back from formencode directly to sqlobject, we include filter_extra_fields to remove anything that our sqlobject isn’t going to like.

The form schema needs to include all the fields the database is going to take, since we’re stripping off anything it doesn’t mention. The Int validator not only ensures that the value is an int, but will change it into a Python integer in the process.


Now that we’ve handled validation and value coercion, lets have a look at the sqlobject class. I’m going to “cheat”, and assume your database for this was created like so (in Postgresql):

create table user_info (
    id serial primary key,
    username varchar(50),
    age int(2)

Since I’m feeling lazy, we’ll rely on SQLObject to pull the table info from the database giving us a SQLObject class like so:

from sqlobject import *
class UserInfo(SQLObject):
    class sqlmeta:
        fromDatabase = True

    Personally, I think if you just leave the whole thing empty and put pass in for the body, it should assume you want it populated from the database…. but the above will do the trick. Please note I'm using the sqlmeta class to define this, which is used in the recent svn builds of sqlobject. If you use the release on the site, you could replace those two lines with _fromDatabase = True instead.

    Putting It All Together

    Now that our form, validation, and sqlobject schema is all done its time for the meat of the matter… the web page controller. Getting this function called will vary depending on your web framework, so I'll just assume you can figure out how to get it called, here's what it looks like in Myghty using implicit module components:from formencode import htmlformfrom ourschema import UserInfoSchemafrom oursqlstuff import UserInfo

    def mypage(m):
    html = m.scomp('/myform.myt')   # load the form into a string
    form = htmlform.HTMLForm(html, UserInfoSchema())
    if m.request_args:
        form_result, errors = form.validate(m.request_args)
        if errors:
            errorForm = form.render(m.request_args, errors)
            UserInfo(**form_result)  # database insert

And there you have it. In a brief 12 lines, we handle displaying a new form to a user, and handle form submission, validation, and database insertion while ensuring that the string values are coerced as needed before database insertion. This task is done quite often in web sites, so making this task as painless as possible is a real time saver.

Hopefully this will help out anyone out there, who was wondering about quicker and easier ways to handle cases like this. If you have any thoughts/suggestions on how to streamline this further, be sure to leave a comment.

Thu, 25 Aug 2005 00:00:00 -0700 <![CDATA[Fragmenting A Framework Userbase]]>

Fragmenting A Framework Userbase

I’ve been thinking a lot lately about web programmers and the web frameworks they choose, or don’t choose, and why. I’m mainly going to talk about Python Web Frameworks as the majority of them have small communites, and possible reasons this could be.

I only started using Python for web development about a year ago, and it took me about a month to settle down on a web framework. In that time, I looked over at least a dozen different frameworks. There’s so many python web frameworks, quite a few people have actually setup entire pages and sections of their site just to covering them all.

I think part of the reason for the proliferation of frameworks is because of the nature of many Python programmers, as I briefly mentioned in a prior post on Making Decisions for Others.

The recent appearence of Django on the Python web framework scene I’m sure has quite a few other Python web framework developers wondering, “Why isn’t the web framework I made getting this much attention and use?”

A Common Base

Many of these same people would like to blame it on hype and good marketing. While that will certaainly boost initial usage, I don’t believe it will create a lasting user base. I think a huge driving factor behind Rails and Django, besides for the hype and marketing, is the fact that both of them make a lot of decisions for you. These decisions start the users all off at a common base of understanding.

The linear progression from:

  1. Never used the framework
  2. Wrote the tutorial app
  3. Wrote their own basic webapp
  4. Wrote an advanced web application

Makes it easy for people a step or two up, to help other new users join them. Because the steps they all take are the same steps to achieve greater understanding of the web framework, they can easily help new users get to where they are. Most, if not all the other Python web frameworks I’ve seen are so flexible its hard to have a common base of understanding amongst new users. The process looks more like this:

  1. Never used the framework
  2. Researched the frameworks options and choices to find a possible starting point
  3. Wrote a basic web application using method X
  4. Wrote an advanced web app using method X

The flexibility of the web framework becomes an obstacle to a strong user-base in this case, as it fragments the users by the methodology they’re using to build their webapp. It also reduces the common re-usable components available, since different users will utilize different options of the framework and have possibly very different starting points.

Have a Tutorial Application

Also lacking from many Python web frameworks is a clear and obvious Tutorial application. Ideally the front page of a Python web framework should be an obvious path to become an experienced user of said framework. Such as:

  1. Install the framework
  2. Write a basic tutorial application
  3. Look here/there for instruction as need to write your own more complex application

A good tutorial should leave a user feeling confident that they know how to install and start with a common base for writing their own web applications. It’s also amazing how many problems people can have just getting a framework installed and running in a minimal configuration. Having a tutorial that leaves them with a functioning web application gives them a big leap forward.

Since many users will do the first tutorial web application, other new users can give help to even newer users that run into a problem. This is where the common base effect really provides some power.

Methods of Fragmentation

The Python frameworks I’ve tried and used have fragmented their starting points and users in various ways. All of them as a result of their “flexibility and power”. Here are a few common trends of fragmentation I’ve seen:

  • Let the user choose various template language schemes (Use ZPT, or Cheetah, or…)
  • Let the user choose from web paradigm (MVC, page-driven, pipelined…)
  • No base or example configuration for a fully working webapp (So everyone sets up their first application slightly differently)

The last one I listed, is probably the easiest to solve, especially with useful web framework template creators like Python Paste. Obviously, removing the first two will be seen by many Python web framework developers as undesirable. I think it’d really help the users though, as it gives them more in common with each other. If they all use the same paradigm, and the same template language with your framework, their ability to help each other increases and they feel confident they made the “right” choice as well.


I’ve assumed for the purpose of this post, that Python web framework makers are interested in having a large user-base. This isn’t always the case, I’m sure some just want a small, very experienced user-base that isn’t going to be asking basic questions like, “I can’t connect to my database like you show in the tutorial”.

I can understand that, but for the other Python web framework makers out there, try and consider some of the things I mentioned. There are a lot of Python coders out there, and a lot of them can live without having 4 template language choices and 2 different design paradigms. So when adding that feature that’d let people get so much “power and flexibility”, will it fragment your user-base?

Mon, 22 Aug 2005 00:00:00 -0700 <![CDATA[Web Design for Programmers?]]>

Web Design for Programmers?

Designing web pages is a pretty annoying task for most web programmers I’ve met, including myself. I’ll be first to admit it, I’m not too hot when it comes to web design. I haven’t even bothered changing the default theme for the blog engine I’m using.

What would be really slick is a sort of Web Design for Programmers book, ala Joel’s User Interface Design for Programmers writings and book. I’ve read quite a few books on web design that try to go over basics, but in the end for some reason it always feels like as a programmer, I’m missing those creative juices that result in an unique and interesting website.

There’s a fairly nice series with this exact title over at that goes over some of the fundamentals. What I think would be ideal, is a website entirely devoted to this topic, with examples and layouts that are good starting points for your own web projects.

I searched around on Google quite a bit, and only came up with the PeachPit articles I cited above. A List Apart is a rather nice site to learn specific web design techniques, but doesn’t really focus on application of the techniques as related to the website on the whole. This part is left up to the designer, which doesn’t help a programmer much.

Having a beautiful website, or even just an aesthetically pleasing one can make a big difference even to other programmers, whether they want to acknowledge it or not. One of the clearest examples of this I’ve seen is with web frameworks and toolkits. A great looking website can get people into a toolkit/framework that might’ve otherwise moved onto a site that looked more “interesting”.

Thu, 18 Aug 2005 00:00:00 -0700 <![CDATA[Language Specific Comparisons]]>

Language Specific Comparisons

I’ve read quite a bit of Paul Graham’s articles regarding Lisp, how awesome Lisp is, how much of a dufus one might be for using a language programmed for dufuses.

Now, before I start, I should mention that these posts are quite old, and I don’t really want to start a flame fest over this again. However, I couldn’t help but notice today with a problem I had in my code, how easy Python made the solution (Zach points out the Lisp solution is quite succinct as well).

Without a doubt, Lisp excels at recursion, function/code generation, and closures. This frequently leaves Lisp looking like a god when you see how many lines of code other languages take to replicate the examples Paul chooses to compare (which always revolve around the things Lisp excels at as Paul Prescod points out)

Today, I came across a fairly common case, where I had a function taking keyword arguments and collecting them all. That would look like this in Python:

def somefunc(**kargs):

Now, if I want to take two of those keyword being passed in, and set some defaults so they’re not required but will always have something set in the function body, its rather easy:

def somefunc(keyone='default', keytwo='anotherdefault', **kargs):

So in one line, I have now added two defaults that will be available in my function body for use. [STRIKEOUT:How succinct is this in Lisp?] What about in Ruby?

I cite two languages that came off very well in the Accumulator Generator shoot-off. [STRIKEOUT:They] Ruby doesn’t do quite as well in this case, which I’ve actually encountered far more than the code generation cases Graham is apt to cite. (I actually like Ruby and am now using it quite a bit, I’ll be quite happy when it has keyword args)

What’s even worse is where in this thread that I referenced above, Graham at the end says he has no clue how one would create a basic class to handle accumulation in Lisp. [STRIKEOUT:I find that rather disturbing that something so easy in Python has Graham saying, “God only knows.” how to do it in Common Lisp] Richard points out below that this is because PaulG is rather keen on macros, and not so big on OO. That makes complete sense to me as Lisp did not start out with OO features, those were added later when it was all the rage. When working in an elegant functional language like Lisp I can see why one would never have to consider OO.

That alone should indicate that many technical comparisons between languages can be easily skewed towards a language by using examples that heavily favor built-in abilities of the language one chooses to boast about.

In the end, I’m left with the belief that different languages have different applications. Claiming one language is the be-all, and is always better for any task is about as false as claiming that a language has no problems or issues.

Within certain realms it does make sense to compare languages, scripting vs scripting, functional vs functional, etc. But leaping to a comparison of functional/dynamic-typed vs non-functional/static-typed is typically going to result in some strange claims.

Anyways, if you feel like commenting, try and come up with an example of where Language X (that you use) has a very succinct solution compared to Language Z (all the others). It’d be great to compare some examples and see areas in which different languages fall flat on their face when it comes to succinctness. (Ie, in most dynamic languages, you’d have to add several lines of code to ensure variables are the type you want. A feature/annoyance of static-typed languages)

Update: An anonymous user kindly informs me that there’s no foundation for my claim that some languages are better in certain realms than others, unfortunately the anonymous user fails to say why.

Ruby has no keyword arguments currently (Ruby 2 will have them and keyword collectors ** as Python does). To even approximate my Python example in Ruby, you’d first need to declare the argument as optional which has the side effect of packaging it into an Array. Wheras in Python **kargs packages up the rest of the key/vals under a dictionary. If someone would like to write out the full translation in Ruby, I’d be happy to put it up here, but I doubt its going to be pretty (until Ruby 2).

Zach was helpful and provided an example showing that the specific task I cited is fairly short in Lisp as well, looking like this:

(defun somefunc (&rest kwargs
                 &key (keyone "default") (keytwo "anotherdefault")
                 ; ...

I would like to make it very clear that my point is not that Python is better, but that technical comparisons can be warped to favor certain languages. This is the same point Paul Prescod makes, and what I’d actually like to see is more technical comparisons that make this point obvious.

Thu, 18 Aug 2005 00:00:00 -0700 <![CDATA[Where's Single Sign-On??]]>

Where’s Single Sign-On??

Really. I mean it. Where’s Single Sign-On?

Liberty Allienace was started 4 years ago to provide a competing alternative to Microsoft’s Passport. It’s taken Liberty Alliance years to get to any sort of usable starting point, while Microsoft has actually had developer code out there and usable since 2001. I’ve even toyed with some Passport login code in Perl 4 years ago.

Since that time, Passport is used rarely if ever on the web, and the Liberty Alliance has tons of docs to show, implementation specs, and even a Java toolkit for Liberty Alliance. Yet I don’t see many (or any?) websites using the federated login of Liberty Alliance. Where’s my single sign-on??

In the time of Web 2.0 a toolkit for the major scripting languages should exist. Why isn’t there one for PHP? Perl? Python? Rails? Has anyone done any calculations in how “good” a website or web service has to be before a user is ready to take the plunge to setup yet another username/password? This is an obstacle that gets in the way of a website having more members, and anyone in the business of getting members should want as few obstacles as possible.

I know I’m not even signing up or using some services purely because I can’t deal with yet another username/password to remember. I can’t even remember the ones I’m currently signed up for, so I have to use software that does password/site remembering for me, in my case Password Wallet.

So what are the supposed strenghts of web services/applications?

  1. You don’t need to install any software
  2. You don’t need to worry about upgrades
  3. You can use it anywhere

Out of these, #3 is the one that has been lost nowadays thanks to a password/username program. I can’t get to any website I’ve registered on without my personal computer, because I can’t remember that many usernames and passwords. Some will say, “just use the same user/password everywhere”, and many people do. Even this doesn’t always work though, if someone else signed up using that username first.

It’s been FOUR YEARS since the big hub-bub over Microsoft Password and Liberty Alliance, and we still don’t have handy little toolkits in the most commonly used scripting languages to interact with them. I don’t know about you, but I find this pretty sad.

Tue, 16 Aug 2005 00:00:00 -0700 <![CDATA[Switched to Typo and some MT3 to Typo Migration Notes]]>

Switched to Typo and some MT3 to Typo Migration Notes

So tonight I took the plunge, and converted the blog to Typo. It’s totally slick, and I’m digging it. Plus when I want to tweak stuff, I don’t have to mess around with Perl.

It’s currently lacking a feature I had grown accustomed to, Technorati auto-pinging. I’m thinking this would be a good thing to contribute, so I’ll start digging into the code and see what needs to be updated.

Migration was also a bit of a hassle, as it appears the MovableType 3 migration script only is happy if you use MySQL for your old and new blog. Getting it migrated to Typo required dumping the MovableType database using pg_dump, and loading it into the Typo database. Then the mt3.rb script needed to be edited to remove all the table prefixes. After this, the script happily imported all my prior posts, comments, and trackbacks.

Sat, 13 Aug 2005 00:00:00 -0700 <![CDATA[Explaining Recursion to a Non-Programmer]]>

Explaining Recursion to a Non-Programmer

I was going on a walk the other night with my wife, and I think she noticed my far off expression and silence. She asked me what I was so thoughtful about, and I admitted my brain was a bit stuck on solving a complex algorithm involving recursion. At that point she asked me what recursion was, she’s very smart, just not a programmer. So I figured there must be some way I could explain recursion without referring to code, syntax, functions, etc.

Here’s the little explanation I came up with on a moments notice:

Let’s say we have a guy named Joe. Joe has some problems and special abilities.

Joe has the neat ability to make temporary clones of himself, when they’re done doing what he asks them, they disappear in a puff of smoke. Joe unfortunately has issues with math. If you hand him a basket of apple’s, he couldn’t tell you how many apples are in the basket, just seeing all those apples at once confuses him.

If you tell Joe a number though, he can easily add 1 to it. So you tell Joe, “Hey, if someone gives you a basket of apples, take one out, clone yourself, and hand the basket to your clone. If the basket is empty after taking the apple out, tell the one that created you there’s 1. When you get a number from the clone you made, add 1 to it, and tell that to the clone that created you.”

Since Joe’s clones know what he knows, they all know these directions as soon as Joe clones himself. So you proceed to hand a basket of 3 apples to Joe.

Joe takes one out, and hands the basket of 2 apples to his clone. Clone 1 takes an apple out, and handes the basket of 1 apple to a new Clone (Clone 2). Clone 2 takes out the apple, sees there’s no more left, and tells Clone 1, “Hey, there’s 1”. Clone 1 adds 1 to it, and tells Joe, “Hey, there’s 2”.

Joe adds 1 to this, and tells you, the basket has 3 apples in it.

You can hand Joe a basket of however many doodad’s as you want, and he’ll happily keep cloning himself until sooner or later a clone takes out the last doodad.

After this little story, you should be able to explain that:

  • A Terminating Condition is when Joe should stop cloning himself
  • Joe is recursive because he clones himself and repeats the task

If anyone else out there has a more elegant or shorter way to explain recursion to a non-programmer, I’d love to hear it. This story could also set you up to explain such fun concepts as tail-end recursion or the difference between truly recursive functions and one that could be easily re-written in a loop. I’ll leave that to someone else to translate to non-computer speak.

Thu, 11 Aug 2005 00:00:00 -0700 <![CDATA[Rails Routes port project page is up]]>

Rails Routes port project page is up

I’ve gotten a project site setup for the Routes port I created. Currently the port passes all the applicable unit tests I copied from the Rails route testing package. It’s very rough right now, but a few people expressed an interest in seeing the current code so now its up.

I will be putting it under a BSD license shortly to make it easier to use with whatever other Python Web Frameworks are interested in using it.

The noticable area its not working in right now, is handling GET args and generating them with extra keys. So I wouldn’t consider this code ready for use in an actual website yet. Hopefully the remaining features to implement will be done shortly, and I’ll have some examples of using Routes with Myghty.

Wed, 10 Aug 2005 00:00:00 -0700 <![CDATA[Porting Routes from Rails]]>

Porting Routes from Rails

I’ve begun porting the Routes system from Rails over the last weekend. It’s definitely been a good brain workout, as the one Nicholas Seckar wrote that drives Rails is quite a brain-fuck. His version actually does quite a bit of code generation based on the routes one sets up. At first I tried to do a fairly direct port of the Ruby code behind this to Python, but my brain just couldn’t wrap itself around what he was doing (He’s told me that quite a few others have said the same).

This left me kind of discouraged, and I left the project (more of a thought at that point) alone and proceed to replicate most of Rails ActionMailer in Python instead. Partly because I liked how ActionMailer assembled mail, and partly because after hearing a talk by Alex Martelli on Metaclasses and other Python “black magic” I just had to find some excuse to use a metaclass.

Why not just use Rails?

Now, before anyone starts yelling “Just use Rails!” at me, I should explain something. I am using Rails, and I rather like it. However, most companies wouldn’t be too thrilled with their employee’s changing the entire codebase of the company every year when the latest, greatest, way to write webapps comes along. Plus, since I already use Python extensively, a lot of stuff that makes life pleasant in Rails-land can be fairly easily adopted to Python-land.

Do I really want to re-write 5k or more lines of Python libraries in Ruby just so I can use Rails? Should the entire web team learn a new language and framework just because its a bit better than the current way? The time spent doing all this wouldn’t be paid off for at least a year or more, if that (and maybe some other language+framework will be the new thing by then). When it comes down to normal Python libraries, I don’t save any lines of code by moving to Ruby. So in the end, it makes sense to adopt part of Rails that make my life easier.

In Case You Didn’t Know

The Routes system from Rails is used by Rails in a lot of functions. Virtually all functions that require a URL to be written (url_for, link_to, form_tag, etc) all use the Routes system to figure out what URL to put in place.

In the Python world, as far as I know (and I’ve looked quite a bit), there is absolutely no web framework that will convert from controller/action -> URL.

Update (9/26/05): Phillip J. Eby rightfully pointed out that other Python web frameworks can do this. I should’ve been more specific in that none of them operate in this manner. Since putting word out about Routes I also found out Aquarium can designate routes for dispatch/generation in this manner, however Aquarium cannot do automatic route lookup to find the shortest URL. This ends up hard-coding the template URL to the route.

Plenty of web frameworks are happy to convert a regular expression into a dispatch command and let you tweak it (Myghty, Django, etc.) but they are all helpless if you want to generate a URL from the commands they used to dispatch.


Currently, I have URL generation (controller/action -> URL) and URL recognition (URL -> controller/action) working and passing all the tests that I’ve ported over from the Rails route testing. Since my version of Routes is independent of the framework, its my hope that multiple Python web frameworks will be able to incorporate the finished Route system I’m building. The URL -> controller/action results in a dict, that the web framework can choose to deal with as it pleases.

The first framework I plan on integrating this with is Myghty as the resolver system is fully user-customizable, no hacking on the framework source is needed to plug it in. The dispatch model will then follow Rails and make the same decisions for the user (controller goes in appdir/controllers/,, etc.) that Rails does.

Once I add a few more tests, and try and clean up my code some, I’ll put it out there. I’m considering a BSD license right now, in the meantime, here’s a sample:

>>> from routes import Mapper
>>> m = Mapper()
>>> m.connect('page/:id', controller='content', action='show_page', id =1)
>>> m.connect(':controller/:action/:id')
>>> m.generate(controller='content',action='index')
>>> m.generate(controller='content', action='show_page')
>>> m.generate(controller='content',action='show_page',id=4)


My implementation choice was to rely heavily on the speed of set operations using the built-in set/frozenset functions in Python 2.4. Under Python 2.3 the sets module provides an ImmutableSet which I’ve also tested my port with. I was actually quite impressed with how fast the sets module is in 2.3, but there is a three-fold speed increase using Python 2.4 built-in sets.

Even under 2.3 the speed should be more than sufficient for most applications, plus, this is a first revision of the code. I’d hardly expect it to be the peak of efficiency right now.

What’s Missing

I currently haven’t implemented memory of the match that got to the current page. Rails does this so if you’re inside a template that was matched by ‘post/show/20’ and you call link_to :id => 30 it would give you a url of ‘post/show/30’. It does this by remembering the match that got you there. Since this match would be setup inside the web framework, I’m considering how to implement it without tying one to a specific framework.

The other noticeable feature missing is Named Routes which act as a sort of macro on top of the normal url generation calls. This shouldn’t be too tough to add at a later point.

Parting Thoughts (ala Jerry Springer)

Before you submit some comment saying I’m an idiot for porting this and should drink more Ruby kool-aid, don’t, please. I’m already drinking it, and it tastes quite good. But just because I like Ruby, doesn’t mean my taste for other languages has changed. Languages have their strong points, and they have things they lack (Yes, they both lack some things). When I’m writing Python, my web framework of choice is so very close to perfection for me, Routes is one of those things that would make it that much sweeter.

If you’re interested in adapting this to your Python web framework and want to start playing with the code I have so far, send me an email and remember boys and girls, be safe.

Tue, 09 Aug 2005 00:00:00 -0700 <![CDATA[Technorati Doesn't Know Your Username Either]]>

Technorati Doesn’t Know Your Username Either

Update: Technorati informs me that they’re working on fixing this problem.

So I figured I’d log into Technorati, update my profile tonight. Problem is, I apparently forgot my username. So I go to the login screen and don’t see anything about what to do if you forget your username. That’s ok, there’s a Lost Password link, that might help. Clicking the Lost Password brings you to a screen that lets you enter your username OR your email address to receive instructions to reset your password. (Still nothing about if you forget your username)

The email looks like this:

Greetings, !

    We received a request to reset your Technorati password.

    To create a new password, please visit this URL:

    If the above link is not clickable, please copy and paste the link into your web browser.

    Thank you for using Technorati.

How useful, clicking this link brings up a very clean, simplistic interface that lets you enter a new password and confirming that. Once you enter these two words and hit enter…. you get the same exact screen, only with the two fields empty .

Did it work? Did you change your password? No feedback whatsoever is given, except that if you try and do it again, you get an error….. which is even worse, because does this mean you did changed it or not?

Of course, at this point, lets remember the real issue is that I don’t remember my username. So I go back to the helpful screen asking me to enter my username OR email for password reset, and start guessing possible usernames I might’ve used (and for me, that could be quite a bit). Technorati helpfully returns an error screen until I finally enter a username that exists, at which point it tells me it sent the email.

Technorati, if [STRIKEOUT:you’re listening] (they are), please fix this…

Sat, 06 Aug 2005 00:00:00 -0700 <![CDATA[Making Decisions for Others]]>

Making Decisions for Others

Please Note: Reading the last post would help greatly for this one, and this post does end up comparing aspects of Django with Rails in terms of making decisions rather than actual usage.

As I mentioned in my last post, Rails has been greatly helped by the decisions it makes for you. It decides the layout of your application directory, the directories you put stuff in, where you put what parts, etc. These decisions thankfully were done by someone familiar with good application design and had most likely even read Martin Fowler’s Refactoring to Patterns.

What results is a clean, well thought out application that you didn’t even need to think out very much. Before you know it, you fall into these useful and powerful design patterns without even thinking about it (I’d hope you would at least, they’re good patterns for a reason).

A lot of web frameworks make the assumption you are familiar with these patterns already, will setup your application to follow them, and use them in a way that makes sense. Part of the reason for this is because some web framework developers are rather appalled at the thought of making these decisions for the framework user. To be making those decisions for the framework user is looked at as holding their hand, and typically the programmers who write frameworks don’t need any hand holding and don’t expect their users to either (I’m assuming this based on the frameworks I’ve tried).

For example, during Ian Bicking’s test run with Rails he noted that having such a directory hierarchy setup for him put him off a little and mentions people arguing about file layout. I’ve seen very little, if anyone else, even comment on the directory structure that Rails lays out for you. The other irony of this particular post is that a lot of the stuff Ian finds “wrong” about Rails is actually Ruby’s “fault” (Implicit passing of names, mix-in’s, etc.). But moving on….

Ian clearly doesn’t want to lay down the law for users of his software, neither do other Python web framework developers. Having choice in how you lay out your application, with no clear direction what goes where, is considered a good thing. I don’t think there’s anything wrong with this approach, if the users you’re aiming your framework at are expected to be experienced, professional, programmers that are quite familiar with web applications and what works best for them. This does mean the audience will not be that large, and I think thats clearly reflected in the small communities most Python web frameworks have. While their communities are small, I’ve found them to be highly skilled and you get great answers to questions on their respective mailing lists.

The one Python web framework that has almost overnight gone from hardly any users, to at least a few dozen (probably more, but even a few dozen is quite good for a Python web framework), is Django. Django makes a lot of decisions for the user, more so even than Rails by a pretty good amount. While Rails was designed as a web application toolkit, and extracted form a large webapp, Django is designed more as a CMS web application toolkit as its been extracted from newspaper-style web sites. Thus, many of the decisions Django makes for you assume you will want users, user-access to the database objects, etc.

The question is, if someone else is making decisions for you, how important is it that they be good decisions?

One of the decisions the Django team made originally irritated me quite a bit. The Django framework is probably about 95% MVC in nature. It has database files that clearly are acting as Models (which it calls models), logic oriented files traditionally called Controllers (that it calls views), and template files that display the results to the user in a typical View role (though Django calls them templates).

The FAQ on the Django site notes that if you squint, sure enough the observations I just made do pop out, however because of the 5% non-MVC I mentioned they decided the names didn’t work. Rather than calling them by what they obviously were (or closest to), they swapped around names in a way that I think is bound to confuse Django programmers when they go into a real MVC environment and tell someone they edited the view when in fact it was the controller. If they had come up with completely different names that didn’t relate to the MVC paradigm at all, I think that would help to avoid later confusion on the part of a Django user.

Despite the naming of these parts, the decision made for the user is clearly for a MVC-style (or influenced) environment which I think is great ( please change the naming! ;) ). So what is the point of this whole thing?

I think its a good thing for developers of toolkits and especially frameworks to make some more decisions for the end-programmer of their work. This makes it easier for not so experienced programmers to get started quickly, and if you’re using a good design pattern even the experienced dev’s wont have anything to complain about. The two frameworks that currently are getting a lot of attention make a lot of decisions for the programmer using them, and this definitely increases the user-base (assuming you want a large user-base). Even though I know good practices on laying out stuff, it makes my life that much easier when its done for me.

Sat, 06 Aug 2005 00:00:00 -0700 <![CDATA[Who's Attracted By Simplicity?]]>

Who’s Attracted By Simplicity?

I’ve been reading a lot of various blogs lately, quite a bit of loud thinking and some 37 signals vs noise as well. A mantra I see over and over regards simplicity, and focusing on doing a few things extremely well. This is reflected in the 37 Signals products, Basecamp, and Backpack. It also bubbles over into the web framework these are all built on, Ruby on Rails.

The Rails framework has a lot of appeal, in large part because of how easy and simple it makes the vast majority of mundane web programming. The sacrifice is completely intentional, and summed up with “Convention over configuration”. This mantra is in the Agile Development with Rails book, and has been uttered on Loud Thinking as well.

Ironically, the people I know that dislike Rails do so because of the “lack of flexibility”. Many decisions have definitely been made for you when you create a Rails application, though I see no harm in this because every decision follows what most regard as the best practice and pattern for that context.

While Rails and Ruby are skills I’m currently learning, I spend a lot of time working with Python. The Python programmers I’ve talked to generally regard lack of flexibility as very unappealing, and would happily sacrifice simplicity or making any decisions for the programmer (I’m referring to Python toolkits/frameworks intended for use by programmers). The assumption is that the programmer will read how to use the tool, see that there’s a half dozen ways to use it, and plug it in as he best sees fit.

Having more ways to use it is a good thing, with no clear direction on any sort of best practice or pattern provided typically. Strangely enough the majority of toolkits and web frameworks for Python generally have rather small groups of users. Is this a coincidence?

My favorite framework, Myghty recently got the very cool ability to call module components implicitly . This added two more mechanisms of dispatch control to the resolver, very powerful stuff. It’s also rather confusing, not because how it works is actually difficult, but because now the way you should use it has become even less certain with the added choices.

The fact that there’s so many ways to use it has definitely hampered the ability of people to understand it and use it effectively. The powerful flexibility has made it powerful though, for totally different paradigms to be built on top of it. For example, you could replicate the pipelining methodology that AxKit uses by configuring the resolver to pass the different extensions through recursively until an “end” is signaled. Or you could configure Myghty to dispatch control to a Controller exactly how Rails does (which I’ve worked on doing).

In the end though, why should I waste my time replicating what another framework already has? I’ve got to admit, even though I know how to pick up the tools and plug them in, I’m still attracted to simplicity.

Fri, 05 Aug 2005 00:00:00 -0700 <![CDATA[Podcasts replacing Radio?]]>

Podcasts replacing Radio?

Was listening to another Daily Source Code by Adam Curry today, it was an older show from a few days ago. I seem to have a love/hate relation to his show, on some days its really cool, and other days I’m tempted to hit un-subscribe.

This is partly because of the premise of Podcasts in general that I hear, and partly from the problems I see in Podcasting. If you haven’t listened to the Daily Source Code, its fairly good. It’s part informative (usually techie), part random mumbling about what happened to his shoulder today, and occasionally he plays some music.

One of the mantras I hear a lot, especially on this show, is how Podcasts are retaliation to the control and limits of the FCC. If Podcasts are going to challenge radio seriously, they need to effectively replace what so many people enjoy right now. The radio I’m referring to is full of music. Podcasts for the most part are full of talking.

Adam is well aware of this problem, and has helped (I’m not sure of the level of involvement) launched the podsafe music network. This is very very cool, and I look forward to when I’ll be able to get Podcasts that are 95% music, 3% ads (gotta have some way to fund the bandwidth), and 2% DJ’s telling you what you just heard (Even though it’ll be in the podcast shownotes).

I wonder what he’s doing to bring the major labels on board, and even the larger indie labels as they’ve already locked up a bunch of great talent that needs more exposure that radio just isn’t providing. The PMN is a great first step, and I can see its definitely aimed at getting artists interested in putting their stuff on it for more play.

In the end, I’m not even sure how much longer a potential market of significant size exists for Podcasts. The known market is people driving somewhere, typically to work (when I listen). The vast majority of Podcasts aren’t too hot on quality, typically have a poor script (if any), and don’t play any music that interests me. With the increasing subscriptions to XM and Sirius (both free of the FCC), and the coming of digital radio, Podcasts have quite a bit of competition for listeners.

While some people love hearing others talk or mashups (I’ll admit there’s some cool ones), will it be enough to reach the 100 million listener march?

Thu, 04 Aug 2005 00:00:00 -0700 <![CDATA[Dissing Programming Langauges...]]>

Dissing Programming Langauges...

A friend recently brought to my attention that I seem to have a knack for insulting a language based on a quick overview without actually getting into it. After thinking about this awhile I’ve noticed it definitely seems to be more of a trend than I’d like.

The latest example of this was Ruby, but before that Python (Lisp before that). With each language, I’d find new things to nitpick on before really diving in (Which I always seemed to do anyways). As a result of this, I think its time for me to really grow up and stop picking on new languages till I’ve at least had the time to write a few apps in them and truly understand where they’re lacking.

As a quick side-note, here are the things I found to nitpick on before actually learning these languages:

  • Objective-C: God, wtf is up with these [] all over the place?
  • Lisp: God, wtf is up with these () all over the place?
  • Python: God, wtf is up with these _____ all over the place, and who in their right mind uses white-space to designate blocks?
  • Ruby: God, wtf is up with these {|| } things all over the place?

Obviously I had issues with syntax, I hope to have purged myself of these issues as of this writing. Remember, sometimes a language has a damn good reason to have crazy characters all over the place.

Wed, 03 Aug 2005 00:00:00 -0700 <![CDATA[The Myghty Python Web Framework]]>

The Myghty Python Web Framework

Myghty is a powerful web application framework that builds on the strengths of Perl’s Mason using my favorite language, Python. While Myghty also handles page templating, it has some very powerful features that make it a full fledged framework in my opinion.

Wondering if Myghty is something you should check out?

I would highly recommend looking into Myghty if you’re looking for any of the following:

  • Clean, unobtrusive templating language for integrating Python data and HTML
  • An easy way to write pages that inherit their look/feel without needing .include statements
  • Re-usable code that you can carry from one project to the next
  • MVC programming paradigm , or page-driven paradigm (Myghty doesn’t care, it can be anywhere in between as well)
  • Advanced caching ability, tunable to just the time intensive sections of a page
  • Run as WSGI, FCGI, SCGI, mod_python, or stand-alone with hardly a single change to your code (to switch between them)
  • Architecture that scales, from a single developer’s personal project, to a multi-developer’s company website


Myghty first came into existence a little over a year ago when Mike Bayer looked around at the Python web application frameworks and couldn’t find one that really sucked him in. Having used Mason for various projects, but not finding anything as elegant as it for Python he decided to port Mason to Python.

This ported code was based on the Perl code almost line by line, resulting in a Python web application framework that worked almost exactly like Mason, except with Python as the base language. It wasn’t very long before Mike started adding to Myghty, such as Module Components, and threaded features not found in Mason.

Terms Used

MVC – Model, View, Controller paradigm is typically used to describe an approach that separates the code in these 3 steps rather than mingle them all in one place. Model usually refers to the database and its abstraction, the Controller handles the logic of dealing with the request, getting the data from the Model, and passing it to the View for display to the user.

Component – A component in Mason is what a template is known as. A template might display a full web-page, just one small section, or send data to other components. Components typically correspond to files on a filesystem, and appear as normal HTML pages with a few lines of Python in them, although they’re far from normal HTML pages…

Resolver – Used to describe the process of determining what to do with a given URL. Myghty has some incredible flexibility here.

Distinctive Features

Module Components

In Python, your modules contain all your Python code. In Myghty, the Component contains a mixture of Python with HTML. To handle a MVC approach to development, Myghty has Module Components. These are a cross of a Myghty Component with a Python Module, that leaves you with a fairly normal Python Module. It’s called directly from Myghty in a similar way to how Myghty would call a Component. These Module Components can be explicitly defined in the configuration, or called implicitly.

The Module Component allows you to directly handle URL’s, and respond from within the Module by sending data back to the client or processing data then sending it on to a normal Component for processing. Module Components work within the Resolving process, and have control passed to them according to a configuration mapping . The mapping uses regular expressions to match URL’s to modules that contain the method to be called, like so:

module_components = [
    {r'myapplication/home/.*' : 'myapp.home:HomeHandler'},
    {r'myapplication/login/.*' : 'myapp.login:LoginHandler'},
    {r'.*/cart' : 'myapp.cart:process_cart'},
    {r'.*' : 'myapp.home:HomeHandler'}

This would map the URL http://yourapp/myapplication/home/blah to the HomeHandler component class, in the home module file, under the myapp package. The method in the HomeHandler class called can be done in two different ways, implicitly, or explicitly. For more details on this, read the Myghty docs on Module Components

While other frameworks implement an MVC approach, Myghty’s goes a step farther by letting you tie it into the resolution options used by the Resolver, and even use your own resolvers…

Advanced Resolver

In version 0.97alpha2 (maybe a better naming scheme could be used? :) the resolver implementation was heavily rewritten to allow massive flexibility in how a URL is handled. When Myghty handles a request, it goes through a fairly complex process to determine what to do with it. This is called the default strategy:

  1. Translate the URL against the path translation options – This can act as an internal URL rewriting scheme, changing a URL to be handled differently
  2. Resolve a dhandler – This rule matches conditionally. It is only called if the rules under it don’t find a match, it then strips off part of the URL its looking for and adds dhandler. So a search for /article/view/38 becomes a search for /article/view/dhandler if 38 isn’t found. Read more about dhandlers
  3. Check the URI Cache – The URI cache is searched for the incoming URI at this point, to see if a component has matched it before. This rule is also executed on a conditional basis.
  4. Upwards Search – The 3rd conditional rule of the bunch, this rule is called during inheritance when a component searches parent paths for an autohandler. Read more about autohandlers
  5. Resolve a Module – Matches a Module Component to the incoming URI.
  6. Resolve a File – Matches a file under the component roots specified for a match to the incoming URI. This will directly call a Myghty template file on the system in response to a URL.

As soon as any of the non-conditional rules matches, the file or module component method is served up. Myghty gets an additional speed boost by keeping compiled versions of all your template files around, so that they’re only compiled the first time they’re called. By using the basic Python way of compiling the templates as .pyc files, it also ensures that they’re re-compiled the second you update one of them. This way you’re always sure you’re viewing the latest update of your template.

So where’s the real power in all this?

These are just the default rules. They can be completely, and utterly customized. You can switch the order around, you can remove resolver rules entirely, and you can even write your very own resolver rules.

Maybe you’ve decided you want to serve some of the components from a database instead of the file system. You could write a custom resolver rule that does just that, and loads the entire component from the database as well. You could write your own version of the Path Translate module to get its rewrite rules from a database table. As if this wasn’t powerful enough, Myghty also lets you have different resolver strategies for different contexts.


Myghty is a very powerful web framework. It has features that go beyond any pure “templating” language I’ve encountered. If you’re not quite happy with what you’re using right now, or something I described above makes you giddy with thoughts of power, maybe its time to take Myghty for a test drive.

In future posts, I’ll describe more of the features I use daily in Myghty, and share code tidbits that let you get complex tasks done easily.

Wed, 06 Jul 2005 00:00:00 -0700 <![CDATA[Sending RSS items to Growl via AppleScript / NetNewsWire]]>

Sending RSS items to Growl via AppleScript / NetNewsWire

A friend alerted me the other day to some useful sites for tracking packages via RSS. There’s one for FedEx and one for UPS.

While these are quite cool, and nice to have in NetNewsWire, the same friend told me its even cooler to have updates to the feeds sent to Growl. That way you get a nice little pop-up window when one of your packages has moved on.

He wrote a Python script that sent RSS updates over the LAN to Growl (Note: The Python script requires While that was cool, his method of adding feeds to watch (editing a text file on a remote machine) didn’t seem that cool. It was about that time that I started thinking, well, why not have Net News Wire manage the feeds? That is what its for after all… so I started looking to see how I could tie this togther.

Searching through quickly led me to a page on calling scripts as subscriptions. So I figure, if I write an AppleScript, that searches a NetNewsWire (NNW) folder for unread items then sends them to Growl via the Applescript interface, that should do the trick.

So without further ado, here’s my little Applescript, based on Brent Simmons Folder Watch example, that searches a NNW folder for unread items, sends them to Growl, then marks them as read.



Download this where desired, add it as a script in NNW using the File menu -> New Special Subscription -> Script. Then locate the GrowlRSS script, it will then open the Show Info window. Expand the Script Settings option, and put the name of the NNW folder to watch as the Args. Close, and hit refresh, and it should send any unread items in that folder to Growl.


You should have Growl 0.6 from svn installed. It’s possible it’ll work with 0.5, but I haven’t tested that.

Sun, 02 Jan 2005 00:00:00 -0800 <![CDATA[Fast CGI with HTML::Mason]]>

Fast CGI with HTML::Mason

I decided the other day for a variety of reasons, to switch from the mod_perl web-app style to using Fast CGI with HTML::Mason. I only found one application out there that uses Mason with Fast CGI, which was the RT Ticket tracking system. However, the way it was setup didn’t match my requirements, nor the way their handler was configured.

After scouring the Internet some more, I hobbled together a working config section for my Apache+mod_fastcgi along with a template for the mason handler. Since I haven’t seen anything going over these basic steps anywhere else, here’s how I did it.

First, the things you should have installed on your system (which I’m assuming is *nix):

  • HTML::Mason
  • CGI::Fast
  • Apache w/mod_fastcgi

Fast CGI runs your CGI application for you, and handles connections between your application, and the webserver. To configure your application, you just use one of the Fast CGI development kits for the response event loop within which your application will process requests.

Since we’re using Perl, there’s already a handy module available from CPAN that I chose to use called CGI::Fast. Then we just wrap our Mason call within a CGI::Fast loop to handle requests. This Perl script will be kept running persistently so that you get the advantage of not having to startup the Perl interpreter on every web page request.

Why Fast CGI instead of mod_perl?

  • You can use the connection feature of Fast CGI to run your app on a different machine than the webserver
  • Your application takes up only as much ram as Perl + your modules
  • It’s more portable, you don’t even need to use Apache, any webserver supporting Fast CGI will work
  • Less configuration files to maintain

Before showing my standard handler template, you should be aware of how I organize a Mason site using Fast CGI. Here’s what my directory structure looks like under the fictional GroovieWebapp:

  • GroovieWebapp/
    • site/
    • modules/
    • libs/
    • mdata/
    • mason-handler.fcgi

Site is where all of our Mason code (and images, css, etc.) is going to go. Modules is for putting any of your own Perl modules into that you made specifically for this webapp. libs for Mason libs, mdata for your Mason data dir, and finally the handler itself. This structure puts your whole webapp under one directory which is also useful under version control systems like Subversion.

Here’s what my standard handler template looks like (mason-handler.fcgi):

#!/usr/bin/perluse FindBin qw($Bin);use lib “$Bin/modules”;

    use strict;use HTML::Mason::CGIHandler;use MIME::Lite;  # Need this to e-mail the error

    Enter CGI::Fast mode, which should also work as a vanilla CGI script.
    require CGI::Fast;

    my $h;my $show_error = 1;  # Should we e-mail the error, or dump to the web?

    if ($show_error) {
    $h = HTML::Mason::CGIHandler->new
        ( comp_root => [
              [main => “/usr/local/www/GroovieWebapp/site”],
              [libs => “/usr/local/www/GroovieWebapp/libs”]
          data_dir => “/usr/local/www/GroovieWebapp/mdata”,
          allow_globals => [qw($Schema $Session)]
        );} else {
    $h = HTML::Mason::CGIHandler->new
        ( comp_root => [
              [main => “/usr/local/www/GroovieWebapp/site”],
              [libs => “/usr/local/www/GroovieWebapp/libs”]
          data_dir => “/usr/local/www/GroovieWebapp/mdata”,
          error_format => 'text',
          error_mode => 'fatal',
          allow_globals => [qw($Schema $Session)]

    Setup our global vars for the request, preload our modules
    { package HTML::Mason::Commands;
  use Apache::Session::File ();    # My favorite for session tracking
  use CGI::Cookie;                 # Also for session tracking
  use DBI;   # It's nice to load and establish our db connection in advance
  DBI->install_driver(“mysql”);  # Put in whatever connection you use, etc.

  # This allows our $Schema variable to be accessed from anywhere in our Mason site
  our $Schema = DBI->connect_cached(“dbi:mysql:DB_NAME”,“DB_USER”, “DB_PASSWORD” );

     These two subroutines add Mason functionality that was lost by using CGIHandler instead
        of the ApacheHandler.
    sub HTML::Mason::FakeApache::document_root {
    my $self = shift;
    return $ENV{'DOCUMENT_ROOT'};}sub HTML::Mason::FakeApache::param {
    my $self = shift;
    my $key = shift;
    my %args = HTML::Mason::Utils::cgi_request_args($self->query,$self->query->request_method);
    my @keys = (keys %args);

    if ($key) {
        return $args{$key};
    } else {
        return @keys;

    For e-mailing the error
    sub send_email {
    my $error = shift;
    my $cgi = shift;
    my $site = $cgi->virtual_host();
    my $time = localtime;
    my $body = “————————————————————\n”;
    $body   .= “———— ERROR OCCURRED\n”;
    $body   .= “————————————————————\n”;
    $body   .= “\n\n”;
    $body   .= “WEBSITE INFO\n”;
    $body   .= “Site: $site\n”;
    $body   .= “Full URI Request: “.$ENV{'REQUEST_URI'}.”\n”;
    $body   .= “Time: $time\n”;
    $body   .= “\n\n”;
    $body   .= “USER INFO\n”;
    $body   .= “Referrer: “.$cgi->referer().”\n”;
    $body   .= “User-agent: “.$cgi->user_agent().”\n”;
    $body   .= “IP Address: “.$cgi->remote_host().”\n”;
    $body   .= “Cookie info: “.$cgi->raw_cookie().”\n”;
    $body   .= “\n\n”;
    $body   .= “MASON ERROR\n”;
    $body   .= $error;

    my $msg = MIME::Lite->new( From    => 'YOUR SITE ',
                               To      => 'YOUR_EMAIL_ADDRESS',
                               Subject => “$site Mason Website Error occured at $time”,
                               Data    => $body,

    while (my $cgi = new CGI::Fast) {
    # the whole point of fastcgi requires the env to get reset here..
    # So we must squash it again (This was copied from RT's Fast CGI handler)
    $ENV{'PATH'}   = ‘bin:/usr/bin';
    $ENV{'CDPATH'} = '' if defined $ENV{'CDPATH'};
    $ENV{'SHELL'}  = ‘bin/sh' if defined $ENV{'SHELL'};
    $ENV{'ENV'}    = '' if defined $ENV{'ENV'};
    $ENV{'IFS'}    = '' if defined $ENV{'IFS'};

    # The Mason $r->uri includes the script name unless we do this
    $ENV{'SCRIPT_NAME'} = '';

    # This deals with someone entering to make sure it serves index.html
    if ( ( !$h->interp->comp_exists( $cgi->path_info ) )
        && ( $h->interp->comp_exists( $cgi->path_info . “index.html” ) ) ) {
        $cgi->path_info( $cgi->path_info . “index.html” );
    #$cgi->header(-charset=>'utf-8');  # If your site uses utf-8 characters

    # Ping our db handle

    # Lets try and handle this…
    eval {

    # Something happened, if it couldn't find the component, don't fatal
    # we'll re-process as a 404
    if ($@) {
        if ($@ =~ /could not find component/) {
            # If you want Mason to handle the error so you can keep your site appearence,
            # this will call Mason to display your error page. If the error was in the
            # code that is called on every page, this will cause trouble.
        } else {

This template has the option of either showing the Mason debugging output to the screen, or capturing the error, displaying a pretty 404 of your desire, and e-mailing you the error that occured.

Now for the Apache httpd.conf:

Tell FastCGI to put its temporary files somewhere sane.
FastCgiIpcDir /tmp

Increases processes if you have heavy loads to deal with
FastCgiServer /usr/local/www/GroovieWebapp/mason-handler.fcgi -idle-timeout 120 -processes 2

ServerAdmin webmaster@YOUR_SERVER

AddHandler fastcgi-script fcgi
ScriptAliasMatch (.*\.html$) /usr/local/www/GroovieWebapp/mason-handler.fcgi$1
ScriptAliasMatch (.*\/$) /usr/local/www/GroovieWebapp/mason-handler.fcgi$1

DocumentRoot /usr/local/www/GroovieWebapp

Restart apache, and check apache’s error_log to see that Fast CGI started up your application properly. Obviously, tailor these samples as needed. Another useful trick when running your webapp this way, is that if you kill the perl processes the apache Fast CGI process manager starts for you, it’ll re-spawn them. I find this a quick way to re-load individual webapps if I made changes to a module they’re using.


Sat, 18 Dec 2004 00:00:00 -0800 <![CDATA[A Mason component to fill-in error messages]]>

A Mason component to fill-in error messages

How often have you seen those sites with a form, and after you miss a field, it pops up a little message in red under the text box indicating what went wrong? I’m guessing a lot, and when writing web applications this can be a pain to code in.

Typically in Perl/PHP, you’d add this by going through your form, adding little bits of code under each and every form element to display the error on invalid input. I quickly got tired of this, and decided to simplify the process.

In examining HTML::FillInForm, I noticed I needed a slightly different tool to easily parse the HTML form. After a little bit of shopping about, I decided on HTML::Parser to do the heavy lifting. Here’s a snippet from a Mason page using my error_fill.mas component to toss in the error messages:

% if ($form_status eq 'errors') {
<&| /lib/error_fill.mas, errors => $errors_hash_ref &>
<p style="color: red; font-weight:bold;">Errors Found: Please correct the fields indicated.
<& /forms/myform.mas, %ARGS &>
% } else {
<& /forms/myform.mas, %ARGS &>
% }

$errors_hash_ref is a hash ref that is keyed by the form field name, with the value set to the error message you want displayed. Right now, error_fill.mas does not have custom pre/post-fix variables for you to tweak how the error message is inserted. It currently puts a <br> followed by a red span element containing your message.

myform.mas is the form I want to display, I prefer to split sections out in Mason to keep my web application modular and the sections of my pages more reusable.

Here’s error_fill.mas:

$errors%args>$errors is a hash reference keyed as:
            'field name' > 'Error message'
    use HTML::Parser;my $form = $m->content;

    my $p = HTML::Parser->new(api_version > 3);$p->handler(default => sub { print _ }, "text");
$p->handler(end => sub {
              my ($tagname, $text, $attr) = _;
              if (($tagname eq ‘select') && (exists $errors->{$attr->{name}})) {
                print $text;
                print ‘’;
                foreach my $err (@{$errors->{$attr->{name}}}) {
                  print ‘’.$err;
                print ‘’;

              } else {
                print $text;
            “tagname, text, attr”);
$p->handler(start => sub {
              my ($tagname, $text, $attr) = _;
              if (($tagname eq 'input') && (exists $errors->{$attr->{name}})) {
                print $text;
                print '<span style="color:red;">';
                foreach my $err ({$errors->{$attr->{name}}}) {
                  print ‘’.$err;
                print ”;

              } else {
                print $text;
            “tagname, text, attr”);

Hopefully this will help some people out.

Fri, 26 Nov 2004 00:00:00 -0800 <![CDATA[XML-RPC interface for Growl]]>

XML-RPC interface for Growl

Growl is a rather spiffy little notifier for OS X I’ve started using lately. Currently however, it’s lacking a nice way to send messages over the network. Since a project I’m working on would be even slicker with a way to communicate updates, I quickly tacked a Python XML-RPC Server onto Growl using Growl’s Python bindings.

In the future, Growl will have its own network interface, I’ve heard. Until then, this does the job quite nicely.

Please note: You will need to install the latest svn of Growl, the PyObjC bridge, and have Xcode 1.5 to get this working. Everything in ALL_CAPS should be changed as needed, this also assumes you want to use icons and they’re in a myIcons dir.


    import Growlimport SimpleXMLRPCServerfrom AppKit import NSImageimport sys, os.path

    iconPath = 'myIcons/'Icons = { 'MESSAGE_1' : 'MESSAGE_1.tiff', \
          'MESSAGE_2' : 'MESSAGE_2.tiff' }

    class MemberFunctions:
    def init(self,Icons):
        gn = Growl.GrowlNotifier()
        gn.applicationName = 'YOUR_APP_NAME'
        gn.applicationIcon = Icons['appIcon']
        gn.notifications = ['MESSAGE_1','MESSAGE_2']
        gn.register() = gn
        self.Icons = Icons

    def notify(self, ntype, title, description, sticky):
        gn =
        Icons = self.Icons
        nIcon = Icons['appIcon']
        if ntype  'MESSAGE_1':
            nIcon = Icons['MESSAGE_1']
        elif ntype  'MESSAGE_2':
            nIcon = Icons['MESSAGE_2']
            icon=nIcon, sticky=sticky)
        return 1

    def loadIcons(icons):
    global iconPath
    for icon in icons.keys():
        filename = icons[icon]
        nImagePath = os.path.abspath(iconPath+filename)
        nIcon = NSImage.alloc().initWithContentsOfFile_(nImagePath).autorelease()
        if not nIcon:
            sys.stderr.write(“Couldn't load file %s\n” % nImagePath)
        icons[icon] = nIcon
    return icons

    Icons = loadIcons(Icons)

    Make our XMLRPC Server, create our instance, and assign it
    server = SimpleXMLRPCServer.SimpleXMLRPCServer((“IP_TO_BIND_TO”, 7300))memfun = MemberFunctions(Icons)server.register_function(memfun.notify)

     Start the server

If you’d like to see a running example of this, check out that this example is based heavily on.

Sat, 20 Nov 2004 00:00:00 -0800