be Groovie

Sacrificing readability for automated doc tests

I’ve tried several times in the past to try out zc.buildout, a fairly neat sounding package that automates the buildout process for a Python app. The promise of fairly easy to write recipes that can setup external processes like nginx in addition to ensuring my webapp is put together with all the things it needs sounded great.

It occurred to me that the docs definitely didn’t help at all. In fact, they’re noticeably bizarre unless you actually realize why they’re written the way they are. Here’s a sample of the zc.buildout docs about how to make a new buildout and bootstrapping.

You’ll notice that it almost looks like command line interactions of some sort are occurring, yet the author of the docs is clearly at an interactive Python prompt. Note that none of the commands shown there will work if you copy them into your Python interpreter, nor is there any indication what you would need to do to get such commands available. As a user trying to follow the docs, that leaves me wondering… am I supposed to be in a Python interpreter? What do these variables get expanded to so that I can do that at my shell prompt? Why can’t you just give me the damn command line I’m supposed to run so I can copy/paste???

Yes, it definitely got me a bit frustrated. I believe the only logical reason the docs were in this bizarre fashion is so that they could be automatically doc-tested. Its a shame that the result of this is docs that make me want to close the web page as soon as I stumble upon the ‘samples’, since there’s no way I can handle wading through the command line abstractions.

Doctests can be useful, but turning command line interactions into a Python interactive session is a massive readability issue. People know and recognize command line interactions, lets stick with them please.

Pylons @PyCon 2008 Wrap-up

Sprints!

Wow, they were great. A lot got done, quite a bit more than I was expecting. I’d like to give a big shout-out to the Pylons sprinters (in random order, sorry if I missed anyone):

• Karen Lo
• Mike Orr
• David Montgomery
• Mike Verdone
• Wes Devauld
• Ian Bicking
• Phil Jenvey
• and many more TG2 sprinters

WebHelpers saw some significant gains, including the addition of a literal object for safe HTML escaping, and default Mako auto-escaping in Pylons 0.9.7.

There will definitely be more Pylons/TG2 sprints in the future, in multiple locations around the country.

Tutorials

Ouch… setuptools came through, unfortunately the network didn’t, nor did my first built egg of Pylons (unfortunately I got a new laptop recently that was missing some things…). Mark Ramm and I did get the show under way, but the delay in starting up definitely affected how far the tutorial attendee’s made it in the basic Wiki project and proved frustrating for all.

I made it through all the Pylons information I had in the Mastering Pylons/TG2 section, but unfortunately had no time left to go into more detail on various aspects of it in a more hands-on style. I definitely won’t be doing any more tutorials without ensuring lots of USB thumb-drives are handy with virtualenv Pylons/TG2 ready to go.

Sessions and the Conference

They seemed ok, several of them seemed very, “And this person is giving this talk…. why?”. Note to session presenters, honesty is great but I really do wonder why you’re presenting when your opening remarks are, “What I’m about to show you, anyone who’s used this already knows, so there’s nothing to see here.” Show us something with some zing!

The lightning talks, despite a first day of some awkward sponsored ones, were actually the highlight for me. I almost spit water laughing on the “Speech Recognition does war”, followed up by “His base of God would have I done now” (Originally intended to be ‘Speech recognition does work. oh god, what have I done now?’). Unfortunately Ian Bicking failed to give us an update on ZhangoPyloGears, I can only imagine it has fully attained consciousness and freed itself of his clutches.

Coming up…

Pylons 0.9.7 is close at hand, just waiting for some final bits of WebHelpers to drop into place and it should be ready to pop. And one last side-note, I think I finally found the documentation tool of my dreams… Sphinx!

PyCon 2008 Social Network

Normally I’m not a fan of social networks, I always seem to get bored quickly. The one time that I actually find myself using them however, is with conferences. The relevance of the immediately upcoming event makes it quite a bit more interesting, and I’ve found it a great way to find people at conferences that I’d like to meet and remember who I met afterwards.

As Doug mentioned earlier, there’s several social sites for PyCon 2008 up. I suggested a Crowdvine network as well, and have set one up for all PyCon 2008 attendees.

Sign Up!

For those attending PyCon 2008, consider joining the PyCon 2008 Crowdvine network. Some handy things to do after joining:

• Add a picture. That’ll help people put a name to a face.
• Fill out your profile. This will help people find you.
• Find people you want to meet. From their profile you can let them know that you are a “fan” or that you “want-to-meet.” Or you leave a comment with something more specific.

Cheers!

It’s just super

No one told me that in New Hampshire, the Democrats have a hidden power. They apparently have a mysterious force known as superdelegates. Yea, I know.

Getting Inspired

I’ve never been to TED, but wow are the talks there inspiring and insightful. They also cover a wide range of topics far beyond just IT. The majority of the talks are also available online, some in high quality 480 progressive mp4 as well.

The best thing I’ve always found about going to conferences, is the inspiration that I can get and find in the talks others give. TED inspires in a different way, so here’s a few of my favorites and why:

Dr. Dean Ornish: The world now eats (and dies) like Americans
Only 3 minutes, WATCH IT. The power of your diet to kill you is rather amazing, if you want an even more comprehensive background on this, read The China Study

Might You Live a Great Deal Longer?
A very fascinating theme, which includes several talks, including the short 3 minute one I cited separately in the hopes more people would watch it. :)

Richard Dawkins: An atheist’s call to arms
This one is sure to get a few people in a huff, for Richard Dawkins has few kind words for those who hold religious beliefs dear to their heart. But its a very interesting talk on science, and oddly enough I didn’t find it very scornful (which some do), probably my background in Philosophy.

Blaise Aguera y Arcas: Jaw-dropping Photosynth Demo
This one has been posted by people before, I have to include it cause its definitely rather incredible to see. Definitely a tech theme on this one.

And finally, one of the most relevant to our current crises with regards to the environment and sustainability…

Janine Benyus: 12 sustainable design ideas from nature
A very exciting talk about work being done to copy sustainable ideas from nature, also called biomimicry. If you’re looking for an upper, this one definitely left my with a smile and some hope for future developments that don’t require us to devastate places with mining, pollute the environment shipping raw materials around the world, etc.

I’m sure I missed a lot of good videos, if there’s something that really inspired or awed you that can be found in the TED talks, please share!

Misapplying book terms, Pylons, and the ‘end-user’

On the Internet, I frequently see terms misapplied to other contexts. Whether its a basic definition misapplied to a different realm, or an analogy grossly misapplied to a context that is not in fact similar. Since this particular term ‘conceptual integrity’ however was used as a beating stick on Pylons, I felt it’s something worth discussing.

First, its very useful to actually understand the context of the book the term comes from, The Mythical Man-Month. The Mythical Man-Month is about Fred Brooks observations as a manager on the OS/360 project at IBM, and the problems encountered when developing an OS for end-users. Before getting into the question of whether an OS end-user is equivalent to a web framework end-user, its useful to look at the term being referred to, ‘conceptual integrity’.

Wikipedia has a fairly good summary of the term:

Conceptual Integrity

To make a user-friendly system, the system must have conceptual integrity, which can only be achieved by separating architecture from implementation. A single chief architect (or a small number of architects), acting on the user’s behalf, decides what goes in the system and what stays out. A “super cool” idea by someone may not be included if it does not fit seamlessly with the overall system design. In fact, to ensure a user-friendly system, a system may deliberately provide fewer features than it is capable of. The point is that if a system is too complicated to use, then many of its features will go unused because no one has the time to learn how to use them.

Is there Conceptual Integrity?

The Pylons developers, including myself, spend quite a bit of time deciding what goes into Pylons, and what does not. Pylons is in fact very very small at its core due to our refusal to let ‘stuff’ into Pylons that is outside its strictly defined scope. In fact, a great many comments I get are usually about features people want, but that aren’t in the scope of Pylons to provide. I would argue that this does show thought has gone into ‘conceptual integrity’, in the explicit decisions made about what is and isn’t provided.

The scope of Pylons is to provide a small, concise, WSGI-driven, lightweight framework, that provides the flexibility for a web developer to use the best tools for the job at hand for building their web application. I would argue based on the definition, the goal of Pylons, and what Pylons has in it now, that it fully meets the design goal of having conceptual integrity with regards to feature scope.

While the original blog post I cited above starts with conceptual integrity, it delves deeply into framework design issues like, ‘Should a framework pick the ORM for you?’. This is slightly odd in the context of Pylons, as the default ORM, and template language are chosen, Pylons merely emphasizes that the choice is yours should you wish to change it (as does Django).

In another misapplied analogy, a parallel is drawn up between having an opinion on a writing style, to having an opinion on a library. Consider how odd this analogy is, that the fairly minor effect of adding a trailing ‘S’, should be compared to whether a particular ORM library is capable of meeting your demands. Should the framework authors decide whether the ORM meets your demands? Should the framework lose a significant amount of capability the second its stock ORM fails to handle your needs? These are interesting questions, well beyond an attempt to draw a parallel to adding a trailing ‘S’ to a word.

Since its clear that Pylons doesn’t encroach on conceptual integrity regarding features that are unused, the next aspect to look at is the concept of a ‘user-friendly system’. This becomes a little more difficult to analyze, because it brings into question the expectations one has of the ‘user-base’. It also shows how the original term of conceptual integrity was misapplied as another parallel is drawn, this time one that I think all web developers should be somewhat terrified by.

Website end-user == Website Programmer?

The last parallel, is to compare the programming expertise of a web developer, to the internet naivety of the website end-user. (Does that offend any other web developers?) The latter of which is well documented in various web usability books, such as the “Don’t Make Me Think” book. The underlying concept being:

• Website users have short attention spans
• Website users won’t use a website that requires thinking
• Website users have no patience

The thesis of this parallel is that a framework author is somehow in the superior position of knowing the appropriate library that will work for your task, and should therefore decide the tools for you. The parallel applied looks like:

• Website programmers have short attention spans (ok, I somewhat buy this one :)
• Website programmers won’t think about whether the tools at hand will accomplish their task
• Website programmers have no patience (Given the languages you have to learn to do anything, I seriously doubt this)

Now, there are quite a few website programmers that don’t think about whether the tools they use, work best for their task. I have seen many websites that are slow, inefficient, or just not well designed because of a lack of thinking regarding architecture, and tool choices. I would not consider these to be website programmers I would want to work with, nor are they the target audience or ‘end-user’ of Pylons. The door is purposely left open in Pylons, and lit up, with the choice available in libraries to use with Pylons.

Regarding making decisions, Pylons has choices for all the same components as Django has decided on:
ORM: SQLAlchemy
Templating: Mako
URL’s: Routes
Sessions: Beaker
Caching: Beaker

The main problem apparently being, Pylons has emphasized that there is choice, that the framework is built so that they could be swapped, should one of these not be the best tool for your web application.

The Pilot System

When designing a new kind of system, a team will design a throw-away system (whether it intends to or not). This system acts as a pilot plant that reveals techniques that will subsequently cause a complete redesign of the system. This second smarter system should be the one delivered to the customer, since delivery of the pilot system would cause nothing but agony to the customer, and possibly ruin the system’s reputation and maybe even the company’s.

In response to the original post regarding conceptual integrity, I submit the issue of the pilot system. Django and other frameworks that start off with libraries you didn’t analyze in regards to your particular web application, have more than a few times led to pilot systems (I usually hear about the second incarnation, which was then done on Pylons).

Where you realize later down the road, “wow, this ORM just can’t handle the data model I have”, which then leads to, “well, since the ORM was hooked into the XML serializer, the admin UI, the form generator, etc. there’s now no competitive advantage over Pylons… which didn’t have those extras.” Systems with heavy and deep integration of libraries that may or may not meet your demands, where large parts of the system become relatively useless should you stop using one part of it, are extremely prone to being a pilot system.

When your pilot system is done, I invite you to check out Pylons, and think about what is going to work for your website application.

Notes

Please note that I take deliberate care in referring to Pylons for use with developing web applications, its my belief that Django, and similar systems in other languages with an ‘app’ centric approach like Joomla, Drupal, etc. are excellent for making web sites. While you can easily make websites with Pylons apps, this is definitely not the primary use-case emphasized and designed for, so it is harder and less obvious in many respects than Django.

The other thing everyone should’ve learned from these posts, is that The Mythical Man-Month is a rather good book worth reading. ;)

Nice tidbits in Leopard

As many have blogged about, Apple has posted a rather hefty list describing all 300 new features of Leopard. Some specifics that really got the coder in me excited:

(Highlighting emphasis all mine)

Cocoa Bridges

Use Ruby and Python as first-class languages for building Cocoa applications, thanks to Objective-C bridges as well as full Xcode and Interface Builder support.

Scripting Bridge

Use Objective-C, Ruby, and Python programs to automate Mac applications. The new Scripting Bridge enables them to easily generate AppleEvents using a concise, AppleScript-like syntax.

Create Instruments with DTrace

Monitor system activity from high-level application behavior down to the operating system kernel, all thanks to the power of DTrace and the instrument builder.

DTrace

Monitor virtually any aspect of your application with DTrace, integrated into the Darwin kernel. Java, Ruby, Python, and Perl have also been extended to support DTrace, providing unprecedented access for monitoring the performance characteristics of those languages.

While some of the things can be done now (There’s PyObjC to use Python with Cocoa), the tighter integration of Python and Ruby into Leopard make it a very appealing upgrade (Even though I rather hate the new semi-transparent menubar and odd shelf nature of the Dock).

Too Much Happening, Filtering the Overload

Tonight I was drifting through Upcoming to see what’s happening in my area. Turns out the first page of 50 some events only covered today…. and its a ‘boring’ Wednesday. As I flipped through page after page, eventually getting all the way up to this upcoming Sunday… it occurred to me that there is way way too much happening within a mere 30 miles of me.

Of course, being a geek I’ve been thinking about ways to try and narrow down this list so I can actually get a useful list for what I’d prolly be interested in for the next 45 days. There’s just no way I can look through 1200 events to see what’s going on for the next month, and I like to plan ahead.

A few things I think would help to begin with would be keyword filtering, of course the next step that requires would be some way to build a useful keylist of what should be flagged for my perusal. Trying to generate a full list of every keyword I might be interested in is going to be pretty tough, though I finally thought of the one application I have which could yield the best list of keywords to filter on… Delicious Library.

It already knows the vast majority of books I own, and all my DVD’s. That’s not a bad starting point, toss in all the music I’ve bought from scouring all the id3 tags on my digital music and that’ll cover my music. I really need a way to pull all my previously watched, and upcoming movies from Netflix, in addition to mining all the data of who was in each flick (IMDB should have that). After which I should have a decent set of keywords to ensure I don’t miss either some special premier with an actor I might like, or a comedian doing standup.

Is there anything like this already that I just haven’t found? Clearly, I’d want to keep this massive keyword list personal, so the software to pull from upcoming would narrow down the result set locally. I’m still a bit too privacy obsessed to consider putting such a massive amount of information into a single vendors hands. :)

SQLAlchemy, Declarative layers, and the ORM ‘Problem’

There’s been a bit of talk on the Pylons devel list regarding the recommended way to use SQLAlchemy with Pylons mainly regarding how to use SA (SQLAlchemy) in a fashion that is well documented and easy to work with (and maintain!).

Prior to Pylons 0.9.6 and SQLAlchemy 0.4 it was a bit of a mess, with the framework needing to load the config (since thats where your db settings are), then setup globals for SA…. eek. Mike Orr had a good intermediary solution for SA 0.3 called SAContext that handled many of the tricky parts. Unfortunately, this actually caused even more confusion as more ways of doing the same thing came about. SAContext solved some of the global config grabbing issues, but the additional layer of indirection made trouble shooting even harder (despite how small of a library it was).

Less is More

So the fix? Less intermediary layers, less indirection… essentially, KISS. Despite how much Pylons was attempting to help a user to get the db going, the additional layers in the end actually caused more problems then they solved. Of course, I shouldn’t have been too surprised…. Mike Bayer did warn me about many of these things at the beginning. Being overly eager to make things “easier” for new users, I ignored him. :)

This is why Pylons does not recommend Elixir, and with SA 0.4 the recommended usage of SA is to use its plain mapper functionality should you need an ORM layer. Yes, that’s right, despite almost every web framework out there pushing its ORM on you (or someone else’s ORM), there are many times when an app doesn’t even need a full-blown ORM.

Declarative vs Basic SA

For a better look at why one might consider additional layers on SA a bad thing lets compare a fairly basic table setup consisting of users and groups. Each user can be in multiple groups, and lets use proper referential integrity to ensure that groups aren’t deleted when users are still in them.

Compare the following two ways of setting up a basic many to many relation and the tables:

SQLAlchemy 0.4

from sqlalchemy import Column, ForeignKey, MetaData, Table, types
from sqlalchemy.orm import mapper, relation

	
metadata = MetaData()

	
person_table = Table('person’, metadata,
    Column('id’, types.Integer, primary_key=True),
    Column('name’, types.String, nullable=False),
    Column('age’, types.String)
)

	
group_table = Table('group’, metadata,
    Column('id’, types.Integer, primary_key=True),
    Column('name’, types.String, nullable=False)
)

	
persongroups_table = Table('person_groups’, metadata,
    Column('person_id’, types.Integer, ForeignKey('person.id’, ondelete='CASCADE’), primary_key=True),
    Column('group_id’, types.Integer, ForeignKey('group.id’, ondelete='RESTRICT’), primary_key=True),
)

	
class Person(object):
    pass

	
class Group(object):
    pass

	
mapper(Person, person_table, properties=dict(
       groups=relation(Group, backref='people’, lazy=False)
))
mapper(Group, group_table)

Elixir

from elixir import *
class Person(Entity):
    has_field('name’, String)
    has_and_belongs_to_many('groups’, of_kind='Group’)

	
class Group(Entity):
    has_field('name’, String)
    has_and_belongs_to_many('people’, of_kind='Person’)

On first glance, its pretty obvious that everyone should love Elixir vs the obviously more tedious SA approach of layout out your tables, then mapping them to the class objects. However, look at these two examples, and try to quickly answer the following questions:

• How do you add a column to the many to many table to store an additional bit of info for the join?
• Do they both enforce referential integrity?
• How do you control whether SA is eager loading the relation? Can you restrict it to just one column of the relation?
• What are the table names used?
• How many tables are in your database?
• Where do you change the id column name?
• Which one is closer to the Zen of Python?

I think the explicit setup makes many of these questions easier to answer just at a glance. Those with enough Elixir experience can fairly easily answer most of these questions, but consider what that implies. Not only do you need to know SQLAlchemy options and parameters, but you need to know Elixir options and how they map to the SQLAlchemy functions. The desire to reduce the up-front setup of the ORM actually increases the amount of knowledge a user has to have in order to use it, and the most worrisome aspect… how to troubleshoot it.

Setups that Grow with You

With Pylons, a goal has been to provide out of the box recommendations that grow with you. That is, using the set of recommended tools may not be as apparently “easy” as some other frameworks. However, the pay-off is that you don’t hit a wall in 2 months when your application inevitably gets a little more advanced and needs to do something the simple tools either can’t do at all, or it’s incredibly difficult to do even slightly complicated things (eager load 2 columns off a related table, but not all of them). This way, the toolset you learned, you can keep using as you get more advanced and you don’t “outgrow” your tools.

While Elixir definitely appears to be easier at first glance, when you need to get more complicated you can’t exactly turn to the SA docs since Elixir has put a layer between you and SA. This can be very crippling when you eventually hit a wall, and so much ‘magic’ is wrapped up in the declarative layer that you have to troubleshoot additional layers of code when something goes wrong. The result of this is that to effectively use Elixir in complicated setups with SQLAlchemy, you need to really really know both of them which actually requires more work for a user than plain SQLAlchemy.

The SA example clearly requires a little more up-front setup, however, are you really adding tables to your database every day? How often are you going to be actually touching the table and mapper code, vs just adding domain model methods to your Person/Group class? Did the layer make it easier or harder to use multiple databases and/or put more between you and advanced SA functionality you might need later?

Adam Gomaa pointed out some interesting issues with Django’s ORM and Elixir but unfortunately tries to do the same thing Elixir and TurboEntity do…. add more layers. More layers more indirection more to wade through when you need to do something that should be pretty basic with SQLAlchemy (and is probably nicely documented on the SA site, which won’t help with these layers until you dig through the layer to find the basic SA objects the SA site refers to…).

What really makes a lot of this even more trouble-some with SA, is that when setting up complex relationships, the order of declaring table objects becomes important, since relations need to refer to them and the ORM classes. This usually results in some interesting metaclass hackery when you have these Entity’s in multiple modules, importing each other, and doing other fairly common stuff.

SQLAlchemy 0.4

In the end, I’ve been using plain SQLAlchemy 0.4 (at beta5 now, but quite stable) a lot lately, and its really great. Yes, setting up the tables (generally a one-time thing) took me probably 15 mins longer than it would’ve with Elixir. But I’m fairly certain I’ve saved myself significantly more time in the long run since I won’t need to worry about diving into Elixir code to try and find SA objects when I need a complex query, or trying to figure out how to hack Elixir’s connection should I need multiple db connections at once, etc.

So please, new users to SQLAlchemy, use just SQLAlchemy. It definitely seems daunting at first, but the flexibility and comprehensive documentation give you a solution that scales to meet your needs, with no walls in sight.

On a side-note, its interesting to compare my position on this issue to the Django team lack of AJAX helpers. The Django team rightfully claims that Javascript isn’t that hard, so “get over it” and learn a nice Javascript library so you can do powerful things. Also note that by including AJAX helpers, Pylons is encouraging one part that doesn’t scale… as the AJAX helpers will have you hitting a wall sooner or later.

Next-Gen DVD Wars Give me the Blues…

I’ve been seeing more and more movies I wouldn’t mind actually buying… except that I have a 56” HD TV set. It looks amazing, pretty good with DVD’s, but really amazing with HD so if I’m going to buy a movie I sure as heck want it in HD.

Unfortunately thanks to the tech companies refusing to come up with a single standard, I can either buy a $200 HD-DVD add-on for my Xbox 360, or a PlayStation 3 which I’ve heard is the cheapest Blu-Ray DVD player available. And of course, I really need to buy both if I want HD movies since some studios are only putting their movies out on one or the other format.

The end result? I’m not buying HD movies for HD-DVD or Blu-ray because its pathetic to have to have 2 players around (no, I don’t care about universal players). In addition, I’m not buying DVD’s anymore since I know that in a year or two I’ll want to re-buy it in whatever HD format wins (please let one of them win in a year or two!!!).

What are other people doing in this situation? Buying one or the other? Still buying DVD’s? Or not buying any movies at all until they get it straight?