be Groovie

Sci-Fi Fiction always on the way to reality

I remember reading a sci-fi book (David Brin’s Earth) awhile back set in the near future where a man-made black-hole fell into the center of the Earth. To say the least, it created a bit of a problem for our fictional heroes.

One of the events (I do hope I’m referring to the right book) that really stood out to me was when one of the main characters was roving around in a city, and a gang of thugs didn’t attack him/her purely because there were old retired people around hooked up with video gear that sent live feedback to the police station. These retired semi-vigilantes roamed around looking for some sort of crime to film, so they could send it in live to the police station. It was an interesting cross between the big-brother scenarios that are quite popular and vigilantism.

Ah, how fun it is as sci-fi I read years ago marches into the real-world. Earlier tonight I read a Cnet article on New York Police using cell phone photographs to help fight crime. I’m sure this will be extended to live video when cell phones and networks here can support it. I’ve found I actually prefer these kind of “big-brother” vs the type that seems to be popular in London; cameras on every street corner.

The problem with cameras all over, is that without some sort of AI, or army of people to watch them, you can only deal with logging the event and responding after the fact. It’s rare that someone watching hundreds of cameras will see some sort of crime in the act, compared to harnessing the ability of individuals to intelligently send relevant (or possibly relevant) video back to police for review.

The only thing in the article that confuses me is why it would cost the NY Police 40k to receive cell phone photos… surely there’s cheaper software that would work?

Low-Contrast Websites

I was talking with my grandfather recently about various technology bits when he mentioned this awful trend he had started to notice on websites. Just the mention of an awful trend with the knowledge that his eyes weren’t what they used to be elicited similar anger from me, mainly, that there seems to be a significant increase in low contrast websites.

Disclaimer: I realize that this blog also commits some of these sins I’m about to rant on, and I’m in the process of changing them.

This trend is very apparent in many blogs, and some are getting so bad I want to scratch my eyes out after reading them. This isn’t the blog author’s fault, as they’re usually selecting a theme that “looks good” on a glance. Trying to read the text when the theme is in use can be a nightmare.

Consider this blog entry, it looks decent contrast-wise to me on first glance. But trying to read the text of the comments at the bottom made my eyes burn. Or the Shopify page which manages to consistently use the same color text as the background behind it. This is a pretty strict Do Not per US Section 508 Code (Web Accessibility Standards). I’m also fairly certain the W3C Accessibility standards frown on low-contrast layouts, especially if no option is given to switch to a high-contrast scheme.

This issue doesn’t just affect old people or those with handicaps. I have no color blindness or issues sorting out colors yet these themes still routinely give me headaches. The thing to keep in mind when using different different shades of the same color on top of each other is that while you might have a wonderful color-calibrated monitor, the vast majority of Internet users do not. Even those with perfect vision are typically at the mercy of the quality of their CRT/TFT and the color settings that may or may not be ‘proper’.

For the designers out there, I realize that these low contrast themes can look pretty, but please put out more high-contrast themes that look great. Now to fix-up my blog…

Ingredients to the Pylons Python Web Framework

As the date edges closer to a Pylons release, I find myself already thinking about future directions of Pylons. I’m obviously rather biased when discussing Pylons as I’m one of its creators, though I still find that thought humorous as the vast vast majority of the code that resulted in an excellent framework is not actually in Pylons, nor did James Gardner (the other Pylons dev) and myself write it.

Matt mentions in a blog entry on Python Web frameworks, maybe its just too easy to write a Python web framework. I didn’t originally see myself writing a framework as such, but it became a fairly logical conclusion to the work I was doing at the time and the framework was originally extracted (like many others) from a large production application. It really is quite easy to write a web framework, and there’s tools available that make it even easier.

WSGI has changed the point at which re-usability is possible in a Python web framework as well, the only possible result I see in the future is more frameworks (Not in the way people think though). Many would argue this is “bad”, however I think due to the different point in re-use made possible by WSGI there will actually be more collaboration and less web developer divisions despite how many ‘frameworks’ are out there (once they’re more WSGI-ish that is). Consider this analogous in some ways to how many linux distributions are out there, and the fact that people can switch between them very easily.

So what went into building Pylons?

The Basics

First, there’s a few things you’re going to need in even the most basic web framework:

• Dispatcher
• Request API/Object

That’s really the utter basics, and there’s frameworks that don’t do much more than this. Obviously many people are going to want more…. like sessions. The question that WSGI raises is why lock a session interface to a framework, when it can be re-used as WSGI middleware? While the term “WSGI Middleware” can be rather intimidating, I previously covered why WSGI is actually quite easy despite the sometimes scary or just annoying term. It’s something to be reckoned with, and there’s no excuse not to get familiar with it if you’re doing web development in Python.

As I needed something rather reliable and sturdy, I went with Myghty’s session and request API. For dispatching, I used a custom resolver utilizing Routes that dispatches to a controller and action (controller method). This results in a fairly MVC’ish style web architecture that’s rather extensible.

So check those off the above list plus sessions:

• Dispatch – Routes based
• Request API – Myghty’s HTTP request object
• Sessions – Myghty’s session interface, which is also now available as WSGI middleware

At this point, we’d have a framework that can interpret a URL, setup an easy to use request object, and call your controller. Your controller would also be able to save data using sessions. We’d have a bit of a problem distributing such a framework though, as it probably wouldn’t be very convenient to setup and install.

In the case of Pylons, so far it means I’ve written just a class to handle dispatch, locating the controllers (actually, Routes will do this), and calling them. A measly hundred or so lines of Python code.

Defaults, Structure Creation, Stand-alone Server

What really helps people get started quickly with a framework is if its easy to install, creates a basic structure of a working web application for them to get started with, and some way to run it. So our new additions:

• Web Framework Installer
• Template for starter Web Application
• Stand-alone Server

This can quickly be a fairly substantial amount of code to write, especially if you want cross-platform installation, and a system that also runs on multiple platforms. Not a problem though, Paste quickly makes the last two of these quite easy, and setuptools handles installing your framework and any requirements it may have. I should note that setuptools isn’t perfect, but it sure beats asking users to go to a half dozen sites and install various packages.

Paste is divided into 3 parts, though it can be tricky to see the relation between them. The core part of Paste contains wsgi middleware parts that will likely appeal to many, and some other basic request handling functionality. PasteScript contains the structure creation bit used in Pylons that generates the starting template. There’s also PasteDeploy which comes in useful later when deploying and running web applications made with a framework using it.

We’re looking at a decent little framework so far, now that we can install it, quickly start a new project, and run it. What’s next?

Templating and Object-Relational Mappers

Some frameworks, typically known as “full-stack” frameworks try and make more choices for you. They aim to fill your needs top-to-bottom, or at least their vision of your needs. While SQLObject is a very popular ORM, it seemed that good use of a layout and basic Python import statements would make using any ORM just as easy.

In Pylons, a models directory is provided, with some commented out suggestions for how you’d go about setting up an ORM. There’s hooks provided in the base application module that’s imported by all your controllers, so its easy to define your ORM classes and use them in your controllers.

One of the other objectives for Pylons was to try and be very ‘Pythonic’. That is, it should re-use as much of a developers Python knowledge as possible. This is used by default for templating since Myghty uses normal Python syntax for its templates in addition to providing powerful caching functionality and great re-use through components.

Not wanting to force anyone into something they couldn’t stand (template languages can be a love-hate thing), we also implemented the TurboGears Template Engines plug-in functionality. This made it very easy to let people use the template language of their choice, in a fairly uniform manner. It was also pretty minimal to implement the template language renderer as the Buffet project provided a great head-start.

Our new list is looking rather complete for a “full-stack” framework:

• Web Framework Installer
• Template for starter Web Application
• Stand-alone Server
• Dispatcher
• Request API/Object
• Sessions
• Templating
• Database Integration

Making the most of Middleware

This is one of the areas where Pylons was able to make some great leaps, with very little actual code. In several areas, thanks to the use of middleware, Pylons is able to offer features other frameworks are still working on.

To start off with, there’s the excellent EvalException middleware which provides an AJAXy exception catching system I have yet to see in any other framework. We’ve formatted it slightly to fit in nicely with Pylons, and it works like a charm. Extremely useful for those times when you want to interactively debug a web application to see what’s what, and it’s a lot quicker than putting print statements all over.

Another important bit, that a lot of frameworks skimp on is unit testing. Using Paste’s fixture middleware, its easy to test your web application. Pylons adds a few objects to the response you can test with, so you can ensure that the session was setup properly, the right template components were called with the right arguments, etc. In the future we’ll likely add some defaults to make using twill an easy option.

The best thing about all of this of course, is that these useful parts can be integrated seamlessly and re-used by other frameworks.

Taking the framework out of Framework

Given how most of these parts are definitely not unique to Pylons, nor are they intended to be, it shouldn’t be long until more frameworks start using the great modularity that Pylons is utilizing. A lot of these parts will likely become standardized to an extent, so that there’s even less barrier to switching frameworks.

At this point, Pylons becomes less of a “framework” in one sense, and more a set of defaults and structure for how a Python web application should be put together. Pylons has more features of course that I haven’t described, the WebHelpers functions are made easily available for use in templates, more Paste middleware is used for slick traceback email’s when you’re in production mode, PasteDeploy makes running your Pylons webapp easy in a variety of situations, etc.

It’s a very easy-to-extend model, with little need to put great amounts of application-y type stuff into the framework itself. It also keeps its components separate for easier testing, out of the actual framework. This means that while Pylons comes with a great set of middleware and parts set up for you, its very easy to swap in your preferred template language, your preferred ORM, a different exception handler, etc. The choice is up to you, but the defaults are set to a good starting point (also called “convention over configuration”).

Upping the Re-usability Ante

Increasing re-usability is what I’d consider the future for Python Web Development. With WSGI middleware driving re-usability, such concepts that “to use Feature X, you must use your Framework Y” just doesn’t apply. Unifying development work on excellent components that can be re-used in any WSGI-compatible framework makes Python Web Development better.

Pylons isn’t alone in aiming for this style of framework, Ian Bicking is working on a project that starts the opposite direction, with just a layout and you add sessions, templating, etc. as you need it. TurboGears is adding more Paste-compatible features that will shortly make it trivial for them to add in the EvalException middleware (assuming they haven’t already, I haven’t checked lately) and other great components. Different frameworks have different levels of re-usability, those that are built with re-usability in mind at the beginning will likely be able to adapt quicker to new demands and requirements, and take maximum advantage of the great middleware being created.

While having more people use Pylons would be great, it isn’t necessary for Pylons to become a better framework. Having more people use the WebHelpers package, or make their framework Paste-compatible, or use Routes, or Myghty’s powerful caching/session API’s all helps Pylons. It also helps any other framework using these components, and that’s what counts the most.

Here’s the final tally of Pylons features and where they came from:

• Web Framework Installer – setuptools
• Template for starter Web Application – PasteScript
• Stand-alone Server – Paste
• Dispatcher – Routes / Myghty
• Request API/Object – Myghty
• Sessions – Myghty
• Caching – Myghty
• Templating – Myghty, or any that support the TurboGears Template Plug-In
• Helper functions/AJAX – WebHelpers
• JSON – simplejson + Pylons decorator
• Global “convenience” objects – Pylons
• Database Integration – SQLObject, SQLAlchemy, anything else
• Interactive Debugging – Paste
• Traceback E-mails – Paste
• Webapp Unit Testing – Paste
• Webapp Deployment – PasteDeploy
• Webapp Distribution/Installation – setuptools

This is just the default set-up, its trivial to add more middleware, which would make this list very, very long and includes such things as OpenID Authentication, authenticated session tickets, along with other great stuff.

The Pylons Code-base:
Pylons Module Reference

Zachary on Routes with CherryPy

Zachary.com has a very nice update on Routes with CherryPy covering not only how to integrate them, but why you’d want to use Routes style dispatch instead of the object publishing approach CherryPy uses by default.

In the future, there’ll also be an independent dispatcher for Routes that handles dispatch at the WSGI level. I believe Ian Bicking already has such code, though he hasn’t released it yet. I’m sure it’d make a great Part 2 in his series(?) on working with WSGI up close and personal. If you haven’t read his Do-It-Yourself Framework, I’d highly suggest giving it a read.

Not only does it help demystify WSGI, but it should hopefully make it more obvious why WSGI is changing the point of competition in the world of web programming.

Why Python?

I first heard about Python from a roommate about 5 or 6 years ago. He was putting together a presentation for some Bio-informatics people, and was going to go over the basics of Python programming as its very popular in the scientific community. I didn’t give it much though as I was mainly using Perl at the time for any programming work, and the white space was of course somewhat of a put-off.

Fast forward a few years… I had been programming rather heavily in Perl at the time, and despite having used Perl literally for years, I still felt like I hadn’t achieved “mastery”. The effects of code in Perl would jump out and surprise me every few days, and sometimes hunting down weird bugs due to ambiguous syntax was rather time consuming. In short, Perl just wasn’t fun anymore.

I started looking around for other languages, and took a look at Ruby. Back then, Ruby had the same things that were annoying me about Perl. Implicit variables that seemingly came out of nowhere (@_, $_, etc), inconsistencies, and since the community was so small it lacked the vibrant and active development Perl had. Ruby also was missing so many of the CPAN modules I had come to rely on. Then browsing through some of Eric Raymond’s writings, I came across an article about Python he wrote titled the same as this blog entry.

White space and Mastery

My white space fears were removed rather quickly.

Of course, this brought me face to face once again with Python's pons asinorum, the
significance of whitespace. This time, however, I charged ahead and roughed out some
code for a handful of sample GUI elements. Oddly enough, Python's use of whitespace
stopped feeling unnatural after about twenty minutes. I just indented code, pretty
much as I would have done in a C program anyway, and it worked.

But this wasn’t what truly piqued my interest. As I mentioned, the feeling that Perl was somehow beyond mastery. To me, mastery means that I can write massive chunks of code with confidence it’ll act exactly as I intended it to. So seeing Raymonds next few paragraphs really got me going.

That was my first surprise. My second came a couple of hours into the project, when I 
noticed (allowing for pauses needed to look up new features in Programming Python) I 
was generating working code nearly as fast as I could type. When I realized this, I 
was quite startled. An important measure of effort in coding is the frequency with 
which you write something that doesn't actually match your mental representation of 
the problem, and have to backtrack on realizing that what you just typed won't 
actually tell the language to do what you're thinking. An important measure of good 
language design is how rapidly the percentage of missteps of this kind falls as you 
gain experience with the language.

This is all summed up very nicely,

I wrote a working, usable fetchmailconf, with GUI, in six working days, of which 
perhaps the equivalent of two days were spent learning Python itself. This reflects 
another useful property of the language: it is compact--you can hold its entire 
feature set (and at least a concept index of its libraries) in your head. C is a 
famously compact language. Perl is notoriously not; one of the things the notion 
``There's more than one way to do it!'' costs Perl is the possibility of compactness.

Being compact is a valuable asset for Python. Not only does it make writing large blocks of code without error possible, but it also means the code is going to be easy to read. After opening up a few other Python projects and scanning through the code, I was hooked and proceeded to buy the Learning Python book (Though the online tutorials would’ve been just fine as well).

Black Magic isn’t so Black

Since then, I’ve been using Python close to non-stop for just a little over a year. In just the first week of using Python, I felt more productive and wrote code I was positive would work (just as ESR said) on the first try. A few months ago, interested in learning even deeper “black magic” of Python, I went to a presentation by Alex Martelli on Python’s Black Magic, Meta-classes and Descriptors.

The most fascinating thing was that meta-classes got their behavior from simple concepts I already knew. It was hard to believe things were so easy. Eric Raymond also noticed this:

It's remarkable enough when implementations of simple techniques work exactly as 
expected the first time; but my first metaclass hack in a new language, six days from 
a cold standing start? Even if we stipulate that I am a fairly talented hacker, this 
is an amazing testament to Python's clarity and elegance of design.

Fields of Use

I have now written quite a few projects in Python, and on several of the open-source ones heard, “I just wanted to add Feature X to this, and I was actually able to figure it out just look at the code. I don’t even know Python!”

These projects span fields: shell scripts, network daemons, web applications, computing, GUI’s, and more. Learning Python is useful for many fields, and I know people learning Python right now so they can tackle problems in different fields with a compact language thats easy to master.

The Choir

I realize that this entry, carried by Planet Python, is mostly preaching to the choir. However, I felt something like this was needed. Partly because some members of the Python community have gotten odd impressions about what we should try and do in Python and what we should just “give up” on, having lost some sort of war (bizarre, I know).

People come to Python for a variety of reasons, they stay for a variety of reasons. Having compelling tools to make tasks easy in many fields is great, and means there’s no need to learn new languages solely to try and make a task in one field slightly easier (Though its always healthy to learn more).

It’s definitely valuable knowledge to know why a task in one field is easy using Language X, and for that alone its good to give it a spin. Learn what was done right, and what was done wrong. In the end though, if that language isn’t a compact language like Python, I’m just not going to enjoy it as much.

I didn’t start learning Python because of Application/Framework Y, I learned it because Python is exceptionally compelling by itself. Others learn it for the same reason, and despite the bizarre claim that book sales is equivalent to language usage/popularity, it should be obvious by now that a book isn’t needed to master Python.

For those demoralized because some other language is getting attention due to a field its had a huge impact on, re-evaluate your feelings. This isn’t the first time its happened, and it won’t be the last. Google uses Python, ILM uses Python, thousands (yes, thousands) of major corporations use Python.

Python doesn’t need to be master of every field, it just needs something compelling enough that you’re as productive using it as you would be using a different tool in a language that isn’t so compact.

Is Rails a DSL? What is a DSL, and is it possible in Python?

I keep seeing blogs and blog comments pop up with some very odd notions. Many of these are summed up in this comment from a recent post by Tim on Ruby books

“It’s doubtful that the Python folks can come up with anything as compelling (or elegant) as Rails. Why? Because Ruby is so good at creating Domain Specific Langauges (DSLs). Ruby’s anonymous code blocks are a big part of what enables DSLs to be written so easily in the language. Python doesn’t have them. Python’s lambda’s (and closures in general) are crippled as well, which also doesn’t help Python’s cause.”

What is a Domain Specific Language (DSL)?

I always like to carefully define my terms before actually talking about them. So first I looked up several common definitions of a DSL. We have Martin Fowler’s entry on a DSL which describes it as:

“The basic idea of a domain specific language (DSL) is a computer language that’s targeted to a particular kind of problem, rather than a general purpose language that’s aimed at any kind of software problem.”

He further clarifies between two types of DSL:

“The most common unix-style approach is to define a language syntax and then either use code generation to a general purpose language, or write an interpreter for the DSL. Unix has many tools that make this easier. I use the term External DSL to refer to this kind of DSL.”

“The lisp and smalltalk communities also have a strong tradition of DSLs, but they tend to do it another way. Rather than define a new language, they morph the general purpose language into the DSL. (Paul Graham describes this well in Programming Bottom-Up.) This Internal DSL (also referred to as an embedded DSL) uses the constructs of the programming language itself to define to DSL.”

To summarize, an Internal DSL satisfies one of the following:

• Uses a general purpose language, then morphs it into a language to fit the domain
• Adds functionality to a general purpose language such that solving a specific domains problem is easier

An External DSL:

• Completely new language syntax
• Typically requires code generation or interpreter for DSL

Martin Fowler espouses on how very dynamic languages like Lisp, Smalltalk, and Ruby are more conducive to internal DSLs as the meta programming capabilities and dynamic nature make it easy to extend and customize the functionality for your domain. I completely agree with his contention that an Internal DSL is limited by the syntax and structure of the language.

DSL’s in Python and Ruby

While the anonymous code blocks in Ruby are useful and syntactically nice, similar functionality can be done in Python through the use of generators and iterators (more powerful generator functionality similar to Ruby’s anonymous blocks are planned for Python 2.5).

Ruby also lets you extend built-in types, which is utilized by Rails to add a few helper functions to core Ruby data types. Python and Ruby both have closures that operate slightly differently, Ruby’s closure is definitely more “full featured”. However this typically isn’t a problem because of list comprehension’s, generator expressions, and generators/iterators.

Zope is a very clear example that its quite possible to build an internal DSL in Python. Whether they built it in a clear, easy to use way is up for someone else to debate. SQLObject makes dealing with databases as easy as ActiveRecord, and there’s many other examples of internal DSL’s built in Python (Take a look at twill, a web application testing language implemented in Python).

Is Rails a DSL?

First, it should be obvious that if Rails is a DSL, its an Internal DSL. The most noticeable strides towards being a web programming DSL in Rails:

• Over a dozen core Ruby classes are extended/modified
• Dozens of helper functions
• New classes (syntactic sugar) defining database access and web paradigms

Given how loose the definition of internal DSL is, I think its clear that Rails qualifies, so does Django, TurboGears, Aquarium, etc.

If you prefer to be more strict about an internet DSL, and argue that a lay programmer needs to be able to do a reasonable amount of ‘work’ without knowing the base language implementing it, Rails starts to move apart. You can do a reasonably large amount of work in Rails without knowing Ruby. A lot can also be done in Zope without knowing Python.

In either case, you can’t really start to make advanced applications in any of the frameworks without knowing the general purpose language its built on. DSL’s can and have been built in Python, and I believe if you go out and start counting them up, you’ll see more DSL’s implemented in Python than in Ruby.

Anyways, hopefully for people that run around re-iterating something they heard somewhere about DSL’s and Python vs. Ruby, this clarified something. It’s easy to create DSL’s in Python, lots of people do, and there’s lots of them. Python and Ruby are both great for making them, outside of Rails I don’t know any other DSL’s in Ruby. Perhaps someone can compile a list of all the internal DSL’s written in both Ruby and Python?

Project/Code Re-Use, TurboGears, and Django

TurboGears has been making some impressive strides in both features, documentation, and possible user acquisition lately. Where it gets somewhat interesting is regarding its user-base though. The approach TG takes – building glue on top of other projects – is not new, as Subway also utilizes this, however the popularity TG has been enjoying has resulted in some interesting by-products.

First, its a bit interesting to take a look at some numbers. It’s hard to be precise, as users don’t exactly announce themselves, but many of them do usually join the mailing lists for the projects.

Django-users:

• Users: 546
• Activity: Medium
• Average messages per month (5 months+): 340

Update: Eugene pointed out that there is Django-developers as well. Many of the Django-devel members are also on the Django-users list, so adding them together wouldn’t be too accurate. Here’s the Django-devel information, along with the total traffic if you take the two combined.

Django-developers:

• Users: 323
• Activitiy: Low
• Average messages per mont (5 months+): 177
• Average messages of Django-devel+users: 517

TurboGears:

• Users: 668
• Activity: High
• Average messages per month (3 months+): 1,124

Information taken from Google Groups as of Nov. 14th, 2005

While each list is bound to pick up users that contribute and lurk who do not actually use the framework, I think this trend is significantly higher with TurboGears. I believe this is because TurboGears builds on several successful projects that are even used in fields outside of web development.

Project/Code Re-Use and Explaining the Mail Traffic

Django takes a somewhat interesting approach, especially when compared to TurboGears, in that re-use is non-existent.The reason given for the amount written from scratch does make sense to a certain extent, especially when you remember that Django was created 2 years ago (some existing projects weren’t too hot back then). It’s also interesting to note that in some cases, the syntax and decisions being applied to Django now, reflect those made quite awhile ago in other projects.

TurboGears pushes re-use to the extreme, and only resorts to writing it from scratch if there’s no clean way to integrate a similar project that does the same thing. Only the new Identity framework, and the crud stuff was written from scratch for example. But many of the largest, most heavily used pieces have large communities of their own, i.e. SQLObject, FormEncode, CherryPy, MochiKit, and setuptools (I likely missed some).

Also interesting to note, several of the top posters to the TG mailing list happen to be the project developers for the individual projects TG is comprised of. I doubt they’re heavy users (or perhaps haven’t even used TG), however they have plenty to contribute as the questions often relate to the individual parts. This creates a powerful community and further fuels development and use of the individual projects TG pulls together.

When considering the use of one framework vs the other, I think this re-use issue is definitely something to be considered. Building on other stable, established projects that are used widely and extensively leads to a more mature platform. The size of each individual community also means more people to work on solutions that benefit the whole (and even those using the parts).

For example, the TG community is starting to tackle some basic CRUD/administration type stuff. I don’t actually use TG myself, but I do use SQLObject and FormEncode which means I can plug my SQLObject classes into a TG project, and use their web based admin tool with my database. This is rather powerful and extremely useful code re-use, and in the future it’ll be even easier to do the same thing (likely using a WSGI app).

Code re-use is important, and the project re-use that TG employs leads to many benefits. Despite the reason cited on the Django page, I think Django is still ripe for re-use cases. The Django syntax is now so close to SQLObject, I wish Django would just switch to it and extended it where needed, rather than continue to re-implement the rest of it.

To be fair, Django does make it possible to use other templating languages, and Zope3 Page Templates are available (along with a Django ZPT error formatter). If part of the argument for OOP and Python is code re-use, projects that re-invent not just the wheel but the entire car aren’t exactly shining examples of what’s possible in Python.

The Wacky World of Ruby

Ruby is a fairly interesting programming language, from the “expressive” syntax to some of the absolutely bizarre documentation. For a Python programmer, the lack of predictability and almost excessively concise syntax (when just one more line would really make things a lot clearer) can be a bit of a downer. Overall though, I’m rather enjoying my experiences with Ruby but not enough that I’d want to use it exclusively.

The “Wacky” bit I cite, comes from some of the strange directions the language seems to go and wacky documentation and books available. Now, the creator of Ruby is completely aware about some of the ways in which Ruby sucks as he put it, and 3 of those are of particular importance to Python programmers since we enjoy a concise (and not complex), predictable, and consistent language. Ruby is working to address these issues in the upcoming 2.0 release which I’m hoping will make it more pleasant to work with.

It is also somewhat wacky the way the Ruby community has compared themselves to other programming language communities to which this blogger has a fairly friendly reply. Sometimes these little jabs run into the printed documentation regarding Ruby as well, which is a big turn-off for me, especially since I really disagree with the reasons that are cited for insulting other languages.

Beyond Java?

Take the Beyond Java book for example. About half-way in it becomes very clear that the author’s view of what is Beyond Java…. is Ruby, and for web programming, Ruby on Rails. The author then goes on to smash Perl, Python, and PHP (ok, I agree with him on PHP :). He puts Ruby in the mix as well, but the only bad thing he says about it is that its lacking commercial backing (except for later on where a new reason emerges).

His comments regarding Python are truly wacky though, as he jumps back to the very old “white-space reliance sucks” argument that hardly ever comes up in the real world. I’ve been using Python for over a year now, and have worked on quite a few collaborative projects, and have yet to see a single error related to mismatched white-space. This is probably because coding standards are well known and Python programmers actually follow them for the most part. It’s hard to express how wonderful this has made it when I’ve jumped into code written by other people, and have been able to easily scan it and add functionality after just a few minutes of looking it over.

When I’ve jumped into other people’s Ruby code looking to make a quick-fix, the syntax quickly became a massive chore to decipher as Perl’s motto of TMTOWTDI holds very true in Ruby as well.

The Beyond Java book also cites Python’s lack of a “killer app” when it comes to web programming and specifically references Ian Bicking’s article on web programming frameworks. Of course, since that article Ian has put out Python Paste which solves a host of problems he mentioned there, and the Python web community’s move to WSGI is helping to standardize methods of running Python web applications.

Even more wacky, later in the Beyond Java book, in yet another comparison of whats for and against the languages, a different reason pops up for Ruby not doing so good. What is it this time?

The biggest strike against Ruby right now is the lack of a strong project that lets Ruby run on the JVM. – Page 163

He goes on to cite how much support Ruby would get if Microsoft was able to woo the Ruby founders over to .NET’s CLR. On the very next page (165) another for and against argument comes out for Python. Since the author just mentioned the JVM and .Net CLR as major drawbacks to Ruby, I was actually expecting him to mention Jython, or even IronPython which is actually being developed by a programmer now at Microsoft. Amazingly enough, neither of these projects is mentioned here, though Jython was mentioned back near the beginning as being too slow (which IronPython apparently solves).

It gets even more wacky a few more pages in, as his reasons against Perl come out.

Perl does have a downside. When you look at overall productivity of a language, you’ve also got to take things like maintenance and readability into account. Perl tends to rate very poorly among experts on a readability scale. – Page 174

Who these experts are is never mentioned, and I’ve seen “experts” for and against the Perl syntax. While Ruby is easier to read than Perl in my personal opinion, its nowhere near as easy to read as Python.

Programming Ruby, The Pragmatic Programmers Guide

I’ve been reading this book for awhile now, and the author’s decision to do something different is admirable but has really been a bane to reading the book. I’ve also read the Agile Web Development with Rails book, which I think was done in a most excellent manner (written by Dave Thomas, the author of Programming Ruby). So this isn’t an attack on Dave, as I really enjoy his writing, I just believe this approach didn’t work so well.

The approach taken in Programming Ruby is to breeze over high-level uses of the language without actually explaining much about why things acted as they do. This quickly drove me nuts, and I stopped reading the entire intro as seeing syntax for no reason wasn’t helping me learn anything. If you want to learn Ruby in the way most programming books teach a language, by carefully and completely going over all the parts then doing more advanced things; skip to page 317 in the Ruby Crystallized Part.

Once I started reading here, everything fell into place very nicely and the language really started to make sense. If the sections were reversed as most programming books have it, I’d consider this book pretty much perfect for a programming book. The thing I think Dave might have missed here, is that most programming books follow this convention because it works. Being different, just to be different, isn’t very good unless there’s a real and practical reason for doing it.

The wackiness doesn’t end though, and I have yet to complete the entire book so I can’t say how many more examples of this are there. Here’s the latest gem though, which actually prompted this entry (though I’ve been thinking these thoughts for awhile).

On Page 330, going over the details of Variables and Constants, I came across this:

Ruby, unlike less flexible languages, lets you alter the value of a constant, although this will generate a warning message.

HUH? Errr, then why the heck do they call it a Constant?? Seriously, maybe its because I’m picky on language terms used but I think they should’ve called it a semi-Constant, or a mostly-Constant Constant. For me, this goes against the entire notion of what a Constant is. Then, on top of that, it insults other “less flexible languages” that (_gasp_) don’t let you change constants.

It’s a Wacky World

There’s many more examples of the wackiness present in the Ruby world. Perhaps its because the language is from Japan, home to so many wacky things by Western standards. Though the writers I mentioned are all non-Japanese, so this explanation doesn’t really cut it. To be fair, the Beyond Java book is well written and the reasons cited in many of the comparisons are valid to an extent.

Ruby in Rails also has its share of wackiness, though it seems so abundant I’ll have to save that for another post entirely. If you’re wondering after all this, why I’m still using Ruby… well, it’s a wacky world, and I do kind of like wacky (I think I’ve used up all allowed uses of the word ‘wacky’ by now). Ruby 2.0 looks to be quite appealing and it’ll be interesting to see how Rails adapts to so many breakage’s that 2.0 appears to introduce over 1.8.

Python isn’t perfect either, and I’m not going to claim it is. There’s plenty of people in both the Python world and the Ruby world who are very forthcoming about failures and successes of the language, so the views expressed by the authors in these books should not be taken to represent the whole. They are some of the most visible speakers though, so I hope that they can someday be as forthcoming as Matz has been.

Python Web Framework Niches

I’ve come to the belief lately that the web frameworks available in Python are increasingly fine-tuned to specific application requirements. Of course, anyone reading the ‘About’ sections for these frameworks should realize this as well. I wonder how many people actually read that section as I’ve seen people latch onto web frameworks without knowing the task it was originally made for.

Without knowing the reason the framework was created, its common for many people to leap to the conclusion that its another Rails wanna-be just because its a ‘full-stack’ web framework. I was playing around with a nice full-stack framework called WebObjects years ago which made it easy to setup database objects, generate CRUD, etc. Zope’s been doing the same stuff for what now seems like eons as well, yet I don’t see people declaring RoR a Zope clone (It obviously isn’t).

In light of that, I’m inclined to agree with Ian Bicking’s response about the lessons Python web people did learn from RoR.

The concept I want to focus on is that people create these new frameworks because they make their task easier than any of the other frameworks already out there. While they might pick up features from other frameworks, most of them aren’t aspiring to be “Python on Rails”. Sometimes this task is easier when other tools can be integrated to avoid code replication, as is the case in one framework I cover here.

Many people have declared the amount of Python web frameworks a “problem” that should be “solved” somehow, perhaps a Highlander fight with swords to the death (_There can be only one!_). I’d like to suggest the opposite, there’s a lot of Python programmers and I think there’s room for even more web frameworks. The variety is a strength because they make it easier to get specific web applications done.

TurboGears

The TurboGears site has a nice about page describing its purpose, though I feel it doesn’t completely explain the rationale for its creation. There’s some interesting and unique decisions made in TurboGears, like using Kid instead of Cheetah or Myghty for templating. Then there’s the inclusion of Mochikit and the TurboGears decorators for returning output as JSON for use with Mochikit.

So what kind of applications is this web framework geared for? (Please excuse the pun)

The best way to answer this is to look at the application this framework was created for, Zesty News, and the abilities of some of the tools being used. Zesty News is in a rather interesting category of web applications in that the end-users themselves will be installing it, quite likely on their home computer rather than a server. Being able to package it up and easily distribute/upgrade it becomes a key issue along with database portability and code thats database agnostic.

Two tools assist here, setuptools for distribution/packaging and SQLObject for portable database code. Zesty News deals extensively with RSS and XML, so it makes sense that the templating language chosen was actually created for dealing with web services.

These design decisions behind TurboGears should make it fairly obvious when to consider it for your next project. The cohesive toolset you get when you choose TurboGears is ideal for developing portable, easily deployable AJAX-enabled web applications that likely deal with XML frequently and need to stay database agnostic. Even if your web application doesn’t deal with XML frequently, the decisions TurboGears makes for AJAX integration will make it easy to add heavy dynamic interaction to a TG webapp.

Django

Django was created to deal with the requirements of working in the web development department of a news publisher. As such, the framework was created specifically to deal with the requirements placed upon the author. What’s rather interesting is the lack of re-use in Django when it comes to doing things that have been done before in other projects (Database mapper, form validation, etc).

The tools and parts of Django were specifically built to work as one package, and using Django makes that very obvious. One of the things most common when in a newsroom or publishing environment is dealing with CRUD. That is, there is a lot of content and ways to get content into the system and administrate the content is a high priority. As a web framework built for dealing with Content, many of the design decisions reflect the common tasks present in CMS’s (Content Management System).

To start with, you get a slick administration interface for your conntent, that’s miles beyond any of the generated CRUD type stuff in other web frameworks. This differs from the philosophy of other web frameworks that give you basic CRUD (Scaffolding in Rails-speak) in that Django’s admin interface is aimed directly at being production-ready with no modification at all.

Django also makes it fairly easy to make a Django ‘application’ like a Forum or Blog, then slot it into other Django application environments. Again, this makes a lot of sense given the original requirements placed on the creator of Django. If a company has 4 websites, and wants them all to have the new Forum/Classified ability it makes a lot of sense for this task to be optimized.

So what web applications are you going to want to use Django for?

Quite a few, as it turns out dealing with Content is a very common task. If you’re writing a web application heavy on content, that needs a full featured web interface for managing the content it’d be really hard not to recommend Django. It’s easy to get started, and in almost no time you have very powerful functionality running that gives you a lot of usability.

Don’t be Everything for Everyone

Part of the reason I picked these two projects to talk about is that they’re both extracted from a working project (as Rails was). I also haven’t seen many people mention the fact that frameworks developed in such a manner are also inherently going to be optimized for the use-cases that brought them into existence.

Open-sourcing the project lets them grow to an extent, but their design is largely baked in and a useful limitation. Too much expansion past the initial design requirement will make them generic, and with that comes a lot of complexity (sometimes worth it though for the extra re-usability).

Note that the specific things Django gives you don’t help that much if you’re trying to write a Zesty News style application. The same goes in reverse as well, since building an Admin interface of your own isn’t fun and can be time consuming. While it’s possible to make web applications that do this in either framework, compensating for framework design will require extra time when you try to use one framework for everything.

If you’re using one framework for everything, maybe its time to take a look around.

Ghost in the Prius

I’ve owned my 2004 Silver Prius for about 18 months now, and in that time the only ‘problem’ I’ve had is the hilariously named Red Triangle of Death often fondly referred to as the Red Triangle of Doom as well (I shall now refer to it as the RToD).

In the case of the RToD, the problem itself seemed to go away after leaving the car off for a little while, much like this poster describes. I found the whole incident rather humorous and frightening in a way, as the dealer didn’t apparently realize that the Prius Drivers Manual states that in the case of this specific error the owner should call the dealership.

Before I describe the latest incident (quite minor in comparison to the RToD), looking back over the RToD incident would be helpful.

I was on my commute home from work (about 26 miles), and have to go through a fairly shitty interchange where 2 lanes of traffic merges to one, then merges with an on-ramp to one, THEN merges onto a very busy freeway. I’m referring in this case to the interchange in the North Bay Area where West 580 meets 101 North.

While sitting in the stop and go merge, I noticed that the engine turned off. Not unusual in the Prius as the engine goes to sleep on occasion during stop-and-go. However, after awhile it didn’t turn back on… then the RToD went into full effect. A few pretty glowing lights on the dash, the LCD got a nifty little icon as well.

Seeing the RToD definitely gets your attention. It’s not something that can be easily ignored, so I pulled off the road and yanked out the Prius Drivers Manual to see what the icons lighting up actually meant. I was already wondering at this point, why the hell do I need to pull out a manual when its an LCD, why not just print the explanation on the damn thing? The screen says all sorts of other useful text, it seemed quite lame they couldn’t just print a full explanation and what to do next right there.

The drivers manual said that in this particular case, I should pull off the road as soon as its convenient (vs. the error about the braking system which said to pull over immediately). Ok, no problem, I was already pulled over. It then said to contact the dealership immediately. It was 7pm, but I figured, what the hell, it said to contact them so I pulled out my cell phone and called them up. My call went like this:

Me: “Hi, I’m calling to report a problem with my Prius”

Dealer: “Sorry, our Service Center is already closed”

Me: “Well, the manual said when these lights go on, I’m supposed to call the dealship. Any clue what I should do?”

Dealer: “The manual said you should call us?”

Me: “Right”

Dealer: “Really… huh. Toyota never told us that.”

At this point I was a little concerned since the dealership apparently didn’t realize that the drivers manual told people to call them in the event of a half dozen different errors. It didn’t say call the next day, it said call immediately and seemed quite firm about it.

While chatting with the dealer, I was still fiddling with things in the car myself, and I tried turning it on again at which point most of the lights went away, and only the engine light was still lit. Since this light merely indicated “see dealership soon”, I figured it was good enough to drive home and scheduled an appt for the next day with the dealer.

The next day, none of the lights were on. I still took it into the dealership where they did a debug dump of the computer, and installed an ECM software update which they said would prevent it from occurring in the future.

The Problem

The real issue that this incident left me pondering, is what the hell do you do if you’re a long ways from a dealership? As only the dealership is somewhat prepared (most of their techs admit they have no clue how to fix it, they just follow manuals) this means you’ll have to plan trips careful to make sure you’re never too far from a dealership should something go wrong.

Someone on a PriusChat forum mentioned that the RToD happened to him on his property out in the back country of Canada. It took over 3 hours for the tow-truck to get to him, and hours more to tow the Prius to the nearest dealership (As his car never got better after leaving it off and turning it on again). At least he was lucky enough to have cell phone reception or some other way to contact a tow truck.

The Prius was originally unique in just how many systems are entirely computerized, however many modern cars are now catching up in electronic complexity so this is starting to become as likely to happen to any new car owner. More complexity of computers systems increases the likelihood of a possible bug cropping up and the testing process will never be able to eliminate them all. This means the people driving the vehicles are in a way, beta-testers.

I don’t think thats too horrible, as the most tested parts are the most essential ones. If the failures resulted in cars flying off the roads, we all would’ve heard about it by now (Right?).

I’m hoping that more car manufacturers in addition to Ford license and utilize Toyota’s hybrid technology because if there was some sort of standard for these parts you’re likely to see more independent repair-shops drop the big investment for the equipment to repair them. That in itself would make me a lot more comfortable driving long treks where dealerships are scarce.

As I mentioned earlier though, the repair techs at the dealerships don’t actually know how to fix any of these problems. They follow service manuals that indicate they should plug Gizmo-A into Plug-A on the car, hit Button-X to dump the data, then send the data back to Toyota for their computer/car engineers to analyze (More or less). Then they usually will either install a software update, or reset the software on the vehicle and hope the problem goes away. Isn’t that reassuring?

Todays Incident

Well, it was pretty minor now that I’ve fully dredged up the memory of my RToD experience. Today my car decided that I’ve been listening to the stereo so much, so it started being a pain. It was sort of like having a child in the car messing with me.

First the audio cut in and out, then it turned off completely. The LCD then decided it no longer needed to inform me of the outside air temperature (is it hot? cold?), and my steering wheel audio controls didn’t appear to be having any effect. So I hit the stereo power on/off and its back…. sort of. Then it cuts out, and I fiddle with it some more till its back on. It continues to do this throughout my drive in, occasionally coming back up to full functionality, then dropping the audio again.

I’ve already called and scheduled an appt at the dealership for tomorrow, so we’ll find out what the heck it is this time after that. For all I know, it might be my friend again on the drive home and leave me alone. In the meantime, I think I’ll browse the Prius forums and see if there’s some ritual that will align me with my car’s spirit so I can hopefully make peace with it.