be Groovie

Notes on the Pylons & repoze.bfg Merger

Some folks might not have time to follow the Pylons-discuss mail list, so this might be news to them, but I’m thrilled to announce that the Pylons and repoze.bfg web frameworks are merging. If this is the first you’ve heard about it, don’t worry, it was only announced a week ago now on the Pylons mail list.

In the time since the announcement, I’ve heard a lot of varying feedback. Some people took a look at Pyramid (the core package that will be equivilant to ‘Pylons 2.0’) and were quick to respond, usually in a knee-jerk type response. I think some of this was due to a miscommunication, and partly because there was so much already done. When other frameworks have merged in other languages, such as Rails merging with Merb, the announcement was just that. There was no code at the time to show, just a promise that when it was ready, it would be awesome.

This merger in contrast already had a starting foundation for a huge chunk of the core features. As a result, people assumed that what we had was already ‘finished’, or close to it. The polish of much of the documentation made it feel odd that there was no “Porting Pylons 1.0 to Pyramid” guide done. In reality, Pyramid is definitely not done, there is still quite a bit of work left before Pyramid will meet the expectations that many Pylons users have. There’s still refinements to be done to Pyramid, and additional packages that Pylons users will most likely always use with it for the feature-set they’re accustomed to.

I’ve summed up a few thoughts on when Pylons users should port to Pyramid to try and help manage expectations better in the future. I’ll make more announcements when packages are ready to ease the transition and a “Porting Guide” is ready.

What is Pylons?

Many Pylons users don’t realize which features they enjoy come from the package ‘pylons’ vs. the other packages that Pylons depends on. Contrary to popular belief the majority of features present in Pylons actually come from other packages. This mistaken belief that most of the features come from the pylons package led some to think that because a lot of my future development time will be spent on adding features/packages around pyramid, Pylons is somehow dead>. This is not the case.

First, Pylons the web framework is mainly a small (~ 1000 LoC) glue layer between Paste, PasteScript, PasteDeploy, WebOb, WebError, Routes, WebHelpers, Mako, and SQLAlchemy. Some people usually end up swapping out Mako/SQLAlchemy but by and large this is the common ‘Pylons Stack’. Most of the new features in Pylons over the past several years actually came from additions to WebHelpers, WebError, or Routes. All of these packages continue to get the same development as they have, so no ‘death’ is occurring.

Second, for over the past 6 months now, there’s been very little in the way of patches submitted, bugs reported, or other feature requests. In many ways Pylons is ‘done’ regarding adding more feature to the core package itself. As I announced on the Pylons-discuss mail list, the Pylons code-base hit some design issues. Adding the features I heard requested from quite a few users (and needed myself) regarding extensibility couldn’t be retro-fitted into the existing design. I encourage anyone curious to read my prior entry on sub-classing for extensibility to be a preview of some future blog posts. I’ll be writing more about design patterns in Python that handle extensibility which many popular Python web frameworks are also struggling to handle.

The Future

I’m very excited about the future for the Pylons Project, which is the new over-arching organization that will be developing Python web framework technologies. The core will be Pyramid, with additional features and functionality building around that. We’re already quickly expanding the developer team with some long-time contributors and having a combined team has definitely helped us progress rapidly.

One of my main goals is to encourage and ease contributions from the community. To that extent I’ve been filling in the contributing section for the Pylons Project as much as possible. I believe this is an area that will quickly set us apart from other projects as we emphasize a higher standard of Python development.

Django did a good job setting the bar high for its documentation of how to contribute to Django, which deserves a lot of credit for clearly defining community policies. Its missing a portion we considered extremely valuable which core developers generally get very picky on when accepting patches… how to test your code. The Pylons Project adapted the rather thorough testing dogma noted by Tres Seaver, which I personally can’t recommend highly enough when it comes to writing unit tests. It’d be nice to see more posts expand on exactly how to test your code. Many developers (including myself) can write code that passes 100% test coverage… but is it brittle test code? Prone to failure if some overly clever macro it uses fail? Seeing a well written set of examples on designing unit tests to avoid common gotcha’s is definitely something anyone contributing (and developers in general) should be familiar with.

For those wanting a gentler introduction to Pyramid (the docs are very verbose and detailed, not at all opinionated), I’ll be blogging more about new features and how to utilize them. Please be patient, I think a lot of people are going to be excited at what’s in store.

Why Extending Through Subclassing (a framework’s classes) is a Bad Idea

Ok, I’ll admit it, overly ambitious blog post. So I’ll refine it a little now, this is intended mainly as my thoughts on why as a tool developer (one who makes tools/frameworks that other programmers then use), its a bad idea to implement extensible objects via developer subclassing. This is actually how the web framework I wrote - Pylons - provides its extensibility to developers and lets them change how the framework functions.

Please excuse the short and possibly incomplete description, this is mainly a quick post to illustrate a tweet I recently made.

First, some background…

One of the things that Pylons 1.0 and prior is missing is a way to easily extend a Pylons project. While it can be done, its very ad-hoc, kludgy, and generally not very well thought-out. Or thought-out at all really. What was somewhat thought-out was how a developer was supposed to extend and customize the framework.

In a Pylons project, the project creates a PylonsApp WSGI object, and all the projects controllers subclass WSGIController. This seemed to work quite well, and indeed many users happily imported PylonsApp, subclassed it to extend/override methods they needed for customization, or changed how their WSGIController subclass worked to change how individual actions would be called.

Everything seemed just fine…. until…

Improving Pylons

When I had some free time a little while back, I set about looking into how to extend and improve Pylons to make up for where it was lacking, extensibility. I quickly realized that I’d need to change rather drastically how Pylons dispatch worked, and how controller methods were called to make them more easily extendable. But then with a certain feeling of dread, the subclassing issue nipped me. All my implementations of PylonsApp and WSGIController were effectively frozen.

Since every single developer using Pylons sub-classes WSGIController, and to a much lesser extent, PylonsApp, any change to any of the main methods would result in immediate breakage of every single Pylons users app that happened to customize them (the very reason subclassing was used!). This meant that I couldn’t very well change the implementation of the actual classes to fix their design, because that would just cause complete breakage. Ugh!

So after looking into it more, I’ve ended up with this short list of the obvious bad reasons this shouldn’t be done. BTW, in Pylons 2, controllers don’t subclass anything, and customization is all with hooks into the framework, no subclassing in sight!

Short List of Why It’s Bad

From a framework maintainers point of view…

1. Implementations of the classes are effectively frozen, because all the class methods are the API.
2. Correcting design flaws or implementation flaws are much more difficult if not impossible without major breakage due to point #1.
3. Heavily sub-classed/large hierarchy classes can have performance penalties.

From a developer ‘extending’ the classes point of view…

1. Figuring out how to unit-test is more difficult as the full implementation is not in your own code… its in the framework code.
2. When using mix-in’s and other classes that also subclassed the framework, strange conflicts or overrides occur that aren’t at all obvious or easy to debug/troubleshoot.

I think there were a few more reasons I came across as well, but I can’t recall them at the moment. In short, I’m now of the rather firm opinion that the only classes you should ever subclass are your own classes. Preferably in the same package, or nearby.