be Groovie

Notes on the Pylons & repoze.bfg Merger

Some folks might not have time to follow the Pylons-discuss mail list, so this might be news to them, but I’m thrilled to announce that the Pylons and repoze.bfg web frameworks are merging. If this is the first you’ve heard about it, don’t worry, it was only announced a week ago now on the Pylons mail list.

In the time since the announcement, I’ve heard a lot of varying feedback. Some people took a look at Pyramid (the core package that will be equivilant to ‘Pylons 2.0’) and were quick to respond, usually in a knee-jerk type response. I think some of this was due to a miscommunication, and partly because there was so much already done. When other frameworks have merged in other languages, such as Rails merging with Merb, the announcement was just that. There was no code at the time to show, just a promise that when it was ready, it would be awesome.

This merger in contrast already had a starting foundation for a huge chunk of the core features. As a result, people assumed that what we had was already ‘finished’, or close to it. The polish of much of the documentation made it feel odd that there was no “Porting Pylons 1.0 to Pyramid” guide done. In reality, Pyramid is definitely not done, there is still quite a bit of work left before Pyramid will meet the expectations that many Pylons users have. There’s still refinements to be done to Pyramid, and additional packages that Pylons users will most likely always use with it for the feature-set they’re accustomed to.

I’ve summed up a few thoughts on when Pylons users should port to Pyramid to try and help manage expectations better in the future. I’ll make more announcements when packages are ready to ease the transition and a “Porting Guide” is ready.

What is Pylons?

Many Pylons users don’t realize which features they enjoy come from the package ‘pylons’ vs. the other packages that Pylons depends on. Contrary to popular belief the majority of features present in Pylons actually come from other packages. This mistaken belief that most of the features come from the pylons package led some to think that because a lot of my future development time will be spent on adding features/packages around pyramid, Pylons is somehow dead>. This is not the case.

First, Pylons the web framework is mainly a small (~ 1000 LoC) glue layer between Paste, PasteScript, PasteDeploy, WebOb, WebError, Routes, WebHelpers, Mako, and SQLAlchemy. Some people usually end up swapping out Mako/SQLAlchemy but by and large this is the common ‘Pylons Stack’. Most of the new features in Pylons over the past several years actually came from additions to WebHelpers, WebError, or Routes. All of these packages continue to get the same development as they have, so no ‘death’ is occurring.

Second, for over the past 6 months now, there’s been very little in the way of patches submitted, bugs reported, or other feature requests. In many ways Pylons is ‘done’ regarding adding more feature to the core package itself. As I announced on the Pylons-discuss mail list, the Pylons code-base hit some design issues. Adding the features I heard requested from quite a few users (and needed myself) regarding extensibility couldn’t be retro-fitted into the existing design. I encourage anyone curious to read my prior entry on sub-classing for extensibility to be a preview of some future blog posts. I’ll be writing more about design patterns in Python that handle extensibility which many popular Python web frameworks are also struggling to handle.

The Future

I’m very excited about the future for the Pylons Project, which is the new over-arching organization that will be developing Python web framework technologies. The core will be Pyramid, with additional features and functionality building around that. We’re already quickly expanding the developer team with some long-time contributors and having a combined team has definitely helped us progress rapidly.

One of my main goals is to encourage and ease contributions from the community. To that extent I’ve been filling in the contributing section for the Pylons Project as much as possible. I believe this is an area that will quickly set us apart from other projects as we emphasize a higher standard of Python development.

Django did a good job setting the bar high for its documentation of how to contribute to Django, which deserves a lot of credit for clearly defining community policies. Its missing a portion we considered extremely valuable which core developers generally get very picky on when accepting patches… how to test your code. The Pylons Project adapted the rather thorough testing dogma noted by Tres Seaver, which I personally can’t recommend highly enough when it comes to writing unit tests. It’d be nice to see more posts expand on exactly how to test your code. Many developers (including myself) can write code that passes 100% test coverage… but is it brittle test code? Prone to failure if some overly clever macro it uses fail? Seeing a well written set of examples on designing unit tests to avoid common gotcha’s is definitely something anyone contributing (and developers in general) should be familiar with.

For those wanting a gentler introduction to Pyramid (the docs are very verbose and detailed, not at all opinionated), I’ll be blogging more about new features and how to utilize them. Please be patient, I think a lot of people are going to be excited at what’s in store.

Why Extending Through Subclassing (a framework’s classes) is a Bad Idea

Ok, I’ll admit it, overly ambitious blog post. So I’ll refine it a little now, this is intended mainly as my thoughts on why as a tool developer (one who makes tools/frameworks that other programmers then use), its a bad idea to implement extensible objects via developer subclassing. This is actually how the web framework I wrote - Pylons - provides its extensibility to developers and lets them change how the framework functions.

Please excuse the short and possibly incomplete description, this is mainly a quick post to illustrate a tweet I recently made.

First, some background…

One of the things that Pylons 1.0 and prior is missing is a way to easily extend a Pylons project. While it can be done, its very ad-hoc, kludgy, and generally not very well thought-out. Or thought-out at all really. What was somewhat thought-out was how a developer was supposed to extend and customize the framework.

In a Pylons project, the project creates a PylonsApp WSGI object, and all the projects controllers subclass WSGIController. This seemed to work quite well, and indeed many users happily imported PylonsApp, subclassed it to extend/override methods they needed for customization, or changed how their WSGIController subclass worked to change how individual actions would be called.

Everything seemed just fine…. until…

Improving Pylons

When I had some free time a little while back, I set about looking into how to extend and improve Pylons to make up for where it was lacking, extensibility. I quickly realized that I’d need to change rather drastically how Pylons dispatch worked, and how controller methods were called to make them more easily extendable. But then with a certain feeling of dread, the subclassing issue nipped me. All my implementations of PylonsApp and WSGIController were effectively frozen.

Since every single developer using Pylons sub-classes WSGIController, and to a much lesser extent, PylonsApp, any change to any of the main methods would result in immediate breakage of every single Pylons users app that happened to customize them (the very reason subclassing was used!). This meant that I couldn’t very well change the implementation of the actual classes to fix their design, because that would just cause complete breakage. Ugh!

So after looking into it more, I’ve ended up with this short list of the obvious bad reasons this shouldn’t be done. BTW, in Pylons 2, controllers don’t subclass anything, and customization is all with hooks into the framework, no subclassing in sight!

Short List of Why It’s Bad

From a framework maintainers point of view…

1. Implementations of the classes are effectively frozen, because all the class methods are the API.
2. Correcting design flaws or implementation flaws are much more difficult if not impossible without major breakage due to point #1.
3. Heavily sub-classed/large hierarchy classes can have performance penalties.

From a developer ‘extending’ the classes point of view…

1. Figuring out how to unit-test is more difficult as the full implementation is not in your own code… its in the framework code.
2. When using mix-in’s and other classes that also subclassed the framework, strange conflicts or overrides occur that aren’t at all obvious or easy to debug/troubleshoot.

I think there were a few more reasons I came across as well, but I can’t recall them at the moment. In short, I’m now of the rather firm opinion that the only classes you should ever subclass are your own classes. Preferably in the same package, or nearby.

Pylons 0.10 and 1.0 Beta 1 Released

Without further ado,

I’m pleased to announced that Pylons 0.10b1 and 1.0b1 are now out. I have not put them on Cheeseshop to ensure they’re not downloaded accidentally.

Upgrading / Installing

I have updated upgrading instructions here: http://pylonshq.com/docs/en/1.0/upgrading/

The instructions to install from scratch on Pylons 1.0b1: http://pylonshq.com/docs/en/1.0/gettingstarted/#installing

The upgrading page covers the important upgrading instructions that Mike Orr touched briefly on before.

Note that these are beta releases, intended for us to discover remaining issues and continue updating any other documentation where applicable. Very little has actually changed in Pylons since 0.9.7, apart from 1.0 dropping all of the legacy functionality and a few explicit clean-ups.

Updates

Routes, Beaker, and WebHelpers however have been seeing quite a bit of updates through the life of Pylons 0.9.7 so no one should think that the developers working on Pylons and its related parts have been hanging out doing nothing. :)

Since Pylons 0.9.7 was released on February 23, 2009, almost one year ago now:

• Routes 1.11 was released, and 1.12 with some great updates will be out shortly
• Beaker has gone from 1.2.2 -> 1.5 with 3 major updates substantially increasing its ease of use and reliability
• WebHelpers is now at 1.0b4 with major updates, core functions rewritten, and new docs up
• SQLAlchemy has gone from 0.4 to 0.5 (with 0.6 in beta)

I believe this speaks a great deal about the benefits of keeping the core Pylons functionality separate from other parts, as a variety of bug fixes and features can be improved without requiring new Pylons releases to quickly address bug reports.

How to Help!

To bring Pylons to 1.0, many docs likely need very small changes. Also, it would be great to take care of reference docs where people have commented about problems/tips. Helping is fairly easy, especially if you’re familiar with restructured text.

First: Clone the Pylons repository on Bitbucket: http://bitbucket.org/bbangert/pylons/

Then: Edit the documentation files under pylons/docs/en/ to read as appropriate, commit the fix, and push it to bitbucket.

Finally: Issue a pull request on bitbucket so that we’ll know your fix is ready. Ideally you should include a note in it about what your fix remedies.

Bug Reports

Did your upgrade not go according to plan? Was there something missing that you needed to do from the upgrading docs?

Let us know by filing a bug report (mark component as documentation, and milestone as 0.10: http://pylonshq.com/project/pylonshq/newticket

You’ll need to login to file a bug report, or feel free to reply to this announcement with the issue.

Thanks (in alphabetical order) to Mike Bayer, Ian Bicking, Mike Burrows, Graham Higgins, Phil Jenvey, Mike Orr, and anyone else I missed for all their hard work on making Pylons and its various components what they are today.

Deploying Python web apps with toppcloud

Ian Bicking recently released a rather interesting package called toppcloud that aims to tackle what I see as a growing need for those of us deploying Python webapps.

I was actually interested in easing my own deployment woes before I saw Ian’s announcement about his package, and was halfway through a rather hefty amount of research on automating server deployments with tools like Chef and Puppet, but toppcloud is a bit different. It not only tackles provisioning a new system on the fly from the ‘cloud’ (using libcloud), but it also handles easy Python (and now PHP) web application deployments.

With such a tantalizing set of goals, I couldn’t really resist getting my feet wet. Boy am I glad I did. PylonsHQ is now running with toppcloud, and I won’t be surprised when more people get it running for them. When many shared hosting providers are $5-10 a month, its rather nice to pay $10/mth for an automatically configured VPS, with one-command deployments. Though of course, unless you go to quite a bit of work yourself, most shared hosting providers don’t have one-command deployments for you.

Before I continue on to describe how I setup Kai - the source code behind PylonsHQ - I should provide a few caveats about using toppcloud:

• toppcloud is alpha software, there’s no releases of it yet, you will be checking it out from source code
• currently, only Rackspace Cloud is known to be working, though since it uses libcloud, in theory, any cloud providers that it supports should be usable
• toppcloud is changing rapidly, be ready to keep up on the commit log to see whats changing
• there are no unit tests, most likely because its very tedious to make the rather significant amount of mock objects required to test the various local/remote commands and the fact that its changing so fast the tests would probably be obsolete in a week

toppcloud philosophy

I’m half guessing here, based on talking with Ian and reading the docs myself, but the philosophy of toppcloud is around providing a common deployment platform, ala Google App Engine, except of course with Postgres or other ‘services’ that you can request to use. At the moment the only services that toppcloud comes with is CouchDB, Files (to store/serve files for your app on the filesystem), and Postgres w/postgis extensions. For those diving into the source code, it shouldn’t be too hard to see how to create additional services and I hope to see more get added as people get more interested.

Therefore, toppcloud is not expected to be everything to everyone, it is expected to be at least an 80+% solution to deploying web apps in a Google App Engine style ease-of-deployment process. toppcloud itself then ensures that depending on what service you asked for, its setup and ready for use on the server when your app is deployed.

If that sounds like something you’re dying to try out, you’re in luck!

Setting up a Pylons App

I’m not going to mention how to check out the toppcloud source, except to mention the directions are in the

/docs/index.txt

file in the toppcloud source (which should be read in its entirety!). Once you have toppcloud installed on your computer, getting from zero to running website on VPS is remarkably quick:

1. $ toppcloud create-node --image-id=14362 baseimage

   Wait until email arrived indicating that server is up and ready

2. $ toppcloud setup-node baseimage

   Server is now all setup to run web apps!

3. $ toppcloud init myapp

   Create an app to deploy to our server

4. $ source myapp/bin/activate

   Activate the virtualenv used for this app

5. $ pip install Pylons

6. $ cd myapp/src

7. $ paster create -t pylons awesomeapp

   Create our Pylons app, or check out an existing one from your VCS here

8. $ cd ../..

9. $ ln -s myapp/src/awesomeapp/awesomeapp/public/ myapp/static

   toppcloud will make available things in the static directory available without hitting the webapp
10. Configure myapp/app.ini similarly to (taken from pylonshq site):

    		[production]
    		app_name = pylonshq
    		runner = src/kai/production.ini
    		version = 1
    		update_fetch = /sync_app
    		service.couchdb =
    		service.files =
    		default_host = pylonshq.com
    	

    In this case, the Pylons site needs CouchDB setup, and the files service (all Pylons sites should use the files service to store cached files, templates, etc.)

11. $ toppcloud update myapp/

    Note that we’re one directory above myapp, and we have a trailing slash on myapp, this is needed because an rsync is done to copy it to the server, don’t leave off the trailing slash! (This will prolly be fixed at some point)

That’s it! 10 easy steps to go from zero to a running deployed website.

I should also note that you will need to make a production.ini file, and that it should have a few important changes in it. All the references to %(here)s should be changed to %(CONFIG_FILES)s since that’s the persistent location that files can be stored for a toppcloud app between deployments. Other configuration information provided by services (Couch supplies the db/host, Postgres has its host/user/pass info) can be accessed via CONFIG_ vars as well. The services docs have some more info.

More apps can be added to a host as desired, until the ram runs out of course. At the moment toppcloud is using a process pool of 5 with 1 thread under mod-wsgi for each application. This can use a bit of ram if you have multiple heavy Pylons processes, hopefully there will shortly be a way to ask toppcloud to use a single process with multiple threads which will help cut the ram profile a bit.

Using Django and PHP

Unfortunately I haven’t actually tried this myself, but there’s nothing preventing it. You’ll need to change the app.ini so that instead of using ‘src/kai/production.ini’ as the runner, it uses a Python file in the directory, say main.py that then loads the Django app as a WSGI application and returns it. Sort of like this. Note that the config vars Django needs for its database should then be present in os.environ when setting up the settings.py that Django uses.

If you’re looking through the toppcloud source code by now, you may have also noticed there’s an example app that uses PHP. There’s nothing holding back toppcloud from setting up mod_passenger and deploying Ruby apps at some point either should someone wish to add that feature.

dumb pipes

There’s been numerous mentions on various blogs and in the news about how much the cell phone carriers hate the concept of being nothing more than “dumb pipes” for wireless Internet and phone use. That means of course, that they’d no longer be competing on what phones you could use but instead solely on service quality and price… I think the same type of transition is in store for shared hosting providers and some of the boutique app deployment shops like heroku.

It’ll take awhile, toppcloud is very rough right now. But when you can d/l a nice little open-source package, run it, choose your choice of cloud provider (based on price + quality!), have it automatically setup for you to deploy your apps to, then deploy apps with a single command… you’ve already done the vast majority of what a service like heroku does, except you could still modify toppcloud if there was something lacking you really needed. And it’s a reason like that, that toppcloud exists to begin with.

Oh, and thanks Ian for writing this before I wasted more time making it myself.

The Myghty Python Web Framework

Myghty is a powerful web application framework that builds on the strengths of Perl’s Mason using my favorite language, Python. While Myghty also handles page templating, it has some very powerful features that make it a full fledged framework in my opinion.

Wondering if Myghty is something you should check out?

I would highly recommend looking into Myghty if you’re looking for any of the following:

• Clean, unobtrusive templating language for integrating Python data and HTML
• An easy way to write pages that inherit their look/feel without needing .include statements
• Re-usable code that you can carry from one project to the next
• MVC programming paradigm , or page-driven paradigm (Myghty doesn’t care, it can be anywhere in between as well)
• Advanced caching ability, tunable to just the time intensive sections of a page
• Run as WSGI, FCGI, SCGI, mod_python, or stand-alone with hardly a single change to your code (to switch between them)
• Architecture that scales, from a single developer’s personal project, to a multi-developer’s company website

History

Myghty first came into existence a little over a year ago when Mike Bayer looked around at the Python web application frameworks and couldn’t find one that really sucked him in. Having used Mason for various projects, but not finding anything as elegant as it for Python he decided to port Mason to Python.

This ported code was based on the Perl code almost line by line, resulting in a Python web application framework that worked almost exactly like Mason, except with Python as the base language. It wasn’t very long before Mike started adding to Myghty, such as Module Components, and threaded features not found in Mason.

Terms Used

MVC – Model, View, Controller paradigm is typically used to describe an approach that separates the code in these 3 steps rather than mingle them all in one place. Model usually refers to the database and its abstraction, the Controller handles the logic of dealing with the request, getting the data from the Model, and passing it to the View for display to the user.

Component – A component in Mason is what a template is known as. A template might display a full web-page, just one small section, or send data to other components. Components typically correspond to files on a filesystem, and appear as normal HTML pages with a few lines of Python in them, although they’re far from normal HTML pages…

Resolver – Used to describe the process of determining what to do with a given URL. Myghty has some incredible flexibility here.

Distinctive Features

Module Components

In Python, your modules contain all your Python code. In Myghty, the Component contains a mixture of Python with HTML. To handle a MVC approach to development, Myghty has Module Components. These are a cross of a Myghty Component with a Python Module, that leaves you with a fairly normal Python Module. It’s called directly from Myghty in a similar way to how Myghty would call a Component. These Module Components can be explicitly defined in the configuration, or called implicitly.

The Module Component allows you to directly handle URL’s, and respond from within the Module by sending data back to the client or processing data then sending it on to a normal Component for processing. Module Components work within the Resolving process, and have control passed to them according to a configuration mapping . The mapping uses regular expressions to match URL’s to modules that contain the method to be called, like so:

module_components = [
    {r'myapplication/home/.*' : 'myapp.home:HomeHandler'}, 
    {r'myapplication/login/.*' : 'myapp.login:LoginHandler'}, 
    {r'.*/cart' : 'myapp.cart:process_cart'}, 
    {r'.*' : 'myapp.home:HomeHandler'} 
]

This would map the URL http://yourapp/myapplication/home/blah to the HomeHandler component class, in the home module file, under the myapp package. The method in the HomeHandler class called can be done in two different ways, implicitly, or explicitly. For more details on this, read the Myghty docs on Module Components

While other frameworks implement an MVC approach, Myghty’s goes a step farther by letting you tie it into the resolution options used by the Resolver, and even use your own resolvers…

Advanced Resolver

In version 0.97alpha2 (maybe a better naming scheme could be used? :) the resolver implementation was heavily rewritten to allow massive flexibility in how a URL is handled. When Myghty handles a request, it goes through a fairly complex process to determine what to do with it. This is called the default strategy:

1. Translate the URL against the path translation options – This can act as an internal URL rewriting scheme, changing a URL to be handled differently
2. Resolve a dhandler – This rule matches conditionally. It is only called if the rules under it don’t find a match, it then strips off part of the URL its looking for and adds dhandler. So a search for /article/view/38 becomes a search for /article/view/dhandler if 38 isn’t found. Read more about dhandlers
3. Check the URI Cache – The URI cache is searched for the incoming URI at this point, to see if a component has matched it before. This rule is also executed on a conditional basis.
4. Upwards Search – The 3rd conditional rule of the bunch, this rule is called during inheritance when a component searches parent paths for an autohandler. Read more about autohandlers
5. Resolve a Module – Matches a Module Component to the incoming URI.
6. Resolve a File – Matches a file under the component roots specified for a match to the incoming URI. This will directly call a Myghty template file on the system in response to a URL.

As soon as any of the non-conditional rules matches, the file or module component method is served up. Myghty gets an additional speed boost by keeping compiled versions of all your template files around, so that they’re only compiled the first time they’re called. By using the basic Python way of compiling the templates as .pyc files, it also ensures that they’re re-compiled the second you update one of them. This way you’re always sure you’re viewing the latest update of your template.

So where’s the real power in all this?

These are just the default rules. They can be completely, and utterly customized. You can switch the order around, you can remove resolver rules entirely, and you can even write your very own resolver rules.

Maybe you’ve decided you want to serve some of the components from a database instead of the file system. You could write a custom resolver rule that does just that, and loads the entire component from the database as well. You could write your own version of the Path Translate module to get its rewrite rules from a database table. As if this wasn’t powerful enough, Myghty also lets you have different resolver strategies for different contexts.

Conclusion

Myghty is a very powerful web framework. It has features that go beyond any pure “templating” language I’ve encountered. If you’re not quite happy with what you’re using right now, or something I described above makes you giddy with thoughts of power, maybe its time to take Myghty for a test drive.

In future posts, I’ll describe more of the features I use daily in Myghty, and share code tidbits that let you get complex tasks done easily.

XML-RPC interface for Growl

Growl is a rather spiffy little notifier for OS X I’ve started using lately. Currently however, it’s lacking a nice way to send messages over the network. Since a project I’m working on would be even slicker with a way to communicate updates, I quickly tacked a Python XML-RPC Server onto Growl using Growl’s Python bindings.

In the future, Growl will have its own network interface, I’ve heard. Until then, this does the job quite nicely.

Please note: You will need to install the latest svn of Growl, the PyObjC bridge, and have Xcode 1.5 to get this working. Everything in ALL_CAPS should be changed as needed, this also assumes you want to use icons and they’re in a myIcons dir.

#!/usr/bin/python

	
import Growl
import SimpleXMLRPCServer
from AppKit import NSImage
import sys, os.path

	
iconPath = 'myIcons/’
Icons = { 'MESSAGE_1’ : 'MESSAGE_1.tiff’, \
          'MESSAGE_2’ : 'MESSAGE_2.tiff’ }

	
class MemberFunctions:
    def init(self,Icons):
        gn = Growl.GrowlNotifier()
        gn.applicationName = 'YOUR_APP_NAME’
        gn.applicationIcon = Icons['appIcon’]
        gn.notifications = ['MESSAGE_1’,'MESSAGE_2’]
        gn.register()
        self.gn = gn
        self.Icons = Icons

    def notify(self, ntype, title, description, sticky):
        gn = self.gn
        Icons = self.Icons
        nIcon = Icons['appIcon’]
        if ntype  'MESSAGE_1':
            nIcon = Icons['MESSAGE_1']
        elif ntype  'MESSAGE_2’:
            nIcon = Icons['MESSAGE_2’]
        gn.notify(noteType=ntype,title=title,description=description,
            icon=nIcon, sticky=sticky)
        return 1

	
def loadIcons(icons):
    global iconPath
    for icon in icons.keys():
        filename = icons[icon]
        nImagePath = os.path.abspath(iconPath+filename)
        nIcon = NSImage.alloc().initWithContentsOfFile_(nImagePath).autorelease()
        if not nIcon:
            sys.stderr.write(“Couldn’t load file %s\n” % nImagePath)
        icons[icon] = nIcon
    return icons

	
Icons = loadIcons(Icons)

	
Make our XMLRPC Server, create our instance, and assign it
	

server = SimpleXMLRPCServer.SimpleXMLRPCServer((“IP_TO_BIND_TO”, 7300))
memfun = MemberFunctions(Icons)
server.register_function(memfun.notify)

	
	
Start the server
	

server.serve_forever()

If you’d like to see a running example of this, check out hellagrowler.py that this example is based heavily on.