be Groovie

Routes planning and the road to Routes 2.0

Routes has been a wonderfully successful project of mine as its used not just in Pylons but quite a few other small WSGI apps. It’s even been integrated as a CherryPy URL dispatching option for those using TurboGears 1.x or plain CherryPy. It’s come a long way since 1.0 and has diverged on quite a few fronts from the Rails version that it was originally duplicating in functionality, which has me thinking that perhaps its time to consider 2.0 and what that will mean.

First, I think there’s quite a few behaviors that don’t make sense and probably never have. The Route minimizing functionality sounds neat and fulfills the Rails ideal of ‘keeping urls pretty’ yet suffers from a fundamental flaw… they result in multiple valid URL’s for the same page. James Tauber covers some of the implications of this in addition to the issue with relative URL’s not working right either. The example he cites is even worse for Routes since Routes can easily result in multiple URL’s mapping to the same controller action.

Routes 2.0

To solve the multiple URL issue, and also address some Named Routes confusion that recently hit the Pylons mailing list, Routes 2.0 will do a few things differently than Routes 1.X. Many things will be significantly more explicit and more predictable as a result. In addition, I want to add more functionality so that Routes can be the end-all of URL generation in addition to URL dispatching, even for use purely generating links conditionally that aren’t even used for URL dispatching.

Some of the key changes planned for Routes 2.0:

• Routes recognition and generation will always be explicit

Currently, there’s an explicit option that removes the keyword controller/action/id defaults, 2.0 will not have these implicit defaults.

• Defaults don’t cause minimization

Defaults just mean they’ll be used, they won’t result in Route minimization which increases the amount of URL’s that end up matching to a single controller action.

• Trailing slashes shouldn’t be an issue

Without routes being minimized, a route such as ‘home/index’ will always be ‘home/index’ instead of being minimized to /home/. This will resolve the trailing slash issue.

• Named Routes will always use the route named

A named route currently just means that the defaults for that route will be used during route generation. This currently can cause confusion because people believe that the named route actually forces that route to be used during generation.

• Generation-only routes

A new option, which will result in routes that are used purely for generation. This option will likely be used primarily for static resources which may be on other servers, or may need the domain rotated so that the browser can do parallel resource loading. For example, one would be able to provide a list of domains to be used, and the generated links will rotate as desired on the page to split page resources over multiple domains.

• Redirect routes

Sometimes, (especially when replacing legacy apps), its desirable to slowly migrate URL scheme’s from the old to the new rather. While URL’s should never change, sometimes the system being replaced has horrid URL’s that violate all URL recommendations. Being able to provide a smooth migration path from the old URL’s to the new ones is handy, and permanent redirects are respected by many search crawlers as well.

Migration and Compatibility

Routes 1.8 will include options to turn on behavior that will be the default in Routes 2.0, and if you like how Routes 1.X works there’s no need to worry, it will still be maintained for the foreseeable future. It’s currently extremely stable, and has a massive unit test suite to ensure it operates as designed.

Add Your Desired Feature Here

What other features are Routes users currently looking for?

Routes functional conditions and WSGI Middleware

Sometimes, it amazes me whats possible fully utilizing WSGI middleware in an application stack. While it likely isn’t something totally unique to the framework, the relative ease with which it can be done still sometimes gets me to grin.

Tonight, a Pylons user on the IRC channel (irc.freenode.net, #pylons) asked if it was possible to get a URL laid out so that /s/SOMETHING would map into their ‘s’ controller, with the second part passed in as a variable. That alone is pretty easy, however the additional requirement was that the controller action would change depending on the user’s “type”.

There’s two ways to deal with this, the first of which is the only possible way in many frameworks. Have every request to the URL map to a single function, and in that function load up the session and call the appropriate function to handle the request based on their user type. This way works fine in Pylons too, but thanks to Routes and WSGI middleware we have another option.

Routes has a lot of capabilities to it, there’s been numerous additions to the Python implementation that the Rails version is not capable of. One of them is the ability to alter the resulting URL match dictionary in various conditional functions. To toggle the controller action used, we’ll be using the ability to pass in a function to Routes conditions that can alter the resulting match.

This condition checking function has full access to the WSGI environ so if you wanted to restrict a specific controller/action combination to people referred from Slashdot, no problem! You can carefully fine-tune the conditions required for dispatch at the same place you define your URL resolution.

Since Pylons uses Beaker for session handling via WSGI middleware, the session object will already be available when our Pylons app gets called. Beaker loads the user session into environ['beaker.session']. Given this knowledge, we can write a conditional function for use with Routes like so:

def check_user(environ, result):
    session = environ['beaker.session']
    user_type = session.get('type')
    if not user_type:
        result['action'] = 'index'
    elif user_type == 'admin':
        result['action'] = 'view_action'
    else:
        result['action'] = 'not_logged_in'
    return True
map.connect('s/:domain', controller='s', conditions=dict(function=check_user))

Viola! Now Routes will run the function provided to see if it returns True before accepting that as a valid match. In the process, the action used will be set as desired. I’ve always thought a good sign something is well designed is when people can use it in ways you didn’t originally anticipate. If that’s the criteria, I think Routes succeeds and then some.

Disclaimer: Yes, I wrote Routes, and a good chunk of Beaker and Pylons, so I might be biased and tooting my own horn. :)

Routes 1.4 Release and Web Services

This is slightly old as Routes 1.4 was released about a week and a half ago, but I thought it deserved some attention. There were a handful of fixes and some slightly major feature enhancements in 1.4.

From the changelog:

• Fixed bug with map.resource related to member methods, found in Rails version.
• Fixed bug with map.resource member methods not requiring a member id.
• Fixed bug related to handling keyword argument controller.
• Added map.resource command which can automatically generate a batch of routes

intended to be used in a REST-ful manner by a web framework.

• Added URL generation handling for a ‘method’ argument. If ‘method’ is specified, it

is not dropped and will be changed to ‘_method’ for use by the framework.

• Added conditions option to map.connect. Accepts a dict with optional keyword args

‘method’ or ‘function’. Method is a list of HTTP methods that are valid for the route. Function is a function that will be called with environ, matchdict where matchdict is the dict created by the URL match.

• Fixed redirect_to function for using absolute URL’s. redirect_to now passes all

args to url_for, then passes the resulting URL to the redirect function. Reported by climbus.

Web Resources

The map.resource command is based directly off the Simply Restful Rails plugin which adds support for various verb-oriented controller actions in a RESTful service style approach. The Simply Restful layout is more or less the exact service style laid out in the Atom Publishing Protocol.

It’s a great approach and it also meant providing a few other features to Routes that I hadn’t implemented previously, the most important being able to limit matching of a URL based on the HTTP method used. This is present in the new conditions clause for a Route:

map.connect('user/:id', controller='user', action='edit', 
    conditions={'method', ['GET', 'HEAD']})
map.connect('user/:id', controller='user', action='update',
    conditions={'method', ['PUT']})

The conditions clause can also accept your own function should you want to restrict the route to matching based off some other criteria (sub-domain, IP address, etc).

def stop_comcast(environ, match):
    if 'comcast.net’ in environ['REMOTE_HOST’]:
        return False
    return True

	
map.connect(’:controller/:action/:id’, conditions={'function’:stop_comcast})

David Heinemeier Hansson recently posted an entry about Resources on Rails discussing how important web services are. The other key point was to make it easier to write controllers that could not only give you easy browser access to your resources, but provide a web service API as well.

The two snippets shown above give you an edit and update capability that restricts matching based off the HTTP method (verb). Writing a huge mess of those for the rest of the functions needed for a full web service API like Atom is a bit of busy-body work, so in the opinionated style of Rails a single command wraps up the whole thing. In Routes it looks like this:

map.resource('user')

That will make the two routes at the top of this entry in addition to routes that handle PUT, and DELETE. It maps them out to a set of actions in the controller, and provides the capability to easily add more methods for specific verbs.

The map.resource command is still getting tuned up, and we’re integrating the additional functionality it provides back into Pylons as well. Josh Triplett also wrote some Python code that will parse HTTP Accept headers fully so that we can add some nice functionality to use in the controller to return the appropriate data given what the client is expecting (HTML, XML, JSON, etc.)

If you’re using a Python web framework that doesn’t use Routes… maybe its time to put a request in. :)

Pylons Related News for the 4th of July

It’s been a busy month for Pylons, with lots of changes for the big internal API change in 0.9. The great news for those making Pylons apps right now with 0.8.2 is how few backwards compatibility issues there are. Most of the big changes take place under the hood and compatibility objects are present to mitigate the massive breakage that I’ve seen happen in other framework upgrades.

This is going to remain a big focus on future development in Pylons too, and the API in 0.9 is solid enough that I don’t see anything but minor tweaks in the future (1.0 and beyond). Pylons based apps will be easy to maintain and upgrade thanks primarily to WSGI.

No NIH Syndrome Here!

We’ve taken some cues from the Django project that we believe make for a cleaner request-cycle. In Pylons 0.9, controllers are expected to return a response object, and for convenience a method is included that renders a template to a new response object and returns it (This will look very familiar to Djangonauts).

The command was integrated with a slightly more enhanced version of Buffet along with the Beaker Session/Caching middleware. The end-result is a powerful command that not only can render templates in any Template-Plugin compatible template language, but the rendered result can also be cached. It’s a great way to utilize template languages which might not be all that quick by themselves.

Sample controller in Pylons 0.9:

from myproj.lib.base import *

	
class UserController(BaseController):
    def show(self, id):
        # Cache based on id in the URL, for 30 seconds
        return render_response(’/user/show.myt’, cache_key=id, cache_expire=30)

    def index(self):
        # Just for fun, use Kid to render the index
        return render_response('kid’, 'user.index’, cache_expire=15)

Being able to easily cache any template in any template language makes it very easy to sprinkle in caching when you need to handle massive loads yet stay dynamic.

Another new feature present in the latest Routes and Pylons is resource mappings, which automatically generate routes for you with HTTP method restrictions. This makes it easy to setup controllers and their actions for specific HTTP verbs (aka, REST-ful URL’s and web services). The implementation of this feature was directly inspired by the Simply Restful Rails plug-in that was also demo’d by David Hansson in his Resources on Rails talk. Being able to discriminate valid routes based on HTTP method was brought up in the past and I’m happy to have seen an implementation that solves the issues I originally had with the idea.

The one feature still in development is to easily discern the content type requested, which was also inspired by Rails. Josh Triplett has written some code that deals with the ugly task of parsing the HTTP Accept headers, and I’m working on adding it into Paste for easy re-use by all. Combined with Routes, it’ll provide a clean and easy way to setup web applications that can serve multiple forms of content from a single action.

Conferences

Pylons was represented at EuroPython by the other lead developer of Pylons, James Gardner. For those still trying to grasp WSGI, he gave a talk on WSGI and middleware that looks like it was quite interesting (I wasn’t there unfortunately.)

Other talks involving Pylons at EuroPython:

• Dr. Rob Collins gave a presentation on Factory monitoring with Pylons, XML-RPC and SVG
• James Gardner’s full talk on Pylons, and slides of his Pylons talk
• Web Framework Shoot-out between Django, Zope, TurboGears, and Pylons (If more were covered, please let me know)

I’ll be at OSCON for those interested in chatting about Pylons, Python, or Python web frameworks later this month.

Pylons 0.9 Release

We’re currently wrapping up new features in 0.9, to make sure the resource mapping capability is present before a feature-freeze for release. Hopefully the release will be out within the next 2 weeks, if you haven’t checked out Pylons before I’d highly suggest taking a look at 0.9!

Routes 1.3.1

Ah yes, time for another release. This time its Routes 1.3.1, nothing too big to see here, thus the minor version increase. However if you were using Routes with the prefix option its highly recommended that you upgrade as there were 2 bugs specifically when used with that option that have been fixed.

Also, for those that want all their URL’s to end in a /, there’s now a append_slash option that will ensure URL’s are generated exactly how you want them to be.

Enjoy!

Routes 1.2 Released

I got Routes 1.2 out the door today, it’s a fairly important update for 2 reasons:

• A bug crept in with 1.1 using the default controller directory scanner. The scanner wasn’t properly retaining the directory prefix which causes mismatches when using controllers underneath a sub-directory.
• url_for can (and should) be used for all your URL needs, including static files

The second one is pretty important if you’re at all interested in creating portable web applications that can be used along-side other applications. While some frameworks provide generation of URL’s that lead to other web pages, hard-linking the other ‘static’ content will cause problems if you try and use the application under a different mount point.

Instead of using a url like /css/source.css it should instead be generated with url_for('/css/source.css'). This way Routes can ensure that if you’re running under a WSGI environment and there’s a SCRIPT_NAME present (this indicates the applications location), it will be pre-pended to your absolute URL’s. When used like this, additional keyword arguments passed in will be used as query args on the URL making url_for a handy way to create URL’s that are properly URL encoded.

Another useful feature that made it into 1.2 allows you to alias URL’s you might want to use throughout your web application, I call these static named routes. An example can be found in the named routes section of the Routes Manual.

Enjoy!

Routes 1.1 Released

I’ve released Routes 1.1 today after extending the unit tests for the new Groupings syntax and updating the docs.

The new syntax is quite powerful and will hopefully make everyone using Routes rather happy. While I’m not about to encourage anyone to use URL’s with .html at the end, there’s plenty of times when you want extensions to mingle with dynamic parts. You can also get some useful abilities like being able to pull out the extension like so:

map.connect('archives/:category/:(section).:(format)', controller='archive', action='by_extension')

This makes it easy to toggle the response depending on the extension, and the regexp business is handled for you.

Integration Enhancements

An additional feature, suggested by a Routes user was to make integration easier in WSGI environments. Earlier, at the beginning of each request you would have to populate the Routes config with the host, protocol, and match result. Now, merely passing the WSGI environ to the Routes config object will run a match, and populate those attributes for you.

The Routes Mapper now can take a function that when called returns a list of valid controllers. If you want to use the directory scanner Routes comes with, all you need to do is pass in the directory you’d like to scan and Routes will scan it for you.

These two new integration improvements make it rather simple to integrate Routes, here’s a basic WSGI app showing this off:

#myapp.py

	
from routes import *

	
map = Mapper(directory=’/my/directory/of/controllers’)
map.connect(’:controller/:action/:id’)
map.connect('home’, '’, controller='home’, action='splash’)

	
class WSGIApp(object):
    def init(self, mapper=map):
        self.mapper = mapper

    def call(self, environ, start_response):
        config = request_config()
        config.mapper = self.mapper
        config.environ = environ

        if not config.mapper_dict:
            start_response(“404 Not Found”, [(“content-type”,“text/html”)])
            return [“No match”]
        else:
            start_response(“200 OK”, [(“content-type”,“text/html”)])
            return [“Match with the following dict: %s” % str(config.mapper_dict)]

That’s it! If you’re not using WSGI, there’s been no backwards breakage so the old style of setting up all the attributes of the config will work fine as well.

So, now I have to figure out if there’s anything else Routes should possibly have… or is the only space for improvement at this point further optimization and perhaps usability improvements?

Routes 1.1 Preview

The major feature I was waiting on for Routes 1.1 is for the most part done, mainly adding more unit tests for the new syntax now. As I mentioned previously when announcing Routes 1.0, this feature is the one quite a few people have been waiting for. The ability to split the route path somewhere other than on a /.

Here’s what a few Routes using the new feature look like:

map.connect(’:category/:(page).html’, controller='stories’, action='view’)

	
map.connect('feeds/:(section).:(extension)’, controller='feeds’, action='formatter’)

	
map.connect('archives/:(year):(month):(day).html’, controller='archives’, action='view’,
          requirements=dict(year=r’\d{4}’,month=r’\d{2}’,day=r’\d{2}’))

The new section dividers, :(something) can be used side-by-side as the last example above shows, however in such cases each path part must have a rigid regexp requirement placed on it to ensure proper collection of the variables. Typically you will have some characters in between each dynamic part so this issue doesn’t arise.

I’ve retained full backward compatibility with the old syntax as well, if you don’t designate the dynamic part using the () modifier it will fall back to looking for the next / instead. So far, all the existing (and extensive) unit tests are passing, in addition to new tests I’ve been adding.

Routes generates a full regular expression for URL matching based off the route path you give it. This makes it a great way to setup URL routing with the power of regexp’s yet avoiding the hassle of writing a large complex regexp yourself. The other very powerful ability you gain is that Routes can turn the keywords back into the URL, like so:

>>> url_for(controller='archives’, action='view’, year=2004, month=10, day=12)
‘archives/20041012.html’

	
>>> url_for(controller='feeds’,action='formatter’,section='computers’,extension='xml’)
‘feeds/computers.xml’

If you’re interested in giving it a spin, it can be checked out and installed easily from the svn repository using setuptools:

sudo easy_install -U http://routes.groovie.org/svn/branches/newsplit

Feedback / Suggestions / Bug Reports greatly appreciated.

Routes 1.0 Released

I’ve finally finished the documentation for Routes and as I mentioned earlier regarded releases am now ready for a 1.0 release. If you’re curious about Routes and want to get up to speed, I’d suggest jumping straight to the Routes Manual.

Routes is currently used in Myghty with the routes Paste template, and has been integrated for use both in Aquarium as well as CherryPy (though CherryPy 2.2 should allow better integration).

Now that Routes is feature-equivilant to the Rails version, its time to start planning for new stuff. The first and most obvious is to allow for more advanced configurability of URL’s by allowing for a new separator. This would allow you to get as creative as you like with URL’s, so you could do something like this:

m.connect('archives/:(article)-:(page).html', controller='blog', action='view')
m.connect('feeds/:section/:(format).xml', controller='feeds', 
          action='xml', format='atom')

This should make for a nice 1.1 feature. For those familiar with the Rails system of Routes, has there been anything you’ve found lacking or were just itching for?

Routes 0.2 released

Routes 0.2 has been released. The change-log is pitifully short:

• Added prefix option
• Fixed Python 2.3 bug with thread-local singleton

But hey, its a small package to begin with so what the heck. Though its only 0.2 I’m rather pleased with it so far, its performance is great and quite reliable so I’m using it in production environments already.

If you haven’t been following my blog long, Routes is a feature-complete implementation of Rails routes system. I talk more about my reasons for re-implementing Routes in Python in an earlier post so I won’t repeat them all here.

It’s fairly unique in the Python world as it will do a route lookup search to turn a dictionary back into the URL (URL Generation) that will ensure the same values are created. This allows you to generate URL’s from inside your web pages and easily add new URL schemes without touching all your web pages.

The Routes package is aimed directly at integration with Python web frameworks that support the MVC style paradigm as it returns a controller and action value with the assumption your framework will know what to do with it.

Tired of writing big regexp’s to match URL’s to a class/method for dispatch in your webapp? Pester the framework creator to integrate Routes. :)

Here’s some Python Web Frameworks that currently or will shortly have Routes support/integration:

• Aquarium has a URL Connector that uses Routes. Not sure if its been added to the core or only exists as an add-on right now.
• Myghty will have Routes integration packaged with it in 0.99 which is on track to be released this week hopefully. It will also bring Python Paste support and integration which will bring a whole bunch of goodies to webapp developers and users. (Note that Python Paste requires Python 2.4)

I’ll be talking more about Python Paste and why you should care later this week. Any comments/suggestions on Routes are greatly appreciated.