be Groovie

Pylons 0.10 and 1.0 Beta 1 Released

Without further ado,

I’m pleased to announced that Pylons 0.10b1 and 1.0b1 are now out. I have not put them on Cheeseshop to ensure they’re not downloaded accidentally.

Upgrading / Installing

I have updated upgrading instructions here: http://pylonshq.com/docs/en/1.0/upgrading/

The instructions to install from scratch on Pylons 1.0b1: http://pylonshq.com/docs/en/1.0/gettingstarted/#installing

The upgrading page covers the important upgrading instructions that Mike Orr touched briefly on before.

Note that these are beta releases, intended for us to discover remaining issues and continue updating any other documentation where applicable. Very little has actually changed in Pylons since 0.9.7, apart from 1.0 dropping all of the legacy functionality and a few explicit clean-ups.

Updates

Routes, Beaker, and WebHelpers however have been seeing quite a bit of updates through the life of Pylons 0.9.7 so no one should think that the developers working on Pylons and its related parts have been hanging out doing nothing. :)

Since Pylons 0.9.7 was released on February 23, 2009, almost one year ago now:

• Routes 1.11 was released, and 1.12 with some great updates will be out shortly
• Beaker has gone from 1.2.2 -> 1.5 with 3 major updates substantially increasing its ease of use and reliability
• WebHelpers is now at 1.0b4 with major updates, core functions rewritten, and new docs up
• SQLAlchemy has gone from 0.4 to 0.5 (with 0.6 in beta)

I believe this speaks a great deal about the benefits of keeping the core Pylons functionality separate from other parts, as a variety of bug fixes and features can be improved without requiring new Pylons releases to quickly address bug reports.

How to Help!

To bring Pylons to 1.0, many docs likely need very small changes. Also, it would be great to take care of reference docs where people have commented about problems/tips. Helping is fairly easy, especially if you’re familiar with restructured text.

First: Clone the Pylons repository on Bitbucket: http://bitbucket.org/bbangert/pylons/

Then: Edit the documentation files under pylons/docs/en/ to read as appropriate, commit the fix, and push it to bitbucket.

Finally: Issue a pull request on bitbucket so that we’ll know your fix is ready. Ideally you should include a note in it about what your fix remedies.

Bug Reports

Did your upgrade not go according to plan? Was there something missing that you needed to do from the upgrading docs?

Let us know by filing a bug report (mark component as documentation, and milestone as 0.10: http://pylonshq.com/project/pylonshq/newticket

You’ll need to login to file a bug report, or feel free to reply to this announcement with the issue.

Thanks (in alphabetical order) to Mike Bayer, Ian Bicking, Mike Burrows, Graham Higgins, Phil Jenvey, Mike Orr, and anyone else I missed for all their hard work on making Pylons and its various components what they are today.

Deploying Python web apps with toppcloud

Ian Bicking recently released a rather interesting package called toppcloud that aims to tackle what I see as a growing need for those of us deploying Python webapps.

I was actually interested in easing my own deployment woes before I saw Ian’s announcement about his package, and was halfway through a rather hefty amount of research on automating server deployments with tools like Chef and Puppet, but toppcloud is a bit different. It not only tackles provisioning a new system on the fly from the ‘cloud’ (using libcloud), but it also handles easy Python (and now PHP) web application deployments.

With such a tantalizing set of goals, I couldn’t really resist getting my feet wet. Boy am I glad I did. PylonsHQ is now running with toppcloud, and I won’t be surprised when more people get it running for them. When many shared hosting providers are $5-10 a month, its rather nice to pay $10/mth for an automatically configured VPS, with one-command deployments. Though of course, unless you go to quite a bit of work yourself, most shared hosting providers don’t have one-command deployments for you.

Before I continue on to describe how I setup Kai - the source code behind PylonsHQ - I should provide a few caveats about using toppcloud:

• toppcloud is alpha software, there’s no releases of it yet, you will be checking it out from source code
• currently, only Rackspace Cloud is known to be working, though since it uses libcloud, in theory, any cloud providers that it supports should be usable
• toppcloud is changing rapidly, be ready to keep up on the commit log to see whats changing
• there are no unit tests, most likely because its very tedious to make the rather significant amount of mock objects required to test the various local/remote commands and the fact that its changing so fast the tests would probably be obsolete in a week

toppcloud philosophy

I’m half guessing here, based on talking with Ian and reading the docs myself, but the philosophy of toppcloud is around providing a common deployment platform, ala Google App Engine, except of course with Postgres or other ‘services’ that you can request to use. At the moment the only services that toppcloud comes with is CouchDB, Files (to store/serve files for your app on the filesystem), and Postgres w/postgis extensions. For those diving into the source code, it shouldn’t be too hard to see how to create additional services and I hope to see more get added as people get more interested.

Therefore, toppcloud is not expected to be everything to everyone, it is expected to be at least an 80+% solution to deploying web apps in a Google App Engine style ease-of-deployment process. toppcloud itself then ensures that depending on what service you asked for, its setup and ready for use on the server when your app is deployed.

If that sounds like something you’re dying to try out, you’re in luck!

Setting up a Pylons App

I’m not going to mention how to check out the toppcloud source, except to mention the directions are in the

/docs/index.txt

file in the toppcloud source (which should be read in its entirety!). Once you have toppcloud installed on your computer, getting from zero to running website on VPS is remarkably quick:

1. $ toppcloud create-node --image-id=14362 baseimage

   Wait until email arrived indicating that server is up and ready

2. $ toppcloud setup-node baseimage

   Server is now all setup to run web apps!

3. $ toppcloud init myapp

   Create an app to deploy to our server

4. $ source myapp/bin/activate

   Activate the virtualenv used for this app

5. $ pip install Pylons

6. $ cd myapp/src

7. $ paster create -t pylons awesomeapp

   Create our Pylons app, or check out an existing one from your VCS here

8. $ cd ../..

9. $ ln -s myapp/src/awesomeapp/awesomeapp/public/ myapp/static

   toppcloud will make available things in the static directory available without hitting the webapp
10. Configure myapp/app.ini similarly to (taken from pylonshq site):

    		[production]
    		app_name = pylonshq
    		runner = src/kai/production.ini
    		version = 1
    		update_fetch = /sync_app
    		service.couchdb =
    		service.files =
    		default_host = pylonshq.com
    	

    In this case, the Pylons site needs CouchDB setup, and the files service (all Pylons sites should use the files service to store cached files, templates, etc.)

11. $ toppcloud update myapp/

    Note that we’re one directory above myapp, and we have a trailing slash on myapp, this is needed because an rsync is done to copy it to the server, don’t leave off the trailing slash! (This will prolly be fixed at some point)

That’s it! 10 easy steps to go from zero to a running deployed website.

I should also note that you will need to make a production.ini file, and that it should have a few important changes in it. All the references to %(here)s should be changed to %(CONFIG_FILES)s since that’s the persistent location that files can be stored for a toppcloud app between deployments. Other configuration information provided by services (Couch supplies the db/host, Postgres has its host/user/pass info) can be accessed via CONFIG_ vars as well. The services docs have some more info.

More apps can be added to a host as desired, until the ram runs out of course. At the moment toppcloud is using a process pool of 5 with 1 thread under mod-wsgi for each application. This can use a bit of ram if you have multiple heavy Pylons processes, hopefully there will shortly be a way to ask toppcloud to use a single process with multiple threads which will help cut the ram profile a bit.

Using Django and PHP

Unfortunately I haven’t actually tried this myself, but there’s nothing preventing it. You’ll need to change the app.ini so that instead of using ‘src/kai/production.ini’ as the runner, it uses a Python file in the directory, say main.py that then loads the Django app as a WSGI application and returns it. Sort of like this. Note that the config vars Django needs for its database should then be present in os.environ when setting up the settings.py that Django uses.

If you’re looking through the toppcloud source code by now, you may have also noticed there’s an example app that uses PHP. There’s nothing holding back toppcloud from setting up mod_passenger and deploying Ruby apps at some point either should someone wish to add that feature.

dumb pipes

There’s been numerous mentions on various blogs and in the news about how much the cell phone carriers hate the concept of being nothing more than “dumb pipes” for wireless Internet and phone use. That means of course, that they’d no longer be competing on what phones you could use but instead solely on service quality and price… I think the same type of transition is in store for shared hosting providers and some of the boutique app deployment shops like heroku.

It’ll take awhile, toppcloud is very rough right now. But when you can d/l a nice little open-source package, run it, choose your choice of cloud provider (based on price + quality!), have it automatically setup for you to deploy your apps to, then deploy apps with a single command… you’ve already done the vast majority of what a service like heroku does, except you could still modify toppcloud if there was something lacking you really needed. And it’s a reason like that, that toppcloud exists to begin with.

Oh, and thanks Ian for writing this before I wasted more time making it myself.

Advanced Caching with Django and Beaker

After seeing more than a few blog posts and packages attempt to provide more advanced caching capability for Django, it occurs to me I should actually just blog how to use Beaker in Django, rather than just keep mumbling about how “Beaker already does that”. So, if you’ve needed caching in Django that goes beyond using just one backend at a time, or maybe can actually cope with the Dog-Pile Effect, this is the blog entry for you (Until I flesh it out further into actual docs on the Beaker site).

Install Beaker

This is simple enough, if you have easy_install available, just:

 easy_install -U Beaker

Or if you prefer to download tar files, grab the Beaker 1.4 tar.gz file

Configuring the Cache

Setting up Beaker’s cache for your Django app is pretty easy. Since only a single cache instance is needed for an app, we’ll set it up as a module global.

Create a beakercache.py file in your Django project with the following contents:

 from beaker.cache import CacheManager
 from beaker.util import parse_cache_config_options

 cache_opts = {
     'cache.type’: 'file’,
     'cache.data_dir’: 'cache/data’,
     'cache.lock_dir’: 'cache/lock’
 }

 cache = CacheManager(**parse_cache_config_options(cache_opts))

There’s a lot more options available, such as memcached, configuring multiple cache backends at once, etc. Now that you know how to provide the configuration options, further customization can be done as needed using the Beaker configuration docs. (Note the very handy cache region configurations which make it easy to toggle cache backend configurations on the fly!)

Using the Cache

Beaker provides a conveinent decorator API to make it easy to cache the results of functions. In this example we’ll just sleep and make a string including the time, add this to your views.py:

 import time
 from datetime import datetime

 from django.http import HttpResponse

 from YOURPROJECT.beakercache import cache

 def hello(request):
     @cache.cache(expire=10)
     def fetch_data():
         time.sleep(4)
         return 'Hello world, its %s’ % datetime.now()
     results = fetch_data()
     return HttpResponse(results)

In this case, the cached data is in a nested function with the decorator. It could of course be in the module elsewhere as well.

Hook the view function up in your urls.py, and hit the view. The first time it will wait a few seconds, then it will return the old time until the cache expires (10 seconds in this case).

The cached function can also accept positional (non-keyword) arguments, which will be used to key the cache. That is, different argument values will result in different copies of the cache that require those arguments to match.

That’s it, it’s really quite easy to use.

Update: It occurs to me this post does say ‘advanced’, and that example wasn’t very advanced, so here’s something a bit more interesting. Let’s configure cache regions to make it easy to toggle how long and where something is cached. Cache regions allow you to arbitrarily configure batches of settings, a ‘region’. Later you can then indicate you want to use that region, and it uses the settings you configured. This also make its easy to setup cache policies and change them in a single location.

In this case, we’ll have ‘long_term’ and ‘short_term’ cache settings, though you can of course come up with as many regions as desired, with the name of your choice. We’ll have the long_term settings use the filesystem, since we want to retain the results for quite awhile and not have them pushed out like memcached does. The short_term settings will use memcached, and be cached for only 2 minutes, long enough to help out on those random slashdog/digg hits.

In the beakercache.py file:

 from beaker.cache import CacheManager
 from beaker.util import parse_cache_config_options

 cache_opts = {
     'cache.type’: 'file’,
     'cache.data_dir’: 'cache/data’,
     'cache.lock_dir’: 'cache/lock’,
     'cache.regions’: 'short_term, long_term’,
     'cache.short_term.type’: 'ext:memcached’,
     'cache.short_term.url’: ’127.0.0.1.11211’,
     'cache.short_term.expire’: '1200’,
     'cache.long_term.type’: 'file’,
     'cache.long_term.expire’: '86400’,
 }

 cache = CacheManager(**parse_cache_config_options(cache_opts))

Now in our views.py:

 import time
 from datetime import datetime

 from django.http import HttpResponse

 from testdjango.beakercache import cache

 def hello(request):
     @cache.region('long_term’)
     def fetch_data():
         time.sleep(15)
         return 'Hello world, its %s’ % datetime.now()
     results = fetch_data()
     return HttpResponse(results)

 def goodbye(request):
     @cache.region('short_term’')
     def fetch_data():
         time.sleep(4)
         return 'Bye world, its %s’ % datetime.now()
     results = fetch_data()
     return HttpResponse(results)     

Beaker 1.4 Released

Beaker 1.4 has now been released, and addresses several fairly important bugs. First, the full changelog:

• Fix bug with hmac on Python 2.4. Patch from toshio, closes ticket #2133 from the TurboGears2 Trac.
• Fix bug with occasional ValueError from FileNamespaceManager.do_open. Fixes #10.
• Fixed bug with session files being saved despite being new and not saved.
• Fixed bug with CacheMiddleware overwriting configuration with default arguments despite prior setting.
• Fixed bug with SyntaxError not being caught properly in entry point discovery.
• Changed to using BlobProperty for Google Datastore.
• Added domain/path properties to the session. This allows one to dynamically set the cookie’s domain and/or path on the fly, which will then be set on the cookie for the session.
• Added support for cookie-based sessions in Jython via the JCE (Java Cryptography Extensions). Patch from Alex Grönholm.
• Update Beaker database extensions to work with SQLAlchemy 0.6 PostgreSQL, and Jython.

Note that the beaker database extension now works on Jython, and the cookies for sessions can be set dynamically during a request (for sites that operate across multiple domains/sub-domains).

Most importantly though, a bug in the import of the Google back-end has been fixed, which caused installation failures on Beaker 1.3.x.

Docs can be found on the Beaker site.

To upgrade your Beaker with easy_install:

 easy_install -U Beaker
 

This release is also notable as the majority of the fixes were contributed by several web framework communities. Thanks for the patches!

Comments and Web Services

I’m trying out Disqus for comments on the blog, for one main reason, I just didn’t feel like implementing comments myself. I am somewhat wary of services that seem integral to a blog, like where the comments are, which is why I’ve been leery of using services like this for some time.

What happens if Disqus runs out of VC and goes belly-up? I’ve seen this with a few other web services, and no one using it is very happy when it occurs. On the other hand, I did find some irony in the fact that one of the features Disqus pitches users is, “Don’t lose your comments if the blog disappears”.

On the other hand, I really appreciate the capabilities a central service-based comment system brings. I’ve inadvertently used it on other blogs, and was very pleasantly surprised to get updates and actually be able to keep up on what was happening on the blogs I commented on.

Are other people worried about using services by new companies or am I just overly paranoid?

I’d almost feel better about it if I could pay five bucks a year or something, as I’d at least have some better reassurance that the company is actually making money like I do with Flickr. Or maybe it’d be nice if companies that are profitable to say so, though it occurs to me that even that isn’t a guarantee, as someone could come along and buy them up, then decide to terminate their functionality (like that one site that people used and were pissed about when Six Apart bought them and shut it down, but for some reason I can’t remember the name of it at all).

Blog upgraded! Now running on Pylons + MongoDB

I’ve now updated the blog software powering my blog, which is very long overdue. In the past, this blog was run off Typo, which apparently now hosts their home site off the github (its moved a dozen times in the past 3+ years).

Typo always worked moderately well for me, however, I found it sluggish (Prolly Rails there), and incredibly horrid on ram usage. It was not at all unusual to see it running past 700 megs of RAM after running for just a few weeks, which is a bit annoying as the machine only has 4GB total and is running quite a few things.

After last weeks SF Python Meetup on MongoDB I figured it was time to get a little actual MongoDB usage under my belt. I also inadvertently implemented enough of the MovableType XMLRPC API as I didn’t want my app to be too extensive, just enough for me to post to my blog.

So in the end, I had a small set of requirements for the replacement:

1. Not be horribly slow
2. Not take up huge amounts of RAM
3. Retain all existing URL’s (It really annoys me when people break their old links)
4. Compatible with my blog software (MovableType / MetaWeblog XMLRPC API)
5. Not screw up the RSS/Atom feeds and cause Planet Python to show all my posts as if they were new (I’ve seen this happen to a few people on occasion)

I wanted to build it myself, because of course, the world definitely needs more blog apps, and I wanted one that used MongoDB. So for those curious, here’s the source code to the blog app.

It’s rather rough, as its fairly custom built just for my needs, nor do I have any plans to expand it into some general purpose blog engine, with themes, etc. The only other thing pending at the moment is to add the ability to comment again, as I haven’t quite gotten that feature in yet. For those trying it out, the README should help get started, but its very rough (thus the name of the package).

Strings, Unidecode, and Slugs

When copying some functionality I needed from the Rails app (to retain URL compatibility), I noticed two things it did which I thought was handy. To convert a title into slug for the article, it used a fairly sophisticated scheme relying on two other packages.

First, was the use of a Ruby port of a Perl package called Text::Unidecode which is pretty cool, and converts UTF-8 chars into their closest ASCII version. I figured someone must’ve ported it to Python as well, and sure enough, someone did! It wasn’t on the cheeseshop though, which was unbearable for me, so I’ve posted it to the Cheeseshop so others can easy_install it.

Next up, was a Ruby library called stringex, which add’s a few things to Rails, including a string method called ‘to_url’. That method does a variety of transformations to remove all those funny characters from a title, and do a bunch of other neat changes of common characters to human readable versions (source for those conversions).

I ported the key module of stringex to Python, and it resides in my blog app. If someone would like to extract it and make it into its own package, or even better, if I somehow missed the fact that someone else has ported it already, let me know (tweet me @benbangert!).

I’ll be writing up my thoughts on making a small app with MongoDB, and how it differs from my experience working with CouchDB for PylonsHQ in a later post for those curious.

Making a better TextArea

I’ve been working on some Javascript to enhance the TextArea elements on the PylonsHQ Snippets section, and have noticed that… well, TextArea’s suck.

The hack I’ve seen is to use one of the newer features of browsers, the editable or ‘design’ attribute’s for div elements I believe. This lets one build a very snazzy amount of features, such as syntax highlighting, code completion, etc., but I don’t think I needed to go that far.

I only have one main design goal, this TextArea is for the user to enter RestructuredText so it’d be awesome if the TextArea acted in a way that made rst a bit nicer. The obvious two things that came to mind:

1. Tab key indents 4-spaces
2. Hitting return on an indented line, will retain the indentation on the next line

I’ve actually gotten some Javascript, hobbled together from various parts of the net, along with an ‘enter’ key handler I wrote myself on bitbucket.

Course, it’d also be nice to have a button or key combo, that will indent/unindent a selection in the TextArea as well.

Anyone else have any Javascript they’ve hobbled together in the past to make TextArea a little nicer for restructured text?