be Groovie

Pylons 0.10 and 1.0 Beta 1 Released

Without further ado,

I’m pleased to announced that Pylons 0.10b1 and 1.0b1 are now out. I have not put them on Cheeseshop to ensure they’re not downloaded accidentally.

Upgrading / Installing

I have updated upgrading instructions here: http://pylonshq.com/docs/en/1.0/upgrading/

The instructions to install from scratch on Pylons 1.0b1: http://pylonshq.com/docs/en/1.0/gettingstarted/#installing

The upgrading page covers the important upgrading instructions that Mike Orr touched briefly on before.

Note that these are beta releases, intended for us to discover remaining issues and continue updating any other documentation where applicable. Very little has actually changed in Pylons since 0.9.7, apart from 1.0 dropping all of the legacy functionality and a few explicit clean-ups.

Updates

Routes, Beaker, and WebHelpers however have been seeing quite a bit of updates through the life of Pylons 0.9.7 so no one should think that the developers working on Pylons and its related parts have been hanging out doing nothing. :)

Since Pylons 0.9.7 was released on February 23, 2009, almost one year ago now:

• Routes 1.11 was released, and 1.12 with some great updates will be out shortly
• Beaker has gone from 1.2.2 -> 1.5 with 3 major updates substantially increasing its ease of use and reliability
• WebHelpers is now at 1.0b4 with major updates, core functions rewritten, and new docs up
• SQLAlchemy has gone from 0.4 to 0.5 (with 0.6 in beta)

I believe this speaks a great deal about the benefits of keeping the core Pylons functionality separate from other parts, as a variety of bug fixes and features can be improved without requiring new Pylons releases to quickly address bug reports.

How to Help!

To bring Pylons to 1.0, many docs likely need very small changes. Also, it would be great to take care of reference docs where people have commented about problems/tips. Helping is fairly easy, especially if you’re familiar with restructured text.

First: Clone the Pylons repository on Bitbucket: http://bitbucket.org/bbangert/pylons/

Then: Edit the documentation files under pylons/docs/en/ to read as appropriate, commit the fix, and push it to bitbucket.

Finally: Issue a pull request on bitbucket so that we’ll know your fix is ready. Ideally you should include a note in it about what your fix remedies.

Bug Reports

Did your upgrade not go according to plan? Was there something missing that you needed to do from the upgrading docs?

Let us know by filing a bug report (mark component as documentation, and milestone as 0.10: http://pylonshq.com/project/pylonshq/newticket

You’ll need to login to file a bug report, or feel free to reply to this announcement with the issue.

Thanks (in alphabetical order) to Mike Bayer, Ian Bicking, Mike Burrows, Graham Higgins, Phil Jenvey, Mike Orr, and anyone else I missed for all their hard work on making Pylons and its various components what they are today.

Deploying Python web apps with toppcloud

Ian Bicking recently released a rather interesting package called toppcloud that aims to tackle what I see as a growing need for those of us deploying Python webapps.

I was actually interested in easing my own deployment woes before I saw Ian’s announcement about his package, and was halfway through a rather hefty amount of research on automating server deployments with tools like Chef and Puppet, but toppcloud is a bit different. It not only tackles provisioning a new system on the fly from the ‘cloud’ (using libcloud), but it also handles easy Python (and now PHP) web application deployments.

With such a tantalizing set of goals, I couldn’t really resist getting my feet wet. Boy am I glad I did. PylonsHQ is now running with toppcloud, and I won’t be surprised when more people get it running for them. When many shared hosting providers are $5-10 a month, its rather nice to pay $10/mth for an automatically configured VPS, with one-command deployments. Though of course, unless you go to quite a bit of work yourself, most shared hosting providers don’t have one-command deployments for you.

Before I continue on to describe how I setup Kai - the source code behind PylonsHQ - I should provide a few caveats about using toppcloud:

• toppcloud is alpha software, there’s no releases of it yet, you will be checking it out from source code
• currently, only Rackspace Cloud is known to be working, though since it uses libcloud, in theory, any cloud providers that it supports should be usable
• toppcloud is changing rapidly, be ready to keep up on the commit log to see whats changing
• there are no unit tests, most likely because its very tedious to make the rather significant amount of mock objects required to test the various local/remote commands and the fact that its changing so fast the tests would probably be obsolete in a week

toppcloud philosophy

I’m half guessing here, based on talking with Ian and reading the docs myself, but the philosophy of toppcloud is around providing a common deployment platform, ala Google App Engine, except of course with Postgres or other ‘services’ that you can request to use. At the moment the only services that toppcloud comes with is CouchDB, Files (to store/serve files for your app on the filesystem), and Postgres w/postgis extensions. For those diving into the source code, it shouldn’t be too hard to see how to create additional services and I hope to see more get added as people get more interested.

Therefore, toppcloud is not expected to be everything to everyone, it is expected to be at least an 80+% solution to deploying web apps in a Google App Engine style ease-of-deployment process. toppcloud itself then ensures that depending on what service you asked for, its setup and ready for use on the server when your app is deployed.

If that sounds like something you’re dying to try out, you’re in luck!

Setting up a Pylons App

I’m not going to mention how to check out the toppcloud source, except to mention the directions are in the

/docs/index.txt

file in the toppcloud source (which should be read in its entirety!). Once you have toppcloud installed on your computer, getting from zero to running website on VPS is remarkably quick:

1. $ toppcloud create-node --image-id=14362 baseimage

   Wait until email arrived indicating that server is up and ready

2. $ toppcloud setup-node baseimage

   Server is now all setup to run web apps!

3. $ toppcloud init myapp

   Create an app to deploy to our server

4. $ source myapp/bin/activate

   Activate the virtualenv used for this app

5. $ pip install Pylons

6. $ cd myapp/src

7. $ paster create -t pylons awesomeapp

   Create our Pylons app, or check out an existing one from your VCS here

8. $ cd ../..

9. $ ln -s myapp/src/awesomeapp/awesomeapp/public/ myapp/static

   toppcloud will make available things in the static directory available without hitting the webapp
10. Configure myapp/app.ini similarly to (taken from pylonshq site):

    		[production]
    		app_name = pylonshq
    		runner = src/kai/production.ini
    		version = 1
    		update_fetch = /sync_app
    		service.couchdb =
    		service.files =
    		default_host = pylonshq.com
    	

    In this case, the Pylons site needs CouchDB setup, and the files service (all Pylons sites should use the files service to store cached files, templates, etc.)

11. $ toppcloud update myapp/

    Note that we’re one directory above myapp, and we have a trailing slash on myapp, this is needed because an rsync is done to copy it to the server, don’t leave off the trailing slash! (This will prolly be fixed at some point)

That’s it! 10 easy steps to go from zero to a running deployed website.

I should also note that you will need to make a production.ini file, and that it should have a few important changes in it. All the references to %(here)s should be changed to %(CONFIG_FILES)s since that’s the persistent location that files can be stored for a toppcloud app between deployments. Other configuration information provided by services (Couch supplies the db/host, Postgres has its host/user/pass info) can be accessed via CONFIG_ vars as well. The services docs have some more info.

More apps can be added to a host as desired, until the ram runs out of course. At the moment toppcloud is using a process pool of 5 with 1 thread under mod-wsgi for each application. This can use a bit of ram if you have multiple heavy Pylons processes, hopefully there will shortly be a way to ask toppcloud to use a single process with multiple threads which will help cut the ram profile a bit.

Using Django and PHP

Unfortunately I haven’t actually tried this myself, but there’s nothing preventing it. You’ll need to change the app.ini so that instead of using ‘src/kai/production.ini’ as the runner, it uses a Python file in the directory, say main.py that then loads the Django app as a WSGI application and returns it. Sort of like this. Note that the config vars Django needs for its database should then be present in os.environ when setting up the settings.py that Django uses.

If you’re looking through the toppcloud source code by now, you may have also noticed there’s an example app that uses PHP. There’s nothing holding back toppcloud from setting up mod_passenger and deploying Ruby apps at some point either should someone wish to add that feature.

dumb pipes

There’s been numerous mentions on various blogs and in the news about how much the cell phone carriers hate the concept of being nothing more than “dumb pipes” for wireless Internet and phone use. That means of course, that they’d no longer be competing on what phones you could use but instead solely on service quality and price… I think the same type of transition is in store for shared hosting providers and some of the boutique app deployment shops like heroku.

It’ll take awhile, toppcloud is very rough right now. But when you can d/l a nice little open-source package, run it, choose your choice of cloud provider (based on price + quality!), have it automatically setup for you to deploy your apps to, then deploy apps with a single command… you’ve already done the vast majority of what a service like heroku does, except you could still modify toppcloud if there was something lacking you really needed. And it’s a reason like that, that toppcloud exists to begin with.

Oh, and thanks Ian for writing this before I wasted more time making it myself.

Beaker 1.4 Released

Beaker 1.4 has now been released, and addresses several fairly important bugs. First, the full changelog:

• Fix bug with hmac on Python 2.4. Patch from toshio, closes ticket #2133 from the TurboGears2 Trac.
• Fix bug with occasional ValueError from FileNamespaceManager.do_open. Fixes #10.
• Fixed bug with session files being saved despite being new and not saved.
• Fixed bug with CacheMiddleware overwriting configuration with default arguments despite prior setting.
• Fixed bug with SyntaxError not being caught properly in entry point discovery.
• Changed to using BlobProperty for Google Datastore.
• Added domain/path properties to the session. This allows one to dynamically set the cookie’s domain and/or path on the fly, which will then be set on the cookie for the session.
• Added support for cookie-based sessions in Jython via the JCE (Java Cryptography Extensions). Patch from Alex Grönholm.
• Update Beaker database extensions to work with SQLAlchemy 0.6 PostgreSQL, and Jython.

Note that the beaker database extension now works on Jython, and the cookies for sessions can be set dynamically during a request (for sites that operate across multiple domains/sub-domains).

Most importantly though, a bug in the import of the Google back-end has been fixed, which caused installation failures on Beaker 1.3.x.

Docs can be found on the Beaker site.

To upgrade your Beaker with easy_install:

 easy_install -U Beaker
 

This release is also notable as the majority of the fixes were contributed by several web framework communities. Thanks for the patches!

Blog upgraded! Now running on Pylons + MongoDB

I’ve now updated the blog software powering my blog, which is very long overdue. In the past, this blog was run off Typo, which apparently now hosts their home site off the github (its moved a dozen times in the past 3+ years).

Typo always worked moderately well for me, however, I found it sluggish (Prolly Rails there), and incredibly horrid on ram usage. It was not at all unusual to see it running past 700 megs of RAM after running for just a few weeks, which is a bit annoying as the machine only has 4GB total and is running quite a few things.

After last weeks SF Python Meetup on MongoDB I figured it was time to get a little actual MongoDB usage under my belt. I also inadvertently implemented enough of the MovableType XMLRPC API as I didn’t want my app to be too extensive, just enough for me to post to my blog.

So in the end, I had a small set of requirements for the replacement:

1. Not be horribly slow
2. Not take up huge amounts of RAM
3. Retain all existing URL’s (It really annoys me when people break their old links)
4. Compatible with my blog software (MovableType / MetaWeblog XMLRPC API)
5. Not screw up the RSS/Atom feeds and cause Planet Python to show all my posts as if they were new (I’ve seen this happen to a few people on occasion)

I wanted to build it myself, because of course, the world definitely needs more blog apps, and I wanted one that used MongoDB. So for those curious, here’s the source code to the blog app.

It’s rather rough, as its fairly custom built just for my needs, nor do I have any plans to expand it into some general purpose blog engine, with themes, etc. The only other thing pending at the moment is to add the ability to comment again, as I haven’t quite gotten that feature in yet. For those trying it out, the README should help get started, but its very rough (thus the name of the package).

Strings, Unidecode, and Slugs

When copying some functionality I needed from the Rails app (to retain URL compatibility), I noticed two things it did which I thought was handy. To convert a title into slug for the article, it used a fairly sophisticated scheme relying on two other packages.

First, was the use of a Ruby port of a Perl package called Text::Unidecode which is pretty cool, and converts UTF-8 chars into their closest ASCII version. I figured someone must’ve ported it to Python as well, and sure enough, someone did! It wasn’t on the cheeseshop though, which was unbearable for me, so I’ve posted it to the Cheeseshop so others can easy_install it.

Next up, was a Ruby library called stringex, which add’s a few things to Rails, including a string method called ‘to_url’. That method does a variety of transformations to remove all those funny characters from a title, and do a bunch of other neat changes of common characters to human readable versions (source for those conversions).

I ported the key module of stringex to Python, and it resides in my blog app. If someone would like to extract it and make it into its own package, or even better, if I somehow missed the fact that someone else has ported it already, let me know (tweet me @benbangert!).

I’ll be writing up my thoughts on making a small app with MongoDB, and how it differs from my experience working with CouchDB for PylonsHQ in a later post for those curious.

Pylons 0.9.7 Released

I’m pleased to announce after a rather lengthy release candidate period, that Pylons 0.9.7 is finally out. Pylons 0.9.7 brings a good amount of changes to Pylons from 0.9.6 while still retaining a fairly hefty amount of backwards compatibility to ensure a mostly painless upgrade.

Some helpful documentation on the new release:

• Upgrading to Pylons 0.9.7
• Pylons 0.9.7 Documentation
• Pylons 0.9.7 Docs in PDF
• Pylons Book (Will be updated soon with the final copy from the printed book)

Major changes in 0.9.7:

• Switched to using WebOb for the request/response object
• Various performance improvements to object initialization
• Beaker and Routes updates
• Middleware improvements, and optimizations

This is a huge step forward for Pylons, and I’d like to thank all of the contributers who have helped make Pylons what it is today. We’ve knocked off more bugs for this release than any before, which shows just how far the Pylons community has come:

• 0.9.5 tickets: 45
• 0.9.6 tickets: 64
• 0.9.7 tickets: 160

And we have finally made a huge dent in the historical “lack of docs” problem that Pylons previously suffered from with the new Sphinx generated docs and a comprehensive Pylons book.

The full changelog which describes the major changes (Look for the bits marked with WARNING that might affect backwards compatibility).

0.9.7 (February 23, 2009)

• WARNING: A new option is available to determine whether or not an actions

arguments should be automatically attached to ‘c’. To turn off this implicit behavior in environment.py: config[‘pylons.c_attach_args’] = False This is set to True by default.

• WARNING: Fixed a minor security hole in the default Pylons error page that

could result in an XSS security hole.

• WARNING: Fixed a security hole in the default project template to use the

StaticURLParser to ensure arbitrary files can’t be sent.

• WARNING: Refactored PylonsApp to remove legacy PylonsApp, moved

session/cache and routes middleware into the project template. This will require projects to be updated to include those 3 middleware in the projects middleware.py.

• Changed to using WebTest instead of paste.fixture for app testing.
• Added render_mako_def to render def blocks within a mako template.
• Changes to cache_decorator and cached_template to support Beaker API

changes in version 1.1. 1.0.3 is still supported.

• Fix HEAD requests causing an Exception as if no content was returned

by the controller. Fixes #507. Thanks mvtellingen, Petr Kobalicek.

• Fix a crash when returning the result of “etag_cache“ in a controller.

Fixes #508.

• “response” flag has been removed from pylons.decorators.cache.beaker_cache,

as it sends all headers along unconditionally including cookies; additionally, the flag was taking effect in all cases previously so prior versions of beaker_cache are not secure. In its place, a new option “cache_headers” is provided, which is a tuple of specific header names to be cached. It defaults to (‘content-type’,’content-length’).

• “invalidate_on_startup” flag added to beaker_cache, which provides a

“starttime” to the cache such that when the application is started or restarted, the cache entry is invalidated.

• Updating host to use 127.0.0.1 for development binding.
• Added option to specify the controller name with a controller variable

in the controller’s module. This name will be used for the controller class rather than the default naming scheme.

• setup.py egg_info now restores projects’ paster_plugins.txt,

allowing paster shell to work again after the egg-info directory was lost. fixes #282. Thanks sevkin.

• The paste_deploy_config.ini_tmpl template is now located at

package/config/deployment.ini_tmpl for new projects.

• Project’s default test fixtures no longer hardcode test.ini; the ini

file used can now be specified via the nosetests —with-pylons argument (defaults to test.ini in setup.cfg). fixes #400.

• validate now defaults to translating FormEncode error messages via Pylons' gettext catalog, then falls back to FormEncode's. fixes #296. Thanks Max Ischenko. * Fixed SQLAlchemy logging not working in paster shell. Fixes #363. Thanks Christoph Haas. * Added optionally engine initialization, to prevent Buffet from loading if there's no 'buffet.template_engines' in the config. * Updated minimal template to work with Tempita and other new templating changes. * Fixed websetup to parse location config file properly when the section isn't 'main'. Fixes #399. * Added default Mako filter of escape for all template rendering. * Fixed template for Session.remove inclusion when using SA. Fixed render_genshi to properly use fragment/format options. Thanks Antonin Enfrun. * Remove template engine from load_environment call. * Removing template controller from projects. Fixes #383. * Added signed_cookie method to WebOb Request/Response sub-classes. * Updated project template to setup appropriate template loader and controller template to doc how to import render. * Added documentation for render functions in pylons.templating. * Adding specific render functions that don't require Buffet. * Added forward controller.util function for forwarding the request to WSGI apps. Fixes #355. * Added default input encoding for Mako to utf-8. Suggested in #348. * Fixed paster controller to raise an error if the controller for it already exists. Fixes #279. * Added __init__.py to template dir in project template if the template engine is genshi or kid. Fixes #353. * Fixed jsonify to use application/json as its the proper mime-type and now used all over the net. * Fixed minimal template not replacing variables properly. Fixes #377. * Fixed validate decorator to no longer catch exceptions should they be

raised in the action that is supposed to display a form. Fixes #374.

• Fixed paster shell command to no longer search for egg_info dir. Allows

usage of paster shell with installed packages. Suggested by Gavin Carothers.

• Added mimetype function and MIMETypes class for registering mimetypes.
• WARNING: Usage of pylons.Response is now deprecated. Please use

pylons.response instead.

• Removed use of WSGIRequest/WSGIResponse and replaced with WebOb subclasses

that implement methods to make it backwards compatible with the Paste wsgiwrappers.

• Fixed missing import in template controller.
• Deprecated function uses string substitution to avoid Nonetype error when

Python optimization is on. Fixes #334.

• E-tag cache no longer returns Content-Type in the headers. Fixes #323.
• XMLRPCController now properly includes the Content-Length of the response.

Fixes #310, thanks Nicholas.

• Added SQLAlchemy option to template, which adds SQLAlchemy setup to the

project template.

• Switched project templating to use Tempita.
• Updated abort/redirect_to to use appropriate Response object when WebOb is

used.

• Updated so that 404’s properly return as Response objects when WebOb is in

use instead of WSGIResponse.

• Added beaker_cache option to avoid caching/restoring global Response values

that were present during the first cache operation.

• Adding StatusCodeRedirect to handle internal redirects based on the status

code returned by the app. This replaces the use of ErrorDocuments in projects.

• Refactored error exceptions to use WebError.
• WSGIController now uses the environ references to response, request, and

the c object for higher performance.

• Added optional use of WebOb instead of paste.wsgiwrapper objects.
• Fixed bug with beaker_cache defaulting to dbm rather than the beaker

cache app-wide default.

• The —with-pylons nose plugin no longer requires a project to have been

registered with setuptools to work.

• The config object is now included in the template namespace.
• StaticJavascripts now accepts keyword arguments for StaticURLParser.

Suggested by Marcin Kasperski.

• Fix pylons.database.AutoConnectHub’s doInTransaction not automatically

connecting when necessary. Fixes #327.

New PylonsHQ Site Launches

The new PylonsHQ site has now launched!

The new site is running on the latest Pylons 0.9.7 code-base backed by the CouchDB database. New features that have been added:

• Snippet repository
• Traceback posting
• Easier to read buildbot status page
• Trac user integration, so your login to the main PylonsHQ site is carried into Trac seamlessly
• Feeds for the various sections, including the new blog
• All new documentation using Sphinx
• OpenID registration and sign-up
• All PylonsHQ code open-sourced and available on BitBucket

Unfortunately, we were unable to integrate the Wiki’s auth, so that will still require a separate login for now.

Comments are through-out the site, to ensure that feedback isn’t missed and of course there’s many more features planned that are coming soon. The site isn’t quite 100% complete, as a few links here and there are likely broken yet (like the tutorial links on the front page). I’ll be putting out frequent updates to remedy this and any other little bits that need more polish.

Enjoy!