Wednesday, December 14, 2011

Using ReportLab to create a PDF from an image.

If you want to create a page in a PDF that contains an image that fills the entire page, here's a snippet that'll do exactly that. It requires PIL and ReportLab. Simply pass the function the path to an image and the canvas, and it'll fill the current page with the image (rotated properly including honoring EXIF orientation attributes).

Due to it handling EXIF orientation values, it's compatible with images taken from the camera of iPhone and iPad devices.

Wednesday, November 16, 2011

Another strange Django error

If you stumble across the error in Django 1.3:

  File "/Users/brad/work/django-formwizard/.env/lib/python2.7/site-packages/django/core/", line 368, in reverse
    app_list = resolver.app_dict[ns]
  File "/Users/brad/work/django-formwizard/.env/lib/python2.7/site-packages/django/core/", line 241, in _get_app_dict
  File "/Users/brad/work/django-formwizard/.env/lib/python2.7/site-packages/django/core/", line 198, in _populate
    p_pattern = pattern.regex.pattern
AttributeError: 'str' object has no attribute 'regex'

Or in Django 1.4:


  File "/brew/Cellar/python/2.7.1/lib/python2.7/site-packages/django/core/handlers/", line 101, in get_response
  File "/brew/Cellar/python/2.7.1/lib/python2.7/site-packages/django/core/", line 300, in resolve
    sub_match = pattern.resolve(new_path)
AttributeError: 'str' object has no attribute 'resolve'

You have probably accidentally left a trailing comma in the URLCONF_ROOT setting, i.e.:

ROOT_URLCONF = 'project.urls',

Sunday, November 13, 2011

Strange UTF-8 decoding error with Jenkins + Python

I've come across a strange problem while setting up Jenkins to build Python projects. For some reason I get the following error:

Traceback (most recent call last):
  File "", line 32, in <module>
    'Topic :: Software Development :: Libraries :: Python Modules',
  File "/usr/local/lib/python2.7/distutils/", line 152, in setup

  File "/home/jenkins/.jenkins/jobs/django-console/workspace/django_attest-0.1.1-py2.7.egg/django_attest/", line 0
SyntaxError: 'NoneType' object has no attribute 'utf_8_decode'

In any case it's fixed by adding the following as an environment variable (I added it to the Properties Content section):

Thursday, November 10, 2011

Upstart + Jenkins

I've been playing with Jenkins for CI of Python projects. I run Jenkins on, and then use Apache2 to proxy a domain to it. Here's a simple virtual host configuration I found somewhere:

<VirtualHost *:80>
    ServerName jenkins
    ProxyPass         /  http://localhost:8081/
    ProxyPassReverse  /  http://localhost:8081/
    ProxyRequests     Off
    # Local reverse proxy authorization override
    # Most unix distribution deny proxy by default (ie
    # /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
    <Proxy http://localhost:8081*>
      Order deny,allow
      Allow from all

Lastly I wrote an Upstart job in /etc/init/jenkins.conf to keep it running across restarts:

description "Jenkins"

start on started network-services
stop on stopping network-services

cd /home/jenkins
sudo -Hu jenkins java -jar jenkins.war -Djava.awt.headless=true --httpPort=8081 --httpListenAddress=
end script

Sunday, October 16, 2011

Upstart job for PostgreSQL 9.1 on Ubuntu 11.10

In line with the recent Upstart theme, he's a script for PostgreSQL 9.1 on Ubuntu 11.10:

Save this to /etc/init/postgresql.conf and delete the symlinks from /etc/rc#.d/ to disable the SysV scripts.

Using Upstart with RabbitMQ on Ubuntu 11.10

I've been tinkering with the idea of using Upstart to control celery processes for a Web site. I plan on using user jobs and hooking the rabbitmq-server and postgresql events to start and stop my celery instances. Unfortunately, on Ubuntu 11.10 RabbitMQ does not come with an Upstart job, but rather a SysV script.

Here's some instructions I've put together to convert it to using Upstart.

Install rabbitmq:

$ sudo apt-get install rabbitmq-server

It will automatically be started, so we first want to shut it down:

$ sudo /etc/init.d/rabbitmq-server stop

Now swap out the built-in /etc/init.d scripts for the Upstart job:

$ sudo rm /etc/rc0.d/K20rabbitmq-server \
/etc/rc1.d/K20rabbitmq-server \
/etc/rc2.d/S20rabbitmq-server \
/etc/rc3.d/S20rabbitmq-server \
/etc/rc4.d/S20rabbitmq-server \
/etc/rc5.d/S20rabbitmq-server \

Put the following in /etc/init/rabbitmq-server.conf:

description "RabbitMQ Server"
author  "RabbitMQ"

start on runlevel [2345]
stop on runlevel [016]

exec /usr/sbin/rabbitmq-server > /var/log/rabbitmq/startup_log \
                              2> /var/log/rabbitmq/startup_err
post-start exec /usr/sbin/rabbitmqctl wait >/dev/null 2>&1

And you're done. You can now use:

sudo start rabbitmq-server

Notable differences between this job and the SysV script:
  • A lock file is not support (it was disabled by default in the SysV script anyway)
  • Shutdown is achieved via SIGTERM, rather than using rabbitmqctl stop. As a side effect, the /var/log/rabbitmq/shutdown_{log, err} files are not used.

Saturday, October 15, 2011

Upstart user jobs on Ubuntu 11.10

Recently I've been exploring Upstart's user jobs functionality. User jobs allow non-root users to have their own jobs in ~/.init/ that they can control.

The first thing I did was to create a simple job:


    sleep 5
end script

This job simply blocks for five seconds, which allows me to test whether user jobs are working properly. To use it I saved it to ~/.init/my-test-job.conf, and then started it via start my-test-job.

Unfortunately by default on Ubuntu 11.10, user jobs are disabled, which meant the start command failed with the error:

start: Rejected send message, 1 matched rules; type="method_call", sender=":1.5" (uid=1000 pid=1655 comm="start my-test-job ") interface="com.ubuntu.Upstart0_6.Job" member="Start" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")

To enable user jobs, I had to edit /etc/dbus-1/system.d/Upstart.conf and change it to:

The original DBus configuration is far more restrictive in what messages it allows to reach Upstart (basically read/write for root and read for everyone else). The above configuration (taken from Upstart's source code) allows any user to control Upstart. This does seem like a security problem, but apparently Upstart can handle user permissions internally.

After making this change, I was able to start my job successfully:

$ start my-test-job
my-test-job stop/waiting

Documentation is pretty scarce, but there's a small section in the man page that's worth checking out (search for User Jobs).


If you want user job start on stanzas to be honored, check out my more recent blog post about it

Monday, July 25, 2011

Repairing PostgreSQL after upgrading to Mac OSX Lion

I upgraded to Mac OSX 10.7 (Lion) today and had some issues with PostgreSQL. It
would seem that PostgreSQL is now bundled with OSX, and the upgrade process appears to have
caused some issues with my previous version.

The problem is this: I'd try to connect to PostgreSQL via the unix socket during (python syncdb
), and I'd get the following error:

psycopg2.OperationalError: could not connect to server: Permission denied
 Is the server running locally and accepting
 connections on Unix domain socket "/var/pgsql_socket/.s.PGSQL.5432"?

After a little trial and error the following seems to work reliably.
  1. Edit PostgreSQL's config to define the unix_socket_directory setting: (for me this was line 68)
$ sudo vim /Library/PostgreSQL/9.0/data/postgresql.conf

In my case I changed the setting to unix_socket_directory = '/var/pgsql_socket/'.
  1. Exceute the following commands.
$ sudo dscl . append /Groups/_postgres GroupMembership postgres
$ sudo chmod g+w,o+rx /var/pgsql_socket/

And finally restart PostgreSQL to apply the config changes:

$ sudo -u postgres /Library/PostgreSQL/9.0/bin/pg_ctl -D /Library/PostgreSQL/9.0/data/ restart

Wednesday, June 29, 2011

Handling line endings with Python 2 csv module

This is another note to self style post, this time about cross-platform handling of CSV files with the Python 2 csv module. The typical boilerplate for processing CSV files is the following:

with open("sample.csv", "r") as handle:
    reader = csv.reader(handle)
    fieldnames =
    for row in reader:
        print row

In general this code works, however when the CSV file uses a single \r (Mac Classic style) the following error will be raised:

new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

I ran some tests across different platforms testing this behaviour, and it seems quite consistent:

End of line marker Mac OSX 10.6.8 Windows 7 Windows XP Ubuntu 10.10
\n Yes Yes Yes Yes
\r\n Yes Yes Yes Yes
\r No No No No

The solution is to open the file using the mode "rU" rather than just "r".

Thursday, June 23, 2011

Mac OSX + IE CSS behaviour (.htc) + Django's runserver

Unfortunately on Mac OSX (10.6 at least) Python's mimetypes.guess_type() function is unable to guess the MIME type for .htc files (they should be text/x-component). This is significant because Internet Explorer won't use a behaviour if the Content-Type header in the response for a .htc file is incorrect.

Python's mimetypes.guess_type() has a list of it's own known MIME types, which is supplemented with records from various external files:

knownfiles = [ 
    "/etc/httpd/mime.types",                # Mac OS X
    "/etc/httpd/conf/mime.types",           # Apache
    "/etc/apache/mime.types",               # Apache 1
    "/etc/apache2/mime.types",              # Apache 2
    "/usr/local/etc/httpd/conf/mime.types", # Apache 1.2
    "/usr/local/etc/mime.types",            # Apache 1.3

So to fix this problem, you could add the mapping to one of those files, however I opted to add it explicitly to my file:

import mimetypes
mimetypes.add_type("text/x-component", ".htc")

Of course, this is only relevant if you're using Django's runserver command to serve static files. If you're using Apache you'll need to modify one of the mime.types file, and if you're using nginx, it should just work!

The other hassle is that the URL for a behaviour in a CSS file is not relative to the CSS file, it's relative to the URL of the page the browser is rendering. For this reason, I always specify absolute paths, e.g.:

behaviour: url(/static/core/css/;

Wednesday, June 22, 2011


I've just released django-protocolify on Github. It's basically a Django app that has a template tag that allows you to dynamically change the protocol of URLs within a section of a template. This can be useful if you want to force all the links in an existing block to use HTTPS rather than HTTP.

Consider the following base.html:

<!DOCTYPE html>
    <title>django-protocolify demo</title>
    {% block navigation %}
        <li><a href="{% url home %}">Home</a></li>
        <li><a href="{% url shop %}">Shop</a></li>
        <li><a href="{% url about_us %}">About Us</a></li>
    {% endblock navigation %}
    <div class="content">
    {% block content %}
        <p>No content here</p>
    {% endblock content %}

Now consider being on a "Payments" page that uses HTTPS. If the user clicks on
one of the navigation links from the base template, they'll still be using
HTTPS. django-protocolify solves this:

{% extends "base.html %}
{% load protocolify %}

{% block navigation %}
{% protocolify to "http" %}
{{ block.super }}
{% endprotocolify %}
{% endblock %}


If it sounds useful, go and check it out.

Saturday, June 4, 2011

OSX + homebrew + gevent

I ran into some trouble installing gevent on OSX, but the solution was pretty straight forward. Unfortunately a simple pip install gevent fails:

$ pip install gevent-0.13.6.tar.gz 
Unpacking ./requirements/src/gevent-0.13.6.tar.gz
  Running egg_info for package from

    building 'gevent.core' extension
    /usr/bin/cc -fno-strict-aliasing -O3 -march=core2 -msse4.1 -w -pipe -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/brew/bin/../Cellar/python/2.7.1/include/python2.7 -c gevent/core.c -o build/temp.macosx-10.4-x86_64-2.7/gevent/core.o
    In file included from gevent/core.c:225:
    gevent/libevent.h:9:19: error: event.h: No such file or directory
    gevent/libevent.h:38:20: error: evhttp.h: No such file or directory
    gevent/libevent.h:39:19: error: evdns.h: No such file or directory
    gevent/core.c:361: error: field ‘ev’ has incomplete type


    gevent/core.c:15344: error: dereferencing pointer to incomplete type
    gevent/core.c:15358: error: dereferencing pointer to incomplete type
    gevent/core.c:15367: error: dereferencing pointer to incomplete type
    gevent/core.c:15385: error: dereferencing pointer to incomplete type
    gevent/core.c: At top level:
    gevent/core.c:21272: error: expected ‘)’ before ‘val’
    error: command '/usr/bin/cc' failed with exit status 1

The tail of the error message isn't particularly useful, it suggests there's an error in the C code. However toward the top we see the event.h: No such file or directory complaint which seems to indicate that libevent is needed (which makes sense). The solution is straight forward, just install libevent, then use pip to install gevent:

$ brew install libevent
$ export CFLAGS=-I/brew/include
$ pip install gevent-0.13.6.tar.gz

And then everything just worked. You'll notice my homebrew installation is at (the non-standard location of) /brew/, so you'll probably need to tailor that to your own situation.

Creating public/private key for SSH

Run the following command replacing <email> with the desired email address.

ssh-keygen -t rsa -b 2048 -C <email> -f id_rsa

Two files will be created: id_rsa and Copy the content out of and add it to a ~/.ssh/authorized_keys on the server. Place id_rsa in ~/.ssh/ on your computer.

Sunday, May 29, 2011

Retrieving the URL of a namespaced Django view

I've finally got around to learning the new class based views approach in Django 1.3 with the goal of writing a mixin that can be used with django-tables to avoid some of the boilerplate code that's currently required. I ended up with a handy SingleTableMixin class which I'll be merging into the development branch when I get a chance.

However I find learning-by-doing is optimal so I started by refactored some legacy CRUD CMS-like views to use the class based views approach. I became a little side-tracked while looking over the old code and found myself cleaning them up. During this process I noticed that it might be a good idea to create URLs via reverse(view_func) rather than reverse(urlname). However it did not work.

After some painful debugging it seems that view functions that are hooked into the URLconf via a namespaced URL can not be reversed via their function reference.

Wednesday, May 25, 2011

Creating a separate linux user for each Web site

For added security it's a good idea to run each Web site as a separate user. For this, I use the following command:

sudo adduser --system --no-create-home <username>

You can then instruct mod_wsgi to use this user:

WSGIDaemonProcess user=<username>

Obviously in both snippets, you replace <username> with an actual username.

Forcing SSH to authenticate via public key for all but one user

When setting up a Ubuntu Web server I generally want to disable password authentication for SSH (and instead use public key), which has been all well and good until now. While experimenting with automatic deployment solutions for Django, I wanted to be able to use password authentication for a the deployment account, whilst enforcing public key authentication for everyone else.

The solution is very simple. In my /etc/ssh/sshd_config I have the typical:

RSAAuthentication no
PubkeyAuthentication yes

And to enable password authentication for a single user I added the following to the end of the file:

# Allow the 'deployment' user to login
# using their password
Match User deployment
PasswordAuthentication no

It's documented in detail in the Match section of the man page.