Using ReportLab to create a PDF from an image.

If you want to create a page in a PDF that contains an image that fills the entire page, here's a snippet that'll do exactly that. It requires PIL and ReportLab. Simply pass the function the path to an image and the canvas, and it'll fill the current page with the image (rotated properly including honoring EXIF orientation attributes).

Due to it handling EXIF orientation values, it's compatible with images taken from the camera of iPhone and iPad devices.

Another strange Django error

If you stumble across the error in Django 1.3:

  ...
  File "/Users/brad/work/django-formwizard/.env/lib/python2.7/site-packages/django/core/urlresolvers.py", line 368, in reverse
    app_list = resolver.app_dict[ns]
  File "/Users/brad/work/django-formwizard/.env/lib/python2.7/site-packages/django/core/urlresolvers.py", line 241, in _get_app_dict
    self._populate()
  File "/Users/brad/work/django-formwizard/.env/lib/python2.7/site-packages/django/core/urlresolvers.py", line 198, in _populate
    p_pattern = pattern.regex.pattern
AttributeError: 'str' object has no attribute 'regex'


Or in Django 1.4:


...

  File "/brew/Cellar/python/2.7.1/lib/python2.7/site-packages/django/core/handlers/base.py", line 101, in get_response
    request.path_info)
  File "/brew/Cellar/python/2.7.1/lib/python2.7/site-packages/django/core/urlresolvers.py", line 300, in resolve
    sub_match = pattern.resolve(new_path)
AttributeError: 'str' object has no attribute 'resolve'



You have probably accidentally left a trailing comma in the URLCONF_ROOT setting, i.e.:


ROOT_URLCONF = 'project.urls',

Strange UTF-8 decoding error with Jenkins + Python

I've come across a strange problem while setting up Jenkins to build Python projects. For some reason I get the following error:


Traceback (most recent call last):

  File "setup.py", line 32, in <module>
    'Topic :: Software Development :: Libraries :: Python Modules',
  File "/usr/local/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()

  ...

  File "/home/jenkins/.jenkins/jobs/django-console/workspace/django_attest-0.1.1-py2.7.egg/django_attest/__init__.py", line 0
SyntaxError: 'NoneType' object has no attribute 'utf_8_decode'

In any case it's fixed by adding the following as an environment variable (I added it to the Properties Content section):
LANG=en_AU.UTF-8

Upstart + Jenkins

I've been playing with Jenkins for CI of Python projects. I run Jenkins on 127.0.0.1:8081, and then use Apache2 to proxy a domain to it. Here's a simple virtual host configuration I found somewhere:


<VirtualHost *:80>
    ServerName jenkins
    ProxyPass         /  http://localhost:8081/
    ProxyPassReverse  /  http://localhost:8081/
    ProxyRequests     Off
    # Local reverse proxy authorization override
    # Most unix distribution deny proxy by default (ie
    # /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
    <Proxy http://localhost:8081*>
      Order deny,allow
      Allow from all
    </Proxy>
</VirtualHost>


Lastly I wrote an Upstart job in /etc/init/jenkins.conf to keep it running across restarts:


description "Jenkins"


respawn
start on started network-services
stop on stopping network-services


script
cd /home/jenkins
sudo -Hu jenkins java -jar jenkins.war -Djava.awt.headless=true --httpPort=8081 --httpListenAddress=127.0.0.1
end script

Upstart job for PostgreSQL 9.1 on Ubuntu 11.10

In line with the recent Upstart theme, he's a script for PostgreSQL 9.1 on Ubuntu 11.10:

Save this to /etc/init/postgresql.conf and delete the symlinks from /etc/rc#.d/ to disable the SysV scripts.

Using Upstart with RabbitMQ on Ubuntu 11.10

I've been tinkering with the idea of using Upstart to control celery processes for a Web site. I plan on using user jobs and hooking the rabbitmq-server and postgresql events to start and stop my celery instances. Unfortunately, on Ubuntu 11.10 RabbitMQ does not come with an Upstart job, but rather a SysV script.

Here's some instructions I've put together to convert it to using Upstart.

Install rabbitmq:

$ sudo apt-get install rabbitmq-server

It will automatically be started, so we first want to shut it down:

$ sudo /etc/init.d/rabbitmq-server stop

Now swap out the built-in /etc/init.d scripts for the Upstart job:

$ sudo rm /etc/rc0.d/K20rabbitmq-server \
/etc/rc1.d/K20rabbitmq-server \
/etc/rc2.d/S20rabbitmq-server \
/etc/rc3.d/S20rabbitmq-server \
/etc/rc4.d/S20rabbitmq-server \
/etc/rc5.d/S20rabbitmq-server \
/etc/rc6.d/K20rabbitmq-server

Put the following in /etc/init/rabbitmq-server.conf:

description "RabbitMQ Server"
author  "RabbitMQ"

start on runlevel [2345]
stop on runlevel [016]
respawn

exec /usr/sbin/rabbitmq-server > /var/log/rabbitmq/startup_log \
                              2> /var/log/rabbitmq/startup_err
post-start exec /usr/sbin/rabbitmqctl wait >/dev/null 2>&1

And you're done. You can now use:

sudo start rabbitmq-server

Notable differences between this job and the SysV script:

• A lock file is not support (it was disabled by default in the SysV script anyway)
• Shutdown is achieved via SIGTERM, rather than using rabbitmqctl stop. As a side effect, the /var/log/rabbitmq/shutdown_{log, err} files are not used.

Upstart user jobs on Ubuntu 11.10

Recently I've been exploring Upstart's user jobs functionality. User jobs allow non-root users to have their own jobs in ~/.init/ that they can control.

The first thing I did was to create a simple job:

task

script
    sleep 5
end script

This job simply blocks for five seconds, which allows me to test whether user jobs are working properly. To use it I saved it to ~/.init/my-test-job.conf, and then started it via start my-test-job.

Unfortunately by default on Ubuntu 11.10, user jobs are disabled, which meant the start command failed with the error:

start: Rejected send message, 1 matched rules; type="method_call", sender=":1.5" (uid=1000 pid=1655 comm="start my-test-job ") interface="com.ubuntu.Upstart0_6.Job" member="Start" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init")

To enable user jobs, I had to edit /etc/dbus-1/system.d/Upstart.conf and change it to:


The original DBus configuration is far more restrictive in what messages it allows to reach Upstart (basically read/write for root and read for everyone else). The above configuration (taken from Upstart's source code) allows any user to control Upstart. This does seem like a security problem, but apparently Upstart can handle user permissions internally.

After making this change, I was able to start my job successfully:

$ start my-test-job
my-test-job stop/waiting

Documentation is pretty scarce, but there's a small section in the man page that's worth checking out (search for User Jobs).

Edit:

If you want user job start on stanzas to be honored, check out my more recent blog post about it

Repairing PostgreSQL after upgrading to Mac OSX Lion

I upgraded to Mac OSX 10.7 (Lion) today and had some issues with PostgreSQL. It
would seem that PostgreSQL is now bundled with OSX, and the upgrade process appears to have
caused some issues with my previous version.

The problem is this: I'd try to connect to PostgreSQL via the unix socket during (python
manage.py syncdb), and I'd get the following error:

psycopg2.OperationalError: could not connect to server: Permission denied
 Is the server running locally and accepting
 connections on Unix domain socket "/var/pgsql_socket/.s.PGSQL.5432"?

After a little trial and error the following seems to work reliably.

1. Edit PostgreSQL's config to define the unix_socket_directory setting: (for me this was line 68)

$ sudo vim /Library/PostgreSQL/9.0/data/postgresql.conf

In my case I changed the setting to unix_socket_directory = '/var/pgsql_socket/'.

2. Exceute the following commands.

$ sudo dscl . append /Groups/_postgres GroupMembership postgres
$ sudo chmod g+w,o+rx /var/pgsql_socket/

And finally restart PostgreSQL to apply the config changes:

$ sudo -u postgres /Library/PostgreSQL/9.0/bin/pg_ctl -D /Library/PostgreSQL/9.0/data/ restart

Handling line endings with Python 2 csv module

This is another note to self style post, this time about cross-platform handling of CSV files with the Python 2 csv module. The typical boilerplate for processing CSV files is the following:

with open("sample.csv", "r") as handle:
    reader = csv.reader(handle)
    fieldnames = reader.next()
    for row in reader:
        print row

In general this code works, however when the CSV file uses a single \r (Mac Classic style) the following error will be raised:

new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

I ran some tests across different platforms testing this behaviour, and it seems quite consistent:

End of line marker	Mac OSX 10.6.8	Windows 7	Windows XP	Ubuntu 10.10
\n	Yes	Yes	Yes	Yes
\r\n	Yes	Yes	Yes	Yes
\r	No	No	No	No

The solution is to open the file using the mode "rU" rather than just "r".

Mac OSX + IE CSS behaviour (.htc) + Django's runserver

Unfortunately on Mac OSX (10.6 at least) Python's mimetypes.guess_type() function is unable to guess the MIME type for .htc files (they should be text/x-component). This is significant because Internet Explorer won't use a behaviour if the Content-Type header in the response for a .htc file is incorrect.

Python's mimetypes.guess_type() has a list of it's own known MIME types, which is supplemented with records from various external files:

knownfiles = [ 
    "/etc/mime.types",
    "/etc/httpd/mime.types",                # Mac OS X
    "/etc/httpd/conf/mime.types",           # Apache
    "/etc/apache/mime.types",               # Apache 1
    "/etc/apache2/mime.types",              # Apache 2
    "/usr/local/etc/httpd/conf/mime.types",
    "/usr/local/lib/netscape/mime.types",
    "/usr/local/etc/httpd/conf/mime.types", # Apache 1.2
    "/usr/local/etc/mime.types",            # Apache 1.3
]

So to fix this problem, you could add the mapping to one of those files, however I opted to add it explicitly to my settings.py file:

import mimetypes
mimetypes.add_type("text/x-component", ".htc")

Of course, this is only relevant if you're using Django's manage.py runserver command to serve static files. If you're using Apache you'll need to modify one of the mime.types file, and if you're using nginx, it should just work!

The other hassle is that the URL for a behaviour in a CSS file is not relative to the CSS file, it's relative to the URL of the page the browser is rendering. For this reason, I always specify absolute paths, e.g.:

behaviour: url(/static/core/css/PIE.htc);

django-protocolify

I've just released django-protocolify on Github. It's basically a Django app that has a template tag that allows you to dynamically change the protocol of URLs within a section of a template. This can be useful if you want to force all the links in an existing block to use HTTPS rather than HTTP.

Consider the following base.html:

<!DOCTYPE html>
<html>
<head>
    <title>django-protocolify demo</title>
</head>
<body>
    <ul>
    {% block navigation %}
        <li><a href="{% url home %}">Home</a></li>
        <li><a href="{% url shop %}">Shop</a></li>
        <li><a href="{% url about_us %}">About Us</a></li>
    {% endblock navigation %}
    </ul>
    <div class="content">
    {% block content %}
        <p>No content here</p>
    {% endblock content %}
    </div>
</body>
</html>

Now consider being on a "Payments" page that uses HTTPS. If the user clicks on
one of the navigation links from the base template, they'll still be using
HTTPS. django-protocolify solves this:

{% extends "base.html %}
{% load protocolify %}

{% block navigation %}
{% protocolify to "http" %}
{{ block.super }}
{% endprotocolify %}
{% endblock %}

...

If it sounds useful, go and check it out.

OSX + homebrew + gevent

I ran into some trouble installing gevent on OSX, but the solution was pretty straight forward. Unfortunately a simple pip install gevent fails:

$ pip install gevent-0.13.6.tar.gz 
Unpacking ./requirements/src/gevent-0.13.6.tar.gz
  Running setup.py egg_info for package from

  ...
    
    building 'gevent.core' extension
    /usr/bin/cc -fno-strict-aliasing -O3 -march=core2 -msse4.1 -w -pipe -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/brew/bin/../Cellar/python/2.7.1/include/python2.7 -c gevent/core.c -o build/temp.macosx-10.4-x86_64-2.7/gevent/core.o
    In file included from gevent/core.c:225:
    gevent/libevent.h:9:19: error: event.h: No such file or directory
    gevent/libevent.h:38:20: error: evhttp.h: No such file or directory
    gevent/libevent.h:39:19: error: evdns.h: No such file or directory
    gevent/core.c:361: error: field ‘ev’ has incomplete type

    ...

    gevent/core.c:15344: error: dereferencing pointer to incomplete type
    gevent/core.c:15358: error: dereferencing pointer to incomplete type
    gevent/core.c:15367: error: dereferencing pointer to incomplete type
    gevent/core.c:15385: error: dereferencing pointer to incomplete type
    gevent/core.c: At top level:
    gevent/core.c:21272: error: expected ‘)’ before ‘val’
    error: command '/usr/bin/cc' failed with exit status 1

The tail of the error message isn't particularly useful, it suggests there's an error in the C code. However toward the top we see the event.h: No such file or directory complaint which seems to indicate that libevent is needed (which makes sense). The solution is straight forward, just install libevent, then use pip to install gevent:

$ brew install libevent
$ export CFLAGS=-I/brew/include
$ pip install gevent-0.13.6.tar.gz

And then everything just worked. You'll notice my homebrew installation is at (the non-standard location of) /brew/, so you'll probably need to tailor that to your own situation.

Creating public/private key for SSH

Run the following command replacing <email> with the desired email address.

ssh-keygen -t rsa -b 2048 -C <email> -f id_rsa

Two files will be created: id_rsa and id_rsa.pub. Copy the content out of id_rsa.pub and add it to a ~/.ssh/authorized_keys on the server. Place id_rsa in ~/.ssh/ on your computer.

Retrieving the URL of a namespaced Django view

I've finally got around to learning the new class based views approach in Django 1.3 with the goal of writing a mixin that can be used with django-tables to avoid some of the boilerplate code that's currently required. I ended up with a handy SingleTableMixin class which I'll be merging into the development branch when I get a chance.

However I find learning-by-doing is optimal so I started by refactored some legacy CRUD CMS-like views to use the class based views approach. I became a little side-tracked while looking over the old code and found myself cleaning them up. During this process I noticed that it might be a good idea to create URLs via reverse(view_func) rather than reverse(urlname). However it did not work.

After some painful debugging it seems that view functions that are hooked into the URLconf via a namespaced URL can not be reversed via their function reference.

Creating a separate linux user for each Web site

For added security it's a good idea to run each Web site as a separate user. For this, I use the following command:

sudo adduser --system --no-create-home <username>

You can then instruct mod_wsgi to use this user:

WSGIDaemonProcess bradleyayers.com user=<username>

Obviously in both snippets, you replace <username> with an actual username.

Forcing SSH to authenticate via public key for all but one user

When setting up a Ubuntu Web server I generally want to disable password authentication for SSH (and instead use public key), which has been all well and good until now. While experimenting with automatic deployment solutions for Django, I wanted to be able to use password authentication for a the deployment account, whilst enforcing public key authentication for everyone else.

The solution is very simple. In my /etc/ssh/sshd_config I have the typical:

RSAAuthentication no
PubkeyAuthentication yes

And to enable password authentication for a single user I added the following to the end of the file:

# Allow the 'deployment' user to login
# using their password
Match User deployment
PasswordAuthentication no

It's documented in detail in the Match section of the man page.