Wednesday, June 29, 2011

Handling line endings with Python 2 csv module

This is another note to self style post, this time about cross-platform handling of CSV files with the Python 2 csv module. The typical boilerplate for processing CSV files is the following:

with open("sample.csv", "r") as handle:
    reader = csv.reader(handle)
    fieldnames =
    for row in reader:
        print row

In general this code works, however when the CSV file uses a single \r (Mac Classic style) the following error will be raised:

new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

I ran some tests across different platforms testing this behaviour, and it seems quite consistent:

End of line marker Mac OSX 10.6.8 Windows 7 Windows XP Ubuntu 10.10
\n Yes Yes Yes Yes
\r\n Yes Yes Yes Yes
\r No No No No

The solution is to open the file using the mode "rU" rather than just "r".

Thursday, June 23, 2011

Mac OSX + IE CSS behaviour (.htc) + Django's runserver

Unfortunately on Mac OSX (10.6 at least) Python's mimetypes.guess_type() function is unable to guess the MIME type for .htc files (they should be text/x-component). This is significant because Internet Explorer won't use a behaviour if the Content-Type header in the response for a .htc file is incorrect.

Python's mimetypes.guess_type() has a list of it's own known MIME types, which is supplemented with records from various external files:

knownfiles = [ 
    "/etc/httpd/mime.types",                # Mac OS X
    "/etc/httpd/conf/mime.types",           # Apache
    "/etc/apache/mime.types",               # Apache 1
    "/etc/apache2/mime.types",              # Apache 2
    "/usr/local/etc/httpd/conf/mime.types", # Apache 1.2
    "/usr/local/etc/mime.types",            # Apache 1.3

So to fix this problem, you could add the mapping to one of those files, however I opted to add it explicitly to my file:

import mimetypes
mimetypes.add_type("text/x-component", ".htc")

Of course, this is only relevant if you're using Django's runserver command to serve static files. If you're using Apache you'll need to modify one of the mime.types file, and if you're using nginx, it should just work!

The other hassle is that the URL for a behaviour in a CSS file is not relative to the CSS file, it's relative to the URL of the page the browser is rendering. For this reason, I always specify absolute paths, e.g.:

behaviour: url(/static/core/css/;

Wednesday, June 22, 2011


I've just released django-protocolify on Github. It's basically a Django app that has a template tag that allows you to dynamically change the protocol of URLs within a section of a template. This can be useful if you want to force all the links in an existing block to use HTTPS rather than HTTP.

Consider the following base.html:

<!DOCTYPE html>
    <title>django-protocolify demo</title>
    {% block navigation %}
        <li><a href="{% url home %}">Home</a></li>
        <li><a href="{% url shop %}">Shop</a></li>
        <li><a href="{% url about_us %}">About Us</a></li>
    {% endblock navigation %}
    <div class="content">
    {% block content %}
        <p>No content here</p>
    {% endblock content %}

Now consider being on a "Payments" page that uses HTTPS. If the user clicks on
one of the navigation links from the base template, they'll still be using
HTTPS. django-protocolify solves this:

{% extends "base.html %}
{% load protocolify %}

{% block navigation %}
{% protocolify to "http" %}
{{ block.super }}
{% endprotocolify %}
{% endblock %}


If it sounds useful, go and check it out.

Saturday, June 4, 2011

OSX + homebrew + gevent

I ran into some trouble installing gevent on OSX, but the solution was pretty straight forward. Unfortunately a simple pip install gevent fails:

$ pip install gevent-0.13.6.tar.gz 
Unpacking ./requirements/src/gevent-0.13.6.tar.gz
  Running egg_info for package from

    building 'gevent.core' extension
    /usr/bin/cc -fno-strict-aliasing -O3 -march=core2 -msse4.1 -w -pipe -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/brew/bin/../Cellar/python/2.7.1/include/python2.7 -c gevent/core.c -o build/temp.macosx-10.4-x86_64-2.7/gevent/core.o
    In file included from gevent/core.c:225:
    gevent/libevent.h:9:19: error: event.h: No such file or directory
    gevent/libevent.h:38:20: error: evhttp.h: No such file or directory
    gevent/libevent.h:39:19: error: evdns.h: No such file or directory
    gevent/core.c:361: error: field ‘ev’ has incomplete type


    gevent/core.c:15344: error: dereferencing pointer to incomplete type
    gevent/core.c:15358: error: dereferencing pointer to incomplete type
    gevent/core.c:15367: error: dereferencing pointer to incomplete type
    gevent/core.c:15385: error: dereferencing pointer to incomplete type
    gevent/core.c: At top level:
    gevent/core.c:21272: error: expected ‘)’ before ‘val’
    error: command '/usr/bin/cc' failed with exit status 1

The tail of the error message isn't particularly useful, it suggests there's an error in the C code. However toward the top we see the event.h: No such file or directory complaint which seems to indicate that libevent is needed (which makes sense). The solution is straight forward, just install libevent, then use pip to install gevent:

$ brew install libevent
$ export CFLAGS=-I/brew/include
$ pip install gevent-0.13.6.tar.gz

And then everything just worked. You'll notice my homebrew installation is at (the non-standard location of) /brew/, so you'll probably need to tailor that to your own situation.

Creating public/private key for SSH

Run the following command replacing <email> with the desired email address.

ssh-keygen -t rsa -b 2048 -C <email> -f id_rsa

Two files will be created: id_rsa and Copy the content out of and add it to a ~/.ssh/authorized_keys on the server. Place id_rsa in ~/.ssh/ on your computer.