Creating a custom AMI with Postgis and its dependencies in order to deploy Django + GeoDjango on Amazon Elastic Beanstalk

While the installation of PostgreSQL + Postgis on my development machine (my beloved MacBook Pro) has been very easy, thanks to MacPorts, installing the necessary software on Amazon Elastic Beanstalk (in order to move my project from local to the cloud) has been an hard challenge!
Theoretically you can customize an environment by using configuration files in which you can specify packages and other resources to install, but the problem is that in the Amazon 64bit Linux distribution for Python (which is an extremely customized version of Red Hat) you don’t have apt (for which postgis packages are available), instead you have to rely on yum. Is possible to install extra repositories for yum (see here: in order to easily install postgis… but honestly I have no idea which repository should be the right one for Amazon Linux, so… it’s been painful, but I opted for an “old school” style installation, by downloading and compiling the missing packages by myself. So, after launching my EC2 instance I did connect to it via SSH and:

1. Switch to root user:

sudo su -

2. Update all the installed packages (which Amazon doesn’t update very often!):

yum update -y

3. Install development tools and necessary libraries (some of them, like “graphviz” are not required for GeoDjango and you can aovid their installation if you want… I’m reporting all my libraries as a future reference for myself :P)

yum install -y python-devel libpcap libpcap-devel libnet libnet-devel pcre pcre-devel gcc gcc-c++ libtool make libyaml libyaml-devel binutils libxml2 libxml2-devel zlib zlib-devel file-devel postgresql postgresql-devel postgresql-contrib geoip geoip-devel graphviz graphviz-devel gettext libtiff-devel libjpeg-devel libzip-devel freetype-devel lcms2-devel libwebp-devel tcl-devel tk-devel

4. Download and compile proj:

unzip && cd proj-4.8.0
./configure && make && sudo make install
cd ..

5. Download and compile geos:

tar -xvf geos-3.4.2.tar.bz2 && cd geos-3.4.2
./configure && make && sudo make install
cd ..

6. Download and compile gdal (this library is the most SLOW to compile and depending on the type of instance that you have launched it may takes up to a couple of hours… be patient!):

unzip && cd gdal-1.10.1
./configure --with-python=yes && make && sudo make install
cd ..

7. Download and install postgis:

tar -xvf postgis-2.1.1.tar.gz && cd postgis-2.1.1
./configure && make && sudo make install

8. Update installed libraries (this step is necessary to avoid issues related to invalid library paths):

sudo echo /usr/local/lib >> /etc/
sudo ldconfig

It’s also a nice idea to export the environment variable LD_LIBRARY_PATH (as /usr/local/lib/:$LD_LIBRARY_PATH).
Once you have installed all the necessary software on your machine you can create a custom AMI by going to: EC2 > instances > select your instance > create AMI. To use that AMI as the default one for your application you have to specify its id in your Elastic Beanstalk environment configuration.

Creating class based view decorators using simple Django function decorators

I love decorators because they allow to extend class/functions capabilities and to be be added and removed easily. Django offers a set of decorators but unfortunely they are designed for functions or class methods only and instead I want to add them to the whole class in order to avoid to override the orignal method just to add the decorator annotation above it. Fortunately wrapping the simple Django decorators in order to use them against classes (basically View classes) is quite simple and I’m gonna show you what I done.
For example let’s consider we have a view that require a POST method. If we use a simple function view we can write the following:

def myView(request):
	return HttpResponse('hello world!')

but, if we use class views (and we should!), we are forced to write:

class MyView(SomeKindOfParentClassView):
	# ...imagine several methods here

	def dispatch(self, request, *args, **kwargs):
		return HttpResponse('hello world!')

wouldn’t be cleaner and more elegant instead to simply write the following?

class MyView(SomeKindOfParentClassView):
	# ...imagine several methods here

…and this is the class decorator implementation:

from django.views.decorators.http import require_POST as post_only

def require_POST(View):
    View.dispatch = method_decorator(post_only)(View.dispatch)
    return View

Basically I redefined the class method and returned the class itself.
This was really simple since the concrete implementation of the decorator has been added using the existend Django decorators… but what if we want to create a custom decorator? For example in my app I created a decorator that enable access to a view only if it’s requested via ajax, this is how I implemented it:

def require_AJAX(SomeKindOfParentClassView):
    def ajaxOnly(function):
        def wrap(request, *args, **kwargs):
            if not request.is_ajax():
                return HttpResponseForbidden()
            return function(request, *args, **kwargs)

        return wrap

    View.dispatch = method_decorator(ajaxOnly)(View.dispatch)
    return View

And the decorated view looks like this:

class MyView(SomeKindOfParentClassView):
    # ...imagine several methods here

As you can see that annotations are really readable, immediate and self-describing.

How to make AngularJS and Django play nice together

In order to make AngularJS working as I wish in my Django app, these are the settings that I’ve adopted:

1. Differentiate Angular templates symbols from Django ones:

Both Angular than Django use doble curly braces to mark variables and/or expressions ({{ myVar }}).
In order to have the full control on how and by who our templates are rendered, I redefined the Angular interpolations signs in the config() method of my client app.


2. Change the default Angular Content-type header used in POST method:

Angular defines the “Content-Type” header as “application/json” for ajax POST, but Django doesn’t understand that content properly and as result, the POST data is not an object as we expect but rather a string! So, I modified the default content type as “application/x-www-form-urlencoded” (which is the format used by jQuery and other JavaScript libraries).

$['Content-Type'] = 'application/x-www-form-urlencoded';

3. Assign the CSFR token to each ajax call automagically

In Django templates we can use the tag {% csrf_token %} inside a form to print an hidden input containing the token. But when it comes to making an xhr post request, how can we pass the token in an effective and DRY manner? The answer I gave myself is to set a default http header for ajax calls containing the value of the token obtained by reading the session cookie (in this way this stuff is handle 100% by JavaScript).

$['X-CSRFToken'] = $cookies.csrftoken;

Differently from points 1 and 2, this is done in the run() method, since $cookies is an Angular service and can’t be used in config() block (in that function only provider objects can be used).
In order to use $cookies we have also to import “angular-cookies.js” in addition to the base “angular.js“.

The final configuration is the following:

angular.module('myapp', ['ngCookies']).
    function($httpProvider, $interpolateProvider) {
        $['Content-Type'] = 'application/x-www-form-urlencoded';
    function($http, $cookies) {
        $['X-CSRFToken'] = $cookies.csrftoken;


Starting from Angular 1.2, you have also to set default headers in order to use Django helper method is_ajax() in your views:

$httpProvider.defaults.headers.common['X-Requested-With'] = 'XMLHttpRequest';

Effective TDD: Tricks to speed up Django tests (up to 10x faster!)

TDD (Test Driven Development) is one of the best practices to follow in order to write better software and once you get used to this practice you can’t write code without writing tests.
In TDD, tests must be executed very frequently and so it’s fundamental that they run as fast as possible in order to avoid time wasting.
In this post I want to share all the tricks I discovered that will make your Django tests run very quickly (at the time of this writing I have a suite of 250 tests that has been tuned to run in ~5 seconds rather than ~50 seconds of the untuned version… that’s 10 times faster!)
In this article I’m assuming that you know: how to run Django unit tests, how to use a different settings module for testing.
These are my tips:

1) Change password hashing

This is the most effective setting you can use to improve the speed of tests, it may sounds ridicolous, but password hashing in Django is designed to be very strong and it makes use of several “hashers”, but this also means that the hashing is very slow. The fastest hasher is the MD5PasswordHasher, so you can use just that one in this way:


2) Use a faster storage system

Currently the fastest database you can use with Django is SQLite. I was initially doubtful about switching from Postgres (my database of choice) to SQLite, but I’m not testing Django itself, I’m testing my own API and since I don’t use raw SQL statements the underlying storage backend should not make differences!
So, to configure SQLite:

    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'simple_test_db'

3) Remove unnecessary middlewares

Middleware layer is one of the Django features that I like the most, BUT the more middleware classes you use, the more your response time will increase (since all the middleware must be executed sequentially before returning the final HTTPResponse, ALWAYS!). So be sure to include only the stuff you really and absolutely need!
One middleware in particular that is very slow and that I removed for the tests is:


4) Remove unnecessary apps

There are several third party apps that you may remove during testing like the debug toolbar, try to remove all the unused/unnecessary apps.

5) Disable debugging

We don’t need extended traceback during tests, so we can disable debugging:

DEBUG = False

6) Turn off logging

This is a significant modification only if we have a huge amount of logging and/or additional logic involved in logs (such object inspections, heavy string manipulation and so on), but anyway logging is futile during testing, so:


7) Use a faster Email backend (by “patching” Django)

By default Django will use django.core.mail.backends.locmem.EmailBackend, which is an in-memory backend designed for testing, however I had several problems with that backend during my tests, they did block unexplainably for ~30 seconds due to headers checking. So I decided to write my own in-memory backend which mimics the Django one but does not check email headers in order to be blazing fast:

from django.core.mail.backends.base import BaseEmailBackend
from django.core import mail

class TestEmailBackend(BaseEmailBackend):

    def send_messages(self, messages):
        return len(messages)

In order to use this backend I had to use the @override_settings decorator, since defining it in settings file is helpless because that asshole of Django (1.5.x) will use django.core.mail.backends.locmem.EmailBackend anyway!
So, by following DRY phylosophy I created a subclass of Django’s TestCase which includes that overridden setting (so I don’t have to remember to define my own backend for each test):

from django.test import TestCase as DjangoTestCase
from django.test.utils import override_settings

class TestCase(DjangoTestCase):

8) Make use of bulk_create

If you are creating more objects don’t rely on save() for each object, but instead create objects in batch using bulk_create() (it’s a method provided by Django ORM to insert multiple records in a single query)

9) Use an in-memory backend for Celery

If you are using Celery, these are my optimal settings for testing:


That’s all! If you have other advices please let me know ;)

Configuring Django Pipeline by using Closure compiler for JavaScript files and YUI compressor for CSS

A couple of days ago I lost several hours trying to understand how to configure django-pipeline by replacing UglifyJS (the standard compressor library adopted) with Closure compiler and YUI compressor, which are in my opinion the two best tools to minify JavaScript and CSS respectively.
Fortunately Pipeline is pretty flexible and allows us to choose among different compressors and also write our own compressor, but I had to face some aspects that were initially unclear to me:

1. Understand how executable compressor bins are invoked by Pipeline:

In the documentation they say that you have to specify the absolute path to the bin of the compressor you are going to use… but, while UglifyJS ships with an executable bin, other compressors are usually simple jar files (with no bin), so pointing to that jars won’t work… so what? You have to write an executable yourself! But don’t panic it’s just a matter of writing a single line of bash script and make it executable :)
Let’s considering for example to adopt YUI compressor for CSS:
once downloaded the jar to a know directory in our filesystem, we can create a simple bin which will be used by pipeline by writing the following script:

java -jar "$TOOLS_PATH/yui_compressor-2.4.jar" $@

The script invokes the jar (java -jar) by passing all the arguments received from Pipeline (“$@” is the alien way for bash to accept a variable number of arguments… something like *args in Python).
$TOOLS_PATH is an environment variable I defined that contains the path to the jars directory (in order to avoid hard-coding it in the settings file).
After saving the script as “yui_compressor” (or whatever you like) you need to make it executable:

chmod +x yui_compressor

That’s all! Now we can define our CSS compression preferences in this way (in “” or whatever is your setting file… yes, if you didn’t know you can and should use multiple settings files in order to differentiate settings based on the environment you are running on… but this is another story for another post):

PIPELINE_CSS_COMPRESSOR = 'pipeline.compressors.yui.YUICompressor'
PIPELINE_YUI_BINARY = os.path.join(

(getEnvironmentVariable() is a custom helper I wrote to retrieve environemnt variables)

2. Make Closure compiler do its job without raising exceptions using AngularJS and other “sophisticated” JavaScript stuff

One of the good parts of Closure is that it has a lot of configurable options and specifically in order to make it happy while compiling my stuff I had to set “language_in” to ECMASCRIPT5 and “warning_level” to QUIET (according to the Google documentation and what I found on StackOverflow).
In order to pass these settings Pipeline provides an arguments option for each specific compiler, so I added:

PIPELINE_CLOSURE_ARGUMENTS = '--language_in=ECMASCRIPT5 --warning_level=QUIET'

If you don’t want to use “PIPELINE_CLOSURE_ARGUMENTS” you can also hard-code that flags in the exacutable you wrote, for example:

java -jar "$TOOLS_PATH/closure_compiler-20130823.jar --language_in=ECMASCRIPT5 --warning_level=QUIET " $@

But don’t do that!

End of post… happy compression at all!