Effective TDD: Tricks to speed up Django tests (up to 10x faster!)

TDD (Test Driven Development) is one of the best practices to follow in order to write better software and once you get used to this practice you can’t write code without writing tests.
In TDD, tests must be executed very frequently and so it’s fundamental that they run as fast as possible in order to avoid time wasting.
In this post I want to share all the tricks I discovered that will make your Django tests run very quickly (at the time of this writing I have a suite of 250 tests that has been tuned to run in ~5 seconds rather than ~50 seconds of the untuned version… that’s 10 times faster!)
In this article I’m assuming that you know: how to run Django unit tests, how to use a different settings module for testing.
These are my tips:

1) Change password hashing

This is the most effective setting you can use to improve the speed of tests, it may sounds ridicolous, but password hashing in Django is designed to be very strong and it makes use of several “hashers”, but this also means that the hashing is very slow. The fastest hasher is the MD5PasswordHasher, so you can use just that one in this way:


2) Use a faster storage system

Currently the fastest database you can use with Django is SQLite. I was initially doubtful about switching from Postgres (my database of choice) to SQLite, but I’m not testing Django itself, I’m testing my own API and since I don’t use raw SQL statements the underlying storage backend should not make differences!
So, to configure SQLite:

    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'simple_test_db'

3) Remove unnecessary middlewares

Middleware layer is one of the Django features that I like the most, BUT the more middleware classes you use, the more your response time will increase (since all the middleware must be executed sequentially before returning the final HTTPResponse, ALWAYS!). So be sure to include only the stuff you really and absolutely need!
One middleware in particular that is very slow and that I removed for the tests is:


4) Remove unnecessary apps

There are several third party apps that you may remove during testing like the debug toolbar, try to remove all the unused/unnecessary apps.

5) Disable debugging

We don’t need extended traceback during tests, so we can disable debugging:

DEBUG = False

6) Turn off logging

This is a significant modification only if we have a huge amount of logging and/or additional logic involved in logs (such object inspections, heavy string manipulation and so on), but anyway logging is futile during testing, so:


7) Use a faster Email backend (by “patching” Django)

By default Django will use django.core.mail.backends.locmem.EmailBackend, which is an in-memory backend designed for testing, however I had several problems with that backend during my tests, they did block unexplainably for ~30 seconds due to headers checking. So I decided to write my own in-memory backend which mimics the Django one but does not check email headers in order to be blazing fast:

from django.core.mail.backends.base import BaseEmailBackend
from django.core import mail

class TestEmailBackend(BaseEmailBackend):

    def send_messages(self, messages):
        return len(messages)

In order to use this backend I had to use the @override_settings decorator, since defining it in settings file is helpless because that asshole of Django (1.5.x) will use django.core.mail.backends.locmem.EmailBackend anyway!
So, by following DRY phylosophy I created a subclass of Django’s TestCase which includes that overridden setting (so I don’t have to remember to define my own backend for each test):

from django.test import TestCase as DjangoTestCase
from django.test.utils import override_settings

class TestCase(DjangoTestCase):

8) Make use of bulk_create

If you are creating more objects don’t rely on save() for each object, but instead create objects in batch using bulk_create() (it’s a method provided by Django ORM to insert multiple records in a single query)

9) Use an in-memory backend for Celery

If you are using Celery, these are my optimal settings for testing:


That’s all! If you have other advices please let me know ;)

Configuring Django Pipeline by using Closure compiler for JavaScript files and YUI compressor for CSS

A couple of days ago I lost several hours trying to understand how to configure django-pipeline by replacing UglifyJS (the standard compressor library adopted) with Closure compiler and YUI compressor, which are in my opinion the two best tools to minify JavaScript and CSS respectively.
Fortunately Pipeline is pretty flexible and allows us to choose among different compressors and also write our own compressor, but I had to face some aspects that were initially unclear to me:

1. Understand how executable compressor bins are invoked by Pipeline:

In the documentation they say that you have to specify the absolute path to the bin of the compressor you are going to use… but, while UglifyJS ships with an executable bin, other compressors are usually simple jar files (with no bin), so pointing to that jars won’t work… so what? You have to write an executable yourself! But don’t panic it’s just a matter of writing a single line of bash script and make it executable :)
Let’s considering for example to adopt YUI compressor for CSS:
once downloaded the jar to a know directory in our filesystem, we can create a simple bin which will be used by pipeline by writing the following script:

java -jar "$TOOLS_PATH/yui_compressor-2.4.jar" $@

The script invokes the jar (java -jar) by passing all the arguments received from Pipeline (“$@” is the alien way for bash to accept a variable number of arguments… something like *args in Python).
$TOOLS_PATH is an environment variable I defined that contains the path to the jars directory (in order to avoid hard-coding it in the settings file).
After saving the script as “yui_compressor” (or whatever you like) you need to make it executable:

chmod +x yui_compressor

That’s all! Now we can define our CSS compression preferences in this way (in “settings.py” or whatever is your setting file… yes, if you didn’t know you can and should use multiple settings files in order to differentiate settings based on the environment you are running on… but this is another story for another post):

PIPELINE_CSS_COMPRESSOR = 'pipeline.compressors.yui.YUICompressor'
PIPELINE_YUI_BINARY = os.path.join(

(getEnvironmentVariable() is a custom helper I wrote to retrieve environemnt variables)

2. Make Closure compiler do its job without raising exceptions using AngularJS and other “sophisticated” JavaScript stuff

One of the good parts of Closure is that it has a lot of configurable options and specifically in order to make it happy while compiling my stuff I had to set “language_in” to ECMASCRIPT5 and “warning_level” to QUIET (according to the Google documentation and what I found on StackOverflow).
In order to pass these settings Pipeline provides an arguments option for each specific compiler, so I added:

PIPELINE_CLOSURE_ARGUMENTS = '--language_in=ECMASCRIPT5 --warning_level=QUIET'

If you don’t want to use “PIPELINE_CLOSURE_ARGUMENTS” you can also hard-code that flags in the exacutable you wrote, for example:

java -jar "$TOOLS_PATH/closure_compiler-20130823.jar --language_in=ECMASCRIPT5 --warning_level=QUIET " $@

But don’t do that!

End of post… happy compression at all!

How to disable keyboard navigation in jQuery UI tabs

This is a quick tip to disable the keyboard navigation introduced in the latest jQuery UI releases (which at the time of this writing is not configurable), but before to show the snippet I have to say that while jQuery is an helpful and powerful JavaScript library, jQuery UI is really sucky and that the team behind it should take it more seriously… I mean, it’s just ridiculous compared to other js UI libraries and further more them don’t care about retro-compatibility (I had problems on every update in the past!).
Ok, this is how I removed the keyboard navigation (it’s really a dirty way but better than what I found online and it’s working!):

    activate: function(e, ui) {

In practice in the “activate” callback I remove the focus from the tab and thus the listeners related to the keyboard are not being executed later.