Testing Flask subdomain routing locally

I’m working on a project in which each customer gets a subdomain for his personal area and thanks to Flask it’s just a matter of using “subdomain” params in routing config. The tricky part is how to setup Flask and the /etc/hosts file in order to make it working on a local development machine.
So the first step is to map 127.0.0.1 to a custom domain name, and the same for a custom subdomain name:

127.0.0.1 localwebsite
127.0.0.1 peter.localwebsite

Of course you can choose any name you like, the only important thing is that the main “fake domain” must match in the “fake subdomain” mapping! (in the case above “localwebsite”)

Then in Flask you have to specify the server name and its port (required!):

app.config['SERVER_NAME'] = 'localwebsite:8080'

In the above scenario I’m mapping the previously “fake domain” by specifying the port on which the Flask server is running (which in my development settings is 8080 and should be 5000 if not specifically defined).
Finally we can register the routes:

@app.route('/', subdomain='<customer>')
def customer_subdomain(customer):
    return 'Customer is: {}'.format(customer)

and if we call peter.localwebsite:8080 it will return “Customer is: peter” as a response! (If we point to localwebsite:8080 the default home page view will be used as expected)

Python: reading numbers from JSON without loss of precision using Decimal class for data processing

In the project I’m working on, I’m using an external API which returns a JSON response containing conversion rates for currencies. Since I’m dealing with currencies and prices, the precision of numbers plays an important rule in order to calculate values in the application. The good thing about JSON, despite its name is the acronym of “JavaScript Object Notation“, is that it’s a cross-language format, so it’s not limited to the capabilities of a specific language like JavaScript, so numbers in in JSON may have an higher precision than a js float!
This is a quote from wikipedia about JSON numbers (emphasis is mine):

Number — a signed decimal number that may contain a fractional part and may use exponential E notation. JSON does not allow non-numbers like NaN, nor does it make any distinction between integer and floating-point. (Even though JavaScript uses a double-precision floating-point format for all its numeric values, other languages implementing JSON may encode numbers differently)

By default Python’s json module will loads decimal numbers as float, so if we have a JSON like:

{ "number": 1.00000000000000000001 }

the default conversion into python will be {u'number': 1.0} if we just write the following code:

import json

json.loads(json_string)

But, fortunately is dead simple to load numbers in JSON using the decimal module, there is no need to write custom decoders as I saw on the web, it’s just a matter of specify the Decimal class for floats parsing in the loads() function in this way:

import json
from decimal import Decimal

json.loads(json_string, parse_float=Decimal)

In this way the loaded python object will be:
{u'number': Decimal('1.00000000000000000001')}
And we will be able to perform precise arithmetic computations!
It’s also possible to use Decimal even for integer numbers, by specifying parse_int:

json.loads(json_string, 
           parse_int=Decimal, 
           parse_float=Decimal)

Additional reading from official Python docs:

Creating class based view decorators using simple Django function decorators

I love decorators because they allow to extend class/functions capabilities and to be be added and removed easily. Django offers a set of decorators but unfortunely they are designed for functions or class methods only and instead I want to add them to the whole class in order to avoid to override the orignal method just to add the decorator annotation above it. Fortunately wrapping the simple Django decorators in order to use them against classes (basically View classes) is quite simple and I’m gonna show you what I done.
For example let’s consider we have a view that require a POST method. If we use a simple function view we can write the following:

@require_POST
def myView(request):
	return HttpResponse('hello world!')

but, if we use class views (and we should!), we are forced to write:

class MyView(SomeKindOfParentClassView):
	# ...imagine several methods here

	@require_POST
	def dispatch(self, request, *args, **kwargs):
		return HttpResponse('hello world!')

wouldn’t be cleaner and more elegant instead to simply write the following?

@require_POST
class MyView(SomeKindOfParentClassView):
	# ...imagine several methods here

…and this is the class decorator implementation:

from django.views.decorators.http import require_POST as post_only

def require_POST(View):
    View.dispatch = method_decorator(post_only)(View.dispatch)
    return View

Basically I redefined the class method and returned the class itself.
This was really simple since the concrete implementation of the decorator has been added using the existend Django decorators… but what if we want to create a custom decorator? For example in my app I created a decorator that enable access to a view only if it’s requested via ajax, this is how I implemented it:

def require_AJAX(SomeKindOfParentClassView):
    def ajaxOnly(function):
        def wrap(request, *args, **kwargs):
            if not request.is_ajax():
                return HttpResponseForbidden()
            return function(request, *args, **kwargs)

        return wrap

    View.dispatch = method_decorator(ajaxOnly)(View.dispatch)
    return View

And the decorated view looks like this:

@require_POST
@require_AJAX
class MyView(SomeKindOfParentClassView):
    # ...imagine several methods here

As you can see that annotations are really readable, immediate and self-describing.

Effective TDD: Tricks to speed up Django tests (up to 10x faster!)

TDD (Test Driven Development) is one of the best practices to follow in order to write better software and once you get used to this practice you can’t write code without writing tests.
In TDD, tests must be executed very frequently and so it’s fundamental that they run as fast as possible in order to avoid time wasting.
In this post I want to share all the tricks I discovered that will make your Django tests run very quickly (at the time of this writing I have a suite of 250 tests that has been tuned to run in ~5 seconds rather than ~50 seconds of the untuned version… that’s 10 times faster!)
In this article I’m assuming that you know: how to run Django unit tests, how to use a different settings module for testing.
These are my tips:

1) Change password hashing

This is the most effective setting you can use to improve the speed of tests, it may sounds ridicolous, but password hashing in Django is designed to be very strong and it makes use of several “hashers”, but this also means that the hashing is very slow. The fastest hasher is the MD5PasswordHasher, so you can use just that one in this way:

PASSWORD_HASHERS = (
    'django.contrib.auth.hashers.MD5PasswordHasher',
)

2) Use a faster storage system

Currently the fastest database you can use with Django is SQLite. I was initially doubtful about switching from Postgres (my database of choice) to SQLite, but I’m not testing Django itself, I’m testing my own API and since I don’t use raw SQL statements the underlying storage backend should not make differences!
So, to configure SQLite:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'simple_test_db'
    }
}

3) Remove unnecessary middlewares

Middleware layer is one of the Django features that I like the most, BUT the more middleware classes you use, the more your response time will increase (since all the middleware must be executed sequentially before returning the final HTTPResponse, ALWAYS!). So be sure to include only the stuff you really and absolutely need!
One middleware in particular that is very slow and that I removed for the tests is:

django.middleware.locale.LocaleMiddleware

4) Remove unnecessary apps

There are several third party apps that you may remove during testing like the debug toolbar, try to remove all the unused/unnecessary apps.

5) Disable debugging

We don’t need extended traceback during tests, so we can disable debugging:

DEBUG = False
TEMPLATE_DEBUG = False

6) Turn off logging

This is a significant modification only if we have a huge amount of logging and/or additional logic involved in logs (such object inspections, heavy string manipulation and so on), but anyway logging is futile during testing, so:

logging.disable(logging.CRITICAL)

7) Use a faster Email backend (by “patching” Django)

By default Django will use django.core.mail.backends.locmem.EmailBackend, which is an in-memory backend designed for testing, however I had several problems with that backend during my tests, they did block unexplainably for ~30 seconds due to headers checking. So I decided to write my own in-memory backend which mimics the Django one but does not check email headers in order to be blazing fast:

from django.core.mail.backends.base import BaseEmailBackend
from django.core import mail

class TestEmailBackend(BaseEmailBackend):

    def send_messages(self, messages):
        mail.outbox.extend(messages)
        return len(messages)

In order to use this backend I had to use the @override_settings decorator, since defining it in settings file is helpless because that asshole of Django (1.5.x) will use django.core.mail.backends.locmem.EmailBackend anyway!
So, by following DRY phylosophy I created a subclass of Django’s TestCase which includes that overridden setting (so I don’t have to remember to define my own backend for each test):

from django.test import TestCase as DjangoTestCase
from django.test.utils import override_settings

@override_settings(EMAIL_BACKEND='myapp.TestEmailBackend')
class TestCase(DjangoTestCase):
    pass

8) Make use of bulk_create

If you are creating more objects don’t rely on save() for each object, but instead create objects in batch using bulk_create() (it’s a method provided by Django ORM to insert multiple records in a single query)

9) Use an in-memory backend for Celery

If you are using Celery, these are my optimal settings for testing:

CELERY_ALWAYS_EAGER = True
CELERY_EAGER_PROPAGATES_EXCEPTIONS = True
BROKER_BACKEND = 'memory'

That’s all! If you have other advices please let me know ;)

Check Django messages (quantity and type) in unit tests

Ok, in this short (but I hope very helpful post), I’m gonna show how to check messages (Django messages framework) in your tests.
I don’t know if there is a better or “normalized” way to do this, but it’s the simplest and effective way I found by myself… so, let’s suppose we have a test in which we want to check that if the submitted form data is invalid, an error message is sent back with the response context:

def testIfSubmittedDataIsInvalidAnErrorMessageIsReturned(self):
    response = self.client.post(self.url, {})
    messages = list(response.context['messages'])
    self.assertEqual(len(messages), 1)
    self.assertEqual(messages[0].level, MSG.ERROR)

MSG is a reference to messages levels constants imported in this way:

from django.contrib.messages import constants as MSG