Speed-up easy-thumbnails on S3

I was developing a small app as a playground and confidence builder, choosing django, heroku and S3 as resources. One of the packages used is easy-thumbnails.

Behaviour

I chose to use the easy-thumbnails app with S3, a possibility granted by the storages framework. At a glance, everything works OK; thumbnails are created and rendered accordingly. However, on a thumbnail-heavy page, I (and my profiling app) noticed significant load times (3s for a page with 7 thumbnails).

Drilling down, it became evident that something was checking the S3 bucket for each thumbnail (a httpplib call occurred for each one).

Cause

TL;DR: If you use the default ImageField, the thumbnail storage is created by the Thumbnailer framework. Each time the storage object is created, it checks for the bucket’s existence. This means a query to S3. Lots of images equals lots of queries to S3, checking for a bucket’s existence.

Logic goes something like this:

  1. In thumbnail.py:
  2. In easy_thumbnails.files.py executing line 52 results in creating a new ThumbnailerFieldFile() object
  3. easy_thumbnails.fields.py has a ThumbnailerField() object created, which uses thumbnail_storage parameter. This is empty in the above call, which results later in creating the storge object
  4. In easy_thumbnails.files.py, the Thumbnailer class constructor contains:

Solution

I have thought of two options:

  1. Either tweak the easy_thumbnails.files.py, the Thumbnailer class constructor or
  2. Try to find a way not to re-create the storage object

I ended up using in my model something like:

Where the thumbnail_storage is initialised only once at the beginning of the .models file:

This way, the storage object is created only once per Django instance and the bucket existence is also queried only once.

Results

Now, a page query takes about 120ms on average instead of 3s. Wow. Much speedup.


A little experiment: If you find this post and ad below useful, please check the ad out :-)




7 thoughts on “Speed-up easy-thumbnails on S3

  1. We are currently struggling with the same issue and it seems that this solution is not working for us. We have easy-thumbnails 1.5.
    Even though thumbnail_storage is initialised in the beginning of the models file, the Thumbnailer class constructor still creates new thumbnail_storage each time it is called.

  2. We are currently struggling with the same issue and it seems that your solution is not working for us: the Thumbnailer class constructor still creates a new thumbnail_storage each time it is called.

    1. Do you use ThumbnailerImageField in your model? I’n my case it was ImageField that was causing the call. ThumbnailerImageField uses the thumbnail_storage which behaves like a singleton (one per django instance).

      Let me know if this helped.

      1. In our case the constructor of Thumbnailer class had thumbnail_storage always set to None.

        We eventually gave up on easy-thumbnail and switched to sorl-thumbnail instead. The package is extra tuned for work with remote data stores and does not result in dozens of SQL queries while checking if thumbnails exist or not.

        1. Thank you for pointing out sorl-thumbnail. I found that with the easy-thumbnail you really need to set thumbnail_storage manually. Otherwise you get the “initialise every time” effect :(

Leave a Reply to Sergei Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Scroll to top