[Django]-How to configure django-compressor and django-staticfiles with Amazon's S3?

by

10👍

✅

Your settings look correct. You should keep both STATICFILES_STORAGE and COMPRESS_STORAGE set to storage.CachedS3BotoStorage though and not switch back to storages.backends.s3boto.S3BotoStorage.

According to this django-compressor issue, the problem is with the way django-staticfiles saves during the collectstatic process (using shutil.copy2). This issue has been corrected in the newer version of django-staticfiles, which can be used instead of the one that ships with Django 1.3.

pip install django-staticfiles==dev

And in your settings.py, switch to the updated version:

STATICFILES_FINDERS = (
    #"django.contrib.staticfiles.finders.FileSystemFinder",
    #"django.contrib.staticfiles.finders.AppDirectoriesFinder",
    "staticfiles.finders.FileSystemFinder",
    "staticfiles.finders.AppDirectoriesFinder",
    "compressor.finders.CompressorFinder",
)

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    #'django.contrib.staticfiles',
    'staticfiles',
    #...
)

After running python manage.py collectstatic again, both the CACHE directory from django-compressor and the collected staticfiles files should show up on S3.

👤Tyler Spencewood

0👍

Using django_compressor==1.2 worked for me. I am not sure why you need to install django-staticfiles however all the versions of django_compressor except 1.2 has that issue.

👤Andrii Zarubin

0👍

After plenty of days of hard work and research I was finally able to do this and I decided to write a detailed guide about it, including how to also serve them zipped with gzip.

Basically you need to do a few things:

  1. Use AWS_IS_GZIPPED = True
  2. If your S3 is outside of US. You need to create a custom S3Connection class where you override the DefaultHost variable to your S3 url. Example s3-eu-west-1.amazonaws.com
  3. If you’re using a dotted bucket name, example subdomain.domain.tld. You need to set AWS_S3_CALLING_FORMAT = 'boto.s3.connection.OrdinaryCallingFormat'
  4. You have to set non_gzipped_file_content = content.file in your CachedS3BotoStorage

This is the CachedS3BotoStorage class you need:

class CachedS3BotoStorage(S3BotoStorage):
    """
    S3 storage backend that saves the files locally, too.
    """
    connection_class = EUConnection
    location = settings.STATICFILES_LOCATION
    def __init__(self, *args, **kwargs):
        super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
        self.local_storage = get_storage_class(
            "compressor.storage.CompressorFileStorage")()

def save(self, name, content):
    non_gzipped_file_content = content.file
    name = super(CachedS3BotoStorage, self).save(name, content)
    content.file = non_gzipped_file_content
    self.local_storage._save(name, content)
    return name

Note that EUConnection is a custom class where I set DefaultHost to my S3 location. Check the much longer and detailed guide for complete custom storages and settings.py

👤Marcus Lind

0👍

Try this post that complete the above solution with some lines, to fix the problem that create many (multiples) manifest_%.json in Amazon S3.
https://stackoverflow.com/a/31545361/1359475

👤Mateus Padua

Leave a comment