[Django]-Synchronous reading of a multipart upload alongside Django

by

5👍

✅

I believe a lot of these suggestions are overcomplicating things.

You need to change the way Django handles uploaded files? Simply modify the upload handler.

The base class is relatively straightforward, and gives you lots of great hooks. You should be able to extend it to do what you want.

👤Hamms

1👍

While I can’t write all the code for you (it’s complex), here is my recommended setup.

  1. Use Tornado + Django: Tornado can embed WSGI processes, so this gives you the ability to have a single process host both Django, and this one off Tornado handler. Here’s a quick sample from one of my active projects (though this using Tornado as a Socket.io handler, it should give you the gist of the solution (:

    # Socket Server
db = momoko.Pool(DB_DSN, **DB_CONFIG)
router   = tornadio2.TornadioRouter(QueryRouter, user_settings={'db':db})
sock_app = tornado.web.Application(router.urls, flash_policy_port = FL_PORT, flash_policy_file = path.join(PROJECT_ROOT, 'assets/xml/flashpolicy.xml'), socket_io_port = WS_PORT, debug=True)
# Django Server
os.environ['DJANGO_SETTINGS_MODULE'] = 'sever.settings'
application = django.core.handlers.wsgi.WSGIHandler()
container = tornado.wsgi.WSGIContainer(application)


# Start the web servers
if __name__ == "__main__":
    try:
        import logging
        tornado.options.parse_command_line()
        logging.getLogger().setLevel(logging.INFO)
        logging.info('Server started')
        tornado.locale.set_default_locale('us_US')
        http_server = tornado.httpserver.HTTPServer(container)
        http_server.listen(8000)
        tornadio2.SocketServer(sock_app, auto_start=True)
        tornado.ioloop.IOLoop.instance().start()
    except KeyboardInterrupt:
        tornado.ioloop.IOLoop.instance().stop()
        logging.info("Stopping servers.")

This could easily be converted to two server instances running on two different ports, with 80 being reserved for Django and 8080 being used for your upload handler.

  1. I’m recommending Tornado because it supports streaming request body, and is very well suited to this type of use. Here’s a gist that might help you.

  2. Your proxying setup will matter. If you’re using NGINX, make sure to turn proxy_buffering off.

  3. I wouldn’t use a database for the ticket/upload check. Redis or memcache would probably be a much faster way to handle this. A cache would also be great way to bass upload progress back and forth between Django and Tornado, since the overhead for setting/getting a new value would be so small.

This is a big hairy problem that will serious engineering to come up with something elegant, but it’s more than doable.

👤Jack Shedd

Leave a comment