2
Iโd start with integrating a normal task management process with your django project, see celery package:
Celery is a simple, flexible and reliable distributed system to
process vast amounts of messages, while providing operations with the
tools required to maintain such a system.
Also, see how to use celery with django.
Then, Iโd create a celery task where Scrapy
is invoked, see:
- Running Scrapy spiders in a Celery task
- Run Scrapy from a script
- How to run Scrapy from within a Python script
- Scrapy run from Python
Also see:
Hope that helps.
Source:stackexchange.com