[Answer]-Using scrapy command "crawl" from django

by

0👍

✅

Ok i have found the solution myself.

In settings.py I defined:

CRAWLER_PATH = os.path.join(os.path.dirname(BASE_DIR), 'required path')

And did the following.

from django.conf import settings
os.chdir(settings.CRAWLER_PATH)
👤Nabin

1👍

You don’t need to change the working directory, unless you want to use the .cfg which can include default options for the deploy command.

In your first approach you forgot to add the crawler path to the python path and set correctly the scrapy settings module:

# file: myapp/management/commands/bot.py
import os
import sys

from django.core.management.base import BaseCommand
from scrapy import cmdline


class Command(BaseCommand):
    help = "Run scrapy"

    def handle(self, *args, **options):
        sys.path.insert(0, '/home/user/mybot')
        os.environ['SCRAPY_SETTINGS_MODULE'] = 'mybot.settings'
        # Execute expects the list args[1:] to be the actual command arguments.
        cmdline.execute(['bot'] + list(args))
👤R. Max

Leave a comment