[Django]-Are bots different from crawlers from python Django point of view

3👍

Scrapy is framework that can be used to create a bot or a crawler (aka spider). A crawler is specific kind of bot, but a bot isn’t necessarily a crawler. Crawlers are defined by being designed to explore the graph of pages (nodes) and their embedded URLs (edges) although they may be restricted from following particular URLs.

Automating tasks is the work of a bot. Whether Scrapy will work for that will depend on what information is needed and how actions have to be taken. Many sites are heavy on javascript these days, so if the bot can’t execute javascript and correctly provide cookies it may not be able to get the information to it’s task. Some web automation tasks may require a browser plug-in or even GUI automation tools may be needed.

Leave a comment