scrapy run spider from script

It is simple and straightforward 🙂

Just check the official documentation. I would make there a little change so you could control the spider to run only when you do python and not every time you just import from it. Just add an if __name__ == "__main__":

import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider(scrapy.Spider):
    # Your spider definition

if __name__ == "__main__":
    process = CrawlerProcess({
        'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'

    process.start() # the script will block here until the crawling is finished

Now save the file as and run ‘python`.


Leave a Comment