asyncio

New in version 2.0.

Scrapy has partial support for asyncio. After you install the asyncio reactor, you may use asyncio and asyncio-powered libraries in any coroutine.

Warning

asyncio support in Scrapy is experimental, and not yet recommended for production environments. Future Scrapy versions may introduce related changes without a deprecation period or warning.

Installing the asyncio reactor

To enable asyncio support, set the TWISTED_REACTOR setting to 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'.

If you are using CrawlerRunner, you also need to install the AsyncioSelectorReactor reactor manually. You can do that using install_reactor():

install_reactor('twisted.internet.asyncioreactor.AsyncioSelectorReactor')

Using custom asyncio loops

You can also use custom asyncio event loops with the asyncio reactor. Set the ASYNCIO_EVENT_LOOP setting to the import path of the desired event loop class to use it instead of the default asyncio event loop.