Signals¶
Scrapy uses signals extensively to notify when certain events occur. You can catch some of those signals in your Scrapy project (using an extension, for example) to perform additional tasks or extend Scrapy to add functionality not provided out of the box.
Even though signals provide several arguments, the handlers that catch them don’t need to accept all of them - the signal dispatching mechanism will only deliver the arguments that the handler receives.
You can connect to signals (or send your own) through the Signals API.
Here is a simple example showing how you can catch signals and perform some action:
from scrapy import signals
from scrapy import Spider
class DmozSpider(Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
]
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
return spider
def spider_closed(self, spider):
spider.logger.info('Spider closed: %s', spider.name)
def parse(self, response):
pass
Deferred signal handlers¶
Some signals support returning Deferred
objects from their handlers, allowing you to run asynchronous code that
does not block Scrapy. If a signal handler returns a
Deferred
, Scrapy waits for that
Deferred
to fire.
Let’s take an example:
class SignalSpider(scrapy.Spider):
name = 'signals'
start_urls = ['http://quotes.toscrape.com/page/1/']
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super(SignalSpider, cls).from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.item_scraped, signal=signals.item_scraped)
return spider
def item_scraped(self, item):
# Send the scraped item to the server
d = treq.post(
'http://example.com/post',
json.dumps(item).encode('ascii'),
headers={b'Content-Type': [b'application/json']}
)
# The next item will be scraped only after
# deferred (d) is fired
return d
def parse(self, response):
for quote in response.css('div.quote'):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('small.author::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
}
See the Built-in signals reference below to know which signals support
Deferred
.
Built-in signals reference¶
Here’s the list of Scrapy built-in signals and their meaning.
Engine signals¶
engine_started¶
- scrapy.signals.engine_started()¶
Sent when the Scrapy engine has started crawling.
This signal supports returning deferreds from its handlers.
Note
This signal may be fired after the spider_opened
signal,
depending on how the spider was started. So don’t rely on this signal
getting fired before spider_opened
.
engine_stopped¶
- scrapy.signals.engine_stopped()¶
Sent when the Scrapy engine is stopped (for example, when a crawling process has finished).
This signal supports returning deferreds from its handlers.
Item signals¶
Note
As at max CONCURRENT_ITEMS
items are processed in
parallel, many deferreds are fired together using
DeferredList
. Hence the next
batch waits for the DeferredList
to fire and then runs the respective item signal handler for
the next batch of scraped items.
item_scraped¶
- scrapy.signals.item_scraped(item, response, spider)¶
Sent when an item has been scraped, after it has passed all the Item Pipeline stages (without being dropped).
This signal supports returning deferreds from its handlers.
- Parameters
item (item object) – the scraped item
spider (
Spider
object) – the spider which scraped the itemresponse (
Response
object) – the response from where the item was scraped
item_dropped¶
- scrapy.signals.item_dropped(item, response, exception, spider)¶
Sent after an item has been dropped from the Item Pipeline when some stage raised a
DropItem
exception.This signal supports returning deferreds from its handlers.
- Parameters
item (item object) – the item dropped from the Item Pipeline
spider (
Spider
object) – the spider which scraped the itemresponse (
Response
object) – the response from where the item was droppedexception (
DropItem
exception) – the exception (which must be aDropItem
subclass) which caused the item to be dropped
item_error¶
- scrapy.signals.item_error(item, response, spider, failure)¶
Sent when a Item Pipeline generates an error (i.e. raises an exception), except
DropItem
exception.This signal supports returning deferreds from its handlers.
- Parameters
item (item object) – the item that caused the error in the Item Pipeline
response (
Response
object) – the response being processed when the exception was raisedspider (
Spider
object) – the spider which raised the exceptionfailure (twisted.python.failure.Failure) – the exception raised
Spider signals¶
spider_closed¶
- scrapy.signals.spider_closed(spider, reason)¶
Sent after a spider has been closed. This can be used to release per-spider resources reserved on
spider_opened
.This signal supports returning deferreds from its handlers.
- Parameters
spider (
Spider
object) – the spider which has been closedreason (str) – a string which describes the reason why the spider was closed. If it was closed because the spider has completed scraping, the reason is
'finished'
. Otherwise, if the spider was manually closed by calling theclose_spider
engine method, then the reason is the one passed in thereason
argument of that method (which defaults to'cancelled'
). If the engine was shutdown (for example, by hitting Ctrl-C to stop it) the reason will be'shutdown'
.
spider_opened¶
- scrapy.signals.spider_opened(spider)¶
Sent after a spider has been opened for crawling. This is typically used to reserve per-spider resources, but can be used for any task that needs to be performed when a spider is opened.
This signal supports returning deferreds from its handlers.
- Parameters
spider (
Spider
object) – the spider which has been opened
spider_idle¶
- scrapy.signals.spider_idle(spider)¶
Sent when a spider has gone idle, which means the spider has no further:
requests waiting to be downloaded
requests scheduled
items being processed in the item pipeline
If the idle state persists after all handlers of this signal have finished, the engine starts closing the spider. After the spider has finished closing, the
spider_closed
signal is sent.You may raise a
DontCloseSpider
exception to prevent the spider from being closed.This signal does not support returning deferreds from its handlers.
- Parameters
spider (
Spider
object) – the spider which has gone idle
Note
Scheduling some requests in your spider_idle
handler does
not guarantee that it can prevent the spider from being closed,
although it sometimes can. That’s because the spider may still remain idle
if all the scheduled requests are rejected by the scheduler (e.g. filtered
due to duplication).
spider_error¶
- scrapy.signals.spider_error(failure, response, spider)¶
Sent when a spider callback generates an error (i.e. raises an exception).
This signal does not support returning deferreds from its handlers.
Request signals¶
request_scheduled¶
request_dropped¶
request_reached_downloader¶
request_left_downloader¶
bytes_received¶
New in version 2.2.
- scrapy.signals.bytes_received(data, request, spider)¶
Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is received for a specific request. This signal might be fired multiple times for the same request, with partial data each time. For instance, a possible scenario for a 25 kb response would be two signals fired with 10 kb of data, and a final one with 5 kb of data.
This signal does not support returning deferreds from its handlers.
Note
Handlers of this signal can stop the download of a response while it
is in progress by raising the StopDownload
exception. Please refer to the Stopping the download of a Response topic
for additional information and examples.
Response signals¶
response_received¶
- scrapy.signals.response_received(response, request, spider)¶
Sent when the engine receives a new
Response
from the downloader.This signal does not support returning deferreds from its handlers.
Note
The request
argument might not contain the original request that
reached the downloader, if a Downloader Middleware modifies
the Response
object and sets a specific request
attribute.
response_downloaded¶
- scrapy.signals.response_downloaded(response, request, spider)¶
Sent by the downloader right after a
HTTPResponse
is downloaded.This signal does not support returning deferreds from its handlers.