Expiration#

By default, cached responses will be stored indefinitely. There are a number of options for specifying how long to store responses, either with a single expiration value, glob patterns, or cache headers.

The simplest option is to initialize the cache with an expire_after value, which will apply to all reponses:

>>> # Set expiration for the session using a value in seconds
>>> session = CachedSession(expire_after=360)

Expiration Precedence#

Expiration can be set on a per-session, per-URL, or per-request basis, in addition to cache headers (see sections below for usage details). When there are multiple values provided for a given request, the following order of precedence is used:

Cache-Control response headers (if enabled)
Cache-Control request headers
Per-request expiration (expire_after argument for CachedSession.request())
Per-URL expiration (urls_expire_after argument for CachedSession)
Per-session expiration (expire_after argument for CachedSession)

Expiration Values#

expire_after can be any of the following time values:

A positive number (in seconds)
A timedelta
A datetime

Or one of the following special values:

DO_NOT_CACHE: Skip both reading from and writing to the cache
EXPIRE_IMMEDIATELY: Consider the response already expired, but potentially usable
NEVER_EXPIRE: Store responses indefinitely

Note

A value of 0 or EXPIRE_IMMEDIATELY will behave the same as Cache-Control: max-age=0. Depending on other settings and headers, an expired response may either be cached and require revalidation for each use, or not be cached at all. See Conditional Requests for more details.

Examples:

>>> from datetime import timedelta
>>> from requests_cache import DO_NOT_CACHE, NEVER_EXPIRE, EXPIRE_IMMEDIATELY, CachedSession

>>> # Specify a simple expiration value in seconds
>>> session = CachedSession(expire_after=60)

>>> # To specify a unit of time other than seconds, use a timedelta
>>> session = CachedSession(expire_after=timedelta(days=30))

>>> # Or expire on a specific date and time
>>> session = CachedSession(expire_after=datetime(2023, 1, 1, 0, 0))

>>> # Update an existing session to store new responses indefinitely
>>> session.settings.expire_after = NEVER_EXPIRE

>>> # Disable caching by default, unless enabled by other settings
>>> session = CachedSession(expire_after=DO_NOT_CACHE)

>>> # Override for a single request: cache the response if it can be revalidated
>>> session.request(expire_after=EXPIRE_IMMEDIATELY)

>>> urls_expire_after = {
...     '*.site_1.com': 30,
...     'site_2.com/resource_1': 60 * 2,
...     'site_2.com/resource_2': 60 * 60 * 24,
...     re.compile(r'site_2.com/resource_\d'): 60 * 60 * 24 * 7,
...     'site_2.com/resource_*': 60 * 60,
...     'site_2.com/static': NEVER_EXPIRE,
... }
>>> session = CachedSession(urls_expire_after=urls_expire_after)

Notes:

urls_expire_after should be a dict in the format {pattern': expire_after}
expire_after accepts the same types as CachedSession.settings.expire_after
Glob patterns will match request base URLs without the protocol, so the pattern site.com/resource/ is equivalent to http*://site.com/resource/**. For regex patterns, the whole URL will be matched, so you can put restrictions on the protocol, e.g. re.compile(r'https://site.com/.*').
If there is more than one match, the first match will be used in the order they are defined
If no patterns match a request, CachedSession.settings.expire_after will be used as a default
See Filter by URLs for an example of using urls_expire_after as an allowlist

Expiration and Error Handling#

In some cases, you might cache a response, have it expire, but then encounter an error when retrieving a new response. If you would like to use expired response data in these cases, use the stale_if_error option.

For example:

>>> # Cache a test response and wait until it's expired
>>> session = CachedSession(stale_if_error=True)
>>> session.get('https://httpbin.org/get', expire_after=1)
>>> time.sleep(1)

Afterward, let’s say the page has moved and you get a 404, or the site is experiencing downtime and you get a 500. You will then get the expired cache data instead:

>>> response = session.get('https://httpbin.org/get')
>>> print(response.from_cache, response.is_expired)
True, True

Similar to the header Cache-Control: stale-if-error, you may also pass time value representing the maximum staleness you are willing to accept:

# If there is an error on refresh, use a cached response if it expired 5 minutes ago or less
session = CachedSession(stale_if_error=timedelta(minutes=5))

In addition to HTTP error codes, stale_if_error also applies to python exceptions (typically a RequestException). See requests documentation on Errors and Exceptions for more details on request errors in general.

Asynchronous Revalidation#

You can use the stale_while_revalidate option to improve performance when refreshing responses. This will cause an expired cached response to be returned initially, while a non-blocking request is sent to refresh the response for the next time it’s requested.

Note

While the corresponding response header Cache-Control: stale-while-revalidate only applies to Conditional Requests, requests-cache extends this behavior to other refresh requests as well (even if a validator is not available).

You may either set this to True to do this regardless of the cached response’s age:

session = CachedSession(stale_while_revalidate=True)

Or specify a maximum staleness value you are willing to accept:

# Use a cached response while revalidating, if it expired 5 minutes ago or less
session = CachedSession(stale_while_revalidate=timedelta(minutes=5))

Removing Responses#

For better read performance, expired responses won’t be removed immediately by default. Instead, they will be replaced the next time they are requested.

You can manually delete responses according to various conditions, and some backends support automatic removal.

Manual Removal#

To delete all cached responses, use BaseCache.clear():

>>> session.cache.clear()

To delete expired responses, use BaseCache.delete():

>>> session.cache.delete(expired=True)

Or, if you have patched requests using install_cache():

>>> import requests_cache
>>> requests_cache.delete(expired=True)

You can also remove responses older than a certain time:

# Remove responses older than 7 days
session.cache.delete(older_than=timedelta(days=7))

Or apply a new expiration value to previously cached responses:

# Reset expiration for all responses to 30 days from now
>>> session.cache.reset_expiration(timedelta(days=30))

Finally, you can delete individual responses matching specific requests or cache keys:

>>> from requests import Request

# Delete a simple GET request by URL
>>> session.cache.delete(urls=['https://httpbin.org/json'])

# Delete by additional request values
>>> request_1 = Request('GET', 'https://httpbin.org/get', params={'key': 'value'})
>>> request_2 = Request('GET', 'https://httpbin.org/get', headers={'header': 'value'})
>>> session.cache.delete(requests=[request_1, request_2])

# Delete by cache key
>>> session.cache.delete('e25f7e6326966e82')

Automatic Removal#

The following backends have native TTL support, which can be used to automatically remove expired responses:

Request Options#

In addition to the base arguments for requests.request(), requests-cache adds some extra cache-related arguments. These apply to CachedSession.request(), CachedSession.send(), and all HTTP method-specific functions (get(), post(), etc.).

Per-Request Expiration#

The expire_after argument can be used to override the session’s expiration for a single request.

>>> session = CachedSession(expire_after=300)
>>> # This request will be cached for 60 seconds, not 300
>>> session.get('https://httpbin.org/get', expire_after=60)

Manual Refresh#

If you want to manually refresh a response before it expires, you can use the refresh argument.

This is equivalent to F5 in most browsers.
The response will be saved with a new expiration time, according to the normal expiration rules described above.
If possible, this will revalidate with the server to potentially avoid re-downloading an unchanged response.
To force a refresh (e.g., skip revalidation and always send a new request), use the force_refresh argument. This is equivalent to Ctrl-F5 in most browsers.

Example:

>>> response_1 = session.get('https://httpbin.org/get')
>>> response_2 = session.get('https://httpbin.org/get', refresh=True)
>>> assert response_2.from_cache is False

Validation-Only Requests#

If you want to always send a conditional request before using a cached response, you can use the session setting always_revalidate:

>>> session = CachedSession(always_revalidate=True)

Unlike the refresh option, this only affects cached responses with a validator.

Cache-Only Requests#

If you want to only use cached responses without making any real requests, you can use the only_if_cached option. This essentially uses your cache in “offline mode”. If a response isn’t cached or is expired, you will get a 504 Not Cached response instead.

>>> session = CachedSession()
>>> session.cache.clear()
>>> response = session.get('https://httpbin.org/get', only_if_cached=True)
>>> print(response.status_code)
504
>>> response.raise_for_status()
HTTPError: 504 Server Error: Not Cached for url: https://httpbin.org/get

You can also combine this with stale_if_error to return cached responses even if they are expired.

>>> session = CachedSession(expire_after=1, stale_if_error=True)
>>> session.get('https://httpbin.org/get')
>>> time.sleep(1)

>>> # The response will be cached but expired by this point
>>> response = session.get('https://httpbin.org/get', only_if_cached=True)
>>> print(response.status_code)
200