Serializers

Some alternative serializers are included, mainly intended for use with FileCache.

Note

Some of these serializers require additional dependencies, listed in the sections below.

Specifying a Serializer

Similar to Backends, you can specify which serializer to use with the serializer parameter for either CachedSession or install_cache().

Built-in Serializers

JSON Serializer

Storing responses as JSON gives you the benefit of making them human-readable and editable, in exchange for a minor reduction in read and write speeds.

Usage:

>>> session = CachedSession('my_cache', serializer='json')

Example JSON-serialized Response (with decoded JSON content)

{
  "url": "https://httpbin.org/json",
  "status_code": 200,
  "reason": "OK",
  "_decoded_content": {
    "slideshow": {
      "author": "Yours Truly",
      "date": "date of publication",
      "slides": [
        {
          "title": "Wake up to WonderWidgets!",
          "type": "all"
        },
        {
          "items": [
            "Why <em>WonderWidgets<\/em> are great",
            "Who <em>buys<\/em> WonderWidgets"
          ],
          "title": "Overview",
          "type": "all"
        }
      ],
      "title": "Sample Slide Show"
    }
  },
  "cache_key": "4dc151d95200ec91fa77021989f5194e9be47e87f8f228306f3a8d5434b9e547",
  "created_at": "2021-07-21T22:34:50.343095",
  "elapsed": 0.242198,
  "encoding": "utf-8",
  "headers": {
    "Date": "Wed, 21 Jul 2021 22:34:50 GMT",
    "Content-Type": "application/json",
    "Content-Length": "429",
    "Connection": "keep-alive",
    "Server": "gunicorn/19.9.0",
    "Access-Control-Allow-Origin": "*",
    "Access-Control-Allow-Credentials": "true"
  },
  "request": {
    "body": "PH%2y",
    "headers": {
      "User-Agent": "python-requests/2.26.0",
      "Accept-Encoding": "gzip, deflate",
      "Accept": "*/*",
      "Connection": "keep-alive"
    },
    "method": "GET",
    "url": "https://httpbin.org/json"
  },
  "raw": {
    "decode_content": false,
    "reason": "OK",
    "status": 200,
    "version": 11
  }
}

Example JSON-serialized Response (with binary content)

{
  "url": "https://httpbin.org/get",
  "status_code": 200,
  "reason": "OK",
  "_content": "dkP>RB4Ki8b0Rt*dwnb*3LqdNXk}q!WpZ;OIv{%rARr(hB0*zgWpH#NIv^q{FDfD|APOKLARr<^V`F7-bS*`0V{c?>Zf7DoAR=daX>cqcWMyV-VRU68EFcOXARr(jNN;m=B03-<XmoUNVrgzJZ*pfMEFcOXARr(jRdZ!>EkS2xZge6#AR=&ibZBpGEplaXb!BsOb1yP3GBz$SA}k;ZARr(hB3La!ZF+7kRB~ZsWi3f$B03-<Qg3f`JuxjdFlIPmF)(2*GB7hXGcjZ_Gh<>hGc{o{G%#g2VK_22A_^cNeJmgfARr=da%pF2ZX!A$A~82JE-^VSF*GqQGBq+HEFcOXAR={gY$7@!B4~7UaC15@FKBdhaAIk0E^l&YFK1<RA_{#9",
  "cache_key": "4dc151d95200ec91fa77021989f5194e9be47e87f8f228306f3a8d5434b9e547",
  "created_at": "2021-07-21T22:34:50.343095",
  "elapsed": 0.242198,
  "encoding": "utf-8",
  "headers": {
    "Date": "Wed, 21 Jul 2021 22:34:50 GMT",
    "Content-Type": "application/json",
    "Content-Length": "308",
    "Connection": "keep-alive",
    "Server": "gunicorn/19.9.0",
    "Access-Control-Allow-Origin": "*",
    "Access-Control-Allow-Credentials": "true"
  },
  "request": {
    "body": "PH%2y",
    "headers": {
      "User-Agent": "python-requests/2.26.0",
      "Accept-Encoding": "gzip, deflate",
      "Accept": "*/*",
      "Connection": "keep-alive"
    },
    "method": "GET",
    "url": "https://httpbin.org/get"
  },
  "raw": {
    "decode_content": false,
    "reason": "OK",
    "status": 200,
    "version": 11
  }
}

This will use ultrajson if installed, otherwise the stdlib json module will be used. You can install the optional dependencies for this serializer with:

pip install requests-cache[json]

YAML Serializer

YAML is another option if you need a human-readable/editable format, with the same tradeoffs as JSON.

Usage:

>>> session = CachedSession('my_cache', serializer='yaml')

Example YAML-serialized Response (with decoded JSON content)

url: https://httpbin.org/json
status_code: 200
reason: OK
_decoded_content:
  slideshow:
    author: Yours Truly
    date: date of publication
    slides:
    - title: Wake up to WonderWidgets!
      type: all
    - items:
      - Why <em>WonderWidgets</em> are great
      - Who <em>buys</em> WonderWidgets
      title: Overview
      type: all
    title: Sample Slide Show
created_at: '2021-07-21T22:32:17.592974'
elapsed: 0.187586
encoding: utf-8
headers:
  Access-Control-Allow-Credentials: 'true'
  Access-Control-Allow-Origin: '*'
  Connection: keep-alive
  Content-Length: '308'
  Content-Type: application/json
  Date: Wed, 21 Jul 2021 22:32:17 GMT
  Server: gunicorn/19.9.0
request:
  method: GET
  url: https://httpbin.org/json
  body: !!binary |
    Tm9uZQ==
  headers:
    Accept: '*/*'
    Accept-Encoding: gzip, deflate
    Connection: keep-alive
    User-Agent: python-requests/2.26.0
raw:
  decode_content: false
  reason: OK
  status: 200
  version: 11

Example YAML-serialized Response (with binary content)

url: https://httpbin.org/get
status_code: 200
reason: OK
_content: !!binary |
  ewogICJhcmdzIjoge30sIAogICJoZWFkZXJzIjogewogICAgIkFjY2VwdCI6ICIqLyoiLCAKICAg
  ICJBY2NlcHQtRW5jb2RpbmciOiAiZ3ppcCwgZGVmbGF0ZSIsIAogICAgIkhvc3QiOiAiaHR0cGJp
  bi5vcmciLCAKICAgICJVc2VyLUFnZW50IjogInB5dGhvbi1yZXF1ZXN0cy8yLjI2LjAiLCAKICAg
  ICJYLUFtem4tVHJhY2UtSWQiOiAiUm9vdD0xLTYwZjhhMDcxLTBiN2JmN2VjNGMyZTdmNjA2YWI4
  ZDYyNCIKICB9LCAKICAib3JpZ2luIjogIjE3My4xOS4xNDEuMjUyIiwgCiAgInVybCI6ICJodHRw
  czovL2h0dHBiaW4ub3JnL2dldCIKfQo=
cache_key: 4dc151d95200ec91fa77021989f5194e9be47e87f8f228306f3a8d5434b9e547
created_at: '2021-07-21T22:32:17.592974'
elapsed: 0.187586
encoding: utf-8
headers:
  Access-Control-Allow-Credentials: 'true'
  Access-Control-Allow-Origin: '*'
  Connection: keep-alive
  Content-Length: '308'
  Content-Type: application/json
  Date: Wed, 21 Jul 2021 22:32:17 GMT
  Server: gunicorn/19.9.0
request:
  method: GET
  url: https://httpbin.org/get
  body: !!binary |
    Tm9uZQ==
  headers:
    Accept: '*/*'
    Accept-Encoding: gzip, deflate
    Connection: keep-alive
    User-Agent: python-requests/2.26.0
raw:
  decode_content: false
  reason: OK
  status: 200
  version: 11

You can install the extra dependencies for this serializer with:

pip install requests-cache[yaml]

BSON Serializer

BSON is a serialization format originally created for MongoDB, but it can also be used independently. Compared to JSON, it has better performance (although still not as fast as pickle), and adds support for additional data types. It is not human-readable, but some tools support reading and editing it directly (for example, bson-converter for Atom).

Usage:

>>> session = CachedSession('my_cache', serializer='bson')

You can install the extra dependencies for this serializer with:

pip install requests-cache[mongo]

Or if you would like to use the standalone BSON codec for a different backend, without installing MongoDB dependencies:

pip install requests-cache[bson]

Response Content Format

By default, any JSON or text response body will be decoded, so the response is fully human-readable/editable. Other content types will be saved as binary data. To save all content as binary, set decode_content=False:

>>> backend = FileCache(decode_content=False)
>>> session = CachedSession('http_cache', backend=backend)

Serializer Security

See Security for recommended setup steps for more secure cache serialization, particularly when using pickle.

Custom Serializers

If the built-in serializers don’t suit your needs, you can create your own. For example, if you had a imaginary custom_pickle module that provides dumps and loads functions:

>>> import custom_pickle
>>> from requests_cache import CachedSession
>>> session = CachedSession(serializer=custom_pickle)

Serializer Pipelines

More complex serialization can be done with SerializerPipeline. Use cases include text-based serialization, compression, encryption, and any other intermediate steps you might want to add.

Any combination of these can be composed with a SerializerPipeline, which starts with a CachedResponse and ends with a str or bytes object. Each stage of the pipeline can be any object or module with dumps and loads functions. If the object has similar methods with different names (e.g. compress / decompress), those can be aliased using Stage.

For example, a compressed pickle serializer can be built as:

Example

>>> import gzip
>>> from requests_cache import CachedSession, SerializerPipeline, Stage, pickle_serializer
>>> compressed_serializer = SerializerPipeline(
...     [
...         pickle_serializer,
...         Stage(dumps=gzip.compress, loads=gzip.decompress),
...     ],
...     is_binary=True,
... )
>>> session = CachedSession(serializer=compressed_serializer)

Text-based Serializers

If you’re using a text-based serialization format like JSON or YAML, some extra steps are needed to encode binary data and non-builtin types. The cattrs library can do the majority of the work here, and some pre-configured converters are included for several common formats in the preconf module.

For example, a compressed JSON pipeline could be built as follows:

Example

>>> import json, gzip
>>> from requests_cache import CachedSession, SerializerPipeline, Stage, json_serializer, utf8_encoder
>>> comp_json_serializer = SerializerPipeline([
...     json_serializer, # Serialize to a JSON string
...     utf8_encoder, # Encode to bytes
...     Stage(dumps=gzip.compress, loads=gzip.decompress), # Compress
... ])

Note

If you want to use a different format that isn’t included in preconf, you can use CattrStage as a starting point.

Additional Serialization Steps

Some other tools that could be used as a stage in a SerializerPipeline include:

Class	loads	dumps
codecs.*	encode	decode
bz2	compress	decompress
gzip	compress	decompress
lzma	compress	decompress
zlib	compress	decompress
pickle	dumps	loads
itsdangerous.signer.Signer	sign	unsign
itsdangerous.serializer.Serializer	loads	dumps
cryptography.fernet.Fernet	encrypt	decrypt