Serializers#
By default, responses are serialized using pickle
, but some alternative serializers are
also included. These are mainly intended for use with FileCache
, but are compatible
with the other backends as well.
Note
Some serializers require additional dependencies
Specifying a Serializer#
Similar to Backends, you can specify which serializer to use with the serializer
parameter
for either CachedSession
or install_cache()
.
JSON Serializer#
Storing responses as JSON gives you the benefit of making them human-readable and editable, in exchange for a minor reduction in read and write speeds.
Usage:
>>> session = CachedSession('my_cache', serializer='json')
Example JSON-serialized Response
{
"url": "https://httpbin.org/get",
"status_code": 200,
"reason": "OK",
"_content": "dkP>RB4Ki8b0Rt*dwnb*3LqdNXk}q!WpZ;OIv{%rARr(hB0*zgWpH#NIv^q{FDfD|APOKLARr<^V`F7-bS*`0V{c?>Zf7DoAR=daX>cqcWMyV-VRU68EFcOXARr(jNN;m=B03-<XmoUNVrgzJZ*pfMEFcOXARr(jRdZ!>EkS2xZge6#AR=&ibZBpGEplaXb!BsOb1yP3GBz$SA}k;ZARr(hB3La!ZF+7kRB~ZsWi3f$B03-<Qg3f`JuxjdFlIPmF)(2*GB7hXGcjZ_Gh<>hGc{o{G%#g2VK_22A_^cNeJmgfARr=da%pF2ZX!A$A~82JE-^VSF*GqQGBq+HEFcOXAR={gY$7@!B4~7UaC15@FKBdhaAIk0E^l&YFK1<RA_{#9",
"cache_key": "4dc151d95200ec91fa77021989f5194e9be47e87f8f228306f3a8d5434b9e547",
"created_at": "2021-07-21T22:34:50.343095",
"elapsed": 0.242198,
"encoding": "utf-8",
"headers": {
"Date": "Wed, 21 Jul 2021 22:34:50 GMT",
"Content-Type": "application/json",
"Content-Length": "308",
"Connection": "keep-alive",
"Server": "gunicorn/19.9.0",
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Credentials": "true"
},
"request": {
"body": "PH%2y",
"headers": {
"User-Agent": "python-requests/2.26.0",
"Accept-Encoding": "gzip, deflate",
"Accept": "*/*",
"Connection": "keep-alive"
},
"method": "GET",
"url": "https://httpbin.org/get"
},
"raw": {
"decode_content": false,
"reason": "OK",
"status": 200,
"version": 11
}
}
This will use ultrajson if installed, otherwise the stdlib
json
module will be used. You can install the optional dependencies for this serializer with:
pip install requests-cache[json]
YAML Serializer#
YAML is another option if you need a human-readable/editable format, with the same tradeoffs as JSON.
Usage:
>>> session = CachedSession('my_cache', serializer='yaml')
Example YAML-serialized Response
url: https://httpbin.org/get
status_code: 200
reason: OK
_content: !!binary |
ewogICJhcmdzIjoge30sIAogICJoZWFkZXJzIjogewogICAgIkFjY2VwdCI6ICIqLyoiLCAKICAg
ICJBY2NlcHQtRW5jb2RpbmciOiAiZ3ppcCwgZGVmbGF0ZSIsIAogICAgIkhvc3QiOiAiaHR0cGJp
bi5vcmciLCAKICAgICJVc2VyLUFnZW50IjogInB5dGhvbi1yZXF1ZXN0cy8yLjI2LjAiLCAKICAg
ICJYLUFtem4tVHJhY2UtSWQiOiAiUm9vdD0xLTYwZjhhMDcxLTBiN2JmN2VjNGMyZTdmNjA2YWI4
ZDYyNCIKICB9LCAKICAib3JpZ2luIjogIjE3My4xOS4xNDEuMjUyIiwgCiAgInVybCI6ICJodHRw
czovL2h0dHBiaW4ub3JnL2dldCIKfQo=
cache_key: 4dc151d95200ec91fa77021989f5194e9be47e87f8f228306f3a8d5434b9e547
created_at: '2021-07-21T22:32:17.592974'
elapsed: 0.187586
encoding: utf-8
headers:
Access-Control-Allow-Credentials: 'true'
Access-Control-Allow-Origin: '*'
Connection: keep-alive
Content-Length: '308'
Content-Type: application/json
Date: Wed, 21 Jul 2021 22:32:17 GMT
Server: gunicorn/19.9.0
request:
method: GET
url: https://httpbin.org/get
body: !!binary |
Tm9uZQ==
headers:
Accept: '*/*'
Accept-Encoding: gzip, deflate
Connection: keep-alive
User-Agent: python-requests/2.26.0
raw:
decode_content: false
reason: OK
status: 200
version: 11
You can install the extra dependencies for this serializer with:
pip install requests-cache[yaml]
BSON Serializer#
BSON is a serialization format originally created for
MongoDB, but it can also be used independently. Compared to JSON, it has better performance
(although still not as fast as pickle
), and adds support for additional data types. It is not
human-readable, but some tools support reading and editing it directly
(for example, bson-converter for Atom).
Usage:
>>> session = CachedSession('my_cache', serializer='bson')
You can install the extra dependencies for this serializer with:
pip install requests-cache[mongo]
Or if you would like to use the standalone BSON codec for a different backend, without installing MongoDB dependencies:
pip install requests-cache[bson]
Serializer Security#
See Security for recommended setup steps for more secure cache serialization, particularly
when using pickle
.
Custom Serializers#
If the built-in serializers don’t suit your needs, you can create your own. For example, if
you had a imaginary custom_pickle
module that provides dumps
and loads
functions:
>>> import custom_pickle
>>> from requests_cache import CachedSession
>>> session = CachedSession(serializer=custom_pickle)
Serializer Pipelines#
More complex serialization can be done with SerializerPipeline
. Use cases include
text-based serialization, compression, encryption, and any other intermediate steps you might want
to add.
Any combination of these can be composed with a SerializerPipeline
, which starts with a
CachedResponse
and ends with a str
or bytes
object. Each stage
of the pipeline can be any object or module with dumps
and loads
functions. If the object has
similar methods with different names (e.g. compress
/ decompress
), those can be aliased using
Stage
.
For example, a compressed pickle serializer can be built as:
Example code
>>> import gzip
>>> from requests_cache import CachedSession, SerializerPipeline, Stage, pickle_serializer
>>> compressed_serializer = SerializerPipeline(
... [
... pickle_serializer,
... Stage(dumps=gzip.compress, loads=gzip.decompress),
... ],
... is_binary=True,
... )
>>> session = CachedSession(serializer=compressed_serializer)
Text-based Serializers#
If you’re using a text-based serialization format like JSON or YAML, some extra steps are needed to
encode binary data and non-builtin types. The cattrs library can do
the majority of the work here, and some pre-configured converters are included for several common
formats in the preconf
module.
For example, a compressed JSON pipeline could be built as follows:
Example code
>>> import json, gzip
>>> from requests_cache import CachedSession, SerializerPipeline, Stage, json_serializer, utf8_encoder
>>> comp_json_serializer = SerializerPipeline([
... json_serializer, # Serialize to a JSON string
... utf8_encoder, # Encode to bytes
... Stage(dumps=gzip.compress, loads=gzip.decompress), # Compress
... ])
Note
If you want to use a different format that isn’t included in preconf
, you can use
CattrStage
as a starting point.
Additional Serialization Steps#
Some other tools that could be used as a stage in a SerializerPipeline
include:
Class |
loads |
dumps |
---|---|---|
encode |
decode |
|
compress |
decompress |
|
compress |
decompress |
|
compress |
decompress |
|
compress |
decompress |
|
dumps |
loads |
|
sign |
unsign |
|
loads |
dumps |
|
encrypt |
decrypt |