Advanced Usage¶

Session Objects¶

The Session object allows you to persist certain parameters across requests. It also establishes a CookieJar and passes it along to any requests made from the Session instance.

A session object has all the methods of the main Requests API.

Let’s persist some cookies across requests:

s = requests.session()

s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")

print r.content
# '{"cookies": {"sessioncookie": "123456789"}}'

Sessions can also be used to provide default data to the request methods:

headers = {'x-test': 'true'}
auth = ('user', 'pass')

with requests.session(auth=auth, headers=headers) as c:

    # both 'x-test' and 'x-test2' are sent
    c.get('http://httpbin.org/headers', headers={'x-test2': 'true'})

Asynchronous Requests¶

Requests has first-class support for non-blocking i/o requests, powered by gevent. This allows you to send a bunch of HTTP requests at the same

First, let’s import the async module. Heads up — if you don’t have gevent this will fail:

from requests import async

The async module has the exact same api as requests, except it doesn’t send the request immediately. Instead, it returns the Request object.

We can build a list of Request objects easily:

urls = [
    'http://python-requests.org',
    'http://httpbin.org',
    'http://python-guide.org',
    'http://kennethreitz.com'
]

rs = [async.get(u) for u in urls]

Now we have a list of Request objects, ready to be sent. We could send them one at a time with Request.send(), but that would take a while. Instead, we’ll send them all at the same time with async.map(). Using async.map() will also guarantee execution of the response hook, described below.

>>> async.map(rs)
[<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>]

Event Hooks¶

Requests has a hook system that you can use to manipulate portions of the request process, or signal event handling.

Available hooks:

args:

A dictionary of the arguments being sent to Request().

pre_request:

The Request object, directly before being sent.

post_request:

The Request object, directly after being sent.

response:

The response generated from a Request.

You can assign a hook function on a per-request basis by passing a {hook_name: callback_function} dictionary to the hooks request parameter:

hooks=dict(args=print_url)

That callback_function will receive a chunk of data as its first argument.

def print_url(args):
    print args['url']

If an error occurs while executing your callback, a warning is given.

If the callback function returns a value, it is assumed that it is to replace the data that was passed in. If the function doesn’t return anything, nothing else is effected.

Let’s print some request method arguments at runtime:

>>> requests.get('http://httpbin', hooks=dict(args=print_url))
http://httpbin
<Response [200]>

Let’s hijack some arguments this time with a new callback:

def hack_headers(args):
    if not args[headers]:
        args['headers'] = dict()

    args['headers'].update({'X-Testing': 'True'})

    return args

hooks = dict(args=hack_headers)
headers = dict(yo=dawg)

And give it a try:

>>> requests.get('http://httpbin/headers', hooks=hooks, headers=headers)
{
    "headers": {
        "Content-Length": "",
        "Accept-Encoding": "gzip",
        "Yo": "dawg",
        "X-Forwarded-For": "::ffff:24.127.96.129",
        "Connection": "close",
        "User-Agent": "python-requests.org",
        "Host": "httpbin.org",
        "X-Testing": "True",
        "X-Forwarded-Protocol": "",
        "Content-Type": ""
    }
}

Verbose Logging¶

If you want to get a good look at what HTTP requests are being sent by your application, you can turn on verbose logging.

To do so, just configure Requests with a stream to write to:

>>> requests.settings.verbose = sys.stderr
>>> requests.get('http://httpbin.org/headers')
2011-08-17T03:04:23.380175   GET   http://httpbin.org/headers
<Response [200]>