Re-use the same socket connection for multiple requests, with optional
client-side certificate verification. See:
HTTPConnectionPool
and
HTTPSConnectionPool
File posting. See:
encode_multipart_formdata()
Built-in redirection and retries (optional).
Supports gzip and deflate decoding. See:
decode_gzip()
and
decode_deflate()
Thread-safe and sanity-safe.
Tested on Python 2.6+ and Python 3.2+, 100% unit test coverage.
Works with AppEngine, gevent, and eventlib.
Small and easy to understand codebase perfect for extending and building upon. For a more comprehensive solution, have a look at Requests which is also powered by urllib3.
pip install urllib3
or fetch the latest source from
github.com/shazow/urllib3.
>>> import urllib3
>>> http = urllib3.PoolManager()
>>> r = http.request('GET', 'http://google.com/')
>>> r.status
200
>>> r.headers['server']
'gws'
>>> r.data
...
urllib3
tries to strike a fine balance between power, extendability, and
sanity. To achieve this, the codebase is a collection of small reusable
utilities and abstractions composed together in a few helpful layers.
The highest level is the PoolManager(…).
The PoolManager
will take care of reusing
connections for you whenever you request the same host. this should cover most
scenarios without significant loss of efficiency, but you can always drop down
to a lower level component for more granular control.
>>> http = urllib3.PoolManager(10)
>>> r1 = http.request('GET', 'http://google.com/')
>>> r2 = http.request('GET', 'http://google.com/mail')
>>> r3 = http.request('GET', 'http://yahoo.com/')
>>> len(http.pools)
2
A PoolManager
is a proxy for a collection of
ConnectionPool
objects. They both inherit from
RequestMethods
to make sure that their API is
similar, so that instances of either can be passed around interchangeably.
The ProxyManager
is an HTTP proxy-aware
subclass of PoolManager
. It produces a single
HTTPConnectionPool
instance for all HTTP
connections and individual per-server:port
HTTPSConnectionPool
instances for tunnelled
HTTPS connections:
>>> proxy = urllib3.ProxyManager('http://localhost:3128/')
>>> r1 = proxy.request('GET', 'http://google.com/')
>>> r2 = proxy.request('GET', 'http://httpbin.org/')
>>> len(proxy.pools)
1
>>> r3 = proxy.request('GET', 'https://httpbin.org/')
>>> r4 = proxy.request('GET', 'https://twitter.com/')
>>> len(proxy.pools)
3
The next layer is the ConnectionPool(…).
The HTTPConnectionPool
and
HTTPSConnectionPool
classes allow you to
define a pool of connections to a single host and make requests against this
pool with automatic connection reusing and thread safety.
When the ssl
module is available, then
HTTPSConnectionPool
objects can be configured
to check SSL certificates against specific provided certificate authorities.
>>> conn = urllib3.connection_from_url('http://google.com')
>>> r1 = conn.request('GET', 'http://google.com/')
>>> r2 = conn.request('GET', '/mail')
>>> r3 = conn.request('GET', 'http://yahoo.com/')
Traceback (most recent call last)
...
HostChangedError: Connection pool with host 'http://google.com' tried to
open a foreign host: http://yahoo.com/
Again, a ConnectionPool is a pool of connections to a specific host. Trying to
access a different host through the same pool will raise a HostChangedError
exception unless you specify assert_same_host=False
. Do this at your own
risk as the outcome is completely dependent on the behaviour of the host server.
If you need to access multiple hosts and don’t want to manage your own
collection of ConnectionPool
objects, then you
should use a PoolManager
.
A ConnectionPool
is composed of a collection
of httplib.HTTPConnection
objects.
At the very core, just like its predecessors, urllib3
is built on top of
httplib
– the lowest level HTTP library included in the Python
standard library.
To aid the limited functionality of the httplib
module, urllib3
provides various helper methods which are used with the higher level components
but can also be used independently.
BaseHTTPResponse
BaseHTTPResponse.CONTENT_DECODERS
BaseHTTPResponse.DECODER_ERROR_CLASSES
BaseHTTPResponse.REDIRECT_STATUSES
BaseHTTPResponse.close()
BaseHTTPResponse.connection
BaseHTTPResponse.data
BaseHTTPResponse.drain_conn()
BaseHTTPResponse.get_redirect_location()
BaseHTTPResponse.getheader()
BaseHTTPResponse.getheaders()
BaseHTTPResponse.geturl()
BaseHTTPResponse.info()
BaseHTTPResponse.json()
BaseHTTPResponse.read()
BaseHTTPResponse.read_chunked()
BaseHTTPResponse.readinto()
BaseHTTPResponse.release_conn()
BaseHTTPResponse.retries
BaseHTTPResponse.stream()
BaseHTTPResponse.url
BytesQueueBuffer
ContentDecoder
DeflateDecoder
GzipDecoder
GzipDecoderState
HTTPResponse
HTTPResponse.close()
HTTPResponse.closed
HTTPResponse.connection
HTTPResponse.data
HTTPResponse.drain_conn()
HTTPResponse.fileno()
HTTPResponse.flush()
HTTPResponse.isclosed()
HTTPResponse.read()
HTTPResponse.read_chunked()
HTTPResponse.readable()
HTTPResponse.release_conn()
HTTPResponse.stream()
HTTPResponse.supports_chunked_reads()
HTTPResponse.tell()
HTTPResponse.url
MultiDecoder
These modules implement various extra features, that may not be ready for prime time.
Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug. There is a Contributor Friendly tag for issues that should be ideal for people who are not very familiar with the codebase yet.
Fork the urllib3 repository on Github to start making your changes.
Write a test which shows that the bug was fixed or that the feature works as expected.
Send a pull request and bug the maintainer until it gets merged and published.
:) Make sure to add yourself to CONTRIBUTORS.txt
.