A pool manager is an abstraction for a collection of ConnectionPools.
If you need to make requests to multiple hosts, then you can use a
PoolManager
, which takes care of maintaining your pools
so you don’t have to.
>>> from urllib3 import PoolManager
>>> manager = PoolManager(10)
>>> r = manager.request('GET', 'http://google.com/')
>>> r.headers['server']
'gws'
>>> r = manager.request('GET', 'http://yahoo.com/')
>>> r.headers['server']
'YTS/1.20.0'
>>> r = manager.request('POST', 'http://google.com/mail')
>>> r = manager.request('HEAD', 'http://google.com/calendar')
>>> len(manager.pools)
2
>>> conn = manager.connection_from_host('google.com')
>>> conn.num_requests
3
The API of a PoolManager
object is similar to that of a
ConnectionPool, so they can be passed around interchangeably.
The PoolManager uses a Least Recently Used (LRU) policy for discarding old
pools. That is, if you set the PoolManager num_pools
to 10, then after
making requests to 11 or more different hosts, the least recently used pools
will be cleaned up eventually.
Cleanup of stale pools does not happen immediately. You can read more about the
implementation and the various adjustable variables within
RecentlyUsedContainer
.
- class urllib3.poolmanager.PoolManager(num_pools: int = 10, headers: Mapping[str, str] | None = None, **connection_pool_kw: Any)[source]¶
Allows for arbitrary requests while transparently keeping track of necessary connection pools for you.
- Parameters:
num_pools – Number of connection pools to cache before discarding the least recently used pool.
headers – Headers to include with all requests, unless other headers are given explicitly.
**connection_pool_kw – Additional parameters are used to create fresh
urllib3.connectionpool.ConnectionPool
instances.Example:
import urllib3 http = urllib3.PoolManager(num_pools=2) resp1 = http.request("GET", "https://google.com/") resp2 = http.request("GET", "https://google.com/mail") resp3 = http.request("GET", "https://yahoo.com/") print(len(http.pools)) # 2
- clear() None [source]¶
Empty our store of pools and direct them all to close.
This will not affect in-flight connections, but they will not be re-used after completion.
- connection_from_context(request_context: dict[str, Any]) HTTPConnectionPool [source]¶
Get a
urllib3.connectionpool.ConnectionPool
based on the request context.
request_context
must at least contain thescheme
key and its value must be a key inkey_fn_by_scheme
instance variable.
- connection_from_host(host: str | None, port: int | None = None, scheme: str | None = 'http', pool_kwargs: dict[str, Any] | None = None) HTTPConnectionPool [source]¶
Get a
urllib3.connectionpool.ConnectionPool
based on the host, port, and scheme.If
port
isn’t given, it will be derived from thescheme
usingurllib3.connectionpool.port_by_scheme
. Ifpool_kwargs
is provided, it is merged with the instance’sconnection_pool_kw
variable and used to create the new connection pool, if one is needed.
- connection_from_pool_key(pool_key: PoolKey, request_context: dict[str, Any]) HTTPConnectionPool [source]¶
Get a
urllib3.connectionpool.ConnectionPool
based on the provided pool key.
pool_key
should be a namedtuple that only contains immutable objects. At a minimum it must have thescheme
,host
, andport
fields.
- connection_from_url(url: str, pool_kwargs: dict[str, Any] | None = None) HTTPConnectionPool [source]¶
Similar to
urllib3.connectionpool.connection_from_url()
.If
pool_kwargs
is not provided and a new pool needs to be constructed,self.connection_pool_kw
is used to initialize theurllib3.connectionpool.ConnectionPool
. Ifpool_kwargs
is provided, it is used instead. Note that if a new pool does not need to be created for the request, the providedpool_kwargs
are not used.
- request(method: str, url: str, body: bytes | IO[Any] | Iterable[bytes] | str | None = None, fields: Sequence[Tuple[str, str | bytes | Tuple[str, str | bytes] | Tuple[str, str | bytes, str]] | RequestField] | Mapping[str, str | bytes | Tuple[str, str | bytes] | Tuple[str, str | bytes, str]] | None = None, headers: Mapping[str, str] | None = None, json: Any | None = None, **urlopen_kw: Any) BaseHTTPResponse ¶
Make a request using
urlopen()
with the appropriate encoding offields
based on themethod
used.This is a convenience method that requires the least amount of manual effort. It can be used in most situations, while still having the option to drop down to more specific methods when necessary, such as
request_encode_url()
,request_encode_body()
, or even the lowest levelurlopen()
.
- request_encode_body(method: str, url: str, fields: Sequence[Tuple[str, str | bytes | Tuple[str, str | bytes] | Tuple[str, str | bytes, str]] | RequestField] | Mapping[str, str | bytes | Tuple[str, str | bytes] | Tuple[str, str | bytes, str]] | None = None, headers: Mapping[str, str] | None = None, encode_multipart: bool = True, multipart_boundary: str | None = None, **urlopen_kw: str) BaseHTTPResponse ¶
Make a request using
urlopen()
with thefields
encoded in the body. This is useful for request methods like POST, PUT, PATCH, etc.When
encode_multipart=True
(default), thenurllib3.encode_multipart_formdata()
is used to encode the payload with the appropriate content type. Otherwiseurllib.parse.urlencode()
is used with the ‘application/x-www-form-urlencoded’ content type.Multipart encoding must be used when posting files, and it’s reasonably safe to use it in other times too. However, it may break request signing, such as with OAuth.
Supports an optional
fields
parameter of key/value strings AND key/filetuple. A filetuple is a (filename, data, MIME type) tuple where the MIME type is optional. For example:fields = { 'foo': 'bar', 'fakefile': ('foofile.txt', 'contents of foofile'), 'realfile': ('barfile.txt', open('realfile').read()), 'typedfile': ('bazfile.bin', open('bazfile').read(), 'image/jpeg'), 'nonamefile': 'contents of nonamefile field', }When uploading a file, providing a filename (the first parameter of the tuple) is optional but recommended to best mimic behavior of browsers.
Note that if
headers
are supplied, the ‘Content-Type’ header will be overwritten because it depends on the dynamic random boundary string which is used to compose the body of the request. The random boundary string can be explicitly set with themultipart_boundary
parameter.
- request_encode_url(method: str, url: str, fields: Sequence[Tuple[str, str | bytes]] | Mapping[str, str | bytes] | None = None, headers: Mapping[str, str] | None = None, **urlopen_kw: str) BaseHTTPResponse ¶
Make a request using
urlopen()
with thefields
encoded in the url. This is useful for request methods like GET, HEAD, DELETE, etc.
- urlopen(method: str, url: str, redirect: bool = True, **kw: Any) BaseHTTPResponse [source]¶
Same as
urllib3.HTTPConnectionPool.urlopen()
with custom cross-host redirect logic and only sends the request-uri portion of theurl
.The given
url
parameter must be absolute, such that an appropriateurllib3.connectionpool.ConnectionPool
can be chosen for it.