urllib3学习

urllib3.connectionpool.connection_from_url(url, **kw)

Given a url, return an ConnectionPool instance of its host.

This is a shortcut for not having to parse out the scheme, host, and port of the url before creating an ConnectionPool instance.

Parameters:	url – Absolute URL string that must include the scheme. Port is optional. **kw – Passes additional parameters to the constructor of the appropriate `ConnectionPool`. Useful for specifying things like timeout, maxsize, headers, etc.

例子

>>> conn = connection_from_url('http://google.com/')
>>> r = conn.request('GET', '/')

urllib3.util.timeout module

class urllib3.util.timeout.Timeout(total=None, connect=<object object>, read=<object object>)

Bases: object

Timeout configuration.

Timeouts can be defined as a default for a pool:

timeout = Timeout(connect=2.0, read=7.0)
http = PoolManager(timeout=timeout)
response = http.request('GET', 'http://example.com/')

Or per-request (which overrides the default for the pool):

response = http.request('GET', 'http://example.com/', timeout=Timeout(10))

Timeouts can be disabled by setting all the parameters to None:

no_timeout = Timeout(connect=None, read=None)
response = http.request('GET', 'http://example.com/, timeout=no_timeout)

Parameters:	total (integer, float, or None) – This combines the connect and read timeouts into one; the read timeout will be set to the time leftover from the connect attempt. In the event that both a connect timeout and a total are specified, or a read timeout and a total are specified, the shorter timeout will be applied. Defaults to None. connect (integer, float, or None) – The maximum amount of time to wait for a connection attempt to a server to succeed. Omitting the parameter will default the connect timeout to the system default, probably the global default timeout in socket.py. None will set an infinite timeout for connection attempts. read (integer, float, or None) – The maximum amount of time to wait between consecutive read operations for a response from the server. Omitting the parameter will default the read timeout to the system default, probably the global default timeout in socket.py. None will set an infinite timeout.

Parameters:

total (integer, float, or None) –
This combines the connect and read timeouts into one; the read timeout will be set to the time leftover from the connect attempt. In the event that both a connect timeout and a total are specified, or a read timeout and a total are specified, the shorter timeout will be applied.

Defaults to None.
connect (integer, float, or None) – The maximum amount of time to wait for a connection attempt to a server to succeed. Omitting the parameter will default the connect timeout to the system default, probably the global default timeout in socket.py. None will set an infinite timeout for connection attempts.
read (integer, float, or None) –
The maximum amount of time to wait between consecutive read operations for a response from the server. Omitting the parameter will default the read timeout to the system default, probably the global default timeout in socket.py. None will set an infinite timeout.

更详细内容请参考：https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#module-urllib3.util.timeout

urllib3.connectionpool module

class urllib3.connectionpool.ConnectionPool(host, port=None)

Bases: object

Base class for all connection pools, such as HTTPConnectionPool and HTTPSConnectionPool.

QueueCls: alias of LifoQueue

close(): Close all pooled connections and disable the pool.

scheme = None

urlopen(method, url, body=None, headers=None, retries=None, redirect=True, assert_same_host=True, timeout=<object object>, pool_timeout=None, release_conn=None, chunked=False, body_pos=None, **response_kw)

Get a connection from the pool and perform an HTTP request. This is the lowest level call for making a request, so you’ll need to specify all the raw details.

可以把“连接池”当做“快递集散中心”，每个“连接”好比“一次快递事务”。

返回值

　　response对象。

Note

More commonly, it’s appropriate to use a convenience method provided by RequestMethods, such as request().

Note

release_conn will only behave as expected if preload_content=Falsebecause we want to make preload_content=False the default behaviour someday soon without breaking backwards compatibility.

Parameters:	method – HTTP request method (such as GET, POST, PUT, etc.) body – Data to send in the request body (useful for creating POST requests, see HTTPConnectionPool.post_url for more convenience). headers – Dictionary of custom headers to send, such as User-Agent, If-None-Match, etc. If None, pool headers are used. If provided, these headers completely replace any pool-specific headers. retries (`Retry`, False, or an int.) – Configure the number of retries to allow before raising a`MaxRetryError` exception. Pass `None` to retry until you receive a response. Pass a `Retry`object for fine-grained control over different types of retries. Pass an integer number to retry connection errors that many times, but no other types of errors. Pass zero to never retry. If `False`, then retries are disabled and any exception is raised immediately. Also, instead of raising a MaxRetryError on redirects, the redirect response will be returned. redirect – If True, automatically handle redirects (status codes 301, 302, 303, 307, 308). Each redirect counts as a retry. Disabling retries will disable redirect, too. assert_same_host – If `True`, will make sure that the host of the pool requests is consistent else will raise HostChangedError. When False, you can use the pool on an HTTP proxy and request foreign hosts. timeout – If specified, overrides the default timeout for this one request. It may be a float (in seconds) or an instance of`urllib3.util.Timeout`. pool_timeout – If set and the pool is set to block=True, then this method will block for `pool_timeout` seconds and raise EmptyPoolError if no connection is available within the time period. release_conn – If False, then the urlopen call will not release the connection back into the pool once a response is received (but will release if you read the entire contents of the response such as when preload_content=True). This is useful if you’re not preloading the response’s content immediately. You will need to call `r.release_conn()` on the response `r` to return the connection back into the pool. If None, it takes the value of`response_kw.get('preload_content', True)`. chunked – If True, urllib3 will send the body using chunked transfer encoding. Otherwise, urllib3 will send the body using the standard content-length form. Defaults to False. body_pos (int) – Position to seek to in file-like body in the event of a retry or redirect. Typically this won’t need to be set because urllib3 will auto-populate the value when needed. **response_kw – Additional parameters are passed to`urllib3.response.HTTPResponse.from_httplib()`

Parameters:

method – HTTP request method (such as GET, POST, PUT, etc.)
body – Data to send in the request body (useful for creating POST requests, see HTTPConnectionPool.post_url for more convenience).
headers – Dictionary of custom headers to send, such as User-Agent, If-None-Match, etc. If None, pool headers are used. If provided, these headers completely replace any pool-specific headers.
retries (Retry, False, or an int.) –
Configure the number of retries to allow before raising aMaxRetryError exception.

Pass None to retry until you receive a response. Pass a Retryobject for fine-grained control over different types of retries. Pass an integer number to retry connection errors that many times, but no other types of errors. Pass zero to never retry.

If False, then retries are disabled and any exception is raised immediately. Also, instead of raising a MaxRetryError on redirects, the redirect response will be returned.
redirect – If True, automatically handle redirects (status codes 301, 302, 303, 307, 308). Each redirect counts as a retry. Disabling retries will disable redirect, too.
assert_same_host – If True, will make sure that the host of the pool requests is consistent else will raise HostChangedError. When False, you can use the pool on an HTTP proxy and request foreign hosts.
timeout – If specified, overrides the default timeout for this one request. It may be a float (in seconds) or an instance ofurllib3.util.Timeout.
pool_timeout – If set and the pool is set to block=True, then this method will block for pool_timeout seconds and raise EmptyPoolError if no connection is available within the time period.
release_conn – If False, then the urlopen call will not release the connection back into the pool once a response is received (but will release if you read the entire contents of the response such as when preload_content=True). This is useful if you’re not preloading the response’s content immediately. You will need to call r.release_conn() on the response r to return the connection back into the pool. If None, it takes the value ofresponse_kw.get('preload_content', True).
chunked – If True, urllib3 will send the body using chunked transfer encoding. Otherwise, urllib3 will send the body using the standard content-length form. Defaults to False.
body_pos (int) – Position to seek to in file-like body in the event of a retry or redirect. Typically this won’t need to be set because urllib3 will auto-populate the value when needed.
**response_kw – Additional parameters are passed tourllib3.response.HTTPResponse.from_httplib()