gevent.pool – Managing greenlets in a group¶Bases: GroupMappingMixin
Maintain a group of greenlets that are still running, without limiting their number.
Links to each item and removes it upon notification.
Groups can be iterated to discover what greenlets they are tracking, they can be tested to see if they contain a greenlet, and they know the number (len) of greenlets they are tracking. If they are not tracking any greenlets, they are False in a boolean context.
Either gevent.Greenlet (the default) or a subclass.
These are the type of
object we will spawn(). This can be
changed on an instance or in a subclass.
Answer how many greenlets we are tracking. Note that if we are empty, we are False in a boolean context.
alias of Greenlet
Begin tracking the greenlet.
If this group is full(), then this method may block
until it is possible to track the greenlet.
Typically the greenlet should not be started when it is added because if this object blocks in this method, then the greenlet may run to completion before it is tracked.
Rough quivalent of the apply() builtin function blocking until
the result is ready and returning it.
The func will usually, but not always, be run in a way
that allows the current greenlet to switch out (for example,
in a new greenlet or thread, depending on implementation). But
if the current greenlet or thread is already one that was
spawned by this pool, the pool may choose to immediately run
the func synchronously.
Any exception func raises will be propagated to the caller of apply (that is,
this method will raise the exception that func raised).
A variant of the apply() method which returns a Greenlet object.
When the returned greenlet gets to run, it will call apply(),
passing in func, args and kwds.
If callback is specified, then it should be a callable which accepts a single argument. When the result becomes ready callback is applied to it (unless the call failed).
This method will never block, even if this group is full (that is,
even if spawn() would block, this method will not).
Caution
The returned greenlet may or may not be tracked
as part of this group, so joining this group is
not a reliable way to wait for the results to be available or
for the returned greenlet to run; instead, join the returned
greenlet.
Tip
Because ThreadPool objects do not track greenlets, the returned
greenlet will never be a part of it. To reduce overhead and improve performance,
Group and Pool may choose to track the returned
greenlet. These are implementation details that may change.
apply() the given func(*args, **kwds), and, if a callback is given, run it with the
results of func (unless an exception was raised.)
The callback may be called synchronously or asynchronously. If called
asynchronously, it will not be tracked by this group. (Group and Pool
call it asynchronously in a new greenlet; ThreadPool calls
it synchronously in the current greenlet.)
Return a value indicating whether this group can track more greenlets.
In this implementation, because there are no limits on the number of
tracked greenlets, this will always return a False value.
An equivalent of itertools.imap(), operating in parallel.
The func is applied to each element yielded from each
iterable in iterables in turn, collecting the result.
If this object has a bound on the number of active greenlets it can
contain (such as Pool), then at most that number of tasks will operate
in parallel.
maxsize (int) –
If given and not-None, specifies the maximum number of finished results that will be allowed to accumulate awaiting the reader; more than that number of results will cause map function greenlets to begin to block. This is most useful if there is a great disparity in the speed of the mapping code and the consumer and the results consume a great deal of resources.
Note
This is separate from any bound on the number of active parallel tasks, though they may have some interaction (for example, limiting the number of parallel tasks to the smallest bound).
Note
Using a bound is slightly more computationally expensive than not using a bound.
Tip
The imap_unordered() method makes much better
use of this parameter. Some additional, unspecified,
number of objects may be required to be kept in memory
to maintain order by this function.
An iterable object.
Changed in version 1.1b3: Added the maxsize keyword parameter.
Changed in version 1.1a1: Accept multiple iterables to iterate in parallel.
The same as imap() except that the ordering of the results
from the returned iterator should be considered in arbitrary
order.
This is lighter weight than imap() and should be preferred if order
doesn’t matter.
See also
imap() for more details.
Wait for this group to become empty at least once.
If there are no greenlets in the group, returns immediately.
Note
By the time the waiting code (the caller of this
method) regains control, a greenlet may have been added to
this group, and so this object may no longer be empty. (That
is, group.join(); assert len(group) == 0 is not
guaranteed to hold.) This method only guarantees that the group
reached a len of 0 at some point.
raise_error (bool) – If True (not the default), if any greenlet that finished while the join was in progress raised an exception, that exception will be raised to the caller of this method. If multiple greenlets raised exceptions, which one gets re-raised is not determined. Only greenlets currently in the group when this method is called are guaranteed to be checked for exceptions.
A value indicating whether this group became empty. If the timeout is specified and the group did not become empty during that timeout, then this will be a false value. Otherwise it will be a true value.
Changed in version 1.2a1: Add the return value.
Kill all greenlets being tracked by this group.
If the given greenlet is running and being tracked by this group, kill it.
Return a list made by applying the func to each element of the iterable.
See also
A variant of the map() method which returns a Greenlet object that is executing the map function.
If callback is specified then it should be a callable which accepts a single argument.
Begin a new greenlet with the given arguments (which are passed to the greenlet constructor) and add it to the collection of greenlets this group is monitoring.
The newly started greenlet.
Bases: Full
Raised when a Pool is full and an attempt was made to add a new greenlet to it in non-blocking mode.
Bases: Group
Create a new pool.
A pool is like a group, but the maximum number of members is governed by the size parameter.
size (int) –
If given, this non-negative integer is the maximum count of active greenlets that will be allowed in this pool. A few values have special significance:
None (the default) places no limit on the number of
greenlets. This is useful when you want to track, but not limit,
greenlets. In general, a Group
may be a more efficient way to achieve the same effect, but some things
need the additional abilities of this class (one example being the spawn
parameter of gevent.baseserver.BaseServer and
its subclass gevent.pywsgi.WSGIServer).
0 creates a pool that can never have any active greenlets. Attempting
to spawn in this pool will block forever. This is only useful
if an application uses wait_available() with a timeout and checks
free_count() before attempting to spawn.
Begin tracking the given unstarted greenlet, possibly blocking until space is available.
Usually you should call start() to track and start the greenlet
instead of using this lower-level method, or spawn() to
also create the greenlet.
blocking (bool) – If True (the default), this function will block until the pool has space or a timeout occurs. If False, this function will immediately raise a Timeout if the pool is currently full.
timeout (float) – The maximum number of seconds this
method will block, if blocking is True. (Ignored if
blocking is False.)
PoolFull – if either blocking is False and the pool
was full, or if blocking is True and timeout was
exceeded.
Caution
If the greenlet has already been started and blocking is true, then the greenlet may run to completion while the current greenlet blocks waiting to track it. This would enable higher concurrency than desired.
See also
Changed in version 1.3.0: Added the blocking and
timeout parameters.
Return a number indicating approximately how many more members can be added to this pool.
Return a boolean indicating whether this pool is full, e.g. if
add() would block.
False if there is room for new members, True if there isn’t.
Add the unstarted greenlet to the collection of greenlets this group is monitoring and then start it.
Parameters are as for add().
Wait until it’s possible to spawn a greenlet in this pool.
timeout (float) – If given, only wait the specified number of seconds.
Warning
If the pool was initialized with a size of 0, this method will block forever unless a timeout is given.
A number indicating how many new greenlets can be put into the pool without blocking.
Changed in version 1.1a3: Added the timeout parameter.
Next page: gevent.pywsgi – A pure-Python, gevent-friendly WSGI server