`runners`¶

class invoke.runners.Runner(context: Context)¶

Partially-abstract core command-running API.

This class is not usable by itself and must be subclassed, implementing a number of methods such as start, wait and returncode. For a subclass implementation example, see the source code for Local.

New in version 1.0.

__init__(context: Context) → None¶

Create a new runner with a handle on some Context.

Parameters:

context –

a Context instance, used to transmit default options and provide access to other contextualized information (e.g. a remote-oriented Runner might want a Context subclass holding info about hostnames and ports.)

Note

The Context given to Runner instances must contain default config values for the Runner class in question. At a minimum, this means values for each of the default Runner.run keyword arguments such as echo and warn.

Raises:

exceptions.ValueError – if not all expected default values are found in context.

context¶: The Context given to the same-named argument of __init__.

program_finished¶

A threading.Event signaling program completion.

Typically set after wait returns. Some IO mechanisms rely on this to know when to exit an infinite read loop.

read_chunk_size = 1000¶: How many bytes (at maximum) to read per iteration of stream reads.

input_sleep = 0.01¶: How many seconds to sleep on each iteration of the stdin read loop and other otherwise-fast loops.

warned_about_pty_fallback¶: Whether pty fallback warning has been emitted.

watchers: List['StreamWatcher']¶: A list of StreamWatcher instances for use by respond. Is filled in at runtime by run.

run(command: str, **kwargs: Any) → Result | None¶

Execute command, returning an instance of Result once complete.

By default, this method is synchronous (it only returns once the subprocess has completed), and allows interactive keyboard communication with the subprocess.

It can instead behave asynchronously (returning early & requiring interaction with the resulting object to manage subprocess lifecycle) if you specify asynchronous=True. Furthermore, you can completely disassociate the subprocess from Invoke’s control (allowing it to persist on its own after Python exits) by saying disown=True. See the per-kwarg docs below for details on both of these.

Note

All kwargs will default to the values found in this instance’s context attribute, specifically in its configuration’s run subtree (e.g. run.echo provides the default value for the echo keyword, etc). The base default values are described in the parameter list below.

Parameters:

command (str) – The shell command to execute.
asynchronous (bool) –
When set to True (default False), enables asynchronous behavior, as follows:
- Connections to the controlling terminal are disabled, meaning you will not see the subprocess output and it will not respond to your keyboard input - similar to hide=True and in_stream=False (though explicitly given (out|err|in)_stream file-like objects will still be honored as normal).
- run returns immediately after starting the subprocess, and its return value becomes an instance of Promise instead of Result.
- Promise objects are primarily useful for their join method, which blocks until the subprocess exits (similar to threading APIs) and either returns a final Result or raises an exception, just as a synchronous run would.
  - As with threading and similar APIs, users of asynchronous=True should make sure to join their Promise objects to prevent issues with interpreter shutdown.
  - One easy way to handle such cleanup is to use the Promise as a context manager - it will automatically join at the exit of the context block.
New in version 1.4.
disown (bool) –
When set to True (default False), returns immediately like asynchronous=True, but does not perform any background work related to that subprocess (it is completely ignored). This allows subprocesses using shell backgrounding or similar techniques (e.g. trailing &, nohup) to persist beyond the lifetime of the Python process running Invoke.

Note

If you’re unsure whether you want this or asynchronous, you probably want asynchronous!

Specifically, disown=True has the following behaviors:
- The return value is None instead of a Result or subclass.
- No I/O worker threads are spun up, so you will have no access to the subprocess’ stdout/stderr, your stdin will not be forwarded, (out|err|in)_stream will be ignored, and features like watchers will not function.
- No exit code is checked for, so you will not receive any errors if the subprocess fails to exit cleanly.
- pty=True may not function correctly (subprocesses may not run at all; this seems to be a potential bug in Python’s pty.fork) unless your command line includes tools such as nohup or (the shell builtin) disown.
New in version 1.4.
dry (bool) –
Whether to dry-run instead of truly invoking the given command. See --dry (which flips this on globally) for details on this behavior.

New in version 1.3.
echo (bool) –
Controls whether run prints the command string to local stdout prior to executing it. Default: False.

Note

hide=True will override echo=True if both are given.
echo_format –
A string, which when passed to Python’s inbuilt .format method, will change the format of the output when run.echo is set to true.

Currently, only {command} is supported as a parameter.

Defaults to printing the full command string in ANSI-escaped bold.
echo_stdin (bool) –
Whether to write data from in_stream back to out_stream.

In other words, in normal interactive usage, this parameter controls whether Invoke mirrors what you type back to your terminal.

By default (when None), this behavior is triggered by the following:
- Not using a pty to run the subcommand (i.e. pty=False), as ptys natively echo stdin to stdout on their own;
- And when the controlling terminal of Invoke itself (as per in_stream) appears to be a valid terminal device or TTY. (Specifically, when isatty yields a True result when given in_stream.)
  
  Note
  
  This property tends to be False when piping another program’s output into an Invoke session, or when running Invoke within another program (e.g. running Invoke from itself).
If both of those properties are true, echoing will occur; if either is false, no echoing will be performed.

When not None, this parameter will override that auto-detection and force, or disable, echoing.
encoding (str) – Override auto-detection of which encoding the subprocess is using for its stdout/stderr streams (which defaults to the return value of default_encoding).
err_stream – Same as out_stream, except for standard error, and defaulting to sys.stderr.
env (dict) –
By default, subprocesses receive a copy of Invoke’s own environment (i.e. os.environ). Supply a dict here to update that child environment.

For example, run('command', env={'PYTHONPATH': '/some/virtual/env/maybe'}) would modify the PYTHONPATH env var, with the rest of the child’s env looking identical to the parent.

See also

replace_env for changing ‘update’ to ‘replace’.
fallback (bool) – Controls auto-fallback behavior re: problems offering a pty when pty=True. Whether this has any effect depends on the specific Runner subclass being invoked. Default: True.
hide –
Allows the caller to disable run’s default behavior of copying the subprocess’ stdout and stderr to the controlling terminal. Specify hide='out' (or 'stdout') to hide only the stdout stream, hide='err' (or 'stderr') to hide only stderr, or hide='both' (or True) to hide both streams.

The default value is None, meaning to print everything; False will also disable hiding.

Note

Stdout and stderr are always captured and stored in the Result object, regardless of hide’s value.

Note

hide=True will also override echo=True if both are given (either as kwargs or via config/CLI).
in_stream –
A file-like stream object to used as the subprocess’ standard input. If None (the default), sys.stdin will be used.

If False, will disable stdin mirroring entirely (though other functionality which writes to the subprocess’ stdin, such as autoresponding, will still function.) Disabling stdin mirroring can help when sys.stdin is a misbehaving non-stream object, such as under test harnesses or headless command runners.
out_stream – A file-like stream object to which the subprocess’ standard output should be written. If None (the default), sys.stdout will be used.
pty (bool) –
By default, run connects directly to the invoked process and reads its stdout/stderr streams. Some programs will buffer (or even behave) differently in this situation compared to using an actual terminal or pseudoterminal (pty). To use a pty instead of the default behavior, specify pty=True.

Warning

Due to their nature, ptys have a single output stream, so the ability to tell stdout apart from stderr is not possible when pty=True. As such, all output will appear on out_stream (see below) and be captured into the stdout result attribute. err_stream and stderr will always be empty when pty=True.
replace_env (bool) – When True, causes the subprocess to receive the dictionary given to env as its entire shell environment, instead of updating a copy of os.environ (which is the default behavior). Default: False.
shell (str) – Which shell binary to use. Default: /bin/bash (on Unix; COMSPEC or cmd.exe on Windows.)
timeout –
Cause the runner to submit an interrupt to the subprocess and raise CommandTimedOut, if the command takes longer than timeout seconds to execute. Defaults to None, meaning no timeout.

New in version 1.3.
warn (bool) –
Whether to warn and continue, instead of raising UnexpectedExit, when the executed command exits with a nonzero status. Default: False.

Note

This setting has no effect on exceptions, which will still be raised, typically bundled in ThreadException objects if they were raised by the IO worker threads.

Similarly, WatcherError exceptions raised by StreamWatcher instances will also ignore this setting, and will usually be bundled inside Failure objects (in order to preserve the execution context).

Ditto CommandTimedOut - basically, anything that prevents a command from actually getting to “exited with an exit code” ignores this flag.
watchers –
A list of StreamWatcher instances which will be used to scan the program’s stdout or stderr and may write into its stdin (typically bytes objects) in response to patterns or other heuristics.

See Automatically responding to program output for details on this functionality.

Default: [].

Returns:

Result, or a subclass thereof.

Raises:

UnexpectedExit, if the command exited nonzero and warn was False.

Raises:

Failure, if the command didn’t even exit cleanly, e.g. if a StreamWatcher raised WatcherError.

Raises:

ThreadException (if the background I/O threads encountered exceptions other than WatcherError).

New in version 1.0.

make_promise() → Promise¶: Return a Promise allowing async control of the rest of lifecycle.

New in version 1.4.

create_io_threads() → Tuple[Dict[Callable, ExceptionHandlingThread], List[str], List[str]]¶

Create and return a dictionary of IO thread worker objects.

Caller is expected to handle persisting and/or starting the wrapped threads.

generate_result(**kwargs: Any) → Result¶

Create & return a suitable Result instance from the given kwargs.

Subclasses may wish to override this in order to manipulate things or generate a Result subclass (e.g. ones containing additional metadata besides the default).

New in version 1.0.

read_proc_output(reader: Callable) → Generator[str, None, None]¶

Iteratively read & decode bytes from a subprocess’ out/err stream.

Parameters:

reader –

A literal reader function/partial, wrapping the actual stream object in question, which takes a number of bytes to read, and returns that many bytes (or None).

reader should be a reference to either read_proc_stdout or read_proc_stderr, which perform the actual, platform/library specific read calls.

Returns:

A generator yielding strings.

Specifically, each resulting string is the result of decoding read_chunk_size bytes read from the subprocess’ out/err stream.

New in version 1.0.

write_our_output(stream: IO, string: str) → None¶

Write string to stream.

Also calls .flush() on stream to ensure that real terminal streams don’t buffer.

Parameters:

stream – A file-like stream object, mapping to the out_stream or err_stream parameters of run.
string – A Unicode string object.

Returns:

None.

New in version 1.0.

handle_stdout(buffer_: List[str], hide: bool, output: IO) → None¶

Read process’ stdout, storing into a buffer & printing/parsing.

Intended for use as a thread target. Only terminates when all stdout from the subprocess has been read.

Parameters:

buffer – The capture buffer shared with the main thread.
hide (bool) – Whether or not to replay data into output.
output – Output stream (file-like object) to write data into when not hiding.

Returns:

None.

New in version 1.0.

handle_stderr(buffer_: List[str], hide: bool, output: IO) → None¶

Read process’ stderr, storing into a buffer & printing/parsing.

Identical to handle_stdout except for the stream read from; see its docstring for API details.

New in version 1.0.

read_our_stdin(input_: IO) → str | None¶

Read & decode bytes from a local stdin stream.

Parameters:: input – Actual stream object to read from. Maps to in_stream in run, so will often be sys.stdin, but might be any stream-like object.
Returns:: A Unicode string, the result of decoding the read bytes (this might be the empty string if the pipe has closed/reached EOF); or None if stdin wasn’t ready for reading yet.

New in version 1.0.

handle_stdin(input_: IO, output: IO, echo: bool = False) → None¶

Read local stdin, copying into process’ stdin as necessary.

Intended for use as a thread target.

Note

Because real terminal stdin streams have no well-defined “end”, if such a stream is detected (based on existence of a callable .fileno()) this method will wait until program_finished is set, before terminating.

When the stream doesn’t appear to be from a terminal, the same semantics as handle_stdout are used - the stream is simply read() from until it returns an empty value.

Parameters:

input – Stream (file-like object) from which to read.
output – Stream (file-like object) to which echoing may occur.
echo (bool) – User override option for stdin-stdout echoing.

Returns:

None.

New in version 1.0.

should_echo_stdin(input_: IO, output: IO) → bool¶

Determine whether data read from input_ should echo to output.

Used by handle_stdin; tests attributes of input_ and output.

Parameters:

input – Input stream (file-like object).
output – Output stream (file-like object).

Returns:

A bool.

New in version 1.0.

respond(buffer_: List[str]) → None¶

Write to the program’s stdin in response to patterns in buffer_.

The patterns and responses are driven by the StreamWatcher instances from the watchers kwarg of run - see Automatically responding to program output for a conceptual overview.

Parameters:: buffer – The capture buffer for this thread’s particular IO stream.
Returns:: None.

New in version 1.0.

generate_env(env: Dict[str, Any], replace_env: bool) → Dict[str, Any]¶

Return a suitable environment dict based on user input & behavior.

Parameters:

env (dict) – Dict supplying overrides or full env, depending.
replace_env (bool) – Whether env updates, or is used in place of, the value of os.environ.

Returns:

A dictionary of shell environment vars.

New in version 1.0.

should_use_pty(pty: bool, fallback: bool) → bool¶

Should execution attempt to use a pseudo-terminal?

Parameters:

pty (bool) – Whether the user explicitly asked for a pty.
fallback (bool) – Whether falling back to non-pty execution should be allowed, in situations where pty=True but a pty could not be allocated.

New in version 1.0.

property has_dead_threads: bool¶

Detect whether any IO threads appear to have terminated unexpectedly.

Used during process-completion waiting (in wait) to ensure we don’t deadlock our child process if our IO processing threads have errored/died.

Returns:: True if any threads appear to have terminated with an exception, False otherwise.

New in version 1.0.

wait() → None¶

Block until the running command appears to have exited.

Returns:: None.

New in version 1.0.

write_proc_stdin(data: str) → None¶

Write encoded data to the running process’ stdin.

Parameters:: data – A Unicode string.
Returns:: None.

New in version 1.0.

decode(data: bytes) → str¶: Decode some data bytes, returning Unicode.

New in version 1.0.

property process_is_finished: bool¶

Determine whether our subprocess has terminated.

Note

The implementation of this method should be nonblocking, as it is used within a query/poll loop.

Returns:: True if the subprocess has finished running, False otherwise.

New in version 1.0.

start(command: str, shell: str, env: Dict[str, Any]) → None¶

Initiate execution of command (via shell, with env).

Typically this means use of a forked subprocess or requesting start of execution on a remote system.

In most cases, this method will also set subclass-specific member variables used in other methods such as wait and/or returncode.

Parameters:

command (str) – Command string to execute.
shell (str) – Shell to use when executing command.
env (dict) – Environment dict used to prep shell environment.

New in version 1.0.

start_timer(timeout: int) → None¶: Start a timer to kill our subprocess after timeout seconds.

read_proc_stdout(num_bytes: int) → bytes | None¶

Read num_bytes from the running process’ stdout stream.

Parameters:: num_bytes (int) – Number of bytes to read at maximum.
Returns:: A string/bytes object.

New in version 1.0.

read_proc_stderr(num_bytes: int) → bytes | None¶

Read num_bytes from the running process’ stderr stream.

Parameters:: num_bytes (int) – Number of bytes to read at maximum.
Returns:: A string/bytes object.

New in version 1.0.

close_proc_stdin() → None¶

Close running process’ stdin.

Returns:: None.

New in version 1.3.

default_encoding() → str¶

Return a string naming the expected encoding of subprocess streams.

This return value should be suitable for use by encode/decode methods.

New in version 1.0.

send_interrupt(interrupt: KeyboardInterrupt) → None¶

Submit an interrupt signal to the running subprocess.

In almost all implementations, the default behavior is what will be desired: submit to the subprocess’ stdin pipe. However, we leave this as a public method in case this default needs to be augmented or replaced.

Parameters:: interrupt – The locally-sourced KeyboardInterrupt causing the method call.
Returns:: None.

New in version 1.0.

returncode() → int | None¶

Return the numeric return/exit code resulting from command execution.

Returns:: int, if any reasonable return code could be determined, or None in corner cases where that was not possible.

New in version 1.0.

stop() → None¶

Perform final cleanup, if necessary.

This method is called within a finally clause inside the main run method. Depending on the subclass, it may be a no-op, or it may do things such as close network connections or open files.

Returns:: None

New in version 1.0.

kill() → None¶

Forcibly terminate the subprocess.

Typically only used by the timeout functionality.

This is often a “best-effort” attempt, e.g. remote subprocesses often must settle for simply shutting down the local side of the network connection and hoping the remote end eventually gets the message.

property timed_out: bool¶: Returns True if the subprocess stopped because it timed out.

New in version 1.3.

__weakref__¶: list of weak references to the object (if defined)

class invoke.runners.Local(context: Context)¶

Execute a command on the local system in a subprocess.

Note

When Invoke itself is executed without a controlling terminal (e.g. when sys.stdin lacks a useful fileno), it’s not possible to present a handle on our PTY to local subprocesses. In such situations, Local will fallback to behaving as if pty=False (on the theory that degraded execution is better than none at all) as well as printing a warning to stderr.

To disable this behavior, say fallback=False.

New in version 1.0.

__init__(context: Context) → None¶

Create a new runner with a handle on some Context.

Parameters:

context –

a Context instance, used to transmit default options and provide access to other contextualized information (e.g. a remote-oriented Runner might want a Context subclass holding info about hostnames and ports.)

Note

The Context given to Runner instances must contain default config values for the Runner class in question. At a minimum, this means values for each of the default Runner.run keyword arguments such as echo and warn.

Raises:

exceptions.ValueError – if not all expected default values are found in context.

should_use_pty(pty: bool = False, fallback: bool = True) → bool¶

Should execution attempt to use a pseudo-terminal?

Parameters:

pty (bool) – Whether the user explicitly asked for a pty.
fallback (bool) – Whether falling back to non-pty execution should be allowed, in situations where pty=True but a pty could not be allocated.

New in version 1.0.

read_proc_stdout(num_bytes: int) → bytes | None¶

Read num_bytes from the running process’ stdout stream.

Parameters:: num_bytes (int) – Number of bytes to read at maximum.
Returns:: A string/bytes object.

New in version 1.0.

read_proc_stderr(num_bytes: int) → bytes | None¶

Read num_bytes from the running process’ stderr stream.

Parameters:: num_bytes (int) – Number of bytes to read at maximum.
Returns:: A string/bytes object.

New in version 1.0.

close_proc_stdin() → None¶

Close running process’ stdin.

Returns:: None.

New in version 1.3.

start(command: str, shell: str, env: Dict[str, Any]) → None¶

Initiate execution of command (via shell, with env).

Typically this means use of a forked subprocess or requesting start of execution on a remote system.

In most cases, this method will also set subclass-specific member variables used in other methods such as wait and/or returncode.

Parameters:

command (str) – Command string to execute.
shell (str) – Shell to use when executing command.
env (dict) – Environment dict used to prep shell environment.

New in version 1.0.

kill() → None¶

Forcibly terminate the subprocess.

Typically only used by the timeout functionality.

This is often a “best-effort” attempt, e.g. remote subprocesses often must settle for simply shutting down the local side of the network connection and hoping the remote end eventually gets the message.

property process_is_finished: bool¶

Determine whether our subprocess has terminated.

Note

The implementation of this method should be nonblocking, as it is used within a query/poll loop.

Returns:: True if the subprocess has finished running, False otherwise.

New in version 1.0.

returncode() → int | None¶

Return the numeric return/exit code resulting from command execution.

Returns:: int, if any reasonable return code could be determined, or None in corner cases where that was not possible.

New in version 1.0.

stop() → None¶

Perform final cleanup, if necessary.

This method is called within a finally clause inside the main run method. Depending on the subclass, it may be a no-op, or it may do things such as close network connections or open files.

Returns:: None

New in version 1.0.

class invoke.runners.Result(stdout: str = '', stderr: str = '', encoding: str | None = None, command: str = '', shell: str = '', env: Dict[str, Any] | None = None, exited: int = 0, pty: bool = False, hide: Tuple[str, ...] = ())¶

A container for information about the result of a command execution.

All params are exposed as attributes of the same name and type.

Parameters:

stdout (str) – The subprocess’ standard output.
stderr (str) – Same as stdout but containing standard error (unless the process was invoked via a pty, in which case it will be empty; see Runner.run.)
encoding (str) – The string encoding used by the local shell environment.
command (str) – The command which was executed.
shell (str) – The shell binary used for execution.
env (dict) – The shell environment used for execution. (Default is the empty dict, {}, not None as displayed in the signature.)
exited (int) –
An integer representing the subprocess’ exit/return code.

Note

This may be None in situations where the subprocess did not run to completion, such as when auto-responding failed or a timeout was reached.
pty (bool) – A boolean describing whether the subprocess was invoked with a pty or not; see Runner.run.
hide (tuple) –
A tuple of stream names (none, one or both of ('stdout', 'stderr')) which were hidden from the user when the generating command executed; this is a normalized value derived from the hide parameter of Runner.run.

For example, run('command', hide='stdout') will yield a Result where result.hide == ('stdout',); hide=True or hide='both' results in result.hide == ('stdout', 'stderr'); and hide=False (the default) generates result.hide == () (the empty tuple.)

Note

Result objects’ truth evaluation is equivalent to their ok attribute’s value. Therefore, quick-and-dirty expressions like the following are possible:

if run("some shell command"):
    do_something()
else:
    handle_problem()

However, remember Zen of Python #2.

New in version 1.0.

__init__(stdout: str = '', stderr: str = '', encoding: str | None = None, command: str = '', shell: str = '', env: Dict[str, Any] | None = None, exited: int = 0, pty: bool = False, hide: Tuple[str, ...] = ())¶

property return_code: int¶: An alias for .exited.

New in version 1.0.

__str__() → str¶: Return str(self).

__repr__() → str¶: Return repr(self).

property ok: bool¶: A boolean equivalent to exited == 0.

New in version 1.0.

property failed: bool¶

The inverse of ok.

I.e., True if the program exited with a nonzero return code, and False otherwise.

New in version 1.0.

tail(stream: str, count: int = 10) → str¶

Return the last count lines of stream, plus leading whitespace.

Parameters:

stream (str) – Name of some captured stream attribute, eg "stdout".
count (int) – Number of lines to preserve.

New in version 1.3.

__weakref__¶: list of weak references to the object (if defined)

class invoke.runners.Promise(runner: Runner)¶

A promise of some future Result, yielded from asynchronous execution.

This class’ primary API member is join; instances may also be used as context managers, which will automatically call join when the block exits. In such cases, the context manager yields self.

Promise also exposes copies of many Result attributes, specifically those that derive from run kwargs and not the result of command execution. For example, command is replicated here, but stdout is not.

New in version 1.4.

__init__(runner: Runner) → None¶

Create a new promise.

Parameters:

runner –

An in-flight Runner instance making this promise.

Must already have started the subprocess and spun up IO threads.

join() → Result¶

Block until associated subprocess exits, returning/raising the result.

This acts identically to the end of a synchronously executed run, namely that:

various background threads (such as IO workers) are themselves joined;
if the subprocess exited normally, a Result is returned;
in any other case (unforeseen exceptions, IO sub-thread ThreadException, Failure, WatcherError) the relevant exception is raised here.

See run docs, or those of the relevant classes, for further details.

invoke.runners.default_encoding() → str¶

Obtain apparent interpreter-local default text encoding.

Often used as a baseline in situations where we must use SOME encoding for unknown-but-presumably-text bytes, and the user has not specified an override.

`runners`¶

Invoke

Navigation

Donate/support

runners¶

`runners`¶