How Checks are Run
In Flake8 2.x, Flake8 delegated check running to pep8. In 3.0 Flake8
takes on that responsibility. This has allowed for simpler
handling of the --jobs
parameter (using multiprocessing
) and
simplified our fallback if something goes awry with concurrency.
At the lowest level we have a FileChecker
. Instances of FileChecker
are
created for each file to be analyzed by Flake8. Each instance, has a copy
of all of the plugins registered with setuptools in the flake8.extension
entry-point group.
The FileChecker
instances are managed by an instance of Manager
. The
Manager
instance handles creating sub-processes with
multiprocessing
module and falling back to running checks in serial if
an operating system level error arises. When creating FileChecker
instances,
the Manager
is responsible for determining if a particular file has been
excluded.
Processing Files
Unfortunately, since Flake8 took over check running from pep8/pycodestyle,
it also had to take over parsing and processing files for the checkers
to use. Since it couldn’t reuse pycodestyle’s functionality (since it did not
separate cleanly the processing from check running) that function was isolated
into the FileProcessor
class. We moved
several helper functions into the flake8.processor
module (see also
Processor Utility Functions).
API Reference
- class flake8.checker.FileChecker(*, filename, plugins, options)[source]
Manage running checks for a file and aggregate the results.
- Parameters:
filename (str) –
plugins (Checkers) –
options (argparse.Namespace) –
- check_physical_eol(token, prev_physical)[source]
Run physical checks if and only if it is at the end of the line.
- Parameters:
token (TokenInfo) –
prev_physical (str) –
- Return type:
None
- handle_newline(token_type)[source]
Handle the logic when encountering a newline token.
- Parameters:
token_type (int) –
- Return type:
None
- process_tokens()[source]
Process tokens and trigger checks.
Instead of using this directly, you should use
flake8.checker.FileChecker.run_checks()
.- Return type:
None
- report(error_code, line_number, column, text)[source]
Report an error by storing it in the results list.
- class flake8.checker.Manager(style_guide, plugins, argv)[source]
Manage the parallelism and checker instances for each plugin and file.
This class will be responsible for the following:
Determining the parallelism of Flake8, e.g.:
Do we use
multiprocessing
or is it unavailable?Do we automatically decide on the number of jobs to use or did the user provide that?
Falling back to a serial way of processing files if we run into an OSError related to
multiprocessing
Organizing the results of each checker so we can group the output together and make our output deterministic.
- Parameters:
style_guide (StyleGuideManager) –
plugins (Checkers) –
argv (Sequence[str]) –
- report()[source]
Report all of the errors found in the managed file checkers.
This iterates over each of the checkers and reports the errors sorted by line number.
- run()[source]
Run all the checkers.
This will intelligently decide whether to run the checks in parallel or whether to run them in serial.
If running the checks in parallel causes a problem (e.g., #117) this also implements fallback to serial processing.
- Return type:
None
- class flake8.processor.FileProcessor(filename, options, lines=None)[source]
Processes a file and holds state.
This processes a file by generating tokens, logical and physical lines, and AST trees. This also provides a way of passing state about the file to checks expecting that state. Any public attribute on this object can be requested by a plugin. The known public attributes are:
- Parameters:
filename (str) –
options (argparse.Namespace) –
- blank_before
Number of preceding blank lines
- blank_lines
Number of blank lines
- fstring_start(lineno)[source]
Signal the beginning of an fstring.
- Parameters:
lineno (int) –
- Return type:
None
- generate_tokens()[source]
Tokenize the file and yield the tokens.
- Return type:
Generator[TokenInfo, None, None]
- hang_closing
User provided option for hang closing
- indent_level
Current level of indentation
- indent_size
Number of spaces used for indentation
- keyword_arguments_for(parameters, arguments)[source]
Generate the keyword arguments for a list of parameters.
- line_number
Line number in the file
- logical_line
Current logical line
- max_doc_length
Maximum docstring / comment line length as configured by the user
- max_line_length
Maximum line length as configured by the user
- multiline
Whether the current physical line is multiline
- next_logical_line()[source]
Record the previous logical line.
This also resets the tokens list and the blank_lines count.
- Return type:
None
- noqa = False
always
False
, included for compatibility
- previous_indent_level
Previous level of indentation
- previous_logical
Previous logical line
- previous_unindented_logical_line
Previous unindented (i.e. top-level) logical line
- should_ignore_file()[source]
Check if
flake8: noqa
is in the file to be ignored.- Returns:
True if a line matches
defaults.NOQA_FILE
, otherwise False- Return type:
- statistics
Statistics dictionary
- total_lines
Total number of lines in the file
- update_checker_state_for(plugin)[source]
Update the checker_state attribute for the plugin.
- Parameters:
plugin (LoadedPlugin) –
- Return type:
None
- verbose
Verbosity level of Flake8
Utility Functions
- flake8.processor.count_parentheses(current_parentheses_count, token_text)[source]
Count the number of parentheses.
- flake8.processor.expand_indent(line)[source]
Return the amount of indentation.
Tabs are expanded to the next multiple of 8.
>>> expand_indent(' ') 4 >>> expand_indent('\t') 8 >>> expand_indent(' \t') 8 >>> expand_indent(' \t') 16
- flake8.processor.is_eol_token(token)[source]
Check if the token is an end-of-line token.
- Parameters:
token (TokenInfo) –
- Return type:
- flake8.processor.is_multiline_string(token)[source]
Check if this is a multiline string.
- Parameters:
token (TokenInfo) –
- Return type: