303 lines
10 KiB
ReStructuredText
303 lines
10 KiB
ReStructuredText
.. _module-pw_presubmit:
|
||
|
||
============
|
||
pw_presubmit
|
||
============
|
||
The presubmit module provides Python tools for running presubmit checks and
|
||
checking and fixing code format. It also includes the presubmit check script for
|
||
the Pigweed repository, ``pigweed_presubmit.py``.
|
||
|
||
Presubmit checks are essential tools, but they take work to set up, and
|
||
projects don’t always get around to it. The ``pw_presubmit`` module provides
|
||
tools for setting up high quality presubmit checks for any project. We use this
|
||
framework to run Pigweed’s presubmit on our workstations and in our automated
|
||
building tools.
|
||
|
||
The ``pw_presubmit`` module also includes ``pw format``, a tool that provides a
|
||
unified interface for automatically formatting code in a variety of languages.
|
||
With ``pw format``, you can format Bazel, C, C++, Python, GN, and Go code
|
||
according to configurations defined by your project. ``pw format`` leverages
|
||
existing tools like ``clang-format``, and it’s simple to add support for new
|
||
languages. (Note: Bazel formatting requires ``buildifier`` to be present on your
|
||
system. If it's not Bazel formatting passes without checking.)
|
||
|
||
.. image:: docs/pw_presubmit_demo.gif
|
||
:alt: ``pw format`` demo
|
||
:align: left
|
||
|
||
The ``pw_presubmit`` package includes presubmit checks that can be used with any
|
||
project. These checks include:
|
||
|
||
* Check code format of several languages including C, C++, and Python
|
||
* Initialize a Python environment
|
||
* Run all Python tests
|
||
* Run pylint
|
||
* Run mypy
|
||
* Ensure source files are included in the GN and Bazel builds
|
||
* Build and run all tests with GN
|
||
* Build and run all tests with Bazel
|
||
* Ensure all header files contain ``#pragma once``
|
||
|
||
-------------
|
||
Compatibility
|
||
-------------
|
||
Python 3
|
||
|
||
-------------------------------------------
|
||
Creating a presubmit check for your project
|
||
-------------------------------------------
|
||
Creating a presubmit check for a project using ``pw_presubmit`` is simple, but
|
||
requires some customization. Projects must define their own presubmit check
|
||
Python script that uses the ``pw_presubmit`` package.
|
||
|
||
A project's presubmit script can be registered as a
|
||
:ref:`pw_cli <module-pw_cli>` plugin, so that it can be run as ``pw
|
||
presubmit``.
|
||
|
||
Setting up the command-line interface
|
||
=====================================
|
||
The ``pw_presubmit.cli`` module sets up the command-line interface for a
|
||
presubmit script. This defines a standard set of arguments for invoking
|
||
presubmit checks. Its use is optional, but recommended.
|
||
|
||
pw_presubmit.cli
|
||
----------------
|
||
.. automodule:: pw_presubmit.cli
|
||
:members: add_arguments, run
|
||
|
||
Presubmit output directory
|
||
--------------------------
|
||
The ``pw_presubmit`` command line interface includes an ``--output-directory``
|
||
option that specifies the working directory to use for presubmits. The default
|
||
path is ``out/presubmit``. A subdirectory is created for each presubmit step.
|
||
This directory persists between presubmit runs and can be cleaned by deleting it
|
||
or running ``pw presubmit --clean``.
|
||
|
||
Presubmit checks
|
||
================
|
||
A presubmit check is defined as a function or other callable. The function must
|
||
accept one argument: a ``PresubmitContext``, which provides the paths on which
|
||
to run. Presubmit checks communicate failure by raising an exception.
|
||
|
||
Presubmit checks may use the ``filter_paths`` decorator to automatically filter
|
||
the paths list for file types they care about.
|
||
|
||
Either of these functions could be used as presubmit checks:
|
||
|
||
.. code-block:: python
|
||
|
||
@pw_presubmit.filter_paths(endswith='.py')
|
||
def file_contains_ni(ctx: PresubmitContext):
|
||
for path in ctx.paths:
|
||
with open(path) as file:
|
||
contents = file.read()
|
||
if 'ni' not in contents and 'nee' not in contents:
|
||
raise PresumitFailure('Files must say "ni"!', path=path)
|
||
|
||
def run_the_build(_):
|
||
subprocess.run(['make', 'release'], check=True)
|
||
|
||
Presubmit checks functions are grouped into "programs" -- a named series of
|
||
checks. Projects may find it helpful to have programs for different purposes,
|
||
such as a quick program for local use and a full program for automated use. The
|
||
:ref:`example script <example-script>` uses ``pw_presubmit.Programs`` to define
|
||
``quick`` and ``full`` programs.
|
||
|
||
Existing Presubmit Checks
|
||
-------------------------
|
||
A small number of presubmit checks are made available through ``pw_presubmit``
|
||
modules.
|
||
|
||
Code Formatting
|
||
^^^^^^^^^^^^^^^
|
||
Formatting checks for a variety of languages are available from
|
||
``pw_presubmit.format_code``. These include C/C++, Java, Go, Python, GN, and
|
||
others. All of these checks can be included by adding
|
||
``pw_presubmit.format_code.presubmit_checks()`` to a presubmit program. These
|
||
all use language-specific formatters like clang-format or yapf.
|
||
|
||
These will suggest fixes using ``pw format --fix``.
|
||
|
||
#pragma once
|
||
^^^^^^^^^^^^
|
||
There's a ``pragma_once`` check that confirms the first non-comment line of
|
||
C/C++ headers is ``#pragma once``. This is enabled by adding
|
||
``pw_presubmit.pragma_once`` to a presubmit program.
|
||
|
||
Python Checks
|
||
^^^^^^^^^^^^^
|
||
There are two checks in the ``pw_presubmit.python_checks`` module, ``gn_pylint``
|
||
and ``gn_python_check``. They assume there's a top-level ``python`` GN target.
|
||
``gn_pylint`` runs Pylint and Mypy checks and ``gn_python_check`` runs Pylint,
|
||
Mypy, and all Python tests.
|
||
|
||
Inclusive Language
|
||
^^^^^^^^^^^^^^^^^^
|
||
.. inclusive-language: disable
|
||
|
||
The inclusive language check looks for words that are typical of non-inclusive
|
||
code, like using "master" and "slave" in place of "primary" and "secondary" or
|
||
"sanity check" in place of "consistency check".
|
||
|
||
.. inclusive-language: enable
|
||
|
||
These checks can be disabled for individual lines with
|
||
"inclusive-language: ignore" on the line in question or the line above it, or
|
||
for entire blocks by using "inclusive-language: disable" before the block and
|
||
"inclusive-language: enable" after the block.
|
||
|
||
.. In case things get moved around in the previous paragraphs the enable line
|
||
.. is repeated here: inclusive-language: enable.
|
||
|
||
pw_presubmit
|
||
------------
|
||
.. automodule:: pw_presubmit
|
||
:members: filter_paths, call, PresubmitFailure, Programs
|
||
|
||
.. _example-script:
|
||
|
||
Example
|
||
=======
|
||
A simple example presubmit check script follows. This can be copied-and-pasted
|
||
to serve as a starting point for a project's presubmit check script.
|
||
|
||
See ``pigweed_presubmit.py`` for a more complex presubmit check script example.
|
||
|
||
.. code-block:: python
|
||
|
||
"""Example presubmit check script."""
|
||
|
||
import argparse
|
||
import logging
|
||
import os
|
||
from pathlib import Path
|
||
import re
|
||
import sys
|
||
from typing import List, Pattern
|
||
|
||
try:
|
||
import pw_cli.log
|
||
except ImportError:
|
||
print('ERROR: Activate the environment before running presubmits!',
|
||
file=sys.stderr)
|
||
sys.exit(2)
|
||
|
||
import pw_presubmit
|
||
from pw_presubmit import (
|
||
build,
|
||
cli,
|
||
cpp_checks,
|
||
environment,
|
||
format_code,
|
||
git_repo,
|
||
inclusive_language,
|
||
filter_paths,
|
||
python_checks,
|
||
PresubmitContext,
|
||
)
|
||
from pw_presubmit.install_hook import install_hook
|
||
|
||
# Set up variables for key project paths.
|
||
PROJECT_ROOT = Path(os.environ['MY_PROJECT_ROOT'])
|
||
PIGWEED_ROOT = PROJECT_ROOT / 'pigweed'
|
||
|
||
# Rerun the build if files with these extensions change.
|
||
_BUILD_EXTENSIONS = frozenset(
|
||
['.rst', '.gn', '.gni', *format_code.C_FORMAT.extensions])
|
||
|
||
|
||
#
|
||
# Presubmit checks
|
||
#
|
||
def release_build(ctx: PresubmitContext):
|
||
build.gn_gen(PROJECT_ROOT, ctx.output_dir, build_type='release')
|
||
build.ninja(ctx.output_dir)
|
||
|
||
|
||
def host_tests(ctx: PresubmitContext):
|
||
build.gn_gen(PROJECT_ROOT, ctx.output_dir, run_host_tests='true')
|
||
build.ninja(ctx.output_dir)
|
||
|
||
|
||
# Avoid running some checks on certain paths.
|
||
PATH_EXCLUSIONS = (
|
||
re.compile(r'^external/'),
|
||
re.compile(r'^vendor/'),
|
||
)
|
||
|
||
|
||
# Use the upstream pragma_once check, but apply a different set of path
|
||
# filters with @filter_paths.
|
||
@filter_paths(endswith='.h', exclude=PATH_EXCLUSIONS)
|
||
def pragma_once(ctx: PresubmitContext):
|
||
cpp_checks.pragma_once(ctx)
|
||
|
||
|
||
#
|
||
# Presubmit check programs
|
||
#
|
||
OTHER = (
|
||
# Checks not ran by default but that should be available. These might
|
||
# include tests that are expensive to run or that don't yet pass.
|
||
build.gn_quick_check,
|
||
)
|
||
|
||
QUICK = (
|
||
# List some presubmit checks to run
|
||
pragma_once,
|
||
host_tests,
|
||
# Use the upstream formatting checks, with custom path filters applied.
|
||
format_code.presubmit_checks(exclude=PATH_EXCLUSIONS),
|
||
# Include the upstream inclusive language check.
|
||
inclusive_language.inclusive_language,
|
||
# Include just the lint-related Python checks.
|
||
python_checks.gn_pylint.with_filter(exclude=PATH_EXCLUSIONS),
|
||
)
|
||
|
||
FULL = (
|
||
QUICK, # Add all checks from the 'quick' program
|
||
release_build,
|
||
# Use the upstream Python checks, with custom path filters applied.
|
||
# Checks listed multiple times are only run once.
|
||
python_checks.gn_python_check.with_filter(exclude=PATH_EXCLUSIONS),
|
||
)
|
||
|
||
PROGRAMS = pw_presubmit.Programs(other=OTHER, quick=QUICK, full=FULL)
|
||
|
||
|
||
def run(install: bool, **presubmit_args) -> int:
|
||
"""Process the --install argument then invoke pw_presubmit."""
|
||
|
||
# Install the presubmit Git pre-push hook, if requested.
|
||
if install:
|
||
install_hook(__file__, 'pre-push', ['--base', 'HEAD~'],
|
||
git_repo.root())
|
||
return 0
|
||
|
||
return cli.run(root=PROJECT_ROOT, **presubmit_args)
|
||
|
||
|
||
def main() -> int:
|
||
"""Run the presubmit checks for this repository."""
|
||
parser = argparse.ArgumentParser(description=__doc__)
|
||
cli.add_arguments(parser, PROGRAMS, 'quick')
|
||
|
||
# Define an option for installing a Git pre-push hook for this script.
|
||
parser.add_argument(
|
||
'--install',
|
||
action='store_true',
|
||
help='Install the presubmit as a Git pre-push hook and exit.')
|
||
|
||
return run(**vars(parser.parse_args()))
|
||
|
||
if __name__ == '__main__':
|
||
pw_cli.log.install(logging.INFO)
|
||
sys.exit(main())
|
||
|
||
---------------------
|
||
Code formatting tools
|
||
---------------------
|
||
The ``pw_presubmit.format_code`` module formats supported source files using
|
||
external code format tools. The file ``format_code.py`` can be invoked directly
|
||
from the command line or from ``pw`` as ``pw format``.
|