Experimentations on Bazel: Python (3), linter & pytest

In the previous articles,we saw how to launch linter for python with a custom rule, and in the one before how to launch pytest test. In this article, we'll see how to combine both usage in more friendly way (IMHO).

Since the previous article, I did more experimentations (on my side), and I was limited by my current knowledge about how to implement rules (and managed dependency). But with bazel's macro, we could fix (or reduce) some issues:

duplication of pytest's boilerplate every BUILD.bazel and *test.py
duplication between pytest and potential linter
too many custom code to please bazel

Create the macro

Create the empty tools/pytest/BUILD.bazel , just to initiate the package.

Create the macro tools/pytest/defs.bzl, it is like the existing py_test defined in exp_python/webapp/BUILD.bazel but with the call inside a function and some part configurable.

"""experience wrap py_test"""

load("@rules_python//python:defs.bzl", "py_test")
load("@my_python_deps//:requirements.bzl", "requirement")

def pytest_test(name, srcs, deps = [], args = [], **kwargs):
    """
        Call pytest
    """
    py_test(
        name = name,
        srcs = [
            "//tools/pytest:pytest_wrapper.py",
        ] + srcs,
        main = "//tools/pytest:pytest_wrapper.py",
        args = [
            "--capture=no",
        ] + args + ["$(location :%s)" % x for x in srcs],
        python_version = "PY3",
        srcs_version = "PY3",
        deps = deps + [
            requirement("pytest"),
        ],
        **kwargs
    )

load to import definition of symbols
pytest_test is the macro, in fact a function that call py_test
//tools/pytest:pytest_wrapper.py is the wrapper to call pytest (see below)
name, srcs, deps, args are the argument of that will customize
**kwargs allow to forward some args from the caller side for the part that are not overwitten by the macro
["$(location :%s)" % x for x in srcs] is to append at the end of the call the list of source file to process, to have the path of those file (srcs is a list of string at this point), we call $(location ...) to have the path in the execution context of the py_test and and the prefix : is to convert the file name into a label. It's hacky but it works

Then create the wrapper that will call pytest tools/pytest/pytest_wrapper.py

import sys
import pytest

# if using 'bazel test ...'
if __name__ == "__main__":
    sys.exit(pytest.main(sys.argv[1:]))

Replace the current call to py_test by a call to hour macro in exp_python/webapp/BUILD.bazel.

load("//tools/pytest:defs.bzl", "pytest_test")
...

# py_test(
#     name = "test",
#     srcs = [
#         "test.py",
#     ],
#     # main = "test.py",
#     args = [
#         "--capture=no",
#     ],
#     python_version = "PY3",
#     srcs_version = "PY3",
#     deps = [
#         ":webapp",
#         requirement("requests"),
#         requirement("fastapi"),
#         requirement("pytest"),
#     ],
# )

pytest_test(
    name = "test",
    srcs = ["test.py"],
    deps = [
        ":webapp",
        requirement("requests"),
        # requirement("fastapi"),
    ],
)

We keep commented the previous code for comparison, it's shorter and we move into a central shared place code that could be shared with other python package. Give a try...

❯ bazel test //exp_python/webapp:test
ERROR: /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/BUILD.bazel:32:12: no such target '//tools/pytest:pytest_wrapper.py': target 'pytest_wrapper.py' not declared in package 'tools/pytest'; however, a source file of this name exists.  (Perhaps add 'exports_files(["pytest_wrapper.py"])' to tools/pytest/BUILD?) defined by /home/david/src/github.com/davidB/sandbox_bazel/tools/pytest/BUILD.bazel and referenced by '//exp_python/webapp:test'
ERROR: /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/BUILD.bazel:32:12: no such target '//tools/pytest:pytest_wrapper.py': target 'pytest_wrapper.py' not declared in package 'tools/pytest'; however, a source file of this name exists.  (Perhaps add 'exports_files(["pytest_wrapper.py"])' to tools/pytest/BUILD?) defined by /home/david/src/github.com/davidB/sandbox_bazel/tools/pytest/BUILD.bazel and referenced by '//exp_python/webapp:test'
ERROR: Analysis of target '//exp_python/webapp:test' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.704s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (18 packages loaded, 19 targets configured)
FAILED: Build did NOT complete successfully (18 packages loaded, 19 targets configured)

Ok, we need to fix some stuff, because it's a macro we need to expose pytest_wrapper.py to caller of the macro. So just follow the recommendation and change the empty tools/pytest/BUILD.bazel into

exports_files(["pytest_wrapper.py"])

Retry...

❯ bazel test //exp_python/webapp:test
INFO: Analyzed target //exp_python/webapp:test (26 packages loaded, 15198 targets configured).
INFO: Found 1 test target...
Target //exp_python/webapp:test up-to-date:
  bazel-bin/exp_python/webapp/test
INFO: Elapsed time: 2.242s, Critical Path: 0.99s
INFO: 5 processes: 3 internal, 2 linux-sandbox.
INFO: Build completed successfully, 5 total actions
//exp_python/webapp:test                                                 PASSED in 0.7s

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these aINFO: Build completed successfully, 5 total actions

First victory.

We can now remove the boilerblate to call pytest at the end of exp_python/webapp/test.py because it is now provided by the shared tools/pytest/pytest_wrapper.py.

from fastapi.testclient import TestClient
from exp_python.webapp.main import app

client = TestClient(app)


def test_read_main():
    response = client.get("/status")
    assert response.status_code == 200
    assert response.json() == {"status": "UP", "version": "0.1.0"}

You can mofidy the test to force the fail, and check if it'll fail as expected.

Use Pytest to launch linter

Pytest have lot of plugin/extension to launch tool like black, pylint, flake8, mypy,... So instead of create one linter call we can configure pytest run all linter.

Add linter into third_party/requirements.txt

...
#test
requests==2.25.1
pytest==6.1.2

#tools
black==20.8b1
pytest-black==0.3.12
pytest-mypy==0.8.1
pytest-pylint==0.18.0
pylint==2.8.2

Configure tools/pytest/defs.bzl to also launch linter and configure the call to pytest (it's like a script and a configuration)

"""experience wrap py_test"""

load("@rules_python//python:defs.bzl", "py_test")
load("@my_python_deps//:requirements.bzl", "requirement")

def pytest_test(name, srcs, deps = [], args = [], **kwargs):
    """
        Call pytest
    """
    py_test(
        name = name,
        srcs = [
            "//tools/pytest:pytest_wrapper.py",
        ] + srcs,
        main = "//tools/pytest:pytest_wrapper.py",
        args = [
            "--capture=no",
            "--black",
            "--pylint",
            "--mypy",
        ] + args + ["$(location :%s)" % x for x in srcs],
        python_version = "PY3",
        srcs_version = "PY3",
        deps = deps + [
            requirement("pytest"),
            requirement("pytest-black"),
            requirement("pytest-pylint"),
            requirement("pytest-mypy"),
        ],
        **kwargs
    )

Run it

❯ bazel test //exp_python/webapp:test
INFO: Analyzed target //exp_python/webapp:test (20 packages loaded, 2332 targets configured).
INFO: Found 1 test target...
FAIL: //exp_python/webapp:test (see /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/execroot/__main__/bazel-out/k8-fastbuild/testlogs/exp_python/webapp/test/test.log)
INFO: From Testing //exp_python/webapp:test:
==================== Test output for //exp_python/webapp:test:
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-6.1.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/3/execroot/__main__/bazel-out/k8-fastbuild/bin/exp_python/webapp/test.runfiles/__main__
plugins: black-0.3.12, pylint-0.18.0, mypy-0.8.1
collected 5 items
--------------------------------------------------------------------------------
Linting files
....Unable to create directory /home/david/.pylint.d
Unable to create file /home/david/.pylint.d/exp_python.__init__1.stats: [Errno 2] No such file or directory: '/home/david/.pylint.d/exp_python.__init__1.stats'

--------------------------------------------------------------------------------

exp_python/webapp/test.py FFF..

=================================== FAILURES ===================================
______________________ [pylint] exp_python/webapp/test.py ______________________
C:  1, 0: Missing module docstring (missing-module-docstring)
C:  7, 0: Missing function or method docstring (missing-function-docstring)
__________________________ exp_python/webapp/test.py ___________________________
1: error: Cannot find implementation or library stub for module named 'fastapi.testclient'
1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
_________________________________ test session _________________________________
mypy exited with status 1.
===================================== mypy =====================================
exp_python/webapp/main.py:1: error: Cannot find implementation or library stub for module named 'fastapi'
Found 2 errors in 2 files (checked 1 source file)
=========================== short test summary info ============================
FAILED exp_python/webapp/test.py::PYLINT
FAILED exp_python/webapp/test.py::mypy
FAILED exp_python/webapp/test.py::mypy-status
========================= 3 failed, 2 passed in 1.99s ==========================
================================================================================
Target //exp_python/webapp:test up-to-date:
  bazel-bin/exp_python/webapp/test
INFO: Elapsed time: 3.841s, Critical Path: 3.45s
INFO: 5 processes: 3 internal, 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 5 total actions
//exp_python/webapp:test                                                 FAILED in 3.0s
  /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/execroot/__main__/bazel-out/k8-fastbuild/testlogs/exp_python/webapp/test/test.log

INFO: Build completed, 1 test FAILED, 5 total actions

Seems to work (or failed as expected, linter detect issue)

Now extend pytest_test to lint over every *.py of the package (and let pytest detects test). As we check every python file, we also need to add all their dependencies.

pytest_test(
    name = "test",
    srcs = glob(["*.py"]),
    deps = [
        ":run",
        ":webapp",
        requirement("requests"),
        requirement("fastapi"),
    ],
)

To fix the linter issue (I comments mypy, not useful for the purpose of this article), provide a configuration file for pylint and disable a pylint rule localy into run.py (full code available into the repo, see bellow). So the final tools/pytest/defs.bzl looks like (with a configuration file for the demo, provide as data to py_test and as args)

"""Wrap pytest"""

load("@rules_python//python:defs.bzl", "py_test")
load("@my_python_deps//:requirements.bzl", "requirement")

def pytest_test(name, srcs, deps = [], args = [], data = [], **kwargs):
    """
        Call pytest
    """
    py_test(
        name = name,
        srcs = [
            "//tools/pytest:pytest_wrapper.py",
        ] + srcs,
        main = "//tools/pytest:pytest_wrapper.py",
        args = [
            "--capture=no",
            "--black",
            "--pylint",
            "--pylint-rcfile=$(location //tools/pytest:.pylintrc)",
            # "--mypy",
        ] + args + ["$(location :%s)" % x for x in srcs],
        python_version = "PY3",
        srcs_version = "PY3",
        deps = deps + [
            requirement("pytest"),
            requirement("pytest-black"),
            requirement("pytest-pylint"),
            # requirement("pytest-mypy"),
        ],
        data = [
            "//tools/pytest:.pylintrc",
        ] + data,
        **kwargs
    )

And modify tools/pytest/BUILD.bazel to export configuration files.

exports_files([
    "pytest_wrapper.py",
    ".pylintrc",
])

Clean up

Remove code and reference to pytest_check (that we introduce previously), it's no longer needed. Now we have

❯ tree -a -I '.vscode|bazel-*|.venv|.git'                                                                                      21:16:36
.
├── .bazelrc
├── .bazelversion
├── BUILD.bazel
├── exp_genrule
│   ├── a.txt
│   ├── b.txt
│   └── BUILD.bazel
├── exp_python
│   └── webapp
│       ├── BUILD.bazel
│       ├── main.py
│       ├── __pycache__
│       │   └── main.cpython-39.pyc
│       ├── run.py
│       └── test.py
├── .github
│   ├── dependabot.yml
│   └── workflows
│       └── ci.yml
├── .gitignore
├── .python-version
├── setup_localdev.sh
├── third_party
│   ├── BUILD.bazel
│   └── requirements.txt
├── tools
│   ├── pytest
│   │   ├── BUILD.bazel
│   │   ├── defs.bzl
│   │   ├── .pylintrc
│   │   └── pytest_wrapper.py
│   └── workspace_status.sh
└── WORKSPACE.bazel

Limits of the solution

Dependencies of test, runtime, tools are all mixed (no isolation), but it's the common (crappy ?) way to work in python ecosystem.
The macro is harder to extract as a workspace, tools for reuse into an other project, but in the other side the code is small enough to be copied and customized for each need
No split in the output of bazel every linter and test will be show as 1 test into report of bazel.

At the end, currently it's the most pleasant solution I found to have pytest, linters run by bazel without too many (and duplicate) boilerplate into each package. A central configuration place, and pretty simple to extends if pytest as the extension (else we can tweak the pytest_wrapper.py to do complementary stuff by example).

To be continued

It's not the end, we'll continue to setup other stuff like depending of another package (a python lib), make a docker image,...

The sandbox_bazel is hosted on github (not with the same history, due to errors), use tag to have the expected view at end of article: article/6_python_3. I'll be happy to have your comments on this article, or to discuss on github repo.

Blog

Experimentations on Bazel: Python (3), linter & pytest

David Bernard

Create the macro

Use Pytest to launch linter

Clean up

Limits of the solution

To be continued

Join Our Newsletter. No Spam, Only the good stuff.

Related