TheLazy.Dev
Portfolio GitHub

Blog

bazel |python |lint

Experimentations on Bazel: Python (2), linter

davidb31

David Bernard

Posted on May 2, 2021

Experimentations on Bazel: Python (2), linter

Time to add code formatter, checkers, linters and other tools that help to keep python code runnable, less buggy and more resilient to evolution.

Setup formatter for Editor

psf/black: The uncompromising Python code formatter will be used. Add it to thrid_party/requirements.txt

...

#tools
black==20.8b1
Enter fullscreen mode Exit fullscreen mode

Relaunch ./setup_localdev.sh and configure your editor to use it (if its can use virtual env), for example for vscode .vscode/settings.json

{
    "python.pythonPath": ".venv/bin/python",
    "python.formatting.provider": "black"
}
Enter fullscreen mode Exit fullscreen mode

Setup formatter for Bazel with bazel-linting-system

But bazel currently doesn't use it (to format or to check the format). After a quick search, thundergolfer/bazel-linting-system: πŸŒΏπŸ’š Experimental system for registering, configuring, and invoking source-code linters in Bazel. seems to be the solution, it's based on bazel's aspect something that we didn't explore or experiment yet. So give it a try by following instruction

Add to WORKSPACE.bazel

http_archive(
    name = "linting_system",
    sha256 = "",
    strip_prefix = "bazel-linting-system-0.4.0",
    url = "https://github.com/thundergolfer/bazel-linting-system/archive/v0.4.0.zip",
)

load("@linting_system//repositories:repositories.bzl", linting_sys_repositories = "repositories")
linting_sys_repositories()

load("@linting_system//repositories:go_repositories.bzl", linting_sys_deps = "go_deps")
linting_sys_deps()
Enter fullscreen mode Exit fullscreen mode

Create tools/linting/aspect.bzl

load("@linting_system//:generator.bzl", "linting_aspect_generator")

lint = linting_aspect_generator(
    name = "lint",
    linters = [
        "@//tools/linting:python",
    ]
)
Enter fullscreen mode Exit fullscreen mode

Create tools/linting/BUILD.bazel

load("@linting_system//:rules.bzl", "linter")

package(default_visibility = ['//visibility:public'])

linter(
    name = "python",
    executable_path = "/usr/local/bin/black",
)
Enter fullscreen mode Exit fullscreen mode

Run it to test

bazel build //... \
    --aspects //tools/linting:aspect.bzl%lint \
    --output_groups=report

...
bazel-out/k8-fastbuild/bin/exp_python/double/test_linter_exe: line 26: /usr/local/bin/black: No such file or directory
...
Enter fullscreen mode Exit fullscreen mode

As expected πŸ˜‰ it failed because we do not have black installed on the system, and it's better when the build system take care of the installation of tool (for repeatability, and to ease the developer setup (less step and

Create tools/linting/python_linter.py

import black
import sys

if __name__ == "__main__":
    sys.exit(black.main())
Enter fullscreen mode Exit fullscreen mode

manual operation)). But we can try to create a tool that will call black (or any other linter).

Modify tools/linting/BUILD.bazel to build python_linter and to use it as linter.

load("@linting_system//:rules.bzl", "linter")
load("@rules_python//python:defs.bzl", "py_binary")
load("@my_python_deps//:requirements.bzl", "requirement")

package(default_visibility = ["//visibility:public"])

py_binary(
    name = "python_linter",
    srcs = ["python_linter.py"],
    python_version = "PY3",
    srcs_version = "PY3",
    deps = [
        requirement("black"),
    ],
)
...
Enter fullscreen mode Exit fullscreen mode

Try to run it python_linter

❯ bazel run //tools/linting:python_linter -- --version        
...
python_linter.py, version 20.8b1
Enter fullscreen mode Exit fullscreen mode

OK, seems to work, update linter to use python_linter

...

linter(
    name = "python",
    executable = "python_linter",
)
Enter fullscreen mode Exit fullscreen mode

Note that we use executable and not executable_path, after look at the rule description

linter = rule(
    implementation = _linter_impl,
    attrs = {
        "executable_path": attr.string(
            doc="Absolute path to the linter that will run",
            mandatory=False,
        ),
        "executable": attr.label(
            executable=True,
            cfg="host",
            doc="Label for an executable linter",
        ),
        "config": attr.label(
            allow_files=True,
            doc="Configuration file for linter",
        ),
        "config_option": attr.string(
            doc="The option used by the linter to pass a path to a configuration file",
        ),
        "config_str": attr.string(
            doc="Raw string configuration options to be passed to linter",
        )
    },
)
Enter fullscreen mode Exit fullscreen mode

Try again

❯ bazel build //... \                                                                                                          20:48:18
          --aspects //tools/linting:aspect.bzl%lint \
          --output_groups=report
DEBUG: Rule 'linting_system' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "a254c73bdde03214b62cacdb570229ed1a1814a2ed749448a1db4e90b18ac0a1"
DEBUG: Repository linting_system instantiated at:
  /home/david/src/github.com/davidB/sandbox_bazel/WORKSPACE.bazel:41:13: in <toplevel>
Repository rule http_archive defined at:
  /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
INFO: Analyzed 9 targets (0 packages loaded, 0 targets configured).
INFO: Found 9 targets...
ERROR: /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/BUILD.bazel:32:10: MirrorAndLint exp_python/webapp/__linting_system/run/exp_python/webapp/run.py failed: (Exit 1): run_linter_exe failed: error executing command bazel-out/k8-fastbuild/bin/exp_python/webapp/run_linter_exe 'exp_python/webapp/run.py;bazel-out/k8-fastbuild/bin/exp_python/webapp/__linting_system/run/exp_python/webapp/run.py'

Use --sandbox_debug to see verbose messages from the sandbox run_linter_exe failed: error executing command bazel-out/k8-fastbuild/bin/exp_python/webapp/run_linter_exe 'exp_python/webapp/run.py;bazel-out/k8-fastbuild/bin/exp_python/webapp/__linting_system/run/exp_python/webapp/run.py'

Use --sandbox_debug to see verbose messages from the sandbox
Traceback (most recent call last):
  File "/home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/38/execroot/__main__/bazel-out/host/bin/tools/linting/python_linter", line 388, in <module>
    Main()
  File "/home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/38/execroot/__main__/bazel-out/host/bin/tools/linting/python_linter", line 288, in Main
    module_space = FindModuleSpace()
  File "/home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/38/execroot/__main__/bazel-out/host/bin/tools/linting/python_linter", line 118, in FindModuleSpace
    raise AssertionError('Cannot find .runfiles directory for %s' % sys.argv[0])
AssertionError: Cannot find .runfiles directory for bazel-out/host/bin/tools/linting/python_linter
INFO: Elapsed time: 0.151s, Critical Path: 0.04s
INFO: 5 processes: 5 internal.
FAILED: Build did NOT complete successfully
Enter fullscreen mode Exit fullscreen mode

What does it mean ? I don't know and I don't know how to fix it. I tried without success

  • executable = "//tools/linting:python_linter", same error
  • executable_path = "$(location //tools/linting:python_linter)", failed with location: command not found.

I keep the experimentation on a branch exp/linting_system and we'll try something different, make out own tools.

Make a build tool as part of project

It's something interesting to try, create as part of the project a tool to build the project. Start by revert changes made in the previous section, but keep it as inspiration to start.
Create tools/python_check/BUILD.bazel

load("@rules_python//python:defs.bzl", "py_binary")
load("@my_python_deps//:requirements.bzl", "requirement")

package(default_visibility = ["//visibility:public"])

py_binary(
    name = "python_check",
    srcs = ["python_check.py"],
    python_version = "PY3",
    srcs_version = "PY3",
    deps = [
        requirement("black"),
    ],
)
Enter fullscreen mode Exit fullscreen mode

Create tools/python_check/python_check.py

import black
import sys

if __name__ == "__main__":
    sys.exit(black.main())
Enter fullscreen mode Exit fullscreen mode

Try it

❯ bazel run //tools/python_check -- --version 
python_check.py, version 20.8b1
Enter fullscreen mode Exit fullscreen mode

Now try to use it in exp_python/webapp/BUILD.bazel by adding

genrule(
    name = "check",
    srcs = glob(["**/*.py"]),
    outs = ["check.log"],
    cmd_bash = """(
        $(location //tools/python_check) --check --verbose $(SRCS)
        ) | tee $@
    """,
    tools = ["//tools/python_check"],
)
Enter fullscreen mode Exit fullscreen mode

Try

❯ bazel build //exp_python/webapp:check 
...
/home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/main.py already well formatted, good job.
would reformat /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/run.py
would reformat /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/test.py
Oh no! πŸ’₯ πŸ’” πŸ’₯
2 files would be reformatted, 1 file would be left unchanged.
...
Enter fullscreen mode Exit fullscreen mode

Basic seems to work. What happens if instead of "--check", we let black format the code ?

/home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/main.py already well formatted, good job.
error: cannot format /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/run.py: [Errno 30] Read-only file system: '/home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/run.py'
error: cannot format /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/test.py: [Errno 30] Read-only file system: '/home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/test.py'
Oh no! πŸ’₯ πŸ’” πŸ’₯
1 file left unchanged, 2 files failed to reformat.
Enter fullscreen mode Exit fullscreen mode

Bazel block us to change source file, it's a pain for a formatter, but it's not a bad idea and it follows the way that bazel build commands could be run on remote build server (via CI or bazel remote execution), and that source is own and managed by the developer. But maybe bazel run could ?

❯ bazel run //tools/python_check -- --check $PWD/exp_python/**/*.py
...
would reformat /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/run.py
would reformat /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/test.py
Oh no! πŸ’₯ πŸ’” πŸ’₯
2 files would be reformatted, 1 file would be left unchanged.
Enter fullscreen mode Exit fullscreen mode
  • $PWD because when running, the current folder is modified, so without absolute path we got Error: Invalid value for '[SRC]...': Path 'exp_python/webapp/main.py' does not exist.
  • **/*.py should be change if your shell doesn't support this syntax.

Try to format

❯ bazel run //tools/python_check -- $PWD/exp_python/**/*.py
...
INFO: Build completed successfully, 1 total action
reformatted /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/run.py
reformatted /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/test.py
All done! ✨ 🍰 ✨
2 files reformatted, 1 file left unchanged.

sandbox_bazel on ξ‚  development [!?⇑] via 🐍 v3.9.4 
❯ bazel run //tools/python_check -- --check $PWD/exp_python/**/*.py
...
INFO: Build completed successfully, 1 total action
All done! ✨ 🍰 ✨
3 files would be left unchanged.
Enter fullscreen mode Exit fullscreen mode

Hourra, we have a bazel build command able to check the code and some bazel run able to check or to update the source code.

Linting & check as test

The goal all of checkers and linters is to evaluate the quality of the code and to detect issue and to suggest improvement, like for test. So it'll make more sens to run checks via bazel test than bazel build.

❯ bazel test //exp_python/...
INFO: Analyzed 4 targets (0 packages loaded, 0 targets configured).
INFO: Found 3 targets and 1 test target...
INFO: Elapsed time: 0.091s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
//exp_python/webapp:test                                        (cached) PASSED in 0.7s

Executed 0 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these aINFO: Build completed successfully, 1 total action
Enter fullscreen mode Exit fullscreen mode

We can add a test_suite into exp_python/webapp/BUILD.bazel that depends of both py_test(name="test"...) and our genrule(name="check"...).

test_suite(
    name = "quality",
    tests = [
        "check",
        "test",
    ],
)
Enter fullscreen mode Exit fullscreen mode 
❯ bazel test //exp_python/...                                                                                                  12:07:31
ERROR: /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/BUILD.bazel:55:11: in test_suite rule '//exp_python/webapp:quality': expecting a test or a test_suite rule but '//exp_python/webapp:check' is not one.
WARNING: Target pattern parsing failed.
INFO: Analyzed 4 targets (1 packages loaded, 7 targets configured).
INFO: Found 3 targets and 1 test target...
INFO: From Executing genrule //exp_python/webapp:check:
/home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/main.py wasn't modified on disk since last run.
/home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/run.py wasn't modified on disk since last run.
/home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/test.py wasn't modified on disk since last run.
All done! ✨ 🍰 ✨
3 files would be left unchanged.
ERROR: command succeeded, but there were errors parsing the target pattern
INFO: Elapsed time: 0.408s, Critical Path: 0.27s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
FAILED: Build did NOT complete successfully
//exp_python/webapp:test                                        (cached) PASSED in 0.7s

Executed 0 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
All tests passed but there were other errors during the build.
FAILED: Build did NOT complete successfully
Enter fullscreen mode Exit fullscreen mode

check is executed but bazel is not happy in accordance to the doc of test_suite.tests (genrule are like *_binary)

No *_binary targets are accepted however, even if they happen to run a test.

It's time create our first custom rule (inspired by the Rules Tutorial - Bazel and the examples/rules at master Β· bazelbuild/examples.

Create tools/python_check/defs.bzl where we create a rule python_check that try to mimic the genrule that we want to convert

"""Launch the python_check tool"""

def _python_check_test_impl(ctx):
    args = ["--check", "--verbose"] + [f.path for f in ctx.files.srcs]

    ctx.actions.run_shell(
        inputs = ctx.files.srcs,
        outputs = [ctx.outputs.log],
        progress_message = "Check into %s" % ctx.outputs.log.short_path,
        command = "(%s %s) | tee '%s'" %
                  (ctx.executable.tool.path, " ".join(args), ctx.outputs.log.path),
    )

python_check = rule(
    implementation = _python_check_test_impl,
    attrs = {
        "srcs": attr.label_list(allow_files = True),
        "log": attr.output(mandatory = True),
        "tool": attr.label(
            executable = True,
            cfg = "exec",
            allow_files = True,
            default = Label("//tools/python_check"),
        ),
    },
)
Enter fullscreen mode Exit fullscreen mode

A rule is composed of the declaration xxx = rule(...) and its implementation.

Then replace the genrule(name="check"...) by a call to python_check into exp_python/webapp/BUILD.bazel

load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")
load("@my_python_deps//:requirements.bzl", "requirement")
load("//tools/python_check:defs.bzl", "python_check")

...

python_check(
    name = "check",
    srcs = glob(["**/*.py"]),
    log = "check.log",
)
...
Enter fullscreen mode Exit fullscreen mode 
❯ bazel build //exp_python/webapp:check                                                                                        13:31:43
INFO: Analyzed target //exp_python/webapp:check (1 packages loaded, 4 targets configured).
INFO: Found 1 target...
Target //exp_python/webapp:check up-to-date:
  bazel-bin/exp_python/webapp/check.log
INFO: Elapsed time: 0.092s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
Enter fullscreen mode Exit fullscreen mode

now make python_check a test rule:

python_check = rule(
    ...
    test = True,
)
Enter fullscreen mode Exit fullscreen mode

Re-try

❯ bazel test //exp_python/...
ERROR: /home/david/src/github.com/davidB/sandbox_bazel/tools/python_check/defs.bzl:14:20: Invalid rule class name 'python_check', test rule class names must end with '_test' and other rule classes must not
...
Enter fullscreen mode Exit fullscreen mode

OK so we need to rename python_check into python_check_test into tools/python_check/defs.bzl and exp_python/webapp/BUILD.bazel and re-try

❯ bazel test //exp_python/...
ERROR: /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/BUILD.bazel:45:18: in python_check_test rule //exp_python/webapp:check: 
/home/david/src/github.com/davidB/sandbox_bazel/tools/python_check/defs.bzl:3:5: The rule 'python_check_test' is executable. It needs to create an executable File and pass it as the 'executable' parameter to the DefaultInfo it returns.
...
Enter fullscreen mode Exit fullscreen mode

Seems that from the error message and the sample examples/line_length.bzl at master Β· bazelbuild/examples, that we should modify also our _impl function to return a DefaultInfo instead of running the command directly. As a test we can also remove the log attributes because the output of test is the exit code and stdout/stderr managed by bazel directly. So Update tools/python_check/defs.bzl to

"""Launch the python_check tool"""

def _python_check_test_impl(ctx):
    args = ["--check", "--verbose"] + [f.path for f in ctx.files.srcs]
    command = "%s %s" % (ctx.executable.tool.path, " ".join(args))

    # Write the file, it is executed by 'bazel test'.
    ctx.actions.write(
        output = ctx.outputs.executable,
        content = command,
    )

    # To ensure the files needed by the command are available, we put them in
    # the runfiles.
    runfiles = ctx.runfiles(files = [ctx.executable.tool] + ctx.files.srcs)
    return [DefaultInfo(runfiles = runfiles)]

python_check_test = rule(
    implementation = _python_check_test_impl,
    attrs = {
        "srcs": attr.label_list(allow_files = True),
        "tool": attr.label(
            executable = True,
            cfg = "exec",
            allow_files = True,
            default = Label("//tools/python_check"),
        ),
    },
    test = True,
)
Enter fullscreen mode Exit fullscreen mode

Remove the log attribute on the caller side

python_check_test(
    name = "check",
    srcs = glob(["**/*.py"]),
)
Enter fullscreen mode Exit fullscreen mode

Re-try

❯ bazel test //exp_python/...                                                                                                  14:54:01
INFO: Analyzed 4 targets (1 packages loaded, 7 targets configured).
INFO: Found 2 targets and 2 test targets...
FAIL: //exp_python/webapp:check (see /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/execroot/__main__/bazel-out/k8-fastbuild/testlogs/exp_python/webapp/check/test.log)
INFO: From Testing //exp_python/webapp:check:
==================== Test output for //exp_python/webapp:check:
/home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/15/execroot/__main__/bazel-out/k8-fastbuild/bin/exp_python/webapp/check.runfiles/__main__/exp_python/webapp/check: line 1: bazel-out/k8-opt-exec-2B5CBBC6/bin/tools/python_check/python_check: No such file or directory
================================================================================
INFO: Elapsed time: 0.210s, Critical Path: 0.06s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//exp_python/webapp:test                                        (cached) PASSED in 0.7s
//exp_python/webapp:check                                                FAILED in 0.0s
  /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/execroot/__main__/bazel-out/k8-fastbuild/testlogs/exp_python/webapp/check/test.log

Executed 1 out of 2 tests: 1 test passes and 1 fails locally.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these a
INFO: Build completed, 1 test FAILED, 2 total actions
Enter fullscreen mode Exit fullscreen mode

Failed but looks like what we expect check is now part of the test flow. But we still have issues with the path of our python_check executable or/and the way we defined the runfiles.

try to replace ctx.executable.tool.path by ctx.executable.tool.short_path

❯ bazel test //exp_python/...                                                                                                  16:29:55
INFO: Analyzed 4 targets (1 packages loaded, 7 targets configured).
INFO: Found 2 targets and 2 test targets...
FAIL: //exp_python/webapp:check (see /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/execroot/__main__/bazel-out/k8-fastbuild/testlogs/exp_python/webapp/check/test.log)
INFO: From Testing //exp_python/webapp:check:
==================== Test output for //exp_python/webapp:check:
Traceback (most recent call last):
  File "/home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/44/execroot/__main__/bazel-out/k8-fastbuild/bin/exp_python/webapp/check.runfiles/__main__/tools/python_check/python_check", line 388, in <module>
    Main()
  File "/home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/44/execroot/__main__/bazel-out/k8-fastbuild/bin/exp_python/webapp/check.runfiles/__main__/tools/python_check/python_check", line 322, in Main
    assert os.path.exists(main_filename), \
AssertionError: Cannot exec() '/home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/44/execroot/__main__/bazel-out/k8-fastbuild/bin/exp_python/webapp/check.runfiles/__main__/tools/python_check/python_check.py': file not found.
Enter fullscreen mode Exit fullscreen mode

It means that short_path works but the tool is missing some dependencies. In fact we only include the entry file and not its runtime dependencies. So change the way to build runfiles by

    runfiles = ctx.runfiles(files = ctx.files.srcs)
    runfiles = runfiles.merge(ctx.attr.tool[DefaultInfo].default_runfiles)
Enter fullscreen mode Exit fullscreen mode

Then

❯ bazel test //exp_python/...
INFO: Analyzed 4 targets (0 packages loaded, 0 targets configured).
INFO: Found 2 targets and 2 test targets...
INFO: Elapsed time: 0.895s, Critical Path: 0.80s
INFO: 3 processes: 4 linux-sandbox.
INFO: Build completed successfully, 3 total actions
//exp_python/webapp:check                                                PASSED in 0.3s
//exp_python/webapp:test                                                 PASSED in 0.7s

Executed 2 out of 2 tests: 2 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these aINFO: Build completed successfully, 3 total actions
Enter fullscreen mode Exit fullscreen mode

Seems to work, try to introduce issue format issue (like previously) to check if detected.

❯ bazel test //exp_python/...
INFO: Analyzed 4 targets (0 packages loaded, 0 targets configured).
INFO: Found 2 targets and 2 test targets...
FAIL: //exp_python/webapp:check (see /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/execroot/__main__/bazel-out/k8-fastbuild/testlogs/exp_python/webapp/check/test.log)
INFO: From Testing //exp_python/webapp:check:
==================== Test output for //exp_python/webapp:check:
/home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/run.py wasn't modified on disk since last run.
/home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/test.py wasn't modified on disk since last run.
would reformat /home/david/src/github.com/davidB/sandbox_bazel/exp_python/webapp/main.py
Oh no! πŸ’₯ πŸ’” πŸ’₯
1 file would be reformatted, 2 files would be left unchanged.
================================================================================
INFO: Elapsed time: 0.907s, Critical Path: 0.78s
INFO: 3 processes: 4 linux-sandbox.
INFO: Build completed, 1 test FAILED, 3 total actions
//exp_python/webapp:test                                                 PASSED in 0.7s
//exp_python/webapp:check                                                FAILED in 0.3s
  /home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/execroot/__main__/bazel-out/k8-fastbuild/testlogs/exp_python/webapp/check/test.log

Executed 2 out of 2 tests: 1 test passes and 1 fails locally.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these aINFO: Build completed, 1 test FAILED, 3 total actions
Enter fullscreen mode Exit fullscreen mode

πŸŽ‰ issue is detected.

After reading the documentation for the cfg, we thing that the target is maybe a better value (simply because python always mixe all dependencies (tools, test, runtime,...) it's crappy but it's is way to work). So we now have tools/python_check/defs.bzl like

"""Launch the python_check tool"""

def _python_check_test_impl(ctx):
    # TODO find a better way to build the command/script and to escape/enclose args
    args = ["--check"] + [f.path for f in ctx.files.srcs]
    command = "%s '%s'" % (ctx.executable.tool.short_path, "' '".join(args))

    # Write the file, it is executed by 'bazel test'.
    # ctx.outputs.executable is the default value for DefaultInfo.executable
    ctx.actions.write(
        output = ctx.outputs.executable,
        content = command,
    )

    # To ensure the files needed by the command are available, we put them in
    # the runfiles.
    runfiles = ctx.runfiles(files = ctx.files.srcs)
    runfiles = runfiles.merge(ctx.attr.tool[DefaultInfo].default_runfiles)
    return [DefaultInfo(
        runfiles = runfiles,
    )]

python_check_test = rule(
    implementation = _python_check_test_impl,
    attrs = {
        "srcs": attr.label_list(allow_files = True),
        "tool": attr.label(
            executable = True,
            cfg = "target",
            default = Label("//tools/python_check"),
        ),
    },
    test = True,
)
Enter fullscreen mode Exit fullscreen mode

We also remove the test_suite from exp_python/webapp/BUILD.bazel to simplify.

On more thing before the pause, we can enable CI to run test in .github/workflows/ci.yml

name: CI

on: [push, pull_request]

jobs:
  test:
    # virtual environments: https://github.com/actions/virtual-environments
    runs-on: ubuntu-20.04

    steps:
      # Caches and restores the bazelisk download directory, the bazel build directory.
      - name: Cache bazel
        uses: actions/cache@v2.1.4
        with:
          path: |
            ~/.cache/bazelisk
            ~/.cache/bazel
          key: ${{ runner.os }}-bazel-cache

      # Checks-out your repository under $GITHUB_WORKSPACE, which is the CWD for
      # the rest of the steps
      - uses: actions/checkout@v2

      - name: Run the test
        run: bazel test //...
      - name: Build the code
        run: bazel build //...
Enter fullscreen mode Exit fullscreen mode

To be continued

It's not the end, we'll continue to setup linter and other stuff, but we're in a state enough to pause (it'was a long article again).

The sandbox_bazel is hosted on github (not with the same history, due to errors), use tag to have the expected view at end of article: article/5_python_2. I'll be happy to have your comments on this article, or to discuss on github repo.

πŸ’– πŸ’ͺ πŸ™… 🚩
davidb31
David Bernard

Posted on May 2, 2021

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Experimentations on Bazel: Python (2), linter
bazel Experimentations on Bazel: Python (2), linter

May 2, 2021