Following two xkcd issues pretty much sum up the experience of coming new to Python. You start flying at day 1: [XKCD 1987: Python Environment](https://xkcd.com/1987/) ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/398cq7th8f7jhbxszix7.png) Later by the time it comes to ship your code to production, you experience the inevitable crash due to getting entangled into all the different Python environment symlinks: [XKCD 353: Python](https://xkcd.com/353/) ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ua1v42v7u14qudts7y4e.png) Occasionally I see posts on social media of people stumbling into this and asking for some single source, which sums up [how to have a proper Python](https://twitter.com/mschoening/status/1771340970560532880) development environment: ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eqhj3t8nb8jqztivk1go.png) An [iconic reaction](https://twitter.com/steveruizok/status/1771976134814572674) of a user on Twitter after getting bunch of suggestions: ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1sz15elosvw34px9q3rp.png) Recently on my team we went through standardizing bunch of Python data processing projects, so i decided to write this one post, pythonists were asking about. ## TL)DR There is unfortunately still no single out-the-box solution. - Find a way to install multiple pythons - Make your python code a python package - Put all your config into a pyproject.toml - Use a virtual environment - Use a deterministic locking solution ## Installing Pythons Your package manager of choice has a way of installing multiple Python versions. But only one of them will be the default `python3` command. E.g a new Python 3.12 might be already released, but your distro package manager will still point `python3` to Python 3.11. No matter what the default Python is, all Python installs will put a versioned `pythonX.YY` command on your path. Lets say you need now Python 3.12 for some project. On MacOS homebrew you can run ```bash brew install python@3.12 python3.12 -V # -> Python 3.12.2 # check installed pythons brew list | grep python@ -> # python@3.10 # python@3.11 # python@3.12 ``` On other distros this will look similar. This will allow you to work on different projects, requiring different Python versions. However, as you might noticed, we only get one single patch version for each minor version of Python. In the example above for `python@3.12` , we got `Python 3.12.2`. At some point there will be bugfixes for Python 3.12 and after a package upgrade, the `python@3.12` package will be replaced with a newer `3.12 patch` , e.g `Python 3.12.3`. People on your team, might not be as quick upgrading and so probably also your production environment. This is when your local env start slightly defer. A lot of the times, this will get unnoticed: new patches do not break backwards compatibility and you simply might not be running into the fixed bug. Hence you might decide not bothering at all. I must admit being guilty of this most of the time. However not bothering is not ideal solution, as this is exactly what is called a non deterministic build. You end up with a slightly different version, as everyone else and now there is a classic "Works on my machine" scenario. One way solving this are version switcher tools. A popular version switcher for Python is [pyenv](https://github.com/pyenv/pyenv). It will allow to install the exact Python version and switch the default `python` command to the exact version of your choice. ```bash brew install pyenv pyenv install -l | grep " 3.12" # list all 3.12 versions pyenv install 3.12.2 mkdir -p ~/src/my-python-project && cd ~/src/my-python-project pyenv local python 3.12.2 which python # => ~/.pyenv/shims/python python -V # => Python 3.12.2 ``` The above will create a `.python-version` file in your project directory. Every time you switch to this directory the exact Python version will be selected. Installing Pythons like this has also a downside: Every Python version will be compiled from src on your machine. Compiling takes more time than just pulling a binary. Pyenv rarely fails on this in my experience. Still compiling sometimes will run into edge cases because you have some wrong version of a system lib or have some linking path misconfigured. Therefore not bothering with exact versions of Python as I mentioned above is also an option. You might at some point do both. And this is how you end up with Randal's XKCD Python environment diagram. That is also why some people have a hard habit to never run the default `python` or `pip` command. Instead they run them like this: ```bash python3.12 my-python-script.py python3.12 -m pip show pandas ``` Also remember those friends: ```bash which python3 # => /opt/homebrew/bin/python3 echo $PATH # => /opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin ... # the python3 command in the most left dir wins ``` At this point I think I have a nervous habit to run `which python` or `python -V` before typing any Python command. ## Virtual Environments Once you have multiple Pythons, you want to make sure, any different project on your machine is using the respective Python with a respective specific set of Python packages. Python by now probably has over 10 solutions how to isolate a project specific environment, while more are being created at the time of writing. (See [[1]]((https://rye-up.com/)), [[2]](https://github.com/astral-sh/uv)). This is because the default package manager pip has 2 major limitations: 1. Pip can only install a single version of a package at a time. A package update replaces this version. That means for example, you can't have pandas 2.x in one project and pandas 1.x for another in same Python installation. 2. Pip has no lockfile support, or at least no obvious one. Lockfiles are created from requirements policies. It means, if a package you need for your software depends on another package, there is no good build-in way to lock an exact installed configuration which meets this dependency. Multiple versions may meet the dependency, but you want to reproduce the exact single resolution variant on another machine - e.g produciton or your teammates computer. (we are at deterministic builds here again) The importance of deterministic locking is explained in detail in upcoming section. Not long ago, pip even had no real good support to tell you, your mix of installed packages is incompatible with each other. I was pretty surprised reading it on pip's mailing list announcement, that they now support proper dependency resolution, in the year after npm vs yarn already happened and Python being there already way longer. It means, that for a long time, if any two packages needed different versions of a 3rd one, where those different versions have an incompatible api breaking change, pip would not warn you. You would end up with whatever latest version came by the most recent `pip install`, even if that recent install is technically breaking another package requirement. In the end only one of the two packages worked with the installed dependency and other was silently broken. This all lead to a really fragmented ecosystem of Python environment and package managers which solved different aspects of the above shortcomings in different ways - virtualenv, venv, conda, mamba, pdm, Poetry, Pipenv, rey, uv - to name a few. Some of them rely on pip, others ditch it completely and use the [pypi.org](https://pypi.org/) package repository api directly with their own dependency resolution algorithm. Writing a reliable and fast dependency resolution algorithm is hard though - in fact it is [NP-hard](https://en.wikipedia.org/wiki/NP-hardness). See [this](https://nex3.medium.com/pubgrub-2fb6470504f) excellent explanation on a modern resolver - PubGrub. Hence, whatever comes as default tooling from a language, will be regarded as more stable and you need good arguments to convince people to switch to something else. Convincing people is sometimes even harder than NP-hard, so it makes also sense to learn whatever is default. A tool shipped with Python for isolating different project environments is `venv`. Venv is short for virtualenv. Prior to `venv` a tool actually named `virtualenv` was implemented and then `venv` was shipped as default tool with every Python install. Some projects still use virtualenv. The tools also slightly defer in how they symlink or copy the Python version. As Python cannot install multiple versions of packages, `venv` helps you create a separate directory of packages for each project - the virtual environment. The virtual environment also fixates the Python version, it was created with. Then for development on a particular project, you activate the respective environment. This activation just fiddles with your shell path and default `python` and `pip` commands and makes the activated default commands point to the symlinks under `~/venv/my-python-project/bin` folder. ```bash cd ~/src/my-python-project which python3.12 # => ~/.pyenv/shims/python3.12 # creates the virtual env python3.12 -m ~/venv/my-python-project source ~/venv/my-python-project/bin/activate which python # => ~/venv/my-python-project/bin/python which pip # => ~/venv/my-python-project/bin/pip pip install pandas==2.2.1 pip show pandas # => # Name: pandas # Version: 2.2.1 # ... # Location: ~/venv/my-python-project/lib/python3.12/site-packages ``` If Python is your first language, then probably all the above sounds like straightforward solution. However we have systems for over a decade in place, which solve it more elegantly. Languages like ruby, rust or go-lang allow to have multiple packages installed at same time. Then you specify a lockfile - which versions to pick before running or compiling your code. People coming from those languages rather expect to find such solution. In my experience this was a big cause of frustration for a lot of people, when trying to establish a good devops workflow. ## Why is there also pipx? Python is also a common choice for other people's software, not only for your project. Hence you will be tempted to install other Python cmd tooling at some point, e.g [cookiecutte](https://www.cookiecutter.io/), [httpie](https://httpie.io/cli), [pyright](https://microsoft.github.io/pyright/#/command-line). Installing bunch of those via pip might clash with each other due to incompatible requirements and as we saw earlier, there can only be one version of each package. Also you will want those tools be accessible on the PATH regardless of currently activated virtual environment. Pipx does the same as described in the **Virtual Environments** chapter, but for each single package. ``` brew install pipx pipx ensurepath pipx install cookiecutter ``` The above will install `cookiecutter` cmd tools into a separate virtual env in `~/.local/pipx/venvs/cookiecutter` and add its command to `~/.local/bin/cookiecutter` which is now on the PATH. ## Deterministic builds and dependency locking The most known way of somehow reproducing the same set of required packages on another machine in Python is a `requirements.txt` file. You list all your packages in a file together with some version policy. Then you run `pip install -r requirments.txt` ``` pandas >=1.5.2, <2.0.0 pyarrow >=12.0.0, <13.0.0 flask >=2.3.3, <3.0.0 ``` However this is not a complete definition of your environment. For example, `pandas` and `pyarrow` need `numpy` to work. `numpy` can be in whatever version, as long as its api compatible with requirements of both `pandas` as `pyarrow`. Also the above `requirements.txt` file allows for installing any range of package versions as long as its not higher as certain major version. Installing those requirements will lead to different set of packages, depending on the point in time you run it. At some point there is only most recent `pandas==1.5.3` and `numpy==1.25.1`, which satisfy the requirements. At another point there will be `pandas==1.9` and `numpy==1.7`. So you must be asking yourself then, why would I write such requirements policy file at all. I will just write down all the exact versions of packages I need and the package those packages need and the packages those packages need and so on - their so called transitive dependencies. Then I use those exact full recursive dependency tree to reproduce the install on deployment. You probably see already that this is a tedious task, which has to be automated. Also you have to think about maintenance required in upgrading all those packages later. You care only about upgrading packages your software needs. All others should just work with your software and be upgraded to a state where they have no reported security holes (CVEs), bugs or performance regressions. This is where [semver](https://semver.org/) comes into play. I have already used semver terminology across the blogpost. Check out the link above for a full reference. Here is a little recap: You can see above, that `pandas==1.5.3` versioning scheme is a 3 dotted number. Those dots are like special code for all developers and users of the package. The code is deciphered like this: `pandas==..`. All users of the code agree, that as long as MAJOR number stays same, the package will still work without adapting your usage code. Once a package maintainer breaks this backwards compatibility, he bumps up the `MAJOR` version. `MINOR` is changed when new features are added. And `PATCH` is changed when bugs are fixed. While we can agree on some convention code, it is not guaranteed, we all will flawlessly follow it. Some package maintainers don't care about the code. Some will publish early packages with `0.0.x` number and just bump the last number until they believe the package api reached somewhat stability to become `1.0.0`. And some will accidentally introduce a breaking change without bumping up MAJOR versions. It makes following goals valid at same time: 1. I want to have a semver policy definition of my development environment. Because I mostly trust the developers of packages to adhere to semver. This allows to see if updates to any required package are supposed to be backwards compatible. 2. I still want to have to my environment locked with exact versions, to be able to reprduce it as close as possible on another machine or fresh install The above goals allow an optimistic upgrade strategy, where I can just do `pip install -U requirements.txt` . Everything up until the current major version will be upgraded. Now I run my test suite and just ship the change without some sort of manual QA. With a good unit and integration test suite this strategy allowed teams, I was on, to literally press a single button on package upgrades and be done with it. As sometimes things behave different across different minor/patch versions, when you ship a change, you want to be sure, it iss shipped with exact same packages, which the feature was tested with. Also when bugs happen and cause a failing CI or worse broken production, you want to be able to install the exact same set of packages on the local machine to debug the issue. Also you want to simply have a good way to track, which exact packages were shipped. As mentioned earlier `pip` has no good build-in way to create an exact lockfile from a semver `requirements.txt` policy. There are some methods involving a `pip freeze` command. But they will freeze whatever is currently installed. And sometimes you might install packages for development, which you don't want to ship - e.g. `jupyter notebook` and `jupyter lab`. The extra list of requirements for only those two will be huge. A common adopted tool for dependency locking is [pip-tools](https://github.com/jazzband/pip-tools). `pip-tools` is also a `pip` package and can be installed via `pip`. `pip-tools` works with the default `pip` resolver for transitive dependencies. Instead of having a single `requirements.txt` file, you change it to a `requirements.in` file ``` # requirements.in pandas >=1.5.2, <2.0.0 pyarrow >=12.0.0, <13.0.0 fllask >=2.3.3, <3.0.0 ``` Now at same directory of the `requirements.in` run `pip-compile` and it will produce a complete `requirements.txt` with a full list of transitive dependencies and their exact versions. ```bash pip install pip-tools pip-compile ``` Extract of the resulting `requirements.txt` ``` # # This file is autogenerated by pip-compile with Python 3.11 # by the following command: # # pip-compile # blinker==1.7.0 # via flask click==8.1.7 # via flask flask==2.3.3 #... numpy==1.25.1 # via # pandas # pyarrow pandas==1.5.2 # via requirements.in pyarrow==12.0.0 # via requirements.in # ... ``` There are also nice interpretability comments, allowing to trace why certain packages were installed or to be precise, which other packages required them. Another benefit of such a full lockfile workflow is interoperability with automation bots for dependency updates and vulnerability scans -e.g [dependabot](https://docs.github.com/en/code-security/dependabot) or [renovate](https://docs.renovatebot.com/). Those tools can only scan packages for CVEs which are somehow listed in a text file. In other words, if your ternary dependencies are not listed here, they are also not scanned. Now you probably think of various other solutions to make reproducible environments - e.g docker or conda. I will discuss them at the end of the post. ## Python project definition Python code does not have to belong to a project or package definition. Python gives you the flexibility to just do `python3 script.py` and it will run it, allowing to import whatever packages are currently known to pip and pip's current path to `site-packages` folder (PYTHONPATH). Also other Python code can be imported relative to the directory where you run the script from. (Python puts the current dir on the PYTHONPATH). In a larger project `script.py` will need other modules to work. And you will want to split those module definition into other files structured by some namespaces. Because Python is designed to have the flexibility to just run things with couple of quick imports, there is also no single set in stone convention for a Python project. However it is a common source of misunderstanding and frustration, so its important to understand the import system. Easiest way you can split code in modules is just throw in `import` statements to other "files". ``` # ./script.py from my-service import MyService if __name__ == "__main__": MyService().compute() # ./my-service.py class MyService: def compute(): # ... ``` Technically this practice is not just "importing files" though. Python has an import convention, which defines modules within its interpreter based on their definition in text files. Upon calling the `import ` or `from import` statement, Python will lookup either a corresponding file with `.py` name or a directory with name `/` and an empty `__init__.py` file in it. The corresponding `*.py` files will be parsed and all entities in them will be made available under the module - functions, classes, constants etc. In the example above. A statement `import my-service` or `from my-service` loads the corresponding `my-service.py` file. This automatically defines the respective module named `my-service` by convention. All function and class definitions like MyService inside `my-service.py` are now part of `my-service` module. By adding a folder with an empty `__init__.py`, you can introduce a module hierarchie - modules containing other modules. ``` ~/my-python-project ├── products | ├── __init__.py | └── product_service.py └── script.py ``` ```python # ./script.py from products.product_service import ProductService # .. # ./products/product_service.py class ProductService: # .. ``` Now `ProductService` class is part of a `product_service` module, which itself is part of `products` module. `product_service` module could itself be defined as directory of files with `__init__.py` marker containing a sub hierarchy of more product services. Introducing this nesting is sometimes just done for the sake of adding an extra namespace. There can be multiple product services depending on different context. Somebody on the internet could also have defined a nice package called [products](https://pypi.org/project/products/) After installing that package, if you call `from products`, Python has no way to know which `products` to use - the one from your project or the one you installed from the internet. In this case another level of nesting is often introduced just for the sake of an extra namespace, often by using the organisation name - e.g. [google/cloud](https://github.com/googleapis/python-aiplatform/tree/main/google/cloud). While for example in java it is very common to use the organisation name as a namespace, pythonists tend to be rather simplistic. Still at least single flat namespace is used preferably. If you install `pandas`, you import all pandas code always `from pandas`, if you install numpy you import nunmpy code always `from numpy` etc. This has a bet, that names like `pandas` or `numpy` are rather unique. There might be also multiple namesapces to import from via a single package install. For reference, Python also has other import and project layout mechanisms as outlined above: - some import version which does not require empty `__init__.py` markers. [namespace packages](https://docs.python.org/3/reference/import.html#namespace-packages) - [relative imports](https://docs.python.org/3/reference/import.html#package-relative-imports) - a [src layout](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/) ( vs above explained flat layout) We chose to ignore all those options on a recent team and never looked back. Wasn't Pythons credo after all: "there should be one obvious way to do it"? When trying to find some minimal project definition example, one often comes across articles named like "Python Package Layout". A common confusion in my experience often is: "I am not creating a pip installable python package. I just want to create an app". But the answer is actually, that it is best to make your app, exactly such an installable package, even if you are not planning to publish it to some package registry. By making your project installable, it can be installed locally in a so called editable mode, with `pip install -e .` This will make your project known to pip without the sources being copied to the `site-packages` folder. The pip package install will point to the project src folder instead. Hence the `editable` naming. The package live changes, whenever you edit a project src file without any extra installation. One benefit of editable install is, that now `import from my-python-project` works no matter in which directory you run a Python script. This gives a consistent way of importing your code in tests. Also you can put jupyter notebooks at some extra folders and don't have to fiddle with import paths to your project sources. For example VSCode starts jupyter notebooks with import path relative to notebook parent directory but `jupyter notebook` command itself sets the import path to the directory the command was launch from. This will mess up imports relative to current launch directory. Contrary if your project was installed to pip, this does not matter, because the project is already on the `PYTHONPATH` and code from it can be imported from anywhere. Another benefit is that an editable install will undergo same dependency resolution as other pip packages. The requirements policy mentioned in earlier chapter can be placed inside `pyproject.toml` configuration file instead of a `requirements.in` file according to relatively recent [PEP 621](https://peps.python.org/pep-0621/) standard. Now `pip install -e` will check, if other installed packages meet your projects requirements. As you might have installed other packages manually for development - e.g jupyter , there might be some clash, which is good to resolve or its just a verification that your current project requirements are met by whatever is now locally installed in the virtual env. ## Venv + PEP 621 + pip-tools The rationale outlined in previous chapters is to most degree the default Python way to achieve isolated per project environments with deterministic builds. In such a workflow a Python project can be defined like this: ``` ~/src/my-python-project ├── my_python_project │ ├── __init__.py │ ├── products │ │ ├── __init__.py │ │ └── products_service.py │ └── run_products_service.py ├── notebooks │ └── Product.ipynb ├── pyproject.toml ├── requirements.txt └── tests ├── __init__.py └── products ├── __init__.py └── test_products_service.py ``` At the root, there is a pyproject.toml, which makes the project installable with `-e` option. It also specifies a policy for project requirements. A full `requirements.txt`, which includes all transitive dependencies, was generated out of the PEP 621 spec in `pyproject.toml`. A top Python module with same (underscored) name is specified under root. All modules have an empty `__init__.py` marker, if they include other modules. Below is the minimal `pyproject.toml`: ```toml # ./pyproject.toml [build-system] requires = ["setuptools>=68.0.0", "setuptools-scm", "wheel"] build-backend = "setuptools.build_meta" [project] name = "piptools_demo" requires-python = ">=3.12, <3.13" dynamic = ["version"] dependencies = [ "pandas >=1.5.2, <2.0.0", "pyarrow >=12.0.0, <13.0.0", "flask >=2.3.3, <3.0.0", ] [tool.setuptools] packages = ["my_python_project"] ``` Python by now has some democratic policy to build packages by multiple [build backends](https://packaging.python.org/en/latest/tutorials/packaging-projects/#choosing-a-build-backend). It makes things even more confusing for new pythonists, so lets ignore it (as this post is already long). The config contains a minimal spec for using the classic [setuptools](https://setuptools.pypa.io/en/latest/setuptools.html) for this and also for some automatic versioning every time a package is build: `setuptools-scm` and `dynamic = ["version"]`. The `packages = ["my_python_project"]` directive tells `setuptools` build backend to build code only from the `my_python_project` module (directory). In case you want to import from other module names at the root, one could specify a list of top modules here. There can also be other folders under root, which are not a python package, hence the configuration need. The `packages` configuration necessity can be omitted at the cost of adding an extra nesting under a folder named `src` at the top. See [src layout vs flat layout](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/). Having the above project setup with minimal build definition, you can now create a virtual env with a suiting Python version, lock and install all dependencies and install the project in editable mode. ``` cd ~/src/my-python-project python3.12 -m venv ~/venv/my-python-project source ~/venv/my-python-project/bin/activate pip install pip-tools pip-compile pip install -r requirements.txt pip install -e . ``` There is also a `pip-sync` command from pip-tools, but it will wipe out everything not listed in `requirements.txt` (excluding pip-tools itself). Unfortunately there is no build-in easy way to list and lock development dependencies like [jupyter lab](https://jupyter.org/) or [black](https://black.readthedocs.io/en/stable/) with Venv + PEP 621 + pip-tools workflow. After having the project installed in editable mode (`-e`), all its code from any place can be imported consistently like this: ```python # ./run_products_service.py, ./notebooks/Product.ipynb or # ./tests/products/test_products_service.py # ... from my_python_project.proudcts.product_service import ProductService # ... ProductService().list_products() ``` When a full `requirements.txt` was generated and checked in under version control, anyone on the team or any CI workflow can install the exact same environment like this ``` cd ~/src git clone git@github.com:my-org/my-python-project cd ~/src/my-python-project python3.12 -m venv ~/venv/my-python-project pip install -r requirements.txt pip install -e . ``` Note how, for installation, pip-tools is not required. It is only needed to lock dependencies as they change. Also installing project requirements and the project itself, as editable package, changes order. The `requirements.txt` already contains everything needed for the project, hence a subsequent editable project install must not pull any new packages. This setup also allows for one command upgrade of all packages and transitive packages to their latests versions, which do not break your PEP 621 semver policy. ``` pip-compile --upgrade ``` In other words installed packages after this command should stay backwards compatible, if every package maintainer did properly adhere to semver, when bumping up versions. See [updating-requirements](https://github.com/jazzband/pip-tools?tab=readme-ov-file#updating-requirements). To upgrade packages with breaking changes, you can manually bump up major versions in the policy and run `pip-compile` and `pip install -r requirements.txt`. Finally adapt the code and make the tests green. ### Conclusion The `venv+pep621+pip-tools` workflow is mostly using basic Python tooling. To clone and reproduce the environment the extra `pip-tools` package is not even required . Believe it or not, but this flow feels natural to a lot of Python developers. The consistent `from my-python-project` import scheme together with an editable install, saves from running into various import issues. (See [[1]](https://stackoverflow.com/questions/448271/what-is-init-py-for) [[2]](https://stackoverflow.com/questions/21819649/namespace-vs-regular-package) [[3]](https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time).) #### Drawbacks One aspect of this setup that somehow annoyingly catches the eye of a lot of developers (myself included in the beginning) is the project name repetition at the root - the `~/src/my-python-project/my_python_project` directory needed for the module naming inside Python interpreter with same name as the project. It turns out to be actually a common pattern across all languages. (See [[1]](https://github.com/cbeust/klaxon/tree/master/klaxon/src/main/java/com/beust/klaxon), [[2]](https://github.com/diesel-rs/diesel/tree/master/diesel), [[3]](https://github.com/toy/image_optim/tree/main/lib/image_optim)). Probably it jumps the eye, because of the flat layout, thus being immediate repetition after project name or maybe it's because Python somehow does not have one obvious way to do it, so you start thinking about this naming scheme too much. On the jvm , for example there is no other way - all code must belong to some other package. In Python you have this choice of just running a script. But if you choose to standardize your project and create a top package - this is the way. (See [[1]](https://github.com/pandas-dev/pandas/tree/main/pandas),[[2]](https://github.com/numpy/numpy/tree/main/numpy),[[3]](https://github.com/more-itertools/more-itertools/tree/master/more_itertools)) There is actually one way more important downside for the `venv + PEP 621 + pip tools` workflow. It does not allow cross platform building. Python packages have an option to define different requirements depending on the system and cpu architecture which the package will be installed on. It results in a different lockfile depending on a platform you run it. `pip-tools` also does not offer an option to create a lockfile for a different platform. Python has a good way of shipping native binaries - wheels (accessible via native-binding) only for the platform you are currently on. Most packages rather choose to ship a different binary wheel under same package version, than adding different transitive dependencies for different hardware architectures. Those different wheels for each system do not affect the pip-tools lockfile. (They do not appear in the lockfile at all). However sometimes the different archtecture requirements strategy is chosen. A popular package with such issue is [tensorflow](https://github.com/tensorflow/tensorflow/issues/61477#issuecomment-1675754839). Tensorflow installs different depending packages on Apple Silicon, Apple Intel, linux/arm64 and linux/amd64. One practice which became popular recently, is getting Apple Silicon machines for development. Those machines use arm64 architecture. Deployment or CI workers will on the other hand often use linux/amd64 architecture. If you use tensorflow and you run `pip-compile` on Apple Silicon, a pip-tools lockfile turns out uninstallable on a linux/amd64 machine. Further, as briefly mentioned above, there is no good way to specify extra development dependencies for all team members. Those are package which are needed for develoipment only - e.g black or jupyter notebook. This could probably be solved natively by an extra requirements file and other pip-compile commands the project has to document and require everyone to run. Having said that, installing dev tooling, where you care about same versioning for everyone on the project, will be likely covered by another popular tool outside Python ecosystem - [pre-commit](https://pre-commit.com/). In general the `venv+pep621+pip-tools` workflow requires everyone on the team to know and run a mix of various tools and their commands - installing Pythons, creating virtual envs, activating them, locking versions etc. Also it requires you to understand and know the implication of all what you read above in detail. Our primarily motivation is working software, hence its natural to strive for a solution which just solves the Python packaging. Have some kind of single default workflow and mostly not care about this. Imho, this is what you get from bundler or cargo. So lets look at other workflows. ## Poetry Poetry promises to solve almost all of the above issues. It does not take ownership of installing Pythons, but it covers everything else, at least on paper. Poetry brings a single tool, which: - automates virtual envs and makes them an internal implementation detail. - requires a project definition with a spec for project requirements (though it is not using PEP 621 standard). - allows multiple groups of dependencies for different stages: dev, test etc. - creates lockfiles for all possible platforms and stages from a semver policy. - ships with tooling for upgrading dependencies w.r.t to current semver policy and previous lockfile - creates a full transitive dependency resolution lockfile for all possible platform specific variations Poetry is best [installed with pipx](https://python-poetry.org/docs/#installing-with-pipx) mentioned earlier. ``` pipx install poetry ``` A Poetry project setup looks almost same as the default Python project setup outlined above: ``` ~/src/my-python-project ├── my_python_project │ ├── __init__.py │ ├── products │ │ ├── __init__.py │ │ └── products_service.py │ └── run_products_service.py ├── notebooks │ └── Product.ipynb ├── pyproject.toml ├── poetry.lock └── tests ├── __init__.py └── products ├── __init__.py └── test_products_service.py ``` The only difference is the configuration within the `pyproject.toml` file: ```toml name = "my-python-project" version = "0.1.0" description = "" authors = ["Chuck Testa "] readme = "README.md" packages = [{include = "poetry_demo"}] [tool.poetry.dependencies] python = ">=3.12,<3.13" pandas = ">=1.5.2, <2.0.0" pyarrow = ">=12.0.0, <13.0.0" flask = ">=2.3.3, <3.0.0" [tool.poetry.group.test.dependencies] pytest = ">=8.1.1" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" ``` Now running `poetry install` will create a `poetry.lock` file, create a virtualenv under a directory managed by Poetry e.g `~/Library/Caches/pypoetry/virtualenvs/` and install all the locked dependcies into it. When a lockfile already exists, `poetry install` will adhere to it and install the packages at their exact versions listed in it. ``` ... [[package]] name = "flask" version = "2.3.3" description = "A simple framework for building complex web applications." [package.dependencies] blinker = ">=1.6.2" click = ">=8.1.3" ... [[package]] name = "blinker" version = "1.7.0" description = "Fast, simple object-to-object and broadcast signaling" ... [[package]] name = "click" version = "8.1.7" description = "Composable command line interface toolkit" [package.dependencies] colorama = {version = "*", markers = "platform_system == \"Windows\""} ... ``` Running `poetry shell` within the projects directory activates the corresponding virtual environment of the current project. ### Drawbacks While Poetry looks good on paper, some people report it to become either very slow or not working with certain packages. One reason is, that there is really no way in Python to tell with 100% certainty, which other packages a particular package will require, without actually installing that package. PyPi allows to upload metadata about Python package dependencies, but their package server will not enforce the validity of that metadata. Unfortunately some packages do not ensure to have consistent metadata file for all architectures with what their package `setup.py` script actually installs. Some packages have no metadata uploaded at all. In this case Poetry has to download the package. It will then try to find typical ways of requirements configuration within package source code. While common patterns exist in Python, there is no 100% guarantee Poetry heuristics can properly scan every package setup. As mentioned earlier Poetry also has a goal to detect requirement for every hardware architecture, which is even harder. Finally the whole resolution slowness also boils down to the core problem of it being NP Hard. Hence adding all possible architectures, source downloads and source inspections makes the task even harder. Once Poetry has resolved the full spec to a lockfile things usually become way smoother. Poetry resolution speed can often also be improved by constraining the range of possible packages, which can be installed. For example a constraint like: `pyarrow = ">=12.0.0, <13.0.0"` has way less combinations to check for than `pyarrow = ">=5.0.0"`. With more packages, those combinations grow exponentially - hence the NP hardness. A good practice is therefore is bumping also the lowest acceptable version of your semver policy after upgrades. And in case package maintainers did upload wrong metadata as in case of tensorflow, one can often add some manual workaround for poetry into pyproject.toml [Example](https://github.com/python-poetry/poetry/issues/8271#issuecomment-1681622725): ``` [tool.poetry.dependencies] python = ">=3.11,<3.12" tensorflow-macos = { version = "^2.15.0", markers = "platform_system=='Darwin' and platform_machine=='arm64'" } tensorflow = "^2.13.0" ``` ## Editor Integration Now that you have as many Pythons as on the xkcd Diagram from the beginning on the post, it is important to configure the Editor to point to the exact symlink of the Python which was created via respective venv, Poetry or other Python project management tool. Only this particular Python symlink will have a proper `PYTHONPATH` configured to point to a `site_packages` directory with corresponding projects dependencies. For venv and a virtual env created under `~/venv/my-python-project`, the Python path will be `~/venv/my-python-project/bin/python`. For Poetry, the virtual env path can be printed with `poetry env info --path`. It results in a path like `~/Library/Caches/pypoetry/virtualenvs/my-python-project-aMfufBzB-py3.11` . The respective Python path will be `~/Library/Caches/pypoetry/virtualenvs/my-python-project-aMfufBzB-py3.11/bin/python` VSCode, for example, is supposed to pick up the right interpreter by default. Open VSCode from a Python project root directory with the `code .` terminal command for this to happen. When using `venv`, the respective virtual environment must be activated before running the `code .` command. Poetry envs should be picked up automatically, if `poetry` command is on the `PATH`. If everything worked correctly, a prompt like `Python 3.11.9 ('python 3.11': venv)` in the bottom right corner will appear . One can also hover over this prompt and verify the full Python path. In case the above does not work as expected, the [interpreter settings](https://code.visualstudio.com/docs/python/environments#_working-with-python-interpreters) can be adjusted manually per project. One can also quickly run: ``` cmd + opt + P -> Python: Select Interpreter ``` Entering an interpreter Path there changes the current VSCode interpreter temporarily. A similar setting can be configured in [Pycharm](https://www.jetbrains.com/help/pycharm/installing-uninstalling-and-reloading-interpreter-paths.html#:~:text=View%20interpreter%20paths%EF%BB%BF&text=Press%20Ctrl%20Alt%200S,selector%20and%20choose%20Interpreter%20Settings.). A lot of plain text editors will ship a language server plugin allowing to add [pyright](https://github.com/microsoft/pyright) integration. Pyright will use the dafault `python` command. If you start neovim, helix or sublime from terminal with activated virtual env, they should pick up the proper interpreter for the current project. Pyright also allows to change the [default python](https://github.com/neovim/nvim-lspconfig/blob/master/doc/server_configurations.md#pyright) path. If somehow the "launch editor from project directory" strategy does not work for you, you will have to figure out, how to do hardcode the Pyright Python path setting per project. ([Example](https://neovim.io/doc/user/options.html#auto-setting)) ## Automated dependcy updates While semver policy plus lockfile and commands like `pip list --outdated` give some tooling for a package upgrade worfklow, it is still tedious to constantly run these things, bump versions, then run tests to check whether your code still works. Therefore dependency updates automaton bots exist. One popular solution is [dependabot](https://github.com/dependabot), which is shipped by github. For github to start creating automated dependency update pull requests, you just have to check in a `.github/dependabot.yml` configuraton under version control: ``` # ./.github/dependabot.yml version: 2 updates: - package-ecosystem: "pip" directory: "/" schedule: interval: "daily" groups: patches: update-types: - "minor" - "patch" open-pull-requests-limit: 100 ``` Dependabot works with both workflows covered in the post: pep 621 + pip-tools lockfile and Poetry. There is no need to configure which solution you use. Dependabot will recognize which python project management solution you use by either detecting a pip-tools or Poetry lockfile. (See dependabot [package ecosystem](https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#package-ecosystem)) Ideally you also have some CI workflow setup, which runs your tests (especially integration tests) on those package updates. Normally dependabot creates a single pull request with a single dependency update. Dependabot bumps up both - the policy and the lockfile. The policy in pyproject.toml is bumped up when newer major version of a dependency listed in `pyproject.toml` is available. Example: ```diff --- a/pyproject.toml +++ b/pyproject.toml - "pyarrow >=14.0.1,<15.0.0", + "pyarrow >=14.0.1,<16.0.0", --- a/requirements.txt +++ b/requirements.txt -pyarrow==14.0.1 +pyarrow==15.0.1 ``` Otherwise, when there were no new versions with breaking change or no new versions for 1st party requirements in pyproject.toml, just the lockfile is updated with minor/patch package upgrades. The idea of automated package updates is also that you have mostly automated merge strategy. On each update PR you run an integration test suite with good coverage. And you are to a high degree confident to just press the merge button and ship a dependency update to production if those tests are green. Whether you choose to follow this practice and merge dependabot PRs with a single click or rebase and test them on some branch is up to you. In the above `.github/dependabot.yml` there is also a grouping option. It will tell dependabot to create a combined PR for all non breaking (minor/patch) changes. ### Other Tools ### Conda Conda will be mentioned briefly. Spoiler: It neither solves deterministic, nor cross platform builds. I believe conda solved a lot of pain at the time, when Python dependencies had often to be compiled. Nowdays I rarely run into a package which does not ship a binary wheel for its native code together with the package. However if no wheel would be available, Python would fallback to compiling the dependency. For this, certain development system libraries must be available on your machine. It is not always straightforward, where to get them for your package manager and sometimes extra setting like `LD_LIBRARY_PATH ` are required to specify a compatible system lib. When conda came, it took care of providing system libraries and Python distributions itself. Hence installing a package just worked, opposed to pip, which required to install system libs on your own. As long as one is fine with only semver compatible environment definition (instead of deterministic), it is close to an out of the box solution. One can also export the exact env clone with all the transitive dependencies. This is close to a lockfile workflow, but not quite. There is no good way of working hand in hand with a policy and a lockfile as outlined in the post. You either can have a semver policy env definition or the one with all the exact versions but not both, where update to the lockfile honor the policy. Conda also defines its own packaging format with an own package repository. It has way fewer packages, than available for pip on PyPi. Therefore extra support for pip packages was added. However for long time there was no resolution between the tree of conda packages and the tree of pip packages and by adding a pip package one accidentally could override a package which came from conda, because a same ternary dependency might be shipped through different packages. This resolution was [added](https://conda.io/projects/conda/en/latest/user-guide/configuration/pip-interoperability.html) as an optional flag, but even after 3 years passed since I saw that feature, it is still marked as experimental. For devops engineers, using conda, means some extra install scripting, larger build times and still no straightforward deterministic build workflow which works for development and deployment. Also using conda will actually require to understand conda and pip, as pip packages eventually will crumble into the environment. While data scientists like conda, because it gives somewhat seamless experience on a single machine, devops engineers do not as it does not have a good solution for maintaining dependencies through project lifetime and ship deteminisitc builds. For deterministic builds on conda ecosystem - [mamba](https://github.com/mamba-org/mamba) exists. It claims to be cross-platform. I did not try whether it actually allows building for another platform when what you are developing on. ### Docker for development Docker is used to deploy an exact same image in production. A common idea then: so why not also using it to specify the development environment? We can use mostly same image for development as the one being deployed. There is recently also a popular [devcontainers.json](https://code.visualstudio.com/docs/devcontainers/create-dev-container) workflow. Even before that workflow a wide adopted approach was to create a single `docker-compose.yml` for develooment, assembling all Dockerfiles from the project with bind mounting the project source files into this local Docker machine. This way all sources live locally and in Docker, but code runs only within a specified Docker environment. I will try to cover such workflow in another post. There are certain caveats though, I want to share here already. Docker is known for its advertising to solve "Works on my machine!" issues. Therefore there is a widespread wrong belief, that it somehow happens automatically by just adding a Dockerfile and let everyone build dev images from it. Pushing/pulling same docker image is deterministic, building an image from a Dockerfile is not. Also a named docker tag like - `python:3` or `my-project:latest` on two different points in time can point towards two different base images. Building an image with exact same `FROM ` statement at different time therefore also can lead towards two completely different images. In a typical Python project Dockerfile, you find some directives like this: ```Dockerfile FROM python:3.12 COPY requirements.txt ./ RUN pip install -r requirements.txt ``` The `COPY requirements.txt ` statement creates a cache based on the fingerprint of the content of the requirements.txt file. When you bump a version in that requirements.txt file, it's fingerprint changes. Afterwards on running `docker build`, docker will repeat all instructions from that line. The cache for this line was invalidated due to a different fingerprint. Now docker will run the `RUN pip install` line and in case the contents of this `requirements.txt` do not lead to a deterministic build, the result is also not deterministic. You might have bumped up a single top package, but now pip will pull latest possible versions for each other transitive dependency as well. For risk of stating the obvious: The reproducibility in Docker workflows comes from building and uniquely tagging an image, then pushing this same image to a registry and using the exact same tag to pull and deploy that image from the registry. In other words, within Docker workflow the Docker image is the lockfile. In a docker workflow it is up for the devops engineers to establish a link between a source code commit revision and accordingly tagged image. Docker images are not checked in to source code. By checking in a full requirements.txt with all transitive dependencies, this link is automatically made at least for all Python libraries. Another drawback for us, Apple Silicon victims - a docker image build for linux/amd64 will not run on linux/arm64. The latter is what MacOS emulates linux on, to run docker. There is some virtualization to run docker on Mac in linux/amd64 mode, but your image must be mutliplatform and running it in 64 bit CPU op mode is still unsupported at the time of writing. Running in 32 bit makes some Python packages with native extensions not even being able to launch. Docker for Mac also comes with a performance penalty of approx 30% , but virtualization of linux/amd64 comes even with some performance punishment. Devcontainers still have their benefits, even if you just build dev images from scratch for same cpu architecture, without having a commit to docker tag relationship in development env. Even if such workflow does not create an exact reproducible environment, it creates a homogeneous and isolated environment. Also they are a good way to automate installation. In the end, if we already develop on another architecture, the expectation is not to have an exact env copy of production locally. By locking the exact dependencies of our own 1st party software, we get at least a guarantee, that exact same versions will be used on another architecture. Those versions came from a single point in time multiplatform build. ## Summary Setting up a Python dev environment is still a common source of frustrations in 2024. There is no out the box solution that would match feature parity with tools from other languages like bundler, cargo or Go Modules. Especially Apple Silicon trend of switching development to `arm64` architecture but deployment still on `amd64` made the situation worse. Most of the time one can set up good workflow though with pip+venv+piptools or 3rd Party tools like Poetry. This will: - Find a way to install multiple pythons - Make your python code a python package - Put all your config into a pyproject.toml - Use a virtual environment - Use a deterministic locking solution Last but not least, because the situation is still far from perfect, there is again some movement in the world of Python package managers. The author, known by its pseudonym [@mitsuhiko](https://github.com/mitsuhiko), is the creator of the popular Flask framework. He started the movement by questioning again, whether another [standard should exist](https://github.com/astral-sh/rye/discussions/6). Despite there being so manhy packaging solutions already, the answer, among pythonists, was yes and so [rye](https://rye-up.com/) was born. It is unfortunately not cross platform yet and also is not supported by dependabot, so I did not try it yet. But maybe it already will work for you. Also it is written in rust, which I suppose speaks for itself.

drehh

How to Setup Python and the Reasons why it is such a mess

Join Our Newsletter. No Spam, Only the good stuff.