The Importance of Consistent Base Systems

Few things are as anxiety-inducing as watching stable production servers become a chaotic mess after a simple update. Yet, this scenario is one that many sysadmins dread. If you're seeking ways to avoid such scenarios, especially while leveraging Ansible in combination with Debian-based systems, read on.

📦 Linux Package Management: A Double-Edged Sword

One of the greatest advantages of Linux systems is without a doubt its package management. Most "classic" distributions handle application- and operating system updates using the same package management system. This has many advantages, like applying the latest security patches for the OS and applications by just doing a single update transaction. But this of course has its drawbacks.

Just think about application updates: For example, databases like PostgreSQL are not always compatible across major releases. This is one of the reasons Long Term Support (LTS) distributions often stick to specific application versions and just provide bug- and security patches during their lifecycle.

🗄️ The Issue With Third Party Repositories

Third-Party packages (in the sense of not provided in the distribution's repository) are often used, be it to avoid unwanted changes by package maintainers, installing software versions provided by the application vendor or ones that are simply not available in the default repositories. However, these often follow distinct release cycles from the underlying distro, complicating system update processes.

This leads to the necessity to mitigate unwanted upgrades when updating the base OS. Different package managers handle this in various ways.

📌 Ansible and Pinning Debian Packages

Debian-based systems offer means to prevent package version upgrades. Apart from employing APT Pinning, sometimes simply adhering to a specific version suffices, accomplished by "holding" packages using apt-mark.

Now to the issue to why inconsistency in systems can make a difference here, using a real-world example (luckily not in production). The Ansible's ansible.builtin.apt module by default uses Aptitude, an alternative frontend to the apt or apt-get CLI. If not available, it falls back to using said programs. dpkg, which is also used as the backend for apt-mark, stores the state of every installed package in the package status database under /var/lib/dpkg/status. Unfortunately, this data is not taken into account by Aptitude. It uses an own database for its hold function.

What's the upshot? Let's say you update all machines using the Apt module for Ansible. The packages you want to handle separately (e.g. the "main" function of the server) are held by DPKG to avoid unwanted upgrades, so you're safe. There's one issue though: one of the systems has Aptitude installed without you knowing. So during the upgrade, Ansible detects Aptitude, adopts it as the default package management tool, and updates all packages, including those retained in DPKG's database. Not an ideal outcome, right?

🛡️ How to Safeguard Against This

Several approaches exist to tackle such scenarios, like containerizing applications, creating custom repos, or employing diverse methods to pin packages. However, if retaining existing infrastructure is the priority, ensuring Aptitude's presence on all systems or utilizing the force_apt_get flag in the Ansible module can avert inconsistencies between systems.

Moreover, it's important that you know your infrastructure and ensure that servers with similar functions or managed by one team are set up in a consistent manner.

❓ How do you handle similar scenarios? Feel free to share your thoughts or critique in the comments!