I want to learn more about Linux. What better way than to make it my daily driver?

It's been just over a month now so I figured I'd share my experience with using and installing Linux along with some of the surprises I encountered along the way.

Now, for the record, I've used Linux before but mostly for Docker build systems at work but I haven never used it as a daily driver on my desktop machine.

Guru or nooby - I hope you find this post useful or interesting in some way.

Quick summary of my setup

Distro: Void Linux base image

Update Mar 2022: I switched to NixOS

I chose Void Linux as my first image. It's a very bare-bones distro, similar to Arch and Gentoo. I liked the idea of packages being audited by other developers before being allowed to be added to the registry. It also runs runit as its init system which I like the philosophy of (small codebase, single responsibility). That's all I'll say about that.

If all you're looking for is barebones distro but don't know what/how to pick, I'd recommend starting with Arch Linux as it has extensive documentation and a very large community.

The rest of the setup

Okay lightning round, here's some other interesting parts of my setup:

The essentials:

• File system format: F2FS with zstd compression (benchmarks were very fast)
• Default shell: fish

UI stuff:

• Window manager: Swaywm with Waybar
• Shell theme: Tide
• GTK UI theme: Adwaita Dark
• San-serif font: Inter
• Serif font: ... Inter... I'll pick out a good serif font, eventually I promise :3
• Monospace font: Jetbrains Mono
• Cursor theme: Elementary OS

Apps:

• File manager: Nautilus
• VM for Windows: QEMU/KVM with virt-manager
• Media player: Haruna
• Terminal emulator: Alacritty
• VM software: KVM, QEMU, libvirt, virsh and Looking Glass

Okay, let's move on.

You can use any distro's documentation

When it comes to trying to configure Linux, pretty much any distro's doco will do. You just have to understand the differences between distros, which, I've come to believe, are far fewer than some would have you believe, at least, from the user's perspective.

This was certainly the case when it came to using Arch Linux's wiki. I ended up using the Arch wiki more than the Void Linux documentation!

There are a few philosophical differences between distros (E.g. Rolling vs point releases, audited package repositories vs unaudited package respositories, etc) but, generally, there aren't a lot of technical differences in terms of the libraries & software used between distros. However, there are two main differences I found I needed to be aware of when it came to setting up Void:

The package manager: Void Linux uses "X Binary Package Manager" while Arch uses "pacman". Generally speaking, this mostly boils down to different CLI semantics. Also sometimes less popular packages in the AUR (Arch's package repository) don't have any equivalents in Void's package repository.

The init system: Void Linux uses "runit" while Arch uses "systemd". This only tripped me up once in the past month when I was trying to setup swaywm which extracts the parts of "systemd" that most software depends on to work, although you could argue that it sort-of defeats the purpose of using a light-weight init system to begin with. See seatd as a lightweight alternative to elogind.

Anyway, based on my experience, I say just pick a distro and run with it. You can always switch distros later if you want. Just make sure to keep track of your setup/configs in some sort of repository or storage system (here's mine).

Getting Linux to boot was the most frustrating part

It took me a whole week just to get Linux to boot. Partly, because I wanted to avoid using GRUB in favour of an EFI Stub.

As it turns out, some motherboard's UEFI (including mine) won't work properly unless you create a bootloader file called /EFI/boot/bootx64.efi.

In my case, any changes I made to efibootmgr would disappear as soon as I rebooted my machine. It was only started working when I created a bootx64.efi file. It didn't even have to be a valid EFI file, the file just had to exist.

So... I have two GPUs in my desktop now

Update Mar 2022: Nvidia have open source drivers now!
Update Sept 2021: Nvidia drivers may have full support for Sway in its next update

Some readers might know where this is going.

I have an Nvidia graphics card.

Nvidia gets a pretty bad rap from a lot of people in the Linux community. I'm literally required to run my window manager with a flag that says --my-next-gpu-wont-be-nvidia.
.

The topic makes for some pretty spicy reading 🌶️🌶️🌶️🌶️🌶️ but to cut a long story short, at the time of writing (22/08/2021), Nvidia's proprietary drivers don't fully support an API called Generic Buffer Management (GBM) - A more or less universally agreed upon API required most Wayland compositor libraries (like the official version of wlroots, used by swaywm) and supported by more or less every other hardware manufacturer's drivers. This means quite a lot of modern desktop environments aren't supported properly when you're using an Nvidia GPU.

I've linked a few posts that summarise the whole Nvidia<>Linux situation:

• "Nvidia sucks and I'm sick of it" by Drew Devault
• Linus Torvald's (in)famous quote
• Phoronix's articles about Nvidia's Wayland support

Luckily, there's something called PRIME that lets you switch between graphics cards for different applications. The fact that something like this exists is pretty mind-blowing to me - I've never seen GPU switching made so easy.

I scoured the depths of my box of old hardware and found an AMD GPU that I now use as my daily driver to render my desktop environment and I simply run prime-run <application> to switch to my newer Nvidia graphics card for apps that require beefier GPU acceleration like games and Krita.

Nvidia has recently been putting in work toward native Wayland support with their their more recent drivers so let's hope we see the day when Nvidia GPUs works just as well as others on Wayland.

Playing games in a Windows VM

I still wanna play games.

"WE HAVE PROTON!!!" screech the hoard of Linux gamers.

Yes, yes, lower your pitch forks. Install Wine+Proton and you've got yourself a plug and play gaming experience on most games.

However, there's no easy means of supporting UWP apps via Wine. That's every game installed through Microsoft Store or Xbox Game Pass. In this case, I found dual booting or using a VM were pretty good solutions.

If you just want a simple solution I'd recommend setting up dual boot on your machine, it's pretty straight forward and very well documented.

For the Dark Souls fans out there (i.e. Familiar with pain and suffering), read on.

GPU passthrough

If you want to play games you're going to want to remove the overhead of virtualisation for your GPU by passing it directly to the VM.

I decided to go the VM route since I didn't really want to have to reboot my system just to play games.

However this turned out to be much harder than I thought it was going to be.

VFIO drivers

"VFIO" drivers allow GPU PCI passthrough to your VM. This bit was pretty straight forward, you install a few drivers, add a couple kernel parameters to your boot, add a couple settings to your VM manager and... Voila! GPU passthrough!

Except it's not always that easy...

The curse of bad IOMMU groups

To oversimplify the problem: Your motherboard assigns your PCI devices into "IOMMU" groups. If you want to passthrough a single PCI device, everything in an IOMMU group has to passed through.

If we the following script, taken from the Arch wiki:

#!/bin/bash
shopt -s nullglob
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

We get a nicely formatted list of our IOMMU groups and all the devices inside them.

Let's take a look at some of my IOMMU groups shall we?

IOMMU Group 1:
    00:01.0 PCI bridge [0604]: Intel Corporation 6th-9th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
    00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 07)
    01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] [1002:6739]
    01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6790/6850/6870 / 7720 OEM] [1002:aa88]
    02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
    02:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)

Not great, you can see that both my GPUs are in the same IOMMU group. Not great if you want minimal-overhead GPU acceleration on both your Linux and Windows machine.

Generally, if PCI passthrough is something you're interested in, most people recommend you look for motherboards with good IOMMU groups. Unfortunately I didn't have the foresight to do this at the time I built my PC and I didn't want to buy a new motherboard for the sake of playing games in a VM.

And so, with no elegant solution to the problem, our journey takes us deeper still into the Linux ecosystem...

acs_override & compiling my first kernel

For the desperate, acs_override has got your back. It's basically a git patch that you apply to the Linux kernel's source code that allows you to totally disregard IOMMU groups.

This stuff is blood magic. There's a price. By separating your hardware IOMMU groups this way, you're opening yourself to a certain set of virtualization related attacks. For my use case, this wasn't too much of an issue.

I patched my kernel with Linux Zen. In addition to the acs_override patch, it claims to modify Linux a way that better caters it to highly interactive realtime experiences (E.g. Gaming or personal desktop use). There are claims online that it reduces perceived "stutter" and reduces the minimum FPS of games under certain workloads.

I think a lot of people think building the Linux Kernel manually is a really complex task but it's actually pretty simple. I'd recommend giving it a go if you run into the same issue as I did.

If you're interested in custom desktop kernel patches take a look at:

• Arch kernel docs: Explain a few things about different kernels (Also has more links that what I've got here)
• Liquorix: Explains what Zen and Liquorix patches change in the Linux kernel
• Xanmod: Does quite well in Phoronix benchmarks

Windows in a window with Looking Glass

So at this point it's totally possible to using Windows inside your Linux machine but only if you're willing to switch your monitor's cable between the GPU running on Linux and the one running on Windows. Not the best experience.

Luckily, Looking Glass comes to the rescue. Looking Glass, in a nutshell, allows your Windows desktop to be displayed as a window in Linux without near-minimal overhead in terms of performance and latency.

Setting it up is a little bit fiddly to setup but once it works - it works!

How it works is explained well in a Youtube video linked in the Looking Glass questions page.

Results

Anyway, after all that, I can finally play Halo without leaving Linux.

Interestingly, the experience is arguably better than it is running directly in Windows because the game is easily window-able. Many games, normally, from my experience, either have to be fully minimised or fullscreen - Or changing from full screen to a window has to be done through gaming settings.

So... Do you recommend trying Linux?

Yep,

I think my post makes it sound like there's an unending list of problems you'll run into if you try use Linux but that's mostly to do with my original goal: Learn more about Linux. Incidentally, I think I learnt more about what makes a good piece of hardware more than anything else but still a good outcome.

If you want a "it just works" flavour of Linux, there's plenty of those out there, too.

I ended up installing Linux with Gnome 40 on my girlfriend's (who is not an engineer) laptop and so far so good, but I might save that story for another time.

Update: It's been 4 months now and I'm still running Linux

Thanks

Last but not least, shout out to Daniel Playfair Cal (A.K.A hedgepigdaniel) for helping me understanding the Linux ecosystem and pointing me in the right direction while setting up Linux.