Gitpod notes and thoughts

A few months ago, a colleague reposted a Gitpod blog post in my employer's slack, mentioning that if you're a member of some reputable GitHub organization, you get a plan with unlimited hours. Thank you, the past me who spent a weekend packaging an app for nixpkgs!

I knew these kinds of cloud IDEs have been around for some time, but that message inspired me to give it a bit of try, who can resist free lunch after all? Well, I'm now using it on a daily basis, so this post will record some notes and thoughts along the way. The content is not quite well structured, but I hope they can provide a bit of insight for other Gitpod users as well.

Hacking GHC

The first Gitpod use case I tried is hacking GHC. And that's quite a roadbump because GHC is hosted on their own GitLab instance!

Yes, you probably can fiddle with tokens and such, to make Gitpod work with self-hosted GitLab. It's mentioned in Gitpod docs, I haven't tried, since there's another problem ahead:

Gitpod requires you to check in .gitpod.yml into the repo you're working with. That's reasonable for most single-repo use cases, but I don't have any mental energy to upstream a Gitpod config, nor do I want to pollute my GHC working branches with this config file.

So I made a dedicated GitHub repo for hacking GHC. Gitpod allows you to specify scripts to run during workspace initialization, so I can just clone GHC to /workspace and open it later, profit.

Git authentication with SSH

All looks good except when I need to push. For GitHub repos I own, pushes to https remotes should work out of the box, but well, this doesn't work for GHC's self-hosted GitLab.

This hasn't been a problem when I used my employer's remote build machine, since I can always enable SSH agent forwarding, so I can do the pushes on that machine. Gitpod allows you to SSH into an active workspace, but honestly I don't want to open a separate terminal window just for git operations, I prefer to stay inside that browser tab all the time.

So. Given Gitpod allows specifying secret environment variables, I generated a passwordless SSH private key and encoded it as a secret. The gitpod-ghc init script will start an ssh-agent and automatically add that key, so future git operations will work out of the box.

Yes, doesn't sound like a good security story, other than permuting the key frequently and limiting the key's scope, I still need to trust Gitpod not to leak my secret, no less than they trust me not to abuse their machines to run crypto miners.

What's persisted and what's not

There are multiple ingredients in making a Gitpod workspace:

The base Dockerfile or a Docker Hub image. Use whatever you like, but given the Gitpod runtime will yolo a bunch of Linux binaries at runtime, it better be a typical glibc-based FHS distro.
The .gitpod.yml config, which allows you to specify init scripts.

Having two places to write scripts may seem redundant, but the .gitpod.yml scripts have access to the /workplace directory. Across restarts of the same Gitpod workspace, only changes in /workplace is persisted!

It's quite an annoyance since for a long-lived workspace, you often accumulate implicit state (e.g. language-specific caches) not only in your project directory, but also in the home directory.

Luckily, there's a limited workaround for that: bind mounts. In gitpod-ghc config, I bind mount a lot of home directory contents from /workplace instead, things like .config, .cache, .cabal. Even the entire Nix store!

It's a pity I can't bind mount the entire home directory though, likely because of busy resources during workplace initialization. And of course, all system package manager invocations are forgotten after workplace restart.

To think of it in a more positive way, Gitpod is enforcing the erase your darlings philosophy. It doesn't strictly prohibit state, it forces you to make your state explicit.

Prebuilds

Gitpod prebuilds is a CI service that runs on the same infra to power the workspaces. When a prebuild is triggered, the Docker image gets built, a fresh /workplace with a checkout of that commit is mounted, and the expensive (init) step of .gitpod.yml scripts get run. The updated /workplace is saved and reused upon workplace start up, so the init step can be skipped at that time.

For a large project like GHC, enabling Gitpod prebuilds is essential to ensure a sane startup time for new workspaces. Here's a non-comprehensive list of things done during gitpod-ghc prebuild time:

Install latest release versions of ghcup, ghc and cabal
Compile and install haskell-language-server
Check out GHC tree into /workspace
Go through the configure step, build hadrian, build up the .hie-bios cache, so the language server shall work nearly instantly for a fresh workspace

In-browser VSCode caveats

Oops. I've written a lot in this post and haven't even mentioned VSCode even once. Yes, Gitpod supports multiple IDEs, but the one I'm using on a daily basis is the in-browser VSCode. Some caveats to keep in mind:

Most default key bindings work. ^W closes the browser tab instead of the editor tab though, luckily there'll be a prompt about unsaved work. Don't silence that prompt.
Flaky connection is fine, but don't drop offline for more than 3 minutes.
Settings do get synced properly. A minor annoyance is you can't declare editor settings in .gitpod.yml, apart from the extension list.
The default extension store is OpenVSX instead of the Microsoft-hosted one. There do exist extensions that are severely outdated in OpenVSX (like rust-analyzer), but as a fallback measure, you can specify URLs to .vsix artifacts in .gitpod.yml.

Other things known to work (or not)

There are things that work, and things that don't work yet. I didn't really spend time to dig into Gitpod's issue tracker, so I'll be more than happy to be corrected in the comments shall the list requires some update!

sudo works. Of course you're still in a container jail, so you can't really mess with sysctl config and such.
systemd doesn't work. You need to bring your own userspace supervisor if you need such a thing.
nix works with single-user installation and sandbox = false. Multi-user mode with nix-daemon should still be possible, but not worth the effort if sandboxing won't work anyway. So don't ever push to binary caches from a Gitpod workspace.
docker works out of the box as rootless mode, and the state directory is in /workplace. podman doesn't, despite unprivileged user namespace is on.
gdb works. strace works. lldb doesn't. rr doesn't.
proot latest version works.
tailscale userspace mode works, but running the daemon as root seems quite fragile. Haven't tried other VPNs yet.
htop kinda works. The CPU/RAM stats don't reflect the workspace, but reflects the entire metal below it.

Community support

Gitpod development happens on GitHub, they allow self-hosting and the sausages are made in the open.

Apart from that, there's an official discord server. Not a super active one, but the staff are there, and questions do get answered within a day or two, regardless of whether you're a paid user or not.