If you are into Serverless and AWS Lambda, you may already know that the AWS Serverless Application Model (SAM) CLI makes it easy to leverage their Docker build images as development containers. We do exactly this for our Rails & Lambda projects.

Leveraging Docker with SAM ensures we have a Linux environment and versioned dependencies that closely mimic the Lambda Runtime or Container being shipped. The use and The Promise of Docker to solve these problems is nothing new... but something else is.

✨ The Rise of Ephemeral Dev Environments

A few weeks ago GitHub's engineering team released an in-depth article announcing their internal usage of the now generally available GitHub Codespaces. Since Custom Ink shares many of the same problems described in this post, I was curious if our Lambda projects could easily leverage Codespaces. But what is this new tool? Where did it come from? And what is this devcontainer.json file?

As best I can tell this all started in May of 2019 when the VS Code team first mentioned their remote development extensions. About a year later this content was rolled up into the VS Code Remote Development guides we have today.

Prior to Codespaces, we have had a clear leader in the automated development environment space with Gitpod. It was even featured in a January 2021 episode of Containers from the Couch. Gitpod leverages the same technology built into VS Code for remote development.

However, sometimes slow and steady wins the race. If this were ever true for GitHub-based projects, I think we have a huge winner with GitHub Codespaces. Keep reading below on how your company (or you) could get started. I will even cover how well Codespaces has worked for our Lambda projects that use an existing Docker in Docker development pattern.

⚙️ GitHub Settings

GitHub Codespaces is ONLY available now for GitHub Teams & Enterprise Cloud plans. It is not yet available for public repositories. If you are an administrator of such an account, here are a few things I did at the organization level to get started experimenting.

• Enable Codespaces: This can also be disabled completely or enabled for select users.
• Repository Access: You can even limit repositories that are able to use Codespaces. If your GitHub account leverages permissions & teams, remember, Codespaces (via the generated GITHUB_TOKEN will not grant anyone elevated permissions to other repositories.
• Manage Spending Limits: It would have been neat to see a way to limit which VMs (vCPU/Memory) options could have been used here.
• Organizational Secrets: Create any secrets your organization needs to enable individuals to work. Remember, Codespaces secrets can be set at the repository or even user level too. Pick the one(s) that work the best for y'all.

🔰 Developer Tips

It could go without saying but getting good at Codespaces for most may mean getting good at VS Code. Technically you could bring your own editor like Vim or Emacs. But trust me, as a recent Sublime Text convert, switching to VS Code is worth it. Make sure to take the time to Google, learn, and in some cases install packages that make the transition easier.

Dotfiles & Settings

Remote development needs to feel local! Everything that makes your editor & terminal productive needs to be available to you. As described in the Personalizing Codespaces guide setting up your Dotfiles was high on my list.

For years I have maintained a personal Zshkit which had a ton of personal functions and aliases. When moving to Codespaces, I took the time to clean them up and create a github.com/metaskills/dotfiles repository, cloned it locally and hooked it up to my ZSH (default shell on Mac) ~/.zshrc file. Codespaces will automatically clone this repo when creating a Codespace and install it by running the install.sh script. Example.

if [ "$CODESPACES" = "true" ]; then
  echo "source /workspaces/.codespaces/.persistedshare/dotfiles/rc" >> $HOME/.zshrc
  sudo chsh -s /usr/bin/zsh
fi

You can leverage the CODESPACES environment variable to do any customization per environment. Also, do not forget to use Settings Sync. I think this is only needed if you use VS Code's web-based editor. More on that topic later.

Your Codespaces Settings

You can Manage Your Codespaces settings at somewhat the same level as the organization. Here are a few settings I did.

• Access & Security: Set this to "All repositories". Your needs may vary.
• Editor Preference: Set to "Visual Studio Code" vs for web. Ensures the [<> Code] button on repos opens VS Code on my Mac and avoids the need to click redirect in the browser.
• Region: I set this manually to EastUs but I suspect I had no reason to do so.
• Added Secrets: Read below on using SSH with Ruby Bundler or NPM packages.

Codespaces Extension

Install the GitHub Codespaces for VS Code. I think this is done for you automatically if you are using the web-based editor. Installing it on your host machine's VS Code will mean you can use Codespaces without ever browsing to GitHub.com and clicking on a [<> Code] button.

The Integrated Terminal

Assuming you have setup your Dotfiles, VS Code's integrated terminal should feel familiar by mirroring your host machine's prompt, aliases, and more. If your default shell is ZSH, you may need to do a few things to help Codespaces use ZSH by default vs Bash. Here are my settings for the integrated terminal now. Mind you, there was (maybe still is) a bug in VS Code where ZSH would not be respected. I have noticed in some cases Bash is used but it is easy to launch a new profile with ZSH if that happens.

  "terminal.integrated.fontSize": 14,
  "terminal.integrated.defaultProfile.osx": "zsh",
  "terminal.integrated.defaultProfile.linux": "zsh",

Using Command+K to clear the terminal's buffer is second nature to most. By default this key binding will not reach the integrated terminal. You can edit your Keyboard Shortcuts JSON file to solve for that. Below is a screen capture of the magic little button you have to press to edit that raw JSON file. Use the following snippet to fix this.

{
  "key": "cmd+k",
  "command": "workbench.action.terminal.clear",
  "when": "terminalFocus"
}

Terminal visibility and placement. When working on my laptop's smaller screen, I learned that you can use Control+~ to toggle the visibility of the integrated terminal. However, when working at my desk and larger screen, I really want the integrated terminal to be to the right of my editor. Thanks to this this Stack Overflow here are convoluted steps to make this happen. Hopefully one day they will make this easier. 😅

1. At the right top of the integrated terminal, click the + sign to open a 2nd terminal.
2. Within the panel to the right, right click any of the two profiles, select Move into Editor Area.
3. Close the bottom integrated terminal with the x button.
4. Focus the editor tab at the top moved from step 2, click the [|] split editor button.
5. Close the shell tab on the left side of the screen.

🎉 Fun Highlights

Here are a few things I was pleasantly surprised with Codespaces' DX and how it works:

• When learning Codespaces or working on uncommitted code, you may have to rebuild your development container. Codespaces automatically maintains your present working directory, open files, etc when doing this. Amazing!
• You can see all your Codespaces on GitHub by navigating to https://github.com/codespaces. However, I typically use VS Code's extension to navigate, open, and disconnect.
• Leveraging the CODESPACES environment variable set to true is an easy way to integrate your existing tooling into Codespaces allowing your teams to support multiple ways to bootstrap your applications.
• Forwarded ports are automatically detected via the integrated terminal's STDOUT. For example, a .bin/rails server will ouput whatever host/port you are using and Codespaces will see it. If needed you can use the forwardPorts config for devcontainer.json.

⚠️ Difficult Lessons

Some hard lessons learned when dipping into the deep end of using GitHub Codespaces. If you have any to share, please drop some comments below.

Private Packages & SSH

GitHub does a great job at providing your Codespace with a short lived GITHUB_TOKEN. Most package managers including NPM and Bundler can leverage this. However, if your organization has standardized on SSH setting up your projects could be a problem.

Thankfully when I reached out on Twitter, Jonathan Carter on the Codespaces team, seemed to suggest they may be working on a native SSH integration one day. Till then, here is the solution I came up with. This process address some sequencing issues around devcontainer.json's Lifecycle Scripts and when your Dotfiles are installed. Credit to VS Codes Using SSH Keys guide. Also, some things here are pulled directly from the GitHub Action to setup SSH. Again, thanks to Johnathan Carter for the ideas.

1. Create a personal Codespace secret called PERSONAL_SSH_KEY by visiting this page https://github.com/settings/codespaces/secrets/new and adding your private key, typically found in the ~/.ssh/id_rsa file.
2. Add this snippet to your postCreate script. It ensures GitHub is in the known hosts for SSH.

echo "Adding GitHub.com keys to ~/.ssh/known_hosts"
printf "\ngithub.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==\n" >> ~/.ssh/known_hosts
printf "\ngithub.com ssh-dss AAAAB3NzaC1kc3MAAACBANGFW2P9xlGU3zWrymJgI/lKo//ZW2WfVtmbsUZJ5uyKArtlQOT2+WRhcg4979aFxgKdcsqAYW3/LS1T2km3jYW/vr4Uzn+dXWODVk5VlUiZ1HFOHf6s6ITcZvjvdbp6ZbpM+DuJT7Bw+h5Fx8Qt8I16oCZYmAPJRtu46o9C2zk1AAAAFQC4gdFGcSbp5Gr0Wd5Ay/jtcldMewAAAIATTgn4sY4Nem/FQE+XJlyUQptPWMem5fwOcWtSXiTKaaN0lkk2p2snz+EJvAGXGq9dTSWHyLJSM2W6ZdQDqWJ1k+cL8CARAqL+UMwF84CR0m3hj+wtVGD/J4G5kW2DBAf4/bqzP4469lT+dF2FRQ2L9JKXrCWcnhMtJUvua8dvnwAAAIB6C4nQfAA7x8oLta6tT+oCk2WQcydNsyugE8vLrHlogoWEicla6cWPk7oXSspbzUcfkjN3Qa6e74PhRkc7JdSdAlFzU3m7LMkXo1MHgkqNX8glxWNVqBSc0YRdbFdTkL0C6gtpklilhvuHQCdbgB3LBAikcRkDp+FCVkUgPC/7Rw==\n" >> ~/.ssh/known_hosts

1. Add this snippet to your Dotfiles. It will ensure the proper SSH agent is started, if not already, and that the key environment variables are set.

if [ "$CODESPACES" = "true" ]; then
  if [ -z "$SSH_AUTH_SOCK" ]; then
    RUNNING_AGENT="`ps -ax | grep 'ssh-agent -s' | grep -v grep | wc -l | tr -d '[:space:]'`"
    if [ "$RUNNING_AGENT" = "0" ]; then
      # Launch a new instance of the agent
      ssh-agent -s &> $HOME/.ssh/ssh-agent
    fi
    eval `cat $HOME/.ssh/ssh-agent`
  fi
  # Add my SSH key.
  if [ -n "${PERSONAL_SSH_KEY+1}" ]; then
    ssh-add - <<< "${PERSONAL_SSH_KEY}"
  fi
fi

In order to see this all come together with our Docker in Docker Lambda patterns, please read the Serverless Docker Patterns article in this series where we describe how to use the SSH_AUTH_SOCK in a cross platform way for Mac & Linux.

AWS CLI

For our Lambda projects we use Docker in Docker patterns where both the AWS & SAM CLIs are pre-installed on the development image. However, you may need the AWS CLI installed on the developer's host machine too. In this case, Codespaces. Here is a short snippet that you can use in your postCreate script.

echo "Installing AWS CLI"
pushd /tmp
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip -qq awscliv2.zip
sudo ./aws/install
rm -rf awscliv2.zip ./aws
popd

Docker in Docker

I've said this before but cross platform Docker in Docker is really hard. This series aims to talk about most of them, but one I learned the hard way is that sometimes the pain comes from the ones you love... in this case AWS SAM. The team is doing some amazing work but I ran into a few issues where Docker in Docker patterns have broken down. Read here for details.

• No Response from Invoke Container for Lambda Inside docker-compose #2837
• Watch Option for SAM Build Command #921

🚂 Full Lamby Example

Assuming the other patterns were in place like various postCreate hooks for SSH, using GitHub Codespaces with your already Docker'ized project is super easy. Here is a complete .devcontainer/devcontainer.json file for one of our projects. Again, see the Serverless Docker Patterns related post on how we are using COMPOSE_FILE for Mac filesystem performance and why it would be needed here.

{
  "name": "my-application",
  "forwardPorts": [4020],
  "remoteEnv": {
    "COMPOSE_FILE": "docker-compose.yml"
  },
  "postCreateCommand": "./.devcontainer/postCreate"
}

In fact, none of this would be needed for a starter application! Give it a try. Go through our Lamby Quick Start guide, commit your project to GitHub... and give Codespaces a try!

🔐 Security Questions

The Codespaces team was kind enough to write their own Security in Codespaces documentation. I'll highlight their introduction below:

Codespaces is designed to be security hardened by default. Consequently, you will need to ensure that your software development practices do not risk reducing the security posture of your codespace.

This guide describes the way Codespaces keeps your development environment secure and provides some of the good practices that will help maintain your security as you work. As with any development tool, remember that you should only open and work within repositories you know and trust.

Good stuff! Security is a shared responsibility and it appears GitHub is doing their part. Please read over the full documentation for more information, but here are a few things I paid special attention to.

• Audit Logs: Are generated and can be queried.
• Organization & User Secrets: Built on the same technology GitHub draws a line between GitHub standard org/user secrets with the Codespace ones. Again, they can be set at the organization, repository, or user. Providing an immense amount of control and security layers.
• Dotfiles: Remind users that these are public repositories! Tho possible to encrypt secrets, I personally recommend keeping them basic to aliases and functions.
• Secure Networking: Authenticated via GitHub via temporary tokens. Forwarding ports for web servers is done securely over the network between the host. Nothing is public by default.

🔮 What is Coming?

As mentioned above, I would love to see a native SSH solution. For now, the workarounds are minimal and feel secure with GitHub Secrets and Codespaces integration.

In their introductory blog article, the GitHub team put a lot of emphasis on prebuilds ensuring that each Codespaces development environment was super fast to setup. This was critical for their team and as of now Gitpod is making a clear distinction this is a key differentiator for them. I suspect prebuilds are coming soon. 🤔

📚 Resources

Thanks so much for reading! I would love to hear if you found this article helpful or what your organization may be doing with GitHub Codespaces. 💕

• GitHub Codespaces - Blazing fast cloud developer environments with Visual Studio Code backed by high performance VMs that start in seconds.
• GitHub’s Engineering Team has moved to Codespaces - Great description of the business needs for easy development environments. Common for most orgs.
• Getting Started with Rails on Lambda - An quick start guide using Docker for development with GitHub & Codespaces.
• VS Code Remote Development - The architecture behind GitHub Codespaces.
• Gitpod - Spin up fresh, automated dev environments for each task, in the cloud, in seconds.
• DevX Digest: The Rise of Ephemeral Developer Environments - Great post by Pauline P. Narvas on where cloud-based dev environments are headed.