A Window into Docker, minikube, and containerd
wmchurchill3
Posted on September 28, 2021
Like many of you, I received an email from Docker notifying me of their changes to service. Having used Docker Desktop for many years as part of my work, I was a little concerned. My concern was not great enough to do anything... Until a co-worker suggested an article switching from Docker for Windows to containerd. This link from 2018 seemed to suggest containerd could run on Windows.
Spoiler Alert/TL;DR: This is not a post about getting containerd running on Windows. I was able to get a Windows nanoserver image running in containerd. I could not get that image connecting to any network. This post is a survey of the source code, GitHub issues, and dead links chased. All documented to show how close and far away we are to something useful.
Where does minikube fit in here?
In my research and frustration, I wanted to try running something else. I enabled Hyper-V on my machine. Followed the instructions at minikube quickstart. Things worked! Thank you to the maintainers of minikube! Great Job! I definitely will be using this more in the future.
The only place I deviated was in starting the minikube cluster. I used the command minikube start --driver=hyperv --container-runtime=containerd. For fun, I checked the Hyper-V Manager and saw a new virtual machine named 'minikube'. Then it hit me. A Linux VM hosts the minikube cluster complete with its own version of containerd. This means I could not run a Windows image!
The Journey Begins
The first stop was the Container Platform Tools on Windows. This is where the dead links begin (see the Links to CRI Spec). My second stop was the containerd site. I downloaded and installed the requirements and release tarball. When the compiling started, I ran into an issue with make looking for gcc. This seemed odd since 1) it is a Go application, 2) having gcc on Windows seems like a high bar for running containers.
Some more Googling brought me to James Sturtevant's site. This made me aware pre-built Windows containerd binaries exist. Now I was making some progress.
The following code snippet will download and configure containerd as a service. Each line does the following:
- Download the latest (as of 20210924) release of containerd
- Make a directory for the containerd binaries and configs
- Expand the containerd tarball
- Move the binaries to the directory created above
- Add containerd to the Path environment variable
- Create a default containerd configuration in the containerd directory
- Tell Windows Defender not worry about the containerd executable
- Register containerd as a service
- Start containerd
In a Admin PowerShell window,
curl.exe -LO https://github.com/containerd/containerd/releases/download/v1.5.5/containerd-1.5.5-linux-amd64.tar.gz
mkdir "C:\Program Files\containerd"
tar -xzf containerd-1.5.5-linux-amd64.tar.gz
mv .\bin\* "C:\Program Files\containerd"
$env:Path = $env:Path + ';C:\Program Files\containerd'
containerd.exe config default | Set-Content "C:\Program Files\containerd\config.toml" -Force
Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\containerd\containerd.exe"
.\containerd.exe --register-service
Start-Service containerd
To verify containerd is running:
- Open the Task Manager
- Go into the
More Detailsview - Scroll to
Background Processes - You should see a
containerd.exeprocess
Running a Container
Under ideal circumstances, we would pull an image using the ctr command.
.\ctr.exe pull docker.io/library/mcr.microsoft.com/windows/nanoserver:10.0.19042.1165-amd64{% raw %}`
```
Unfortunately, there is some authentication around the Microsoft images. Assuming you have one downloaded using Docker, we can
1. Save the image
1. Import the image using ctr
1. Run the image.
From the Admin PowerShell window,
```
docker save mcr.microsoft.com/nanoserver:10.0.19042.1165-amd64 -o nanoserver.tar
.\ctr.exe image import --all-platforms c:\wherever\you\put\this\nanoserver.tar
.\ctr.exe run -rm mcr.microsoft.com/windows/nanoserver:10.0.19042.1165-amd64 test cmd /c echo hello
```
If you see `hello` on the next line immediately after the command, Success!
That's it, right?
We have a container running a Windows image, but no network.
### Creating A Network for the containers
We need extra setup for networking our pods. CNI (Container Networking Interface) will provide NAT'ing for our dev environment. We also must get a helper script to set up the network. The steps:
1. Get the CNI tools executables
1. Get the helper script hns.psm1
1. Create some directories
1. Expand the CNI tools into the created directories.
1. Allow your machine to execute scripts
1. Unblock the helper script, hns.psm1
1. Import hsn.psm1 for use. Disregard the warning about verbs. This is a naming convention.
From the PowerShell window,
```
curl.exe -LO https://github.com/microsoft/windows-container-networking/releases/download/v.0.2.0/windows-container-networking-cni-amd64-v0.2.0.zip
curl.exe -LO https://raw.githubusercontent.com/microsoft/SDN/master/Kubernetes/windows/hns.psm1
mkdir -force "C:\Program Files\containerd\cni\bin"
mkdir -force "C:\Program Files\containerd\cni\conf"
Expand-Archive windows-container-networking-cni-amd6464-v0.2.0.zip -DestinationPath "C:\Program Files\containerd\cni\bin" -Force
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope LocalMachine
Unblock-File -Path .\hns.psm1
ipmo .\hns.psm1
```
Now to configure the network. From the Admin PowerShell window,
```
$subnet="10.0.0.0/16"
$gateway="10.0.0.1"
New-HNSNetwork -Type Nat -AddressPrefix $subnet -Gateway $gateway -Name "nat"
```
In this case, the name must be `nat`.
Let's check our work. From the PowerShell window:
```
netsh lan show profiles
```
You should see the new 'nat' network.
```
Profile on interface vEthernet (nat)
=======================================================================
Applied: User Profile
Profile Version : 1
Type : Wired LAN
AutoConfig Version : 1
802.1x : Enabled
802.1x : Not Enforced
EAP type : Microsoft: Protected EAP (PEAP)
802.1X auth credential : [Profile credential not valid]
Cache user information : [Yes]
```
If you get an error about dot3svc not running, run `net start dot3svc` and run the `netsh` command again.
Configure containerd to use that network. From the Admin PowerShell window,
```
@"
{
"cniVersion": "0.2.0",
"name": "nat",
"type": "nat",
"master": "Ethernet",
"ipam": {
"subnet": "$subnet",
"routes": [
{
"gateway": "$gateway"
}
]
},
"capabilities": {
"portMappings": true,
"dns": true
}
}
"@ | Set-Content "C:\Program Files\containerd\cni\conf\0-containerd-nat.conf" -Force
```
### Container Runtime Interface (CRI)
We are in the endgame now. I promise. From the [README](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md), crictl provides a CLI for CRI-compatible container runtimes.
The following snippet performs the following:
1. Download the crictl executable.
1. Creates the default location for crictl to look for a configuration
1. Creates the configuration
From a PowerShell,
```
curl.exe -LO https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.20.0/crictl-v1.20.0-windows-amd64.tar.gz
tar -xvf crictl-v1.20.0-windows-amd64.tar.gz
mkdir $HOME\.crictl
@"
runtime-endpoint: npipe://./pipe/containerd-containerd
image-endpoint: npipe://./pipe/containerd-containerd
timeout: 10
#debug: true
"@ | Set-Content "$HOME\.crictl\crictl.yaml" -Force
```
### The Payoff
Using a pod.json of
```
{
"metadata": {
"name": "nanoserver-sandbox",
"namespace": "default",
"uid": "hdishd83djaidwnduwk28bcsb"
},
"logDirectory": "/tmp",
"linux": {}
}
```
The magic happens with this command:
```
$POD_ID=(./crictl runp .\pod.json)
$CONTAINER_ID=(./crictl create $POD_ID .\container.json .\pod.json)
./crictl start $CONTAINER_ID
```
### The Problem
Running the `.\crictl runp .\pod.json` creates a sandbox pod for use in creating a container in the next command. The runp command fails setting up the network adapter for the pod. The output is as follows:
```
time="2021-09-22T09:25:29-04:00" level=debug msg="get runtime connection"
time="2021-09-22T09:25:29-04:00" level=debug msg="connect using endpoint 'npipe://./pipe/containerd-containerd' with '10s' timeout"
time="2021-09-22T09:25:29-04:00" level=debug msg="connected successfully using endpoint: npipe://./pipe/containerd-containerd"
time="2021-09-22T09:25:29-04:00" level=debug msg="RunPodSandboxRequest: &RunPodSandboxRequest{Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:nanoserver-sandbox,Uid:hdishd83djaidwnduwk28bcsb,Namespace:default,Attempt:0,},Hostname:,LogDirectory:,DnsConfig:nil,PortMappings:[]*PortMapping{},Labels:map[string]string{},Annotations:map[string]string{},Linux:&LinuxPodSandboxConfig{CgroupParent:,SecurityContext:nil,Sysctls:map[string]string{},},},RuntimeHandler:,}"
time="2021-09-22T09:25:29-04:00" level=debug msg="RunPodSandboxResponse: nil"
time="2021-09-22T09:25:29-04:00" level=fatal msg="run pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox \"e4cc6fc22dbdf8ccde0035239873cb9f31b074fca4650acc545a8af5a51d814c\": error creating endpoint hcnCreateEndpoint failed in Win32: IP address is either invalid or not part of any configured subnet(s). (0x803b001e) {\"Success\":false,\"Error\":\"IP address is either invalid or not part of any configured subnet(s). \",\"ErrorCode\":2151350302} : endpoint config &{ e4cc6fc22dbdf8ccde0035239873cb9f31b074fca4650acc545a8af5a51d814c_nat 11d59574-13be-4a14-b3e8-11cc0d5a7805 [] [{ 0}] { [] [] []} [{10.0.0.1 0.0.0.0/0 0}] 0 {2 0}}"
```
There is a [GitHub Issue](https://github.com/containerd/containerd/issues/4851) that hints to a problem with the pod network workflow on Windows
### Conclusion
There is a good possibility this issue will remain for a while. It has been around for the better part of a year. If one is running Linux containers, there is a great substitute in [minikube](https://minikube.sigs.k8s.io/docs/). It is easy to setup, well documented, maintained, and simulates a production environment. It appears Windows images will still need to run on Docker. Please leave a comment below if you are able to find a workaround.
### Relevant Links
[GitHub Issue: Windows CNI plugin has no chance to create and configure container VNIC](https://github.com/containerd/containerd/issues/4851)
[James Sturtevant's Windows Containers on Windows 10 without Docker using Containerd](https://www.jamessturtevant.com/posts/Windows-Containers-on-Windows-10-without-Docker-using-Containerd/)
[PowerShell Execution Policies](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_execution_policies?view=powershell-7.1)
[minikube](https://minikube.sigs.k8s.io/docs/)
[crictl README has pod.json samples](https://github.com/containerd/containerd/blob/main/docs/cri/crictl.md)
<a href="https://dev.to/leading-edje">
<a/>
Posted on September 28, 2021
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.
Related
