Rethinking our ASP.NET Docker CI

At Wehkamp we've been using ASP.NET for a long time. Containers brought .NET with us to our microservices architecture. We used a shell script in a special build container to build our service. Today we'll discuss how we turned that shell script into a multi stage build Dockerfile.

Intro
Goals
Project structure
Step 1: Configuration with global arguments
Step 2: Build stage
Step 3: runtime stage
All together
Reading list
Changelog
Comments

Goals

With the new setup we have some goals in mind:

Reproducibility — it should be on my machine as it is in the pipeline. It would be great if I can just do a docker build . and everything should start building as it does in Jenkins.
Cacheability — Docker uses layers, which greatly speeds up the system. When we do a restore as a set of layers, we don't need to generate network traffic repeatedly.
Upgradability — the current solution will focus on .NET 7.0, but we need things to be easily upgradable to a newer (or older) version of .NET.
Readability — the Dockerfile should be treaded as a DevOps manifest, showing how to build and run our application. Let's make sure people understand what's going on. But also the output that Docker shows in the terminal should be easy to read and understand.

If at some point we can add security to the Dockerfile, that would also be great.

Project structure

Most of what we do is based on convention. Our project structure reflects those conventions:

.
├── src/
│   └── Blaze.Prime.Service.Api/
│       ├── Blaze.Prime.Service.Api.csproj
│       └── ...
├── test/
│   └── Blaze.Prime.Service.Api.UnitTests/
│       ├── Blaze.Prime.Service.Api.UnitTests.csproj
│       └── ...
├── .dockerignore
├── .gitignore
├── Blaze.Prime.sln
├── Dockerfile
└── nuget.config

It is important that the name of the C# project directory matches the name of the C# project file.

Step 1: Configuration with global arguments

Our Dockerfile may grow. Let's make it easy to upgrade and changing global settings by using ARG instructions. We will use these arguments in the build and runtime images.

########################
# Configuration settings
########################

ARG API_NAME="Blaze.Prime.Service.Api"
ARG \
   API_PROJECT="./src/$API_NAME/$API_NAME.csproj" \
   API_DLL="$API_NAME.dll" \
   PORT=5000 \
   ASPNET_VERSION="7.0" \
   # options: q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic]
   # minimal keeps your pipeline readable while inforing you what's going on
   VERBOSITY="minimal" \
   APP_DIR="/app" \
   TAG="" \
   EXECUTE_TESTS="true"

Let's construct the project and DLL names by convention. To make debugging easier, we set the VERBOSITY argument centrally. We find that the minimal settings is a great default, as it just gives enough output to understand what's going on.

Our CI fills the TAG with the tag of the Docker image that is being built.

Step 2: Build stage

Now that we have the arguments in place, let's start with the beginning of our build image. We use the standard SDK provided by Microsoft. We don't feel the need to share telemetry and we would like to have our times in local Amsterdam time.

################################################
# Build image, uses ASPNET_VERSION, API_PROJECT, 
# VERBOSITY, TAG, APP_DIR, EXECUTE_TESTS
################################################

FROM mcr.microsoft.com/dotnet/sdk:$ASPNET_VERSION AS build

ARG API_PROJECT VERBOSITY APP_DIR
ENV \
   TZ=Europe/Amsterdam \
   DOTNET_CLI_TELEMETRY_OPTOUT=1 \
   DOTNET_NOLOGO=0

WORKDIR /build

Notice how we're not defining TAG and EXECUTE_TESTS here. If we would, we would lose our caching ability when a new tag is used or when testing is switched off.

2.1 Restore in cacheable layers

In our old script we just executed a dotnet restore as part of the build script, which is fine, but nothing is cached by Docker. Our build script will take longer than necessary. Andrew Lock did an excellent job of writing a solution that makes the restore cacheable, so let's use his solution:

# Let's restore the solution, nuget and project files and do a restore
# in cachable layers. This will speed up the build process greatly

# copy global files to restore
COPY *.sln *.*config ./

# copy src files to restore
COPY src/*/*.csproj .
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done

# copy test files to restore
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done; \
   echo "" \
   && echo "-------------" \
   && echo "1/4 RESTORING" \
   && echo "-------------" \
   && echo "" \
   && dotnet restore --verbosity "$VERBOSITY" || exit 1

This method restores the project structure and makes the dotnet restore cacheable.

Notice how we are using echo to make our output more readable?

Output of the Dockerfile.

2.2 The build script

Now what should we copy? The base files like the solution and the NuGet config are already in there. Let's copy src and test only. This will help us cache builds that are triggered just because some documentation was changed (outside of the directory).

# copy dirs that are only needed for building and testing
COPY src ./src
COPY test ./test

# Note on build: don't use --no-restore, sometimes certain packages cannot be
# restored by the dotnet restore. The build will add them, as it has more context (!?)
# example: Package System.Text.Json, version 6.0.0 was not found

RUN echo "" \
   && echo "------------" \
   && echo "2/4 BUILDING" \
   && echo "------------" \
   && echo "" \
   && dotnet build --configuration Release --verbosity "$VERBOSITY" -nowarn:NETSDK1004 || exit 1

2.3 Optional testing and publishing

At the top of our Dockerfile, we've defined the ARG named EXECUTE_TESTS. Let's use it to test if we should do any testing. You might want to be able to skip testing to run faster locally.

# defining the argument here caches the previous layers when the value switches
ARG EXECUTE_TESTS
RUN echo "" \
   && echo "-----------" \
   && echo "3/4 TESTING" \
   && echo "-----------" \
   && echo ""; \
   if [ "$EXECUTE_TESTS" = "true" ]; then \
      dotnet test --configuration Release --logger "console;verbosity=$VERBOSITY" --no-build || exit 1; \
   else \
      echo "Skipping unit tests"; \
   fi; \
   echo "" \
   && echo "--------------" \
   && echo "4/4 PUBLISHING" \
   && echo "--------------" \
   && echo "" \
   && dotnet publish "$API_PROJECT" --configuration Release --output "$APP_DIR" --no-restore -nowarn:NETSDK1004 || exit 1

Step 3: runtime stage

The hard part is over: we have our application built. But there are still some choices to make!

3.1 Which runtime base image should we use?

So what runtime image are we going to use? We've built the entire project and checked the size of each resulting image:

Base image	Total output size
mcr.microsoft.com/dotnet/aspnet:7.0	218MB
mcr.microsoft.com/dotnet/nightly/aspnet:7.0-jammy-chiseled	114MB
mcr.microsoft.com/dotnet/aspnet:7.0-alpine	112MB

Looks like the chiseled project is very promising, as it promises a secure and minimal Debian image, but they have not released it for .NET 7. Most of our projects will run fine on Alpine.

3.2 Go self contained or not?

If you want to go even smaller, you can go to a self contained (and trimmed) application. It is a great way to limit the attack surface even further and a will result in an even smaller image (which is also great). But there are some caveats: your application may take longer to boot and you can't cache your ASP.NET dependencies, so there might be some more network traffic involved. For now we just stick with the old adage as we try to read more on what the best way forward is.

3.3 The final runtime image

We've decided on Alpine for now, so our runtime image looks like this:

#################################################
# Runtime image, uses PORT, TAG, API_DLL, APP_DIR
#################################################

FROM mcr.microsoft.com/dotnet/aspnet:$ASPNET_VERSION-alpine as runtime

RUN apk add --no-cache icu-libs krb5-libs libgcc libintl libssl1.1 libstdc++ zlib tzdata

ARG PORT API_DLL APP_DIR

# create a new user and change directory ownership
RUN adduser --disabled-password \
   --home "$APP_DIR" \
   --gecos '' dotnetuser && chown -R dotnetuser "$APP_DIR"

# impersonate into the new user
USER dotnetuser

ENV \
   ASPNETCORE_URLS=http://*:$PORT \
   ASPNETCORE_ENVIRONMENT=Production \
   TZ=Europe/Amsterdam \
   DOTNET_CLI_TELEMETRY_OPTOUT=1 \
   DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false

WORKDIR $APP_DIR
COPY --from=build $APP_DIR .

EXPOSE $PORT

ENV PROGRAM="$API_DLL"
ENTRYPOINT dotnet "$PROGRAM"

ARG TAG

For security reasons, we create a new user, so we don't run as root.

All together

When we put it all together, we get the following Dockerfile:

########################
# Configuration settings
########################

ARG API_NAME="Blaze.Prime.Service.Api"
ARG \
   API_PROJECT="./src/$API_NAME/$API_NAME.csproj" \
   API_DLL="$API_NAME.dll" \
   PORT=5000 \
   ASPNET_VERSION="7.0" \
   # options: q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic]
   # minimal keeps your pipeline readable while inforing you what's going on
   VERBOSITY="minimal" \
   APP_DIR="/app" \
   TAG="" \
   EXECUTE_TESTS="true"

################################################
# Build image, uses ASPNET_VERSION, API_PROJECT, 
# VERBOSITY, TAG, APP_DIR, EXECUTE_TESTS
################################################

FROM mcr.microsoft.com/dotnet/sdk:$ASPNET_VERSION AS build

ARG API_PROJECT VERBOSITY APP_DIR
ENV \
   TZ=Europe/Amsterdam \
   DOTNET_CLI_TELEMETRY_OPTOUT=1 \
   DOTNET_NOLOGO=0

WORKDIR /build

# Let's restore the solution, nuget and project files and do a restore
# in cachable layers. This will speed up the build process greatly

# copy global files to restore
COPY *.sln *.*config ./

# copy src files to restore
COPY src/*/*.csproj .
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done

# copy test files to restore
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done; \
   echo "" \
   && echo "-------------" \
   && echo "1/4 RESTORING" \
   && echo "-------------" \
   && echo "" \
   && dotnet restore --verbosity "$VERBOSITY" || exit 1

# copy dirs that are only needed for building and testing
COPY src ./src
COPY test ./test

# Note on build: don't use --no-restore, sometimes certain packages cannot be
# restored by the dotnet restore. The build will add them, as it has more context (!?)
# example: Package System.Text.Json, version 6.0.0 was not found

RUN echo "" \
   && echo "------------" \
   && echo "2/4 BUILDING" \
   && echo "------------" \
   && echo "" \
   && dotnet build --configuration Release --verbosity "$VERBOSITY" -nowarn:NETSDK1004 || exit 1

# defining the argument here caches the previous layers when the value switches
ARG EXECUTE_TESTS
RUN echo "" \
   && echo "-----------" \
   && echo "3/4 TESTING" \
   && echo "-----------" \
   && echo ""; \
   if [ "$EXECUTE_TESTS" = "true" ]; then \
      dotnet test --configuration Release --logger "console;verbosity=$VERBOSITY" --no-build || exit 1; \
   else \
      echo "Skipping unit tests"; \
   fi; \
   echo "" \
   && echo "--------------" \
   && echo "4/4 PUBLISHING" \
   && echo "--------------" \
   && echo "" \
   && dotnet publish "$API_PROJECT" --configuration Release --output "$APP_DIR" --no-restore -nowarn:NETSDK1004 || exit 1


#################################################
# Runtime image, uses PORT, TAG, API_DLL, APP_DIR
#################################################

FROM mcr.microsoft.com/dotnet/aspnet:$ASPNET_VERSION-alpine as runtime

RUN apk add --no-cache icu-libs krb5-libs libgcc libintl libssl1.1 libstdc++ zlib tzdata

ARG PORT API_DLL APP_DIR

# create a new user and change directory ownership
RUN adduser --disabled-password \
   --home "$APP_DIR" \
   --gecos '' dotnetuser && chown -R dotnetuser "$APP_DIR"

# impersonate into the new user
USER dotnetuser

ENV \
   ASPNETCORE_URLS=http://*:$PORT \
   ASPNETCORE_ENVIRONMENT=Production \
   TZ=Europe/Amsterdam \
   DOTNET_CLI_TELEMETRY_OPTOUT=1 \
   DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false

WORKDIR $APP_DIR
COPY --from=build $APP_DIR .

EXPOSE $PORT

ENV PROGRAM="$API_DLL"
ENTRYPOINT dotnet "$PROGRAM"

ARG TAG

Pretty cool, right?

Reading list

While studying the subject we found some interesting blogs to read:

On security: How to build smaller and secure Docker Images for .NET
On dotnet restore & Docker optimizations: Optimising ASP.NET Core apps in Docker - avoiding manually copying csproj files (Part 2)
On the new Chiselled project: Chiselled Ubuntu: the perfect present for your containerised and cloud applications
On Windows you might get the error "ERROR: failed to solve: error from sender: readdir: open src\*: The filename, directory name, or volume label syntax is incorrect." This is due to a bug in the Docker Moby BuildKit, which cannot handle the wildcards. On PowerShell you can execute $env:DOCKER_BUILDKIT=0 to disable BuildKit.

Changelog

2023-05-26: removed the section on compiling git info into the binary.
2023-05-26: installed the tzdata package to fix timezone support.
2023-05-26: notes on how to fix the Windows Docker error: "error from sender: readdir".
2023-05-26: nuget.config is now optional.
2023-05-26: make sure docker build exits on error.

Blog