Optimizely on Kubernetes

Introduction

This article shows you how to set up the Optimizely CMS sample site using Kubernetes. The article also covers topics such as scaling and load balancing, and some gotchas found when installing.

The article is based on Google Cloud for hosting the database
and provisioning a Kubernetes
cluster, but you can of course use Azure or an on-premise
installation if it suits you better.

Pre-requisites

This assumes you has the following:

Kubernetes Cluster (In this article i use Google Kubernetes Engine)
Azure Servicebus (Standard or Premium tier)
Azure Storage Account
SQL Server (2012 or later)
Linux or WSL2

Getting the sample site

The sample site used in this article is found here.
Start by cloning the repository:

git clone git@github.com:episerver/netcore-preview.git
cd netcore-preview

The netcore-preview repository contains 2 sample web sites: Quicksilver and Alloy. This article focus on running the Alloy web site. To speed up building the solution, remove the Quicksilver project by running the following command in the same directory as the solution:

dotnet sln remove  Quicksilver/EPiServer.Reference.Commerce.Site/EPiServer.Reference.Commerce.Site.csproj

Creating the database

To create a new Optimizely database we first need to install the new EPiserver CLI using the following command:

dotnet tool install EPiServer.Net.Cli --global --add-source https://pkgs.dev.azure.com/EpiserverEngineering/netCore/_packaging/beta-program/nuget/v3/index.json --version 1.0.0-pre-020034

When the CLI is done installing you can run the following command to create the Optimizely CMS database:

dotnet-episerver create-cms-database Alloy/AlloyMvcTemplates.csproj -S {SQL SERVER ADDRESS} -U {USER} -P {PASSWORD}

When the installer is done, the EPiServerDB connectionstring is appended to the appSettings.json
configuration file located in the root of the project.

To create an admin user run the following command:

dotnet-episerver add-admin-user Alloy/AlloyMvcTemplates.csproj -u username -p password -e user@email.com -c EPiServerDB

Configuring the website to run multiple instances

There are some things we must do in order to make the site able to run multiple instances

Configuring EventProvider
Configuring BlobStorage
Adding Dataprotection

Adding Azure EventProvider & BlobProvider

In a loadbalanced website events must be distributed to all instances, this is carried out by the EventProvider.
In this article I will use Azure Servicebus to distribute the events. Optimizely provides a implementation for replicating events, AzureEventProvider found in the EPiServer.Azure NuGet package.

However we cannot use AzureEventProvider directly when we run on top of Kubernetes since Servicebus SubscriptionNames has a limitation of 50 characters and AzureEventProvider sets SubscriptionName using the following code snippet: MachineName + Guid.NewGuid().ToString("N").

When running a Kubernetes Deployment, pod names get a hash suffix. This name is used as the machine name, so when using AzureEventProvider the SubscriptionName would be: podname + hash + 32 character guid which is to many characters due to the 50 character restriction in Azure Service Bus.

To circumvent this, we create a custom implementation which inherits AzureEventProvider and uses System.Reflection magic to set SubscriptionName. The implementation can be found here

Install the required NuGet package by running the following command:
dotnet add package EPiServer.Azure

When the NuGet package(s) has been installed you need to add the following code shown below:
Register services in Startup.cs

public void ConfigureServices(IServiceCollection services)
{
  ...
        // register blob storage
        services.AddAzureBlobProvider(o => {
            o.ConnectionString = Environment.GetEnvironmentVariable("BLOBSTORAGE_CONNECTIONSTRING");
        });

        // register event provider
        services.AddEventProvider<CustomAzureEventProvider>();
        services.Configure<CustomAzureEventProviderOptions>(o => {
            o.ConnectionString = Environment.GetEnvironmentVariable("SERVICEBUS_CONNECTIONSTRING");
            o.TopicName = "siteevents";
            o.SubscriptionName =  Guid.NewGuid().ToString("N");
        });

  ...
}

Configuring Dataprotection

In order to encrypt/decrypt and validate view-state, a shared key on each instance of our application needs to be used. This was previously done in ASP.NET Framework by setting the <machineKey> element under <system.web>. In ASP.NET Core, this has been replaced by DataProtection.

To distribute and persist our key to each instance we will use Azure Blobstorage.
Note that in production, you probably want to encrypt the keys at rest aswell. You can read more about this here

Install required NuGet by running the following command: dotnet add package Azure.Extensions.AspNetCore.DataProtection.Blobs

Add the code to Startup.cs

public void ConfigureServices(IServiceCollection services)
{
  ...
  services
    .AddDataProtection()
    .PersistKeysToAzureBlobStorage(Environment.GetEnvironmentVariable("BLOBSTORAGE_CONNECTIONSTRING"), "keys", "epidemo");

  ...
}

Configuring License

Getting a developer license for Optimizely can be done here.

Instead of bundling container specfic configuration with our application, a better way is to mount configuration either as files in an attached volume or as environment variables.
By default, Optimizely checks for the Licence.config file in the app root directory. We need to configure the application to check for the license in another directory where the license will be mounted.
To configure this add the following to Startup.cs

public void ConfigureServices(IServiceCollection services)
{
    ...

    services.Configure<Licensing.LicensingOptions>(o => {
        o.LicenseFilePath = "/etc/app/License.config";
    });

    ...
}

Dockerizing the project

The final thing we need to do in order to run the site on Kubernetes is to dockerize the application.

Create a new Dockerfile in the root directory

FROM mcr.microsoft.com/dotnet/sdk:5.0-buster-slim AS build-env
WORKDIR /app

COPY NuGet.config .
COPY dependencies.props .
COPY ./Alloy/AlloyMvcTemplates.csproj   ./Alloy/
COPY *.sln .
RUN dotnet restore

COPY . .
RUN dotnet publish -c Release -o out

FROM mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim
WORKDIR /app
COPY --from=build-env /app/out .

EXPOSE 80

ENTRYPOINT ["dotnet", "AlloyMvcTemplates.dll"]

Deploying the site on Kubernetes

Since our docker image does not contain any ConnectionStrings or Licensefile we need to inject them into the container. This is done by using Kubernetes Secrets & ConfigMap.
By separating configuration and application the configuration could be reused acros multiple applications.

Creating Connection string Secret

Create a new secret containg our connectionstrings by running the following command:

kubectl create secret generic epi-connectionstring \ 
--from-literal=eventprovider='{EVENT_PROVIDER_CONNECTION_STRING}' \
--from-literal=blobstorage='{BLOB_STORAGE_CONNECTION_STRING}' \
--from-literal=database='{DB_CONNECTION_STRING}'

Creating a License ConfigMap

Create a new configmap by running the following command:

kubectl create configmap epi-demo-license --from-file=License.config=PATH_TO_License.config

Creating Deployment Manifest

Deployments are used to run a specified number of replicas at any given time.

Create a new deployment by running the following command:

kubectl apply -f https://raw.githubusercontent.com/crikke/epi-ha-k8s-article/master/deployment.yml

Run kubectl get deployments epi-k8s to verify that the deploy is created

NAME      READY   UP-TO-DATE   AVAILABLE   AGE
epi-k8s   2/2     2            2           3h11m

Loadbalancing and making the application reachable outside the cluster

Currently, the application is only reachable within the cluster. In order to expose it to the internet, a Service is needed which routes incomming traffic to pods. Selecting which pod a service should route traffic to is done by using matching Pod labels.

Labels are key/value pairs that are attached to objects, such as pods. Labels are intended to be used to specify
identifying attributes of objects that are meaningful and relevant to users, but do not directly imply semantics to the
core system. Labels can be used to organize and to select subsets of objects.

Read more about labels at Kubernetes official documentation here

Create a new service by running the following command:

kubectl apply -f https://github.com/crikke/epi-ha-k8s-article/blob/master/service.yml

Verify that the service is created by running kubectl get service epi-k8s-demo

You should get similar output as the example shown below:

NAME           TYPE           CLUSTER-IP   EXTERNAL-IP    PORT(S)                      AGE
epi-k8s-demo   LoadBalancer   10.44.7.49   34.88.54.254   80:30080/TCP,443:30443/TCP   155m

And thats it, you now have Optimizely CMS running in a Kubernetes cluster!

Your application should now be reachable by the EXTERNAL-IP field from your service. Note that if it´s <PENDING>, wait some minutes and check again.

If you visit the site through EXTERNAL-IP, you could get an 404 error. This is because you first need to add a new site inside the Optimizely Admin UI. Visit EXTERNAL-IP/episerver/cms and go to Admin > Configuration > Manage Sites > Add Site

Conclusion

The Optimizely Devteam has done massive work of migrating Optimizely from .NET Framework and all underlying dependencies. Not only does this open many possibilities on hosting your application, but brings performance boosts aswell.

I hope this post helped you get some insight how to run Optimizely CMS on Kubernetes.
Happy coding!

Blog