Twemproxy is an open source proxy server created by Twitter and is one of the methods recommended by Google for managing a cluster of Memorystore instances in GCP (Google Cloud Platform). Memorystore is the managed service provided by GCP for application caching with either Redis or Memcached. Twemproxy provides a way to evenly distribute cached data between multiple Redis instances to improve the performance, reliability, and resilience of a distributed system.

This blog post is the start of a series that will provide you with the foundational information you need to go from creating a Docker image of a Twemproxy server to deploying a managed instance group of containers created from that image to serve as a proxy for several instances of Memorystore in GCP.

The first step is to download a copy of the distribution tarball using the link provided by Twitter in the Twemproxy Github repo. The link can be found under the "Build" section. This example uses version 0.4.1 of Twemproxy, (also referred to as nutcracker).

Create a directory for your Twemproxy server project and place the nutcracker-0.4.1.tar.gz file in it along with an empty Dockerfile and an empty nutcracker.yml file. The YAML file is where you will write your configuration for the server.

Next, copy and paste the following code in your Dockerfile.

FROM debian:buster as builder
​
COPY nutcracker-0.4.1.tar.gz /
​
RUN apt-get update && apt-get -qy install \
    automake \
    libtool \
    make \
    && rm -rf /var/lib/apt/lists/*

RUN tar xzf /nutcracker-0.4.1.tar.gz
RUN cd nutcracker-0.4.1 && CFLAGS="-ggdb3 -O0" ./configure --enable-debug=full && make && make install
​
​
# Main
​
FROM debian:buster
​
COPY --from=builder /usr/local/sbin/nutcracker /usr/local/sbin/nutcracker
COPY nutcracker.yml /etc/nutcracker.yml
​
EXPOSE 6380
CMD ["nutcracker", "-c", "/etc/nutcracker.yml"]

There are two stages to this Dockerfile. The first stage installs the tools required to build Twemproxy, unzips the nutcracker-0.4.1.tar.gz file, and then uses the make tool to build it.

The second stage copies over the nutcracker directory that was created by the build process as well as the configuration file, nutcracker.yml. Lastly, this stage defines the command that will start the server once the container is running. nutcracker is the name of the executable and the -c flag followed by /etc/nutcracker.yml directs the application to your customized configuration file.

There are a lot of options for customizing the Twemproxy server based on your requirements, most of which can be defined in the configuration file. Other customization options, like changing the size of the mbufs used to ferry requests and responses, can be changed through optional parameters passed to the nutcracker command. Changes to logging must be done at the compilation step as illustrated in the Dockerfile with the --enable-debug=full parameter passed to the ./configure command.

You can find a complete list of configuration options in the Twemproxy README file.

As a starting point, you can use the configuration below in your nutcracker.yml file and adjust as needed. You will need to make sure that the port in the listen field matches the exposed port in your Dockerfile and you will need to change the IP addresses and port numbers listed under the servers property to match your Memorystore instances once they have been created.

pool:
    listen: 0.0.0.0:6380
    hash: fnv1a_64
    distribution: ketama
    redis: true
    auto_eject_hosts: true
    timeout: 2000
    server_retry_timeout: 10000
    server_failure_limit: 3
    server_connections: 40
    preconnect: true
    servers:
        - 127.0.0.1:6379:1
        - 127.0.0.2:6379:1
        - 127.0.0.3:6379:1

To make sure the Dockerfile builds and that the image runs locally in a container, run the following commands from within the same directory as your Dockerfile.

docker build . -t twemproxy-test
docker run --name twemproxy -it --rm twemproxy-test

You will see the following message once the server has successfully started running along with a couple of error messages due to the fact that the proxy is unable to connect with Redis instances that do not exist yet.

nutcracker-0.4.1 build for Linux 4.19.121-linuxkit x86_64 started on pid 1
run, rabbit run / dig that hole, forget the sun / and when at last the work is done / don't sit down / it's time to dig another one

In the next blog post we will use Terraform to define all of the resources required to deploy a managed instance group of Twemproxy servers and a cluster of three Memorystore instances. Once the cloud resources have been provisioned you will be able to replace the arbitrary server IP addresses and ports in the example configuration file above with actual address information for your deployed Redis instances.