Using OpenPolicyAgent to Inject Proxy Configuration into Kubernetes Workloads

It's not unusual to find enterprise services behind a corporate proxy, and in order to get out, must have this proxy access configured to access out through it. The same applies to services running o Kubernetes.

One way to do this is manually, by creating a Secret resource containing a proxy string like the following:

kubectl create secret generic corporate-proxy --from-literal=authentication-string="http://{USERNAME}:{PASSWORD}@{PROXY_ADDRESS}:{PROXY_PORT}" --from-literal=no-proxy-string="localhost,127.0.0.1,0.0.0.0,10.0.0.0/8,cattle-system.svc,10.42.0.0/24,.svc,.cluster.local,example.com"

and, mount it to the environment of a Kubernetes application resource:

apiVersion: v1
kind: Pod
metadata:
  name: appWithProxy
  namespace: default
spec:
  containers:
  - name: appWithProxy
    image: yourApplication
    env:
    - name: HTTP_PROXY
      valueFrom:
        secretKeyRef:
          name: corporate-proxy
          key: authentication-string
    - name: NO_PROXY
      valueFrom:
        secretKeyRef:
          name: corporate-proxy
          key: no-proxy-string

and when you exec into that container, you can use that proxy to access public resources:

curl --proxy $HTTP_PROXY http://ipinfo.io/json

However, this is a tedious, manual process that you would, then, need to repeat for every resource you'd like to have proxy access.

Among the many things OpenPolicyAgent Gatekeeper does, is allow you to mutate Kubernetes API requests based on a variety of scoping rules, and inject things like environment variables into resources that meet said scope.

This is done using a Mutating Admission Webhook. Using Mutating Admission Webhook to inject environment variables like these proxy settings, for example, to modify requests to create a resource like a Pod or Deployment, etc. to inject the proxy configuration data. There are other types of Webhooks, such as (in the linked example) Validating Webhooks, which do things like confirm a workload has a label, or an appropriate name, etc. to dictate some other action by the Admission Webhook.

OPA Gatekeeper, in this scenario, will do much of this for you, and instead, mutation and scoping is dictated by policies. In our case, we want to create a namespace that will contain resources that we wish to mutate, upon request, to include mounting the above corporate-proxy resource Secret as an environment variable.

OPA also provides facilities for testing these policies.

After installing OPA Gatekeeper:

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.7/deploy/gatekeeper.yaml

Or if you're a Rancher user, Gatekeeper can be installed and enabled from the UI.

We can then define a ModifySet resource:

apiVersion: mutations.gatekeeper.sh/v1beta1
kind: ModifySet
metadata:
  name: demo-annotation-owner
  namespace: proxy-test
spec:
    applyTo:
      - groups: [""]
        versions: ["v1"]
        kinds: ["Pod", "Deployment"]
    match:
      scope: Namespaced
      kinds:
        - apiGroups: ["*"]
          kinds: ["Pod"]
    location: "spec.containers[name:*].env"
    parameters:
      values:
        fromList:
          - name: HTTP_PROXY
            valueFrom:
              secretKeyRef:
                name: corporate-proxy
                key: authentication-string
          - name: NO_PROXY
            valueFrom:
              secretKeyRef:
                name: corporate-proxy
                key: authentication-string

which is deployed to the proxy-test namespace, and a couple of lines down, the scope is set to Namespaced, meaning that it will only apply the following to matching resources in that namespace.

If the API server passes a request that is that namespace, and matches any apiGroup ("*"), and is of kind: Pod, it will read a spec like the following:

apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
  namespace: proxy-test
spec:
  volumes:
  - name: shared-data
    emptyDir: {}
  containers:
  - name: nginx
    image: nginx
    env:
    - name: SOME_EXISTING_VAR
      value: "Present"
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
  dnsPolicy: Default

where it will look in spec.containers, and you'll see since this request does not contain an env block, it will inject our Secret from above (secret/corporate-proxy) into this spec appending env represented above in Yaml, but can also be inserted as JSON:

[{"name": "HTTP_PROXY", "valueFrom": {"secretKeyRef": {"name": "corporate-proxy", "key": "authentication-string"}}}]

and create the Pod.

You can test this out by creating two Pods using the above spec, one in the proxy-test namespace and one in another:

cat << EOF >> proxy.yaml
apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
  namespace: proxy-test
spec:
  volumes:
  - name: shared-data
    emptyDir: {}
  containers:
  - name: nginx
    image: nginx
    env:
    - name: SOME_EXISTING_VAR
      value: "Present"
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
  dnsPolicy: Default
EOF

kubectl apply -f proxy.yaml

and one in another:

cat << EOF >> no-proxy.yaml
apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
  namespace: default
spec:
  volumes:
  - name: shared-data
    emptyDir: {}
  containers:
  - name: nginx
    image: nginx
    env:
    - name: SOME_EXISTING_VAR
      value: "Present"
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
  dnsPolicy: Default
EOF

kubectl apply -f no-proxy.yaml

and when you run kubectl describe on the namespaced Pod, you'll see the following:

root@ubuntu-proxy-k3s:~# kubectl describe pod/shell-demo -n proxy-test
Name:         shell-demo
Namespace:    proxy-test
...
Containers:
  nginx:
    Image:      nginx
...
    Environment:
      SOME_EXISTING_VAR:     "Present"
      HTTP_PROXY:  <set to the key 'authentication-string' in secret 'corporate-proxy'>  Optional: false
      NO_PROXY:    <set to the key 'authentication-string' in secret 'corporate-proxy'>  Optional: false

where the one in the default namespace will not have an Envrionment set.

These ModifySet resources can be written to mutate based on a number of other scopes as well, and any part of a manifest that you specify in the location key, and then assign with a value under parameters as we did above. Other examples of using OPA Gatekeeper resources to mutate resources can be found here.

Blog

Using OpenPolicyAgent to Inject Proxy Configuration into Kubernetes Workloads

Joseph D. Marhee

Join Our Newsletter. No Spam, Only the good stuff.

Related