Kubernetes pod - read-once config

January 29, 2021 - 6 min read
kubernetes security
FilesDiffEdit

Table of Contents

Introduction

Sometimes you need to store sensitive information in a configuration-file in kubernetes, but you don't want the information laying around.

This post is about an idea for how to fix this problem using some tricks:

  • Using an encrypted config stored in a persistant volume
  • Making the password for the encrypted config available when starting the container, but then remove it after decryption
  • Remove the decrypted config file once it is read
  • Leaving a running app started with the decrypted config, but without any way of getting to that config if you have access inside the container.

Everything needed is in this post github-files here.

Also note that we use alpine with s6-overlay. This trick should also work with other distroes without too much changes, but some changes are needed.

Heads-up

We use the fact that you usually don't have access using SYS_PTRACE inside a container, hence, there is no way to debug the running application to get to the unencrypted config.

Prerequisites

  • Knowledge about containers, alpine and some s6-overlay
  • An application that doesnt care if it's config is deleted after it is started.

Actual content

The container

There is little you would need to do to the Containerfile. What's important is to include gnupg, s6-overlay and some scripts. Below is a full example of a containerfile that should work out of the box.

Tip

Dockerfile and Containerfile are mostly the same.. This works using docker as well, just use docker build -f Containerfile.

In Containerfile
FROM alpine

ENV S6_OVERLAY_VERSION=v2.1.0.2

RUN apk update \
    && apk add gnupg curl \
    && curl -sSL https://github.com/just-containers/s6-overlay/releases/download/${S6_OVERLAY_VERSION}/s6-overlay-amd64.tar.gz | tar xfz - -C /

COPY root /

ENTRYPOINT ["/init"]
CMD []

The example-image used in the k8s-resources.yaml below was built and uploaded to docker-hub like this:

docker build -f Containerfile -t xeor/k8s-read-once-example:latest .
Sending build context to Docker daemon  10.75kB
Step 1/6 : FROM alpine
# Lots of lines #
Successfully built 53daa1df30ae
Successfully tagged xeor/k8s-read-once-example:latest
docker push xeor/k8s-read-once-example:latest
The push refers to repository [docker.io/xeor/k8s-read-once-example]
# More lines #
1: digest: sha256:fd68012fa17907af7d22fbb693f982c0dfddeadd5ea7ddb8d0de1e550f6b4e7f size: 947

Inside the root directory, we need to put some helper scripts..

/usr/bin/secret_encrypt

This is a simple helper-script to encrypt a file /config/secrets.yaml into the file /config/secrets.yaml.enc. This will be helpfull to test and bootstrap your setup, and in case you want to change the data in the encrypted file.

In root/usr/bin/secret_encrypt
#!/bin/sh

set -eu

# Password-file containing the password used for encryption/decryption.
# This file will be deleted by the scripts unless `DONT_DELETE_PWFILE` is set.
pwfile="/etc/init-secrets/secretfile_pw"

# The unencrypted config
file="/config/secrets.yaml"

# The encrypted version of the config
enc_file="/config/secrets.yaml.enc"

gpg_params="--output ${enc_file} --symmetric --s2k-mode 3 --s2k-count 65011712 --s2k-digest-algo SHA512 --cipher-algo AES256 --no-symkey-cache --armor ${file}"

if [[ ! -e "${file}" ]]; then
  echo "${file} doesnt exists.. Nothing to encrypt"
  exit 1
fi

if [[ -e "${pwfile}" ]]; then
  cat "${pwfile}" | gpg --batch --passphrase-fd - --yes ${gpg_params}
  if [[ "${DONT_DELETE_PWFILE:-}" ]]; then
    echo "All done, kept ${pwfile}"
  else
    rm -f "${pwfile}"
    echo "All done, deleted ${pwfile}"
  fi
else
  gpg ${gpg_params}
fi

/usr/bin/secret_decrypt

Used for decryption when the container starts. It should be called at container start to make sure the decrypted version of the config is ready when the appliation reads it.

In root/usr/bin/secret_decrypt
#!/bin/sh

set -eu

pwfile="/etc/init-secrets/secretfile_pw"

file="/config/secrets.yaml"
enc_file="/config/secrets.yaml.enc"

gpg_params="--output ${file} --no-symkey-cache --decrypt ${enc_file}"

if [[ ! -e "${enc_file}" ]]; then
  echo "${enc_file} dont exists. Halting so you can inspect manually."
  echo ""
  echo "If this is the first time you start this container, and you want to initialize the encrypted config:"
  echo "  * Make sure the unencrypted config (${file}) exists."
  echo "  * Use the 'secret_encrypt' command to make ${enc_file}"
  echo "  * Kill/restart the container"
  echo -n "Sleeping"
  while true; do echo -n .; sleep 60; done
fi

if [[ -e "${pwfile}" ]]; then
  cat "${pwfile}" | gpg --batch --passphrase-fd - --yes ${gpg_params}
  if [[ "${DONT_DELETE_PWFILE:-}" ]]; then
    echo "All done, kept ${pwfile}"
  else
    rm -f "${pwfile}"
    echo "All done, deleted ${pwfile}"
  fi
else
  gpg ${gpg_params}
fi

/usr/bin/secret_delete_waiter

This file might need to be modified to your need. This version is tailored to work with the inotify utility that comes with busybox. See /etc/cont-init.d/00-secrets-handler below for how it is used. It's single goal is work with inotifyd to delete a file once it is read.

In root/usr/bin/secret_delete_waiter
#!/bin/sh

event="${1}"
file="${2}"

echo "Waiting for ${file} to be read so we can delete it"

case "$event" in
  r)
    echo "File (${file}) is read, deleting it"
    rm -f "${file}"
    exit 0
    ;;
  x)
    # Means that the file can't be watched anymore.. Of corse it can't, we just deleted it :p
    ;;
  *)
    echo "Got an unsupported event on file (${file}), got ${event}"
    exit 1
    ;;
esac

/etc/cont-init.d/00-secrets-handler

Placing this script inside cont-init.d when using s6-overlay makes it start before every service. It's goal is simple,

  1. Decrypt the secret
  2. Put a watcher in the background that will delete the config-file once the application is started.
In root/etc/cont-init.d/00-secrets-handler
#!/usr/bin/execlineb -P

if {
    /usr/bin/secret_decrypt
}

background {
    inotifyd /usr/bin/secret_delete_waiter /config/secrets.yaml:r
}

Kubernetes

Now that we have an idea how the image should be modified, lets start with kubernetes.

We will use minikube, but should work fine on a normal kubernetes cluster as well. Just keep in mind that the default /config persistant volume we create uses a volume on the node itself.

Create the resources

This file is used to create the kubernetes resources for the example. Nothing magical, but see comments inline in the file for more info.

In k8s-resources.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-secret
spec:
  containers:
  - name: app

    # The image we built
    image: xeor/k8s-read-once-example

    volumeMounts:
    # Mount shared with the initContainer which will get
    # our temporary secret from it
    - mountPath: "/etc/init-secrets"
      name: init-secrets

    # Persistant mount containing the encrypted and
    # temporary decrypted file
    - name: config
      mountPath: /config

  initContainers:
  # Container responsible for copying our secret from the
  # read-only-mounted Secret to a location that our app can
  # read, but also delete it.
  # There is nothing special here, just a cp from one volume
  # (mounted from a Secret) to a mount that is later shared
  # with our main app.
  - name: base-secret-injector
    image: alpine
    args:
    - "/bin/sh"
    - "-c"
    - "cp /etc/init-secrets-ro/* /etc/init-secrets/"
    volumeMounts:
    # This is the shared mount-point
    - mountPath: "/etc/init-secrets"
      name: init-secrets
    
    # This is the mount-point from the the Secret resource
    - mountPath: "/etc/init-secrets-ro"
      name : init-secrets-ro

  volumes:
  # Since this is just a temporary volume, we will keep it
  # in memory and start it empty.
  - name: init-secrets
    emptyDir:
      medium: Memory

  # Only mounted on the init-container
  - name: init-secrets-ro
    secret:
      secretName: 'encryption-key'

  # This is where you keep your encrypted config. The decrypted
  # config will also be placed here. In this example, we store it on
  # the host, which works great for a simple minikube example.
  - name: config
    hostPath:
      path: /tmp/read-once-config-temp
      type: DirectoryOrCreate

---

# The passphrase used for encrypting/decrypting the config.
# secretfile_pw will be the filename and it will be mounted
# on /etc/init-secrets-ro/ in the init-container
apiVersion: v1
kind: Secret
metadata:
  name: 'encryption-key'
type: Opaque
stringData:
  secretfile_pw: "the-secret-key"

To get everything running, creating the pod and the secret defined.

kubectl apply -f k8s-resources.yaml
pod/pod-with-secret created
secret/encryption-key created

Check the logs of the container after it has started up. You might want to wait a second or two, maybe even check the status while you wait with kubectl get pods --watch pod-with-secret (^c to exit the status-view)

kubectl logs pod-with-secret 
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 00-secrets-handler: executing... 
/config/secrets.yaml.enc dont exists. Halting so you can inspect manually.

If this is the first time you start this container, and you want to initialize the encrypted config:
  * Make sure the unencrypted config (/config/secrets.yaml) exists.
  * Use the 'secret_encrypt' command to make /config/secrets.yaml.enc
  * Kill/restart the container

Creating the initial secrets.yaml

When the pod is created, we will have an empty /config folder. This is where our config lives, both the encrypted and the temporary unencrypted version.

Enter the container to create the secrets.yaml and encrypt it.. Use kubectl exec -it pod-with-secret -- sh to enter the container before executing the following inside the container.

Inside the container
echo "secret: 1234" > /config/secrets.yaml
secret_encrypt  # This works since /etc/init-secrets/secretfile_pw exists
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
All done, deleted /etc/init-secrets/secretfile_pw
ls /config
secrets.yaml      secrets.yaml.enc
rm /config/secrets.yaml
cat /config/secrets.yaml.enc
-----BEGIN PGP MESSAGE-----

jA0ECQMK2zgjW8e3yeH/0koB6uIXd8Z58VaVP2sAPq9YfNrczEyMskP0sfZQRPW6
oUi5JUFFP3GQqGJvccTlRJK6LsRlID0E59fATRXIx77HiD8Ykzhp633q4w==
=bOCo
-----END PGP MESSAGE-----
exit

Trying it out

We can now recreate the resources to see how it behaves. Note that the /config volume is persistant.

Tip

Normally, you could just kill the pod and it will be restarted by kubernetes, but we are running a Kind of Pod, which is just a dumb pod. When they are restarted automatically, they are running under something else, like an ReplicaSet or Deployment.

kubectl delete -f k8s-resources.yaml
pod "pod-with-secret" deleted
secret "encryption-key" deleted
kubectl apply -f k8s-resources.yaml 
pod/pod-with-secret created
secret/encryption-key created

Check the logs to see that it now decrypted our file and deleted the temp-pw-file.

kubectl logs pod-with-secret
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 00-secrets-handler: executing... 
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: AES256.CFB encrypted data
gpg: encrypted with 1 passphrase
All done, deleted /etc/init-secrets/secretfile_pw
[cont-init.d] 00-secrets-handler: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.

Enter the new container (kubectl exec -it pod-with-secret -- sh) when it has started. The secrets.yaml file can now only be read once before it is deleted.

ls /config/
secrets.yaml      secrets.yaml.enc
cat /config/secrets.yaml
secret: 1234
cat /config/secrets.yaml
cat: can't open '/config/secrets.yaml': No such file or directory

Notice that we where only able to read the file once. This is the same as if the application reads it.

The magic that deletes it is the inotify process running in the background started by 00-secrets-handler.

Important

You should make sure that if you app dies, the pod also dies (is killed). Doing the above will remove the ability for the app to restart itself.

Back to Blog