Encrypting User Data with Kairos

Kairos offers the ability to encrypt user data partitions with LUKS. User-data partitions are dedicated to persist data for a running system, stored separately from the OS images. This encryption mechanism can also be used to encrypt additional partitions created during the installation process.

Kairos supports the following encryption scenarios:

1. Offline mode - Encryption key for partitions is stored on the machine inside the TPM chip.
2. Online mode (Automated) - Keypair used to encrypt the partition passphrase is stored on the TPM chip, and an external server is used to store the encrypted passphrases.
3. Online mode (Manually configured) - Plaintext passphrase is stored in the KMS server and returned to the node after TPM challenging.

Kairos uses the TPM chip to encrypt partition passphrases, and for offline encryption, it stores the passphrase in the non-volatile registries of the chip. To enable encryption, you will need to specify the labels of the partitions you want to encrypt, a minimum configuration for offline encryption can be seen below:

#cloud-config

install:
  # Label of partitions to encrypt
  # COS_PERSISTENT is the OS partition
  # dedicated to user-persistent data.
  encrypted_partitions:
  - COS_PERSISTENT

Please note that for online mode, you will also need to specify the key management server address that will be used to store the keys, a complete configuration reference is the following:

#cloud-config

# Install block
install:
  # Label of partitions to encrypt
  # COS_PERSISTENT is the OS partition
  # dedicated to user-persistent data.
  encrypted_partitions:
  - COS_PERSISTENT

# Kcrypt configuration block
kcrypt:
  challenger:
    # External KMS Server address. This must be reachable by the node
    challenger_server: "http://192.168.68.109:30000"
    # (optional) Instructs the client to lookup the KMS using mdns
    mdns: false
    # (optional) PEM certificate the node uses to verify the KMS.
    # Only for an https challenger_server. See "Verifying the KMS" below.
    certificate: ""
  # (optional) Custom Non-Volatile index to use to store encoded blobs
  nv_index: ""
  # (optional) Custom Index for the RSA Key pair
  c_index: ""
  # (optional) Custom TPM device
  tpm_device: ""

Option	Description
install.encrypted_partitions	Label of partitions to encrypt
kcrypt.challenger.challenger_server	External KMS Server address
kcrypt.challenger.mdns	Discover KMS using mdns. Defaults to false
kcrypt.challenger.certificate	PEM certificate used to verify the KMS. See Verifying the KMS
kcrypt.nv_index	Custom Non-Volatile index to use to store encoded blobs
kcrypt.c_index	Custom Index for the RSA Key pair
kcrypt.tpm_device	Custom TPM device

Requirements

The host machine must have a TPM chip version 2.0 or higher to use encryption with Kairos. A list of TPM chips/HW can be found in the list of certified products, however, any modern machine has a TPM 2.0 chip.

Components

The Kairos encryption design involves three components to manage partitions encryption and decryption lifecycle:

• kcrypt runs on the machine and attempts to unlock partitions by using plugins to delegate encryption/decryption business logic.
• kcrypt-discovery-challenger runs on the machine, it is called by kcrypt and uses the TPM chip to retrieve the passphrase as described below.
• kcrypt-challenger is the KMS (Key Management Server) component, deployed in Kubernetes, which manages secrets and partitions of the nodes.

Offline mode

This scenario covers encryption of data at rest without any third party or KMS server. The keys used to encrypt the partitions are stored in the TPM chip.

Scenario: Offline encryption

A high level overview of the interaction between the components can be observed here:

A complete cloud config example for this scenario can be:

#cloud-config

install:
  encrypted_partitions:
  - COS_PERSISTENT

hostname: metal-{{ trunc 4 .MachineID }}
users:
- name: kairos
  # Change to your pass here
  passwd: kairos
  groups:
  - admin
  ssh_authorized_keys:
  # Replace with your github user and un-comment the line below:
  # - github:mudler

Note, we define a list of partition labels that we want to encrypt. In the example above we set COS_PERSISTENT to be encrypted, which in turns will encrypt all the user-data of the machine (this includes, for instance, Kubernetes pulled images, or any runtime persisting data on the machine).

Online mode

Online mode involves an external service (the Key Management Server, KMS) to boot the machines. The KMS role is to enable machine to boot by providing the encrypted secrets, or passphrases to unlock the encrypted drive. Authentication with the KMS is done via TPM challenging.

In this scenario, we need to first deploy the KMS server to an existing Kubernetes cluster, and associate the TPM hash of the nodes that we want to manage. During deployment, we specify the KMS server inside the cloud-config of the nodes to be provisioned.

Requirements

• A Kubernetes cluster
• Kcrypt-challenger reachable by the nodes attempting to boot

Install the KMS (kcrypt-challenger)

To install the KMS (kcrypt-challenger), you will first need to make sure that certificate manager is installed. You can do this by running the following command:

kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml
kubectl wait --for=condition=Available deployment --timeout=2m -n cert-manager --all

To install kcrypt-challenger on a Kubernetes cluster with helm, you can use the commands below:

# Install the helm repository
helm repo add kairos https://kairos-io.github.io/helm-charts
helm repo update

# Install the Kairos CRDs
helm install kairos-crd kairos/kairos-crds

# Deploy the KMS challenger
helm install kairos-challenger kairos/kairos-challenger --set service.challenger.type="NodePort"

# we can also set up a specific port and a version:
# helm install kairos-challenger kairos/kairos-challenger --set image.tag="v0.2.2" --set service.challenger.type="NodePort" --set service.challenger.nodePort=30000

A service must be used to expose the challenger. If using the node port, we can retrieve the address with:

export EXTERNAL_IP=$(kubectl get nodes -o jsonpath='{.items[].status.addresses[?(@.type == "ExternalIP")].address}')
export PORT=$(kubectl get svc kairos-challenger-escrow-service -o json | jq '.spec.ports[0].nodePort')

Register a node

In order to register a node on the KMS, the TPM hash of the node needs to be retrieved first. The TPM hash is a SHA256 sum of the EK public key, which is part of the EK key pair that is present on the TPM from the manufacturer. The EK private key cannot be changed and is unique to the TPM and therefore, the EK public key can be used to uniquely challenge the device. During the challenge, the node will send its public key to the challenger, which will generate a SHA256 checksum and compare it to a local database of checksums. In order to build this database, the checksum needs to be registered with the challenger.

You can get a node TPM hash by running /system/discovery/kcrypt-discovery-challenger as root from the LiveCD:

kairos@localhost:~> ID=$(sudo /system/discovery/kcrypt-discovery-challenger)
kairos@localhost:~> echo $ID
7441c78f1976fb23e6a5c68f0be35be8375b135dcb36fb03cecc60f39c7660bd

This is the TPM hash you should use in the definition of the SealedVolume in the examples below.

Scenario: Automatically generated keys

The TPM chip generates unique RSA keys for each machine during installation, which are used to encrypt a generated secret. These keys can only be accessed by the TPM and not by the KMS, thus ensuring that both the KMS and the TPM chip are required to boot the machine. As a result, even if the machine or its disks are stolen, the drive remains sealed and encrypted. Deployment using this method, will store the encrypted key used to boot into the KMS, and the keypair used to encrypt it in the TPM chip of the machine during installation. This means that, only the TPM chip can decode the passphrase, and the passphrase is stored in the KMS such as it can't be decrypted by it. As such, nodes can boot only with the KMS, and the disk can be decrypted only by the node.

To register a node to kubernetes, use the TPM hash retrieved before (see section "Register a node") and replace it in this example command:

cat <<EOF | kubectl apply -f -
apiVersion: keyserver.kairos.io/v1alpha1
kind: SealedVolume
metadata:
    name: test2
    namespace: default
spec:
  TPMHash: "7441c78f1976fb23e6a5c68f0be35be8375b135dcb36fb03cecc60f39c7660bd"
  partitions:
    - label: COS_PERSISTENT
  quarantined: false
EOF

This command will register the node on the KMS.

A node can use the following during deployment, specifying the address of the challenger server:

Trusted Boot uses a different cmdline mechanism

install.grub_options is read on GRUB installations only. On Trusted Boot the cmdline is part of the signed EFI artifact, so rd.neednet=1 set this way is discarded, the initramfs comes up without networking, and the node cannot reach the challenger server. Build the artifact with the parameter instead, as described in Additional efi entries.

#cloud-config

install:
  encrypted_partitions:
  - COS_PERSISTENT
  grub_options:
    extra_cmdline: "rd.neednet=1"

kcrypt:
  challenger:
    challenger_server: "http://192.168.68.109:30000"
  nv_index: ""
  c_index: ""
  tpm_device: ""

hostname: metal-{{ trunc 4 .MachineID }}
users:
- name: kairos
  # Change to your pass here
  passwd: kairos
  groups:
  - admin
  ssh_authorized_keys:
  # Replace with your github user and un-comment the line below:
  # - github:mudler

Scenario: Static keys

In this scenario the Kubernetes administrator knows the passphrase of the nodes, and sets explicitly during configuration the passphrase for each partitions of the nodes. This scenario is suitable for cases when the passphrase needs to be carried over, and not to be tied specifically to the TPM chip. The TPM chip is still used for authentication a machine. The discovery-challenger needs still to know the TPM hash of each of the nodes before installation. To register a node to kubernetes, replace the TPMHash in the following example with the TPM hash retrieved before, and specify a passphrase with a secret reference for the partition:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: default
type: Opaque
stringData:
  pass: "awesome-plaintext-passphrase"
---
apiVersion: keyserver.kairos.io/v1alpha1
kind: SealedVolume
metadata:
    name: test2
    namespace: default
spec:
  TPMHash: "7441c78f1976fb23e6a5c68f0be35be8375b135dcb36fb03cecc60f39c7660bd"
  partitions:
    - label: COS_PERSISTENT
      secret:
       name: mysecret
       path: pass
  quarantined: false
EOF

The node doesn't need any specific configuration beside the kcrypt challenger, so for instance:

Trusted Boot uses a different cmdline mechanism

As in the scenario above, install.grub_options does not apply on Trusted Boot. Put rd.neednet=1 in the signed artifact at build time, see Additional efi entries.

#cloud-config

install:
  encrypted_partitions:
  - COS_PERSISTENT
  grub_options:
    extra_cmdline: "rd.neednet=1"

kcrypt:
  challenger:
    challenger_server: "http://192.168.68.109:30000"
  nv_index: ""
  c_index: ""
  tpm_device: ""

hostname: metal-{{ trunc 4 .MachineID }}
users:
- name: kairos
  # Change to your pass here
  passwd: kairos
  groups:
  - admin
  ssh_authorized_keys:
  # Replace with your github user and un-comment the line below:
  # - github:mudler

Encrypting on first boot (templates and cloud images)

The scenarios above encrypt partitions during installation, which requires a TPM to be present at install time. A node can instead encrypt its partitions on the first boot: the image carries only the encryption policy, and the LUKS material is created on each machine against that machine's own TPM.

This is aimed at golden images and templates. A typical flow on a virtualization platform looks like this:

1. Install Kairos once in a VM, without a TPM attached and without encrypted_partitions in the install config. Power the VM off and mark it as a template.
2. Add the encryption policy below to the template's OEM partition, for example as /oem/91_encrypt_on_boot.yaml. The policy is plain configuration and contains no key material, so the template stays safe to copy, snapshot and export.
3. For each new node, clone the template, attach a fresh virtual TPM and boot.

On the first boot of each clone, Kairos notices that partitions configured for encryption are still plaintext, encrypts them against the clone's own TPM before they are mounted, unlocks them and continues booting. Every later boot detects the partitions are already encrypted and does nothing.

#cloud-config

install:
  # Same key the install time scenarios use.
  encrypted_partitions:
  - COS_PERSISTENT

kcrypt:
  # Explicit opt-in for encrypting on boot. Defaults to false.
  encrypt_on_boot: true

Without a KMS configured the passphrase is stored in the local TPM, as in offline mode. The online scenarios apply too: add the usual kcrypt.challenger block to the policy and the passphrase is managed by the KMS instead.

warning

Encrypting a partition destroys the data on it. The boot time step only runs when kcrypt.encrypt_on_boot is explicitly set to true, but enabling it on an existing node whose listed partitions hold plaintext data will destroy that data on the next boot. In the template flow this is what you want, since a fresh template holds nothing of value in its persistent partition.

There is no plaintext fallback. If the configuration says a partition must be encrypted and the machine cannot do it, for example because no TPM is attached or the KMS cannot be reached, the boot halts with a message describing the problem, and the partition is left untouched. The same applies when the configuration itself cannot be read on boot: since the node might have opted in, the boot stops rather than continue with a policy it cannot see.

Notes:

• Partitions the running boot depends on cannot be encrypted this way: the OEM partition (the policy itself is read from it), the state and recovery partitions (they hold the system that is booting) and the EFI partition. Listing one of them halts the boot with a message saying so. Encrypt OEM at install time if you need it encrypted.
• Trusted Boot (UKI) images always encrypt during installation, so this flow does not apply to them.

Verifying the KMS

The examples above reach the KMS over plain http. On that path the node sends its TPM attestation, and receives the passphrase that unlocks its disk, to whatever answers on that address. Nothing proves the answer came from your KMS.

Set kcrypt.challenger.certificate to close that. The node then verifies the KMS against the certificate you pin, and refuses to talk to anything else:

#cloud-config

kcrypt:
  challenger:
    challenger_server: "https://kms.example.com"
    certificate: |
      -----BEGIN CERTIFICATE-----
      MIIBkTCB+wIJAJ...
      -----END CERTIFICATE-----

Three things to know about the value:

• It is the PEM text itself, not a path to a file. The node appends it to its system CA pool, so it can be either the KMS server certificate or the CA that signed it.
• It only takes effect when challenger_server is an https URL. With http there is no TLS handshake to verify and the setting does nothing.
• The kcrypt-challenger deployment serves plain HTTP itself. TLS is terminated by whatever fronts it, typically an Ingress, and the certificate you pin here is the one that endpoint presents.

Without this setting the connection is unauthenticated. Treat it as required for any KMS that is not on a network you fully control.

Discoverable Key Management Server (KMS)

By setting kcrypt.challenger.mdns to true in the config, Kairos will try to discover the KMS using the mdns protocol. For that to work, the kcrypt.challenger.challenger_server options should also be a domain ending in .local. If a server exists in the same network and responds to the request, Kairos will use the information from the response to connect to the server.

The server running in the kubernetes cluster (See the Components section) does not implement the mDNS protocol and thus it won't respond to the client's request. That is because the server is running inside Kubernetes, which has its own network and it won't receive the client's broadcast message.

A server is needed that runs in the same network as the Kairos node and responds with the IP address and port where the KMS is reachable. There may be tools that can be configured for the job, but we also provide a little utility that does exactly that: https://github.com/kairos-io/simple-mdns-server/ (archived, but its releases are still downloadable and the tool still works).

The process to deploy the KMS is similar to the Online mode. An example on how to test this feature locally, can be found in this document.

Troubleshooting

• Invoking /system/discovery/kcrypt-discovery-challenger without arguments returns the TPM pubhash.
• Invoking kcrypt-discovery-challenger with 'discovery.password' triggers the logic to retrieve the passphrase, for instance can be used as such:

echo '{ "data": "{ \"label\": \"LABEL\" }"}' | sudo /system/discovery/kcrypt-discovery-challenger "discovery.password"

The url of the KMS server is looked up in the kairos config (In directories/oem and /sysroot/oem).

Notes

If encryption is enabled and COS_PERSISTENT is set to be encrypted, every cloud config file in /usr/local/cloud-config will be protected and can be used to store sensitive data. However, it's important to keep in mind that although the contents of /usr/local are retained between reboots and upgrades, they will not be preserved during a resets.