Troubleshooting

Things can go wrong. This section tries to give guidelines in helping out identify potential issues.

It is important first to check out if your issue was already submitted in the issue tracker.

• Kairos
• Kairos UKI

Root permission

By default, there is no root user set. A default user (kairos) is created and can use sudo without password authentication during LiveCD bootup.

Get back the kubeconfig

On all nodes, which are deployed with the P2P full-mesh feature of the cluster, it’s possible to invoke kairos get-kubeconfig to recover the kubeconfig file.

Reporting issues

If you are reporting a bug, please open an issue on the Kairos GitHub repository

When reporting issues, please provide as much information as possible. This includes:

• The version of Kairos you are using (Attach the full /etc/os-release file)
• The hardware you are running Kairos on (In case of suspecting hardware compatibility issues)
• The steps to reproduce the issue
• Any logs or screenshots you have
• Any other relevant information
• If you are using a custom configuration, please provide the configuration file

Some small things you can do to provide us with the best information possible:

• When running kairos-agent commands, please add the --debug flag and attach the output to the issue
• You can also run kairos-agent state and attach the output to the issue as that provides information about various components of the system
• You can also run kairos-agent config and attach the output to the issue as that provides the current cloud-config for the system (Be aware that any sensitive information should be redacted before attaching the output)
• Immucore logs, available under /run/immucore/ can be useful to attach as well
• Agent logs, available under /var/log/kairos/*.log which are stored on each run of the agent
• Journalctl logs for the following services (use journalctl -u <service> to get the logs) can be useful:
  • cos-setup-network
  • cos-setup-fs
  • cos-setup-boot
  • kairos

Trusted boot

TPM Event Logs

The Trusted Platform Module (TPM) logs contain important security-related measurements that are taken during the boot process. These logs are instrumental in ensuring that the boot process is secure and has not been tampered with.

Viewing TPM Event Logs

To view TPM event logs, you can use the tpm2_eventlog command as follows:

tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements

However, for a more detailed and structured view, especially when analyzing what contributes to a specific PCR (Platform Configuration Register) value like PCR 7, the systemd-pcrlock tool provides enhanced visibility:

/usr/lib/systemd/systemd-pcrlock

This tool is particularly useful for understanding the sequence of events that influence each PCR value during the boot process.

Key Management in Firmware

To ensure the integrity and security of your system, it is critical to manage and verify the keys used by the firmware, especially when dealing with upgrades and security patches.

Listing Enrolled Keys

You can list keys that are currently enrolled in the firmware using sbctl:

sbctl list-enrolled-keys

For further details and usage, refer to the sbctl tool’s GitHub repository: https://github.com/Foxboron/sbctl

Verifying EFI Signature

To verify if the EFI executables are signed with the correct keys, you can use tools like goverify:

go get -u github.com/Foxboron/go-uefi/cmd/goverify
goverify

This step ensures that any upgrades or installations are performed using authorized and validated keys, thereby preserving the security of the system. More information and source code can be found on GitHub: https://github.com/Foxboron/go-uefi/blob/master/cmd/goverify/main.go

Checking if Secure Boot is Enabled

Secure Boot is a security standard that helps to ensure that a device boots using only software that is trusted by the Original Equipment Manufacturer (OEM). When Secure Boot is enabled, the firmware checks each piece of boot software for a valid digital signature.

Command to Check Secure Boot Status

To check if Secure Boot is enabled on your system, use the following command:

dmesg | grep -i secure

The output will display entries related to Secure Boot, indicating whether it is enabled and showing details about the X.509 certificates loaded during the boot process, as seen in the example output provided in your notes.