How to install Fedora on an HP X2 Chromebook

We have been working lately with Enric Balletbo and Dorinda Bassey to improve the support for the HP X2 Chromebook in Fedora. This post explains how to install Fedora on that Chromebook.

The article ended being up being longer than I thought, so for the impatient this is the summary:

  • Switch the Chromebook to Developer Mode.

  • Boot from the internal disk.

  • Go to a virtual terminal with Ctrl+Alt+F2 and login as root.

  • Enable developer mode boot from external disk (USB/microSD):

      $ crossystem dev_boot_usb=1
    
  • Install packages needed by the chromebook-setup.sh script:

      $ sudo dnf install bc curl util-linux gdisk lz4 \
        e2fsprogs uboot-tools udisks2 vboot-utils \
        guestfs-tools qemu-user-static
    
  • Clone Enric’s chromebooks scripts repo:

      $ git clone https://github.com/eballetbo/chromebooks.git
      $ pushd chromebooks
    
  • Flash a Fedora image to a storage media (replace /dev/sda with your block device):

      $ sudo ./chromebook-setup.sh deploy_fedora \
        --architecture=arm64 --storage=/dev/sda \
        --kparams="clk_ignore_unused deferred_probe_timeout=30"
    
  • Plug the USB/microSD device into the Chromebook and choose to boot from an external device.

  • After the Fedora initial setup, install the following packages:

      $ sudo dnf install uboot-tools vboot-utils lz4 -y
    
  • Remove packages that expects the grub2 bootloader to be used:

      $ sudo dnf remove grubby kexec-tools -y
    
  • Enjoy Fedora on the Chromebook ūüôā

Now the longer version…

Challenges supporting the Chromebooks in Fedora

Supporting the Chromebooks is not trivial, because these laptops use a different firmware (Coreboot) and boot stack (Depthcharge) than what is used by all the other aarch64 machines supported by Fedora (UEFI and GRUB). There are good reasons why that is the case, but it poses some challenges and complicates making these laptops to work in Fedora out-of-the-box.

For this reason, a standard Fedora ISO image can’t just be booted on a Chromebook to start an installation process.

Current approach used to install Fedora on the Chromebooks

To overcome this, Enric wrote a set of chromebook scripts that can be used to setup a block device (i.e: USB drive or microSD card) and write a Fedora image that can be booted directly. The script also adds to the system a kernel-install plugin, written by Dorinda, that takes the Linux kernel image being installed and package it in the format (FIT) expected by the Chromebook bootloader.

That way, the Fedora installation would look and behave exactly the same than in any other system.

In the future support for Chromebooks might be added to the Anaconda OS installer used by Fedora, but in the meantime using this script allows us to do experimentation and make further customization that wouldn’t be suitable for a general system installer.

Following are the instructions to install Fedora on a HP X2 Chromebook using the chromebook-setup.sh script.

Switch the Chromebook to Developer Mode

In the default mode the Chromebook can only boot binaries that are trusted by its firmware. This means that nothing besides ChromeOS can be installed. To boot a different OS, the mode should be change to Developer. In this mode, any binary that is signed with the Google’s developer key can be booted. That key is available for anyone so is what is used to sign the Linux images when generating the FIT images during the Fedora kernel packages installations.

The ChromiumOS project has excellent articles explaining the Developer Mode and how to enable it on Chromebooks without a physical keyboard, such as the HP X2.

After enabling developer mode, boot from the internal disk and switch to a virtual terminal with Ctrl+Alt+F2. Then login as root and execute the following to enable booting from an external disk (USB/microSD):

$ crossystem dev_boot_usb=1

Flashing a Fedora image

The chromebook-setup.sh script can be used to flash fedora images to a block device and do all the needed setup to make it bootable.

It supports many different options but it also has reasonable defaults. The list of options can be listed with ./chromebook-setup --help.

Following are the steps to flash a Fedora image.

Install packages needed by the chromebook-setup.sh script:

$ sudo dnf install bc curl util-linux gdisk lz4 \
  e2fsprogs uboot-tools udisks2 vboot-utils \
  guestfs-tools qemu-user-static

Clone Enric’s chromebooks scripts repository:

$ git clone https://github.com/eballetbo/chromebooks.git
$ pushd chromebooks

Execute the script, for example:

$ sudo ./chromebook-setup.sh deploy_fedora \
  --architecture=arm64 --storage=/dev/sda \
  --kparams="clk_ignore_unused deferred_probe_timeout=30"

The deploy_fedora option will install a Fedora image in the specified storage media. By default the latest Fedora Workstation Rawhide image will downloaded and used, but a different image can be chosen using the --image=$image option.

The --architecture and --storage options specify the architecture and block device used respectively.

Finally, the --kparams option allows to set additional kernel command line parameters.

The clk_ignore_unused parameter is currently needed because there is a bug when the MSM/snapdragon DRM driver is built as a module. Some needed clocks are gated before the driver probe function is executed, causing it to fail.

And the deferred_probe_timeout=30 is needed because there are a lot of drivers probe deferrals and the default 10 seconds expires causing drivers to fail to probe due timeouts.

Hopefully these two issues would be fixed soon and the parameters won’t be needed anymore.

One the script finishes flashing the image, plug the USB drive or insert the microSD in the Chromebook and choose "Boot from external media".

The system should boot and start the Fedora initial setup program to configure the system and create a user. Once that is done, start a terminal and install the following packages needed by the kernel-install Chromebook plugin:

$ sudo dnf install uboot-tools vboot-utils lz4 -y

Remove packages that expects the grub2 bootloader to be used:

$ sudo dnf remove grubby kexec-tools -y

And that’s it. Now the Fedora should behave like in any other system. If there are any bugs, please file issues in the chromebooks scripts repository.

Happy hacking!

Automatic LUKS volumes unlocking using a TPM2 chip

I joined Red Hat a few months ago, and have been working on improving the Trusted Platform Module 2.0 (TPM2) tooling, towards having a better TPM2 support for Fedora on UEFI systems.

For brevity I won’t explain in this post what TPMs are and their features, but assume that readers are already familiar with trusted computing in general. Instead, I’ll explain what we have been working on, the approach used and what you might expect on Fedora soon.

For an introduction to TPM, I recommend Matthew Garret’s excellent posts¬†about the topic, Philip Tricca’s presentation¬†about TPM2 and the official Trusted Computing Group (TCG)¬†specifications. I also found¬†“A Practical Guide to TPM 2.0” book to be much easier to digest than the official TCG documentation. The book is an open access one, which means that’s¬†freely available.

LUKS volumes unlocking using a TPM2 device

Encryption of data at rest is a key component of security.  LUKS provides the ability to encrypt Linux volumes, including both data volumes and the root volume containing the OS. The OS can provide the crypto keys for data volumes, but something has to provide the key for the root volume to allow the system to boot.

The most common way to provide the crypto key to unlock a LUKS volume,  is to have a user type in a LUKS pass-phase during boot. This works well for laptop and desktop systems, but is not well suited for servers or virtual machines since is an obstacle for automation.

So the first TPM feature we want to add to Fedora (and likely one of the most common use cases for a TPM) is the ability to bind a LUKS volume master key to a TPM2. That way the volume can be automatically unlocked (without typing a pass-phrase) by using the TPM2 to obtain the master key.

A key point here is that the actual LUKS master key is not present in plain text form on the system, it is protected by TPM encryption.

Also, by sealing the LUKS master key with a specific set of Platform Configuration Registers (PCR), one can make sure that the volume will only be unlocked if the system has not been tampered with. For example (as explained in this post), PCR7 is used to measure the UEFI Secure Boot policy and keys. So the LUKS master key can be sealed against this PCR, to avoid unsealing it if Secure Boot was disabled or the used keys were replaced.

Implementation details: Clevis

Clevis¬†is a plugable framework for automated decryption that has a number of “pins”, where each pin implements an {en,de}cryption support using a different backend. It also has a command line interface to {en,de}crypt data using these pins,¬†create complex security policies and bind a pin to a LUKS volume to later unlock it.

Clevis relies on the José project, which is an C implementation of the Javascript Object Signing and Encryption (JOSE) standard. It also uses the LUKSMeta project to store a Clevis pin metadata in a LUKS volume header.

On encryption, a Clevis pin takes some data to encrypt and a JSON configuration to produce a JSON Web Encryption (JWE) content. This JWE has the data encrypted using a JSON Web KEY (JWK) and information on how to obtain the JWK for decryption.

On decryption, the Clevis pin obtains a JWK using the information provided by a JWE and decrypts the ciphertext also stored in the JWE using that key.

Each Clevis pin defines their own JSON configuration format, how the JWK is created, where is stored and how to retrieve it.

As mentioned, Clevis has support to bind a pin with a LUKS volume. This means that a LUKS master key is encrypted using a pin and the resulting JWE is stored in a LUKS volume meta header. That way Clevis is able to later decrypt the master key and unlock the LUKS volume. Clevis has dracut and udisks2 support to do this automatically and the next version of Clevis will also include a command line tool to unlock non-root (data) volumes.

Clevis TPM2 pin

Clevis provides a mechanism to automatically supply the LUKS master key for the root volume. The initial implementation of Clevis has support to obtain the LUKS master key from a network service, but we have extended Clevis to take advantage of a TPM2 chip, which is available on most servers, desktops and laptops.

By using a TPM, the disk can only be unlocked on a specific system – the disk will neither boot nor be accessed on another machine.

This implementation also works with UEFI Secure Boot, which will prevent the system from being booted if the firmware or system configuration has been modified or tampered with.

To make use of all the Clevis infrastructure and also be able to use the TPM2 as a part of more complex security policies, the TPM2 support was implemented as a clevis tpm2 pin.

On encryption the tpm2 pin generates a JWK, creates an object in the TPM2 with the JWK as sensitive data and binds the object (or seals if a PCR set is defined in the JSON configuration) to the TPM2.

The generated JWE contains both the public and wrapped sensitive portions of the created object, as well as information on how to unseal it from the TPM2 (hashing and key encryption algorithms used to recalculate the primary key, PCR policy for authentication, etc).

On decryption the tpm2 pin takes the JWE that contains both the sealed object and information on how to unseal it,  loads the object into the TPM2 by using the public and wrapped sensitive portions and unseals the JWK to decrypt the ciphertext stored in the JWE.

The changes haven’t been merged yet, since the pin is using features from tpm2-tools master so we have to wait for the next release of the tools. And also there are still discussions on the pull request about some details, but it should be ready to land soon.

Usage

The Clevis command line tools can be used to encrypt and decrypt data using a TPM2 chip. The tpm2 pin has reasonable defaults but one can configure most of its parameters using the pin JSON configuration (refer to the Clevis tpm2 pin documentation for these), e.g:

$ echo foo | clevis encrypt tpm2 '{}' > secret.jwe

And then the data can later be decrypted with:

$ clevis decrypt < secret.jwe
foo

To seal data against a set of PCRs:

$ echo foo | clevis encrypt tpm2 '{"pcr_ids":"8,9"}' > secret.jwe

And to bind a tpm2 pin to a LUKS volume:

$ clevis luks bind -d /dev/sda3 tpm2 '{"pcr_ids":"7"}'

The LUKS master key is not stored in raw format, but instead is wrapped with a JWK that has the same entropy than the LUKS master key. It’s this JWK that is sealed with the TPM2.

Since Clevis has both dracut and udisks2 hooks, the command above is enough to have the LUKS volume be automatically unlocked using the TPM2.

The next version of Clevis also has a clevis-luks-unlock command line tool, so a LUKS volume could be manually unlocked with:

$ clevis luks unlock -d /dev/sda3

Using the TPM2 as a part of more complex security policies

One of Clevis supported pins is the Shamir Shared Secret (SSS) pin, that allows to encrypt a secret using a JWK that is then split into different parts. Each part is then encrypted using another pin and a threshold is chose to decide how many parts are needed to reconstruct the encryption key, so the secret can be decrypted.

This allows for example to split the JWK used to wrap the LUKS mater key in two parts. One part of the JWK could be sealed with the TPM2 and another part be stored in a remote server. By sealing a JWK that’s only one part of the needed key to decrypt the LUKS master key, an attacker obtaining the data sealed in the TPM won’t be able to unlock the LUKS volume.

The Clevis encrypt command for this particular example would be:

$ clevis luks bind -d /dev/sda3 sss '{"t": 2, "pins": \
  {"http":{"url":"http://server.local/key"}, "tpm2": \
  {"pcr_ids":"7"}}}'

Limitations of this approach

One problem with the current implementation is that Clevis is a user-space tool and so it can’t be used to unlock a LUKS volume that has an encrypted /boot directory. The boot partition still needs to remain unencrypted so the bootloader is able to load a Linux kernel and an initramfs that contains Clevis, to unlock the encrypted LUKS volume for the root partition.

Since the initramfs is not signed on a Secure Boot setup, an attacker could replace the initramfs and unlock the LUKS volume. So the threat model meant to protect is for an attacker that can get access to the encrypted volume but not to the trusted machine.

There are different approaches to solve this limitation. The previously mentioned post¬†from Matthew Garret suggests to have a small initramfs that’s built into the signed Linux kernel. The only task for this built-in initramfs would be to unseal the LUKS master key, store it into the kernel keyring and extend PCR7 so the key can’t be unsealed again. Later the usual initramfs can unlock the LUKS volume by using the key already stored in the Linux kernel.

Another approach is to also have the /boot directory in an encrypted LUKS volume and provide support for the bootloader to unseal the master key with the TPM2, for example by supporting the same JWE format in the LUKS meta header used by Clevis. That way only a signed bootloader would be able to unlock the LUKS volume that contains /boot, so an attacker won’t be able to tamper the system by replacing the initramfs since it will be in an encrypted partition.

But there is work to be done for both approaches, so it will take some time until we have protection for this threat model.

Still, having an encrypted root partition that is only automatically unlocked on a trusted machine has many use cases. To list a few examples:

  • Stolen physical disks or virtual machines images can’t be mounted on a different machine.
  • An external storage media can be bind to a set of machines, so it can be automatically unlocked only on trusted machines.
  • A TPM2 chip can be reset before sending a laptop to repair, that way the LUKS volume can’t be automatically unlocked anymore.
  • An encrypted volume can be bound to a TPM2 if there is no risk of someone having physical access to the machine but unbound again when there is risk. So the machine can be automatically unlocked on safe places but allow to require a pass-phrase on unsafe places.

Acknowledgements

I would like to thanks Nathaniel McCallum and Russell Doty for their feedback and suggestions for this article.