Booting Fedora with sd-boot and Secure Boot enabled

Every once in a while there is a discussion in the Fedora development list about replacing the default GRUB bootloader. That latest of these threads was "future of dual booting Windows and Fedora, redux" about a month ago, but searching for GRUB in the mailing list archives one can find many similar conversations.

One of the options to replace GRUB that is always mentioned is systemd-boot (sd-boot), since is quite minimal and enough for simple booting needs such as what is required for Fedora Workstation.

But currently this can’t be done without additional work, because shim only supports using GRUB as a second stage loader. There is a RFE: add support for multiple second stage loaders #472 filed for shim, but there wasn’t a conclusion on that issue about how to move forward.

This means that to use sd-boot instead of GRUB and keeping Secure Boot enabled, users need to create their own key pairs and enroll a cert with the public key into the UEFI firmware db database.

To have an idea of what that would entail, I installed sd-boot and removed shim and GRUB in one of my machines. Below are the steps I followed in case someone wants to replicate it.

To keep the example generic, I mention how a public key can be enrolled with the Open Virtual Machine Firmware (OVMF) UEFI firmware used to boot virtual machines. But a similar procedure can be followed to enroll keys using the UEFI settings of a physical machine. Refer to your hardware vendor documentation for how to do this.

1. Create a key pair to sign the binaries and enroll into the UEFI firmware db key database

$ cat << EOF > configuration_file.config
[ req ]
default_bits = 4096
distinguished_name = req_distinguished_name
prompt = no
string_mask = utf8only
x509_extensions = myexts

[ req_distinguished_name ]
O = Organization
CN = Organization signing key
emailAddress = E-mail address

[ myexts ]
basicConstraints=critical,CA:FALSE
keyUsage=digitalSignature
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid
EOF

$ openssl req -x509 -new -nodes -utf8 -sha256 -days 36500 -batch -config \
configuration_file.config -outform DER -out public_key.der -keyout private_key.priv

$ openssl x509 -in public_key.der -inform DER -outform PEM -out public_key.pem

2. Enroll the public cert in the UEFI firmware db database

$ sudo cp public_key.der /boot/efi/

On reboot, enter into the UEFI firmware settings and go to:

Device Manager
    Secure Boot Configuration
        Secure Boot Mode 

That will allow to enroll a key by choosing:

Custom Secure Boot Options
    DB options
        Enroll Signature
            Enroll Signature Using File

After that, when booting the enrolled key should be present in db and added to the Linux kernel .platform keyring:

$ mokutil --test-key public_key.der 
public_key.der is already in db

$ keyctl list %:.platform
3 keys in keyring:
577585465: ---lswrv     0     0 asymmetric: Microsoft Windows Production PCA 2011: a92902398e16c49778cd90f99e4f9ae17c55af53
137367868: ---lswrv     0     0 asymmetric: Organization signing key: ae6f49657c238754eedc503e98cb97afdb064706
984007323: ---lswrv     0     0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4

3. Install sd-boot and remove shim and grub2 packages

$ sudo bootctl install
$ sudo mkdir /boot/efi/$(cat /etc/machine-id)
$ sudo rm /etc/dnf/protected.d/{shim,grub2}*.conf
$ sudo dnf remove shim* grub2* -y
$ sudo rm -r /boot/loader
$ sudo dnf reinstall kernel-core -y

4. Sign the sd-boot and Linux kernel binaries

$ sudo dnf install -y sbsigntools pesign

$ sudo sbsign --key private_key.priv --cert public_key.pem  /usr/lib/systemd/boot/efi/systemd-bootx64.efi \
--output /boot/efi/EFI/systemd/systemd-bootx64.efi

$ sudo sbsign --key private_key.priv --cert public_key.pem /lib/modules/$(uname -r)/vmlinuz \
--output /boot/efi/$(cat /etc/machine-id)/$(uname -r)/linux
Image was already signed; adding additional signature

Check that the binaries have been signed correctly:

$ sudo pesign -S -i /boot/efi/EFI/systemd/systemd-bootx64.efi
---------------------------------------------
certificate address is 0x7f348c12b1c8
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is Test signing key
The signer's email address is e-mail address
Signing time: Thu Sep 01, 2022
There were certs or crls included.
---------------------------------------------

$ sudo pesign -S -i /boot/efi/$(cat /etc/machine-id)/$(uname -r)/linux
---------------------------------------------
certificate address is 0x7fb81a4b11e8
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is Fedora Secure Boot Signer
No signer email address.
Signing time: Thu Apr 28, 2022
There were certs or crls included.
---------------------------------------------
certificate address is 0x7fb81a4b1b30
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is kernel-signer
No signer email address.
Signing time: Thu Apr 28, 2022
There were certs or crls included.
---------------------------------------------
certificate address is 0x7fb81a4b26f8
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is Organization signing key
The signer's email address is e-mail address
Signing time: Thu Sep 01, 2022
There were certs or crls included.
---------------------------------------------

On reboot the machine should be able to boot the signed sd-boot and Linux kernel with Secure Boot enabled:

$ mokutil --sb-state
SecureBoot enabled

5. Sign an out-of-tree kernel module

This step is mentioned for completeness, since some users need to load out-of-tree kernel modules. And this is only possible for signed modules when Secure Boot is enabled.

If that’s needed, following is an example on how to do it:

$ sudo dnf install kernel-devel -y

$ git clone https://github.com/maK-/SimplestLKM.git

$ pushd SimplestLKM

$ make

$ /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 key-private_key.priv \
public_key.der hello.ko

$ modinfo hello.ko | grep signature
signature: 91:F7:20:16:1C:85:AC:DC:54:25:C2:B2:E9:ED:02:93:79:43:1D:7F:

$ insmod hello.ko

$ dmesg
[ 1653.929819] hello: loading out-of-tree module taints kernel.
[ 1653.930435] Hello world!

A pain point with this setup is that one need to manually sign the sd-boot and Linux kernel binaries each time that those are updated. This could be automated of course but that would require to have the private key in the filesystem and that would defeat the whole purpose of the Secure Boot protection.

Since in that case an attacker that gets root access could be able to sign any binaries they want and make their attack persistent across reboots.

For this reason what I’ve been done is keeping the private key in an external media and only attach it when there is a need to sign a new binary, which makes updating packages more inconvenient.

Hopefully this won’t be a problem anymore once the mentioned shim issue #472 gets resolved. Because then the sd-boot binary could get signed during package build with the Fedora key just like GRUB and there won’t be a need to sign neither sd-boot nor the Linux kernel binaries with a custom key.

And that’s all for today. Happy hacking!

Using an I2C SSD1306 OLED on Fedora with a Raspberry Pi

Linux 5.18 version landed in Fedora 36 and it includes a new ssd130x DRM driver for the Solomon OLED display controllers.

One of the supported devices is SSD1306, which seems to be a quite popular display controller for small and cheap (I bought a pack of 3 on Amazon for less than 15€) monochrome OLED panels.

I do a lot of development and testing on RPi boards and found that these small OLED panels are useful to get console output without the need to either connect a HDMI monitor or a serial console.

This blog post explains how to setup Fedora 36 to use a SSD1306 OLED connected through I2C.

First you need connect the SSD1306 display to the RPi, Adrafruit has an excellent article on how to do that.

Then you need to install Fedora 36, this can be done by downloading an aarch64 Fedora raw image (e.g: Workstation Edition) and executing the following command:

sudo arm-image-installer --image Fedora-Workstation-36-1.5.aarch64.raw.xz --target=rpi4 --media=$device --addconsole --addkey=id_rsa.pub --norootpass --resizefs

where $device is the block device for your uSD (e.g: /dev/sda).

Fedora 36 was released with Linux 5.17, but as mentioned the ssd130x DRM driver landed in 5.18, so you need to update the Fedora kernel:

sudo dnf update kernel -y

Finally you need to configure the RPi firmware to enable the I2C1 pins and load the ssd1306 Device Tree Blob Overlay (dtbo) to register a SSD1306 I2C device, e.g:

sudo cat << EOF >> /boot/efi/config.txt
dtparam=i2c1=on
dtoverlay=ssd1306,inverted
EOF

The ssd1306.dtbo supports many parameters in case your display doesn’t match the defaults. Take a look to the RPi overlays README for more details about all these parameters and their possible values.

The ssd130x DRM driver registers an emulated fbdev device that can be bound with fbcon and use the OLED display to have a framebuffer console. If you want to do that, it’s convenient to change the virtual terminal console font to a smaller one, e.g:

sudo sed -i 's/FONT=.*/FONT="drdos8x8"/' /etc/vconsole.conf
sudo dracut -f

And that’s it. If your SSD1306 uses SPI instead, this is supported since Linux 5.19 but that would be the topic for another post.

Happy hacking!

How to troubleshoot deferred probe issues in Linux

When working on the retro handheld console mentioned in a previous post, I had an issue where the LCD driver was not probed when booting a custom Linux kernel image built.

To understand the problem, first some knowledge is needed about how devices and drivers are registered in the Linux kernel, how these two sets are matched (bound) and what is a probe deferral.

If you are not familiar with these concepts, please read this post where are explained in detail.

The problem is that the st7735r driver (that’s needed for the SPI TFT LCD panel I was using) requires a GPIO based backlight device. To make it more clear, let’s look at the relevant bits in the adafruit-st7735r-overlay.dts that’s used as a Device Tree Blob (DTB) overlay to register the needed devices:

fragment@2 {
...
    af18_backlight: backlight {
        compatible = "gpio-backlight";
...
    };
};

fragment@3 {
...
    af18: adafruit18@0 {
        compatible = "jianda,jd-t18003-t01";
        backlight = <&af18_backlight>;
...
    };
};

We see that the adafruit18@0 node for the panel has a backlight property whose value is a phandle to the af18_backlight label used to refer to the backlight node.

The drivers/gpu/drm/tiny/st7735r.c probe callback then uses the information in the DTB to attempt getting a backlight device:

static int st7735r_probe(struct spi_device *spi)
{
...
	dbidev->backlight = devm_of_find_backlight(dev);
	if (IS_ERR(dbidev->backlight))
		return PTR_ERR(dbidev->backlight);
...
}

The devm_of_find_backlight() function returns a pointer to the backlight device if this could be found or a -EPROBE_DEFER error pointer if there is a backlight property defined in the DTB but this could not be found.

For example, this can happen if the driver that registers that expected backlight device was not yet probed.

If the probe callback returns -EPROBE_DEFER, the kernel then will put the device that matched the driver but failed to probe in a deferred probe list. The list is iterated each time that a new driver is probed (since it could be that the newly probed driver registered the missing devices that forced the probe deferral).

My problem then was that the needed driver (CONFIG_BACKLIGHT_GPIO since the backlight node has compatible = gpio-backlight) was not enabled in my kernel, leading to the panel device to remain in the deferred probe list indefinitely due a missing backlight device that was never registered.

This is quite a common issue on Device Tree based systems and something that it used to take me a lot of time to root cause when I started working on Linux embedded platforms.

A few years ago I added a /sys/kernel/debug/devices_deferred debugfs entry that would expose the list of devices deferred to user-space, which makes much more easier to figure out what devices couldn’t be bound due their driver probe being deferred.

Later, Andrzej Hajda improved that and added to the devices_deferred debugfs entry support to print the reason of the deferral.

So checking what devices were deferred and the reason is now quite trivial, i.e:

$ cat /sys/kernel/debug/devices_deferred 
spi0.0  spi: supplier backlight not ready

Much better than spending a lot of time looking at kernel logs and adding debug printous to figure out what’s going on.

Happy hacking!

Linux drivers and devices registration, matching, aliases and modules autoloading

Every once in a while I have to explain these concepts to someone so I thought that it could be something worth to write about.

Device drivers and devices registration

The Linux kernel documentation covers quite well how device drivers and devices are registered and how these two are bound. But the summary is that drivers and devices are registered independently and each of these specify their given bus type. The Linux kernel device model then uses that information to bind drivers with devices of the same bus type.

Drivers and devices are registered using the driver_register() function which is usually called from either the drivers’ module_init() function or platform code.

Devices are registered using the register_device() function which is usually called by subsystems that parses a list of devices from some hardware topology description, some enumerable bus or platform code that hardcodes the devices to be registered.

Drivers and device matching (binding)

When a driver is registered for a given bus, the list of devices registered for that bus is iterated to find a match.

In the same manner, when a device is registered, the list of drivers registered for the same bus is iterated to find a match.

That way, it doesn’t matter the order in which drivers and devices are registered. They a device will be bound to a driver regardless of which one was registered first.

Drivers’ probe callback

If a match is found, the driver’s probe callback is executed. This function handler contains the driver-specific logic to bind the driver with a device and any setup needed for this. The probe function returns 0 if the driver could be bound to the device successfully or a negative errno code if the driver was not able to bound the device.

Probe deferral

A special errno code -EPROBE_DEFER is used to indicate that the bound failed because a driver could not provide all the resources needed by the device. When this happens, the device is put into a deferred probe list and the probe is retried again at a later time.

That later time is when a new driver probes successfully. When this happens, the device deferred probe list is iterated again and all devices are tried to bind again with their matched driver.

If the newly probed driver provides a resource that was missing by drivers whose probe was deferred, then their probe will succeed this time and their bound devices will be removed from the deferred list.

If all required resources are provided at some point, then all drivers should probe correctly and the deferred list should become empty.

It is a simple and elegant (albeit inefficient) solution to the fact that drivers and devices registration are non-deterministic. This leads to drivers not having a way to know if a resource won’t ever be available or is just that the driver that would provide the resource has just not probed yet.

But even if the kernel probed the drivers in a deterministic order (i.e: by using device dependency information), the driver would have no way to know if for example the missing resource would be provided by a driver that was built as a kernel module and would be loaded much later by user-space or even manually by an operator.

Module device tables

Each driver provides information about what devices can be matched against and usually this information is provided on a per firmware basis. For example, a driver that supports devices registered using both ACPI and Device Tree hardware descriptions, will contain separate ID tables for the ACPI and OpenFirmware (OF) devices that can be matched.

To illustrate this, the drivers/input/touchscreen/hideep.c has the following device ID tables:

static const struct i2c_device_id hideep_i2c_id[] = {
	{ HIDEEP_I2C_NAME, 0 },
	{ }
};
MODULE_DEVICE_TABLE(i2c, hideep_i2c_id);

#ifdef CONFIG_ACPI
static const struct acpi_device_id hideep_acpi_id[] = {
	{ "HIDP0001", 0 },
	{ }
};
MODULE_DEVICE_TABLE(acpi, hideep_acpi_id);
#endif

#ifdef CONFIG_OF
static const struct of_device_id hideep_match_table[] = {
	{ .compatible = "hideep,hideep-ts" },
	{ }
};
MODULE_DEVICE_TABLE(of, hideep_match_table);
#endif

Module aliases

If information defined in these tables are exported using the MODULE_DEVICE_TABLE() macro, then these will be in the drivers kernel modules as alias entries in the module information.

For example, one can check the module aliases for a given module using the modinfo command, i.e:

$ modinfo drivers/input/touchscreen/hideep.ko | grep alias
alias:          i2c:hideep_ts
alias:          acpi*:HIDP0001:*
alias:          of:N*T*Chideep,hideep-tsC*
alias:          of:N*T*Chideep,hideep-ts

Here are listed the legacy I2C platform, ACPI and OF devices that are exported by MODULE_DEVICE_TABLE(i2c, hideep_i2c_id), MODULE_DEVICE_TABLE(acpi, hideep_acpi_id) and MODULE_DEVICE_TABLE(of, hideep_match_table) respectively.

Module autoloading

The module aliases information is only used by user-space, the kernel uses the actual device tables to match the driver with the registered devices. In fact, the MODULE_DEVICE_TABLE() is a no-op if the driver is built-in the kernel image and not built as a module.

The way this work is that when a device is registered for a bus type, the struct bus_type.uevent callback is executed and the bus driver reports a uevent to udev to take some actions. The uevent contains key-value pairs and one of them is the device MODALIAS.

For example, on my laptop when the PCI bus is enumerated and my GPU registered the following uevent MODALIAS will be sent (as shown by udevadm monitor -p):

KERNEL[189823.929341] add   /devices/pci0000:00/0000:00:02.0 (pci)                                                  
ACTION=add                                              
DEVPATH=/devices/pci0000:00/0000:00:02.0                                                                               
SUBSYSTEM=pci                                                                                                          
...
MODALIAS=pci:v00008086d00003EA0sv000017AAsd00002292bc03sc00i00                                                         
...

This information is then used by udev and pass to kmod to load the module if needed. It will do something like:

$ modprobe pci:v00008086d00003EA0sv000017AAsd00002292bc03sc00i00

Since mod{probe,info} can also take a module alias besides the module name. For exapmle, the following should tell the module that matches this alias:

$ modinfo pci:v00008086d00003EA0sv000017AAsd00002292bc03sc00i00 | grep ^name
name:           i915

This information is also present in sysfs, i.e:

$ cat /sys/devices/pci0000\:00/0000\:00\:02.0/uevent 
DRIVER=i915
PCI_CLASS=30000
PCI_ID=8086:3EA0
PCI_SUBSYS_ID=17AA:2292
PCI_SLOT_NAME=0000:00:02.0
MODALIAS=pci:v00008086d00003EA0sv000017AAsd00002292bc03sc00i00

$ /sys/devices/pci0000\:00/0000\:00\:02.0/modalias 
pci:v00008086d00003EA0sv000017AAsd00002292bc03sc00i00

In theory, drivers should only define device ID tables for the firmware interfaces that they support. That is, a driver that supports devices registered through let’s say ACPI should only need a struct acpi_device_id. And also the same table should be used to match the driver with a device and send the MODALIAS information. For example if a device was registered through OF, only the struct of_device_id should be used for both matching and module alias reporting.

But in practice things are more complicated and there are exceptions in some subsystems, although that’s a topic for another time since this post got already too long.

If you are curious about the possible pitfalls though, I wrote about a bug chased some time ago in Fedora where the cause was a driver not reporting the MODALIAS that one would expect.

Happy hacking!

Building a retro handheld console with Fedora and a RPi zero

I built a retro console for my kids some time ago and they asked me if they could have one but that was portable.

Now that I finished the retro handheld console, thought that could be useful to share how it was done in case others wanted to replicate.

Hardware

I used a Waveshare 128×128, 1.44inch LCD display HAT which is a good fit because it contains both a LCD display and GPIO keys that can be used as a gamepad.

The board is a HAT for the Raspberry Pi Zero 2W board, HATs are expansion boards whose connectors are compatible with the RPi Zero 2W pinout.

And that is all the hardware needed if the console will just be powered with a micro USB cable. But if the goal is to power it through a battery, then more components are needed. Instead of listing them here and explaining how to do that part, I will refer to this excellent guide that I followed.

Software

I just used a stock Fedora Server image for this project, no additional software was needed than what is already packaged in the distro.

The image can be flashed using the arm-image-installer tool. There is no support for the RPi Zero 2W but since is quite similar to the RPi3, that can just be used as the target instead, i.e:

sudo arm-image-installer --image=Fedora-Server-36-1.5.aarch64.raw.xz \
--target=rpi3 --media=/dev/$device --addkey=id_rsa.pub --norootpass --resizefs

Where $device is the block device for the uSD card used to install the OS.

Then I followed these steps:

  1. Boot the uSD card, go through Fedora initial setup, create a retroarch user and make it member of the video and input groups.

  2. Install a Libretro emulator (i.e: mGBA for Game Boy Advance) and the Retroarch frontend.

sudo dnf install libretro-mgba retroarch
  1. Create a user service to run the game.
$ mkdir -p ~/.config/systemd/user/

$ cat << EOF > ~/.config/systemd/user/retroarch.service
> [Unit]
Description=Start Doom

[Service]
ExecStart=retroarch -L /lib64/libretro/mgba_libretro.so /home/retroarch/roms/doom.zip
Restart=always

[Install]
WantedBy=default.target
EOF
  1. Enable the service and lingering for the retroarch user, the latter is needed to allow the service to start at boot even when the user was not logged in.
$ systemctl --user enable retroarch.service
$ sudo loginctl enable-linger retroarch
  1. Add RPi config snippets to support the LCD and GPIO keys.

Two Device Tree Blob Overlays (DTBO) are used to drive the SPI controller of the LCD panel and the GPIO keys. These are adafruit-st7735r.dtbo and gpio-key.dtbo.

The overlays support options that can be configured in the RPi config.txt file such as the pins used, display resolution, if the display has to be rotated, the input event code that has to be reported for each GPIO key and so on.

For the HAT mentioned above, the following has to be added to the /boot/efi/config.txt file:

# Enable SPI
dtparam=spi=on

# TFT LCD Hat
dtoverlay=adafruit-st7735r,128x128,dc_pin=25,reset_pin=27,led_pin=24,rotate=90

# GPIO keys configuration

# Directional pad (KEY_UP, KEY_LEFT, KEY_RIGHT, KEY_DOWN, KEY_ENTER)
dtoverlay=gpio-key,gpio=6,active_low=1,gpio_pull=up,label=UP,keycode=103
dtoverlay=gpio-key,gpio=5,active_low=1,gpio_pull=up,label=LEFT,keycode=105
dtoverlay=gpio-key,gpio=26,active_low=1,gpio_pull=up,label=RIGHT,keycode=106
dtoverlay=gpio-key,gpio=19,active_low=1,gpio_pull=up,label=DOWN,keycode=108
dtoverlay=gpio-key,gpio=13,active_low=1,gpio_pull=up,label=PRESS,keycode=28

# Buttons (KEY_X, KEY_C)
dtoverlay=gpio-key,gpio=21,active_low=1,gpio_pull=up,label=KEY_1,keycode=45
dtoverlay=gpio-key,gpio=20,active_low=1,gpio_pull=up,label=KEY_2,keycode=44

# Power (KEY_POWER), useful to power off the console
dtoverlay=gpio-key,gpio=16,active_low=1,gpio_pull=up,label=KEY_3,keycode=116
  1. Workaround a bug in the st7735r driver that prevents the module to be auto loaded.

There is a bug in the st7735r driver which causes the module to not be loading automatically. To workaround this issue, create a modules-load.d config snippet to force the module to be loaded:

$ echo st7735r | sudo tee /etc/modules-load.d/st7735r.conf

Unfortunately this is a quite common bug in SPI drivers. But for this particular driver the workaround should not be needed in the future since it was already fixed by this commit. But at the time of this writing, the Fedora version used (36) still does not contain the fix.

  1. Prevent the simpledrm driver to be initialized.

The firmware seems to add a "simple-framebuffer" Device Tree node even when there is no monitor connected in its mini HDM port. This leads to the simpledrm driver to be probed, so it needs to be denied listed using a kernel command line parameter:

$ sudo grubby --update-kernel=DEFAULT \
--args=initcall_blacklist=simpledrm_platform_driver_init
  1. Update the adafruit-st7735r.dtbo to the latest version.

This was the only change I needed in Fedora 36. The issue is that the adafruit-st7735r.dtbo overlay is for the legacy fb_st7735r driver instead of the DRM st7735r driver. The latter has a different Device Tree node property to specify the display rotation and so the rotate=90 option specified in the config.txt file will be ignored.

$ curl https://github.com/raspberrypi/firmware/raw/master/boot/overlays/adafruit-st7735r.dtbo -O
$ sudo mv adafruit-st7735r.dtbo /boot/efi/overlays/

Happy gaming!

Fedora 36: A brave new (DRM/KMS only) world

The upcoming Fedora release includes a change proposed by Peter Robinson and yours truly, that disables all the Frame Buffer Device (fbdev) drivers in the Linux kernel package.

This means that starting from Fedora 36, only Direct Rendering Manager (DRM) / Kernel Mode Settings (KMS) drivers will be used. No more legacy stuff.

Many Linux developers have been arguing for over a decade now that fbdev is deprecated and it has been almost 7 years since the previous fbdev maintainer said that no new fbdev drivers would be added to the kernel. So this feels like a change long overdue.

This was possible thanks to the work of Thomas Zimmermann @ SUSE, who added the simpledrm driver, that exposes the system framebuffer set-up by the firmware as a DRM device.

Using the simpledrm driver, a framebuffer would be available even if the platform does not have a DRM driver or the user disables it using the nomodeset kernel command-line parameter.

Before this change, the video drivers used when DRM drivers were disabled with the nomodeset parameter, where either efifb (for systems booted with EFI) or vesafb (for systems booted with legacy BIOS and a VESA video mode selected on boot using the vga kernel command line parameter).

The need to rely on the system framebuffer initialized by the firmware was the primary reason why these fbdev drivers were still enabled in the Fedora kernel.

But the simpledrm replaces both, since is able to expose any system framebuffer, as long as platform code pass it as a proper I/O memory region for the driver to use it.

That includes not only the EFI system framebuffer provided through the Graphics Output Protocol (GOP) and the VESA framebuffer configured with the vga parameter, but also framebuffers whose information is passed to the kernel through other hardware description mechanisms. For example, as a "simple-framebuffer" Device Tree node, which is common on aarch64 devices.

Even when no fbdev drivers are used anymore, the fbdev subsystem is still enabled. This is needed because the The Framebuffer Console (fbcon) needs an fbdev device to bind. But fortunately the DRM subsystem provides a generic fbdev emulation layer, so DRM drivers can also provide a fbdev interface for fbcon (or even to user-space that still expects an fbdev device to work).

While working on this, we found that a lot of assumptions made all over the stack were no longer true once simpledrm was used instead of {efi,vesa}fb. So changes were needed in the Linux kernel, Plymouth, GDM, Xorg, etc.

But that will be a topic for another post since this one got too long already!

A framebuffer hidden in plain sight

Soon after I set up my Rockpro64 board, Peter Robinson told me about an annoying bug that happened on machines with a Rockchip SoC.

The problem was that the framebuffer console just went away after GRUB booted the Linux kernel. We started looking at this and Peter mentioned the following data points:

  • Enabling early console output on the framebuffer registered by the efifb driver (earlycon=efifb efi=debug) would get some output but at some point everything would just go blank.
  • The display worked when passing fbcon=map:1 and people were using that as a workaround.
  • Preventing the efifb driver to be loaded (modprobe.blacklist=efifb) would also make things to work.

So the issue seemed to be related to the efifb driver somehow but wasn’t clear what was happening.

What this driver does is to register a framebuffer device that relies on the video output configured by the firmware/bootloader (using the EFI Graphics Output Protocol) until a real driver takes over an re-initializes the display controller and other IP blocks needed for video output.

I read The Framebuffer device and The Framebuffer Console sections in the Linux documentation to get more familiar about how this is supposed to work.

What happens is that the framebuffer console is bound by default to the first framebuffer registered, which is the one registered by the efifb driver.

Later, the rockchipdrm driver is probed and a second framebuffer registered by the DRM fbdev emulation layer but the frame buffer console is still bound to the first frame buffer, that’s using the EFI GOP but this gets destroyed when the kernel re-initializes the display controller and related IP blocks (IOMMU, clocks, power domains, etc).

So why are users left with a blank framebuffer? It’s because the framebuffer is registered but it’s not attached to the console.

Once the problem was understood, it was easy to solve it. The DRM subsystem provides a drm_aperture_remove_framebuffers() helper function to remove any existing drivers that may own the framebuffer memory, but the rockchipdrm driver was not using this helper.

The proposed fix (that landed in v5.14-rc1) then is for the rockchipdrm driver to call the helper to detach any existing early framebuffer before registering its own.

After doing that, the early framebuffer is unbound from the framebuffer console and the one registered by the rockchipdrm driver takes over:

[   40.752420] fb0: switching to rockchip-drm-fb from EFI VGA

The curious case of the ghostly modalias

I was finishing my morning coffee at the Fedora ARM mystery department when a user report came into my attention: the tpm_tis_spi driver was not working on a board that had a TPM device connected through SPI.

There was no /dev/tpm0 character device present in the system, even when the driver was built as a module and the Device Tree (DT) passed to the kernel had a node with a "infineon,slb9670" compatible string.

Peter Robinson chimed in and mentioned that he had briefly looked at this case before. The problem, he explained, is that the module isn’t auto-loaded but that manually loading it make things to work.

At the beginning he thought that this was just a common issue of a driver not having module alias information. This would lead to kmod not knowing that the module has to be loaded, when the kernel reported a MODALIAS uevent as a consequence of the SPI device being registered.

But when checking the module to confirm that theory, he found that there were alias entries:

$ modinfo drivers/char/tpm/tpm_tis_spi.ko | grep alias
alias:          of:N*T*Cgoogle,cr50C*
alias:          of:N*T*Cgoogle,cr50
alias:          of:N*T*Ctcg,tpm_tis-spiC*
alias:          of:N*T*Ctcg,tpm_tis-spi
alias:          of:N*T*Cinfineon,slb9670C*
alias:          of:N*T*Cinfineon,slb9670
alias:          of:N*T*Cst,st33htpm-spiC*
alias:          of:N*T*Cst,st33htpm-spi
alias:          spi:cr50
alias:          spi:tpm_tis_spi
alias:          acpi*:SMO0768:*

Since the board uses DT to describe the hardware topology, the TPM device should had been registered by the Open Firmware (OF) subsystem. And should cause the kernel to report a "MODALIAS=of:NspiTCinfineon,slb9670", which should had matched the "of:N*T*Cinfineon,slb9670" module alias entry.

But when digging more on this issue, things started to get more strange. Looking at the uevent sysfs entry for this SPI device, he found that the kernel was not reporting an OF modalias but instead a legacy SPI modalias: "MODALIAS=spi:slb9670".

But how come? a user asked, the device is registered using DT, not platform code! Where is this modalias coming from? Is this legacy SPI device a ghost?

Peter said that he didn’t believe in paranormal events and that there should be a reasonable explanation. So armed with grep, he wanted to get to the bottom of this but got preempted by more urgent things to do.

Coincidentally, I had chased down that same ghost before many moons ago. And it’s indeed not a spirit from the board files dimension but only an incorrect behavior in the uevent logic of the SPI subsystem.

The reason is that the SPI uevent handler always reports a MODALIAS of the form "spi:foobar" even for devices that are registered through DT. This leads to the situation described above and it’s better explained by looking at the SPI subsystem code:

static int spi_uevent(struct device *dev, struct kobj_uevent_env *env)
{
	const struct spi_device		*spi = to_spi_device(dev);
	int rc;

	rc = acpi_device_uevent_modalias(dev, env);
	if (rc != -ENODEV)
		return rc;

	return add_uevent_var(env, "MODALIAS=%s%s", SPI_MODULE_PREFIX, spi->modalias);
}

Conversely, this is what the platform subsystem uevent handler does (which properly reports OF module aliases):

static int platform_uevent(struct device *dev, struct kobj_uevent_env *env)
{
	struct platform_device	*pdev = to_platform_device(dev);
	int rc;

	/* Some devices have extra OF data and an OF-style MODALIAS */
	rc = of_device_uevent_modalias(dev, env);
	if (rc != -ENODEV)
		return rc;

	rc = acpi_device_uevent_modalias(dev, env);
	if (rc != -ENODEV)
		return rc;

	add_uevent_var(env, "MODALIAS=%s%s", PLATFORM_MODULE_PREFIX,
			pdev->name);
	return 0;
}

Fixing the SPI core would be trivial, but the problem is that there are just too many drivers and Device Trees descriptions that are relying on the current behavior.

It should be possible to change the core, but first all these drivers and DTs have to be fixed. For example, the I2C subsystem had the same issue but has already been resolved.

A workaround then in the meantime could be to add to the legacy SPI device ID table all the entries that are found in the OF device ID table. That way, a platform using for example a DT node with compatible "infineon,slb9670" will match against an alias "spi:slb9670", that will be present in the module.

And that’s exactly what the proposed fix for the tpm_tis_spi driver does.

$ modinfo drivers/char/tpm/tpm_tis_spi.ko | grep alias
alias:          of:N*T*Cgoogle,cr50C*
alias:          of:N*T*Cgoogle,cr50
alias:          of:N*T*Ctcg,tpm_tis-spiC*
alias:          of:N*T*Ctcg,tpm_tis-spi
alias:          of:N*T*Cinfineon,slb9670C*
alias:          of:N*T*Cinfineon,slb9670
alias:          of:N*T*Cst,st33htpm-spiC*
alias:          of:N*T*Cst,st33htpm-spi
alias:          spi:cr50
alias:          spi:tpm_tis_spi
alias:          spi:slb9670
alias:          spi:st33htpm-spi
alias:          acpi*:SMO0768:*

Until the next mystery!

A lethal spurious interrupt

A big part of my work on Fedora/RHEL is to troubleshoot and do root cause analysis across the software stack. Because many of these projects are decades old, this usually feels like being stuck somewhere between being an archaeologist and a detective.

Many bugs are boring but some are interesting, either because the investigation made me learn something new or due to the amount of effort that was sunk into figuring out the problem. So I thought that it would be a nice experiment to share a little about the ones that are worth mentioning. This is the first of such posts, I may write more in the future if have time and remember to do it:

It was a dark and stormy night when Peter Robinson mentioned a crime to me, a Rockpro64 board was found to not boot when the CONFIG_PCIE_ROCKCHIP_HOST option was enabled. He also already had found the criminal, it was the CONFIG_DEBUG_SHIRQ option.

I have to admit that I only knew CONFIG_DEBUG_SHIRQ by name and that it was a debug option for shared interrupts, but didn’t even know what this option was about. So the first step was to read the help text of the Kconfig symbol to learn more on this option.

Enable this to generate a spurious interrupt just before a shared interrupt handler is deregistered (generating one when registering is currently disabled). Drivers need to handle this correctly. Some don’t and need to be caught.

This was the smoking gun: a spurious interrupt!

We now knew what was the weapon used but we still had questions, why would triggering an interrupt lead to the board being hung? The next step then was to figure out where exactly this was happening, it certainly would have to be somewhere in the driver’s IRQ handler code path.

By looking at the pcie-rockchip-host driver code, we see two IRQ handlers registered: rockchip_pcie_subsys_irq_handler() for the "pcie-sys" IRQ and rockchip_pcie_client_irq_handler() for the "pcie-client" IRQ.

Adding some debug printouts to both would show us that the issue was happening in the latter, when calling to rockchip_pcie_read(). This function just hangs indefinitely and never returns.

Peter wrote in the filled bug that this issue was reported before and there was even an RFC patch posted by a Rockchip engineer, who mentioned the assessment of the problem:

With CONFIG_DEBUG_SHIRQ enabled, the irq tear down routine
would still access the irq handler register as a shared irq.
Per the comment within the function of __free_irq, it says
"It’s a shared IRQ — the driver ought to be prepared for
an IRQ event to happen even now it’s being freed". However
when failing to probe the driver, it may disable the clock
for accessing the register and the following check for shared
irq state would call the irq handler which accesses the register
w/o the clk enabled. That will hang the system forever.

The proposed solution was to check in the rockchip_pcie_read() function if a rockchip->hclk_pcie clock was enabled before trying to access the PCIe registers’ address space. But that wasn’t accepted because it was solving the symptom and not the cause.

But it did confirm our findings, that the problem was an IRQ handler being called before it was expected and that the PCIe register access hangs due to a clock not being enabled.

With all of that information and reading once more the pcie-rockchip-host driver code, we could finally reconstruct the crime scene:

  1. "pcie-sys" IRQ is requested and its handler registered.
  2. "pcie-client" IRQ is requested and its handler registered.
  3. probe later fails due to readl_poll_timeout() returning a timeout.
  4. the "pcie-sys" IRQ handler is unregistered.
  5. CONFIG_DEBUG_SHIRQ triggers a spurious interrupt.
  6. "pcie-client" IRQ handler is called for this spurious interrupt.
  7. IRQ handler tries to read PCIE_CLIENT_INT_STATUS with clocks gated.
  8. the machine hangs because rockchip_pcie_read() call never returns.

The root cause of the problem then is that the IRQ handlers are registered too early, before all the required resources have been properly set up.

Our proposed solution then is to move all the IRQ initialization into a later stage of the probe function. That makes it safe for the IRQ handlers to be called as soon as they are registered.

Until the next mystery!

Automatic LUKS volumes unlocking using a TPM2 chip

I joined Red Hat a few months ago, and have been working on improving the Trusted Platform Module 2.0 (TPM2) tooling, towards having a better TPM2 support for Fedora on UEFI systems.

For brevity I won’t explain in this post what TPMs are and their features, but assume that readers are already familiar with trusted computing in general. Instead, I’ll explain what we have been working on, the approach used and what you might expect on Fedora soon.

For an introduction to TPM, I recommend Matthew Garret’s excellent posts about the topic, Philip Tricca’s presentation about TPM2 and the official Trusted Computing Group (TCG) specifications. I also found “A Practical Guide to TPM 2.0” book to be much easier to digest than the official TCG documentation. The book is an open access one, which means that’s freely available.

LUKS volumes unlocking using a TPM2 device

Encryption of data at rest is a key component of security.  LUKS provides the ability to encrypt Linux volumes, including both data volumes and the root volume containing the OS. The OS can provide the crypto keys for data volumes, but something has to provide the key for the root volume to allow the system to boot.

The most common way to provide the crypto key to unlock a LUKS volume,  is to have a user type in a LUKS pass-phase during boot. This works well for laptop and desktop systems, but is not well suited for servers or virtual machines since is an obstacle for automation.

So the first TPM feature we want to add to Fedora (and likely one of the most common use cases for a TPM) is the ability to bind a LUKS volume master key to a TPM2. That way the volume can be automatically unlocked (without typing a pass-phrase) by using the TPM2 to obtain the master key.

A key point here is that the actual LUKS master key is not present in plain text form on the system, it is protected by TPM encryption.

Also, by sealing the LUKS master key with a specific set of Platform Configuration Registers (PCR), one can make sure that the volume will only be unlocked if the system has not been tampered with. For example (as explained in this post), PCR7 is used to measure the UEFI Secure Boot policy and keys. So the LUKS master key can be sealed against this PCR, to avoid unsealing it if Secure Boot was disabled or the used keys were replaced.

Implementation details: Clevis

Clevis is a plugable framework for automated decryption that has a number of “pins”, where each pin implements an {en,de}cryption support using a different backend. It also has a command line interface to {en,de}crypt data using these pins, create complex security policies and bind a pin to a LUKS volume to later unlock it.

Clevis relies on the José project, which is an C implementation of the Javascript Object Signing and Encryption (JOSE) standard. It also uses the LUKSMeta project to store a Clevis pin metadata in a LUKS volume header.

On encryption, a Clevis pin takes some data to encrypt and a JSON configuration to produce a JSON Web Encryption (JWE) content. This JWE has the data encrypted using a JSON Web KEY (JWK) and information on how to obtain the JWK for decryption.

On decryption, the Clevis pin obtains a JWK using the information provided by a JWE and decrypts the ciphertext also stored in the JWE using that key.

Each Clevis pin defines their own JSON configuration format, how the JWK is created, where is stored and how to retrieve it.

As mentioned, Clevis has support to bind a pin with a LUKS volume. This means that a LUKS master key is encrypted using a pin and the resulting JWE is stored in a LUKS volume meta header. That way Clevis is able to later decrypt the master key and unlock the LUKS volume. Clevis has dracut and udisks2 support to do this automatically and the next version of Clevis will also include a command line tool to unlock non-root (data) volumes.

Clevis TPM2 pin

Clevis provides a mechanism to automatically supply the LUKS master key for the root volume. The initial implementation of Clevis has support to obtain the LUKS master key from a network service, but we have extended Clevis to take advantage of a TPM2 chip, which is available on most servers, desktops and laptops.

By using a TPM, the disk can only be unlocked on a specific system – the disk will neither boot nor be accessed on another machine.

This implementation also works with UEFI Secure Boot, which will prevent the system from being booted if the firmware or system configuration has been modified or tampered with.

To make use of all the Clevis infrastructure and also be able to use the TPM2 as a part of more complex security policies, the TPM2 support was implemented as a clevis tpm2 pin.

On encryption the tpm2 pin generates a JWK, creates an object in the TPM2 with the JWK as sensitive data and binds the object (or seals if a PCR set is defined in the JSON configuration) to the TPM2.

The generated JWE contains both the public and wrapped sensitive portions of the created object, as well as information on how to unseal it from the TPM2 (hashing and key encryption algorithms used to recalculate the primary key, PCR policy for authentication, etc).

On decryption the tpm2 pin takes the JWE that contains both the sealed object and information on how to unseal it,  loads the object into the TPM2 by using the public and wrapped sensitive portions and unseals the JWK to decrypt the ciphertext stored in the JWE.

The changes haven’t been merged yet, since the pin is using features from tpm2-tools master so we have to wait for the next release of the tools. And also there are still discussions on the pull request about some details, but it should be ready to land soon.

Usage

The Clevis command line tools can be used to encrypt and decrypt data using a TPM2 chip. The tpm2 pin has reasonable defaults but one can configure most of its parameters using the pin JSON configuration (refer to the Clevis tpm2 pin documentation for these), e.g:

$ echo foo | clevis encrypt tpm2 '{}' > secret.jwe

And then the data can later be decrypted with:

$ clevis decrypt < secret.jwe
foo

To seal data against a set of PCRs:

$ echo foo | clevis encrypt tpm2 '{"pcr_ids":"8,9"}' > secret.jwe

And to bind a tpm2 pin to a LUKS volume:

$ clevis luks bind -d /dev/sda3 tpm2 '{"pcr_ids":"7"}'

The LUKS master key is not stored in raw format, but instead is wrapped with a JWK that has the same entropy than the LUKS master key. It’s this JWK that is sealed with the TPM2.

Since Clevis has both dracut and udisks2 hooks, the command above is enough to have the LUKS volume be automatically unlocked using the TPM2.

The next version of Clevis also has a clevis-luks-unlock command line tool, so a LUKS volume could be manually unlocked with:

$ clevis luks unlock -d /dev/sda3

Using the TPM2 as a part of more complex security policies

One of Clevis supported pins is the Shamir Shared Secret (SSS) pin, that allows to encrypt a secret using a JWK that is then split into different parts. Each part is then encrypted using another pin and a threshold is chose to decide how many parts are needed to reconstruct the encryption key, so the secret can be decrypted.

This allows for example to split the JWK used to wrap the LUKS mater key in two parts. One part of the JWK could be sealed with the TPM2 and another part be stored in a remote server. By sealing a JWK that’s only one part of the needed key to decrypt the LUKS master key, an attacker obtaining the data sealed in the TPM won’t be able to unlock the LUKS volume.

The Clevis encrypt command for this particular example would be:

$ clevis luks bind -d /dev/sda3 sss '{"t": 2, "pins": \
  {"http":{"url":"http://server.local/key"}, "tpm2": \
  {"pcr_ids":"7"}}}'

Limitations of this approach

One problem with the current implementation is that Clevis is a user-space tool and so it can’t be used to unlock a LUKS volume that has an encrypted /boot directory. The boot partition still needs to remain unencrypted so the bootloader is able to load a Linux kernel and an initramfs that contains Clevis, to unlock the encrypted LUKS volume for the root partition.

Since the initramfs is not signed on a Secure Boot setup, an attacker could replace the initramfs and unlock the LUKS volume. So the threat model meant to protect is for an attacker that can get access to the encrypted volume but not to the trusted machine.

There are different approaches to solve this limitation. The previously mentioned post from Matthew Garret suggests to have a small initramfs that’s built into the signed Linux kernel. The only task for this built-in initramfs would be to unseal the LUKS master key, store it into the kernel keyring and extend PCR7 so the key can’t be unsealed again. Later the usual initramfs can unlock the LUKS volume by using the key already stored in the Linux kernel.

Another approach is to also have the /boot directory in an encrypted LUKS volume and provide support for the bootloader to unseal the master key with the TPM2, for example by supporting the same JWE format in the LUKS meta header used by Clevis. That way only a signed bootloader would be able to unlock the LUKS volume that contains /boot, so an attacker won’t be able to tamper the system by replacing the initramfs since it will be in an encrypted partition.

But there is work to be done for both approaches, so it will take some time until we have protection for this threat model.

Still, having an encrypted root partition that is only automatically unlocked on a trusted machine has many use cases. To list a few examples:

  • Stolen physical disks or virtual machines images can’t be mounted on a different machine.
  • An external storage media can be bind to a set of machines, so it can be automatically unlocked only on trusted machines.
  • A TPM2 chip can be reset before sending a laptop to repair, that way the LUKS volume can’t be automatically unlocked anymore.
  • An encrypted volume can be bound to a TPM2 if there is no risk of someone having physical access to the machine but unbound again when there is risk. So the machine can be automatically unlocked on safe places but allow to require a pass-phrase on unsafe places.

Acknowledgements

I would like to thanks Nathaniel McCallum and Russell Doty for their feedback and suggestions for this article.