How to install Fedora on an HP X2 Chromebook

We have been working lately with Enric Balletbo and Dorinda Bassey to improve the support for the HP X2 Chromebook in Fedora. This post explains how to install Fedora on that Chromebook.

The article ended being up being longer than I thought, so for the impatient this is the summary:

  • Switch the Chromebook to Developer Mode.

  • Boot from the internal disk.

  • Go to a virtual terminal with Ctrl+Alt+F2 and login as root.

  • Enable developer mode boot from external disk (USB/microSD):

      $ crossystem dev_boot_usb=1
    
  • Install packages needed by the chromebook-setup.sh script:

      $ sudo dnf install bc curl util-linux gdisk lz4 e2fsprogs \
        uboot-tools udisks2 vboot-utils guestfs-tools
    
  • Clone Enric’s chromebooks scripts repo:

      $ git clone https://github.com/eballetbo/chromebooks.git
      $ pushd chromebooks
    
  • Flash a Fedora image to a storage media (replace /dev/sda with your block device):

      $ sudo ./chromebook-setup.sh deploy_fedora \
        --architecture=arm64 --storage=/dev/sda \
        --kparams="clk_ignore_unused deferred_probe_timeout=30"
    
  • Plug the USB/microSD device into the Chromebook and choose to boot from an external device.

  • After the Fedora initial setup, install the following packages:

      $ sudo dnf install uboot-tools vboot-utils lz4 -y
    
  • Remove packages that expects the grub2 bootloader to be used:

      $ sudo dnf remove kexec-tools -y
    
  • Enjoy Fedora on the Chromebook 🙂

Now the longer version…

Challenges supporting the Chromebooks in Fedora

Supporting the Chromebooks is not trivial, because these laptops use a different firmware (Coreboot) and boot stack (Depthcharge) than what is used by all the other aarch64 machines supported by Fedora (UEFI and GRUB). There are good reasons why that is the case, but it poses some challenges and complicates making these laptops to work in Fedora out-of-the-box.

For this reason, a standard Fedora ISO image can’t just be booted on a Chromebook to start an installation process.

Current approach used to install Fedora on the Chromebooks

To overcome this, Enric wrote a set of chromebook scripts that can be used to setup a block device (i.e: USB drive or microSD card) and write a Fedora image that can be booted directly. The script also adds to the system a kernel-install plugin, written by Dorinda, that takes the Linux kernel image being installed and package it in the format (FIT) expected by the Chromebook bootloader.

That way, the Fedora installation would look and behave exactly the same than in any other system.

In the future support for Chromebooks might be added to the Anaconda OS installer used by Fedora, but in the meantime using this script allows us to do experimentation and make further customization that wouldn’t be suitable for a general system installer.

Following are the instructions to install Fedora on a HP X2 Chromebook using the chromebook-setup.sh script.

Switch the Chromebook to Developer Mode

In the default mode the Chromebook can only boot binaries that are trusted by its firmware. This means that nothing besides ChromeOS can be installed. To boot a different OS, the mode should be change to Developer. In this mode, any binary that is signed with the Google’s developer key can be booted. That key is available for anyone so is what is used to sign the Linux images when generating the FIT images during the Fedora kernel packages installations.

The ChromiumOS project has excellent articles explaining the Developer Mode and how to enable it on Chromebooks without a physical keyboard, such as the HP X2.

After enabling developer mode, boot from the internal disk and switch to a virtual terminal with Ctrl+Alt+F2. Then login as root and execute the following to enable booting from an external disk (USB/microSD):

$ crossystem dev_boot_usb=1

Flashing a Fedora image

The chromebook-setup.sh script can be used to flash fedora images to a block device and do all the needed setup to make it bootable.

It supports many different options but it also has reasonable defaults. The list of options can be listed with ./chromebook-setup --help.

Following are the steps to flash a Fedora image.

Install packages needed by the chromebook-setup.sh script:

$ sudo dnf install bc curl util-linux gdisk lz4 e2fsprogs \
  uboot-tools udisks2 vboot-utils guestfs-tools

Clone Enric’s chromebooks scripts repository:

$ git clone https://github.com/eballetbo/chromebooks.git
$ pushd chromebooks

Execute the script, for example:

$ sudo ./chromebook-setup.sh deploy_fedora \
  --architecture=arm64 --storage=/dev/sda \
  --kparams="clk_ignore_unused deferred_probe_timeout=30"

The deploy_fedora option will install a Fedora image in the specified storage media. By default the latest Fedora Workstation Rawhide image will downloaded and used, but a different image can be chosen using the --image=$image option.

The --architecture and --storage options specify the architecture and block device used respectively.

Finally, the --kparams option allows to set additional kernel command line parameters.

The clk_ignore_unused parameter is currently needed because there is a bug when the MSM/snapdragon DRM driver is built as a module. Some needed clocks are gated before the driver probe function is executed, causing it to fail.

And the deferred_probe_timeout=30 is needed because there are a lot of drivers probe deferrals and the default 10 seconds expires causing drivers to fail to probe due timeouts.

Hopefully these two issues would be fixed soon and the parameters won’t be needed anymore.

One the script finishes flashing the image, plug the USB drive or insert the microSD in the Chromebook and choose "Boot from external media".

The system should boot and start the Fedora initial setup program to configure the system and create a user. Once that is done, start a terminal and install the following packages needed by the kernel-install Chromebook plugin:

$ sudo dnf install uboot-tools vboot-utils lz4 -y

Remove packages that expects the grub2 bootloader to be used:

$ sudo dnf remove kexec-tools -y

And that’s it. Now the Fedora should behave like in any other system. If there are any bugs, please file issues in the chromebooks scripts repository.

Happy hacking!

Booting Fedora with sd-boot and Secure Boot enabled

Every once in a while there is a discussion in the Fedora development list about replacing the default GRUB bootloader. That latest of these threads was "future of dual booting Windows and Fedora, redux" about a month ago, but searching for GRUB in the mailing list archives one can find many similar conversations.

One of the options to replace GRUB that is always mentioned is systemd-boot (sd-boot), since is quite minimal and enough for simple booting needs such as what is required for Fedora Workstation.

But currently this can’t be done without additional work, because shim only supports using GRUB as a second stage loader. There is a RFE: add support for multiple second stage loaders #472 filed for shim, but there wasn’t a conclusion on that issue about how to move forward.

This means that to use sd-boot instead of GRUB and keeping Secure Boot enabled, users need to create their own key pairs and enroll a cert with the public key into the UEFI firmware db database.

To have an idea of what that would entail, I installed sd-boot and removed shim and GRUB in one of my machines. Below are the steps I followed in case someone wants to replicate it.

To keep the example generic, I mention how a public key can be enrolled with the Open Virtual Machine Firmware (OVMF) UEFI firmware used to boot virtual machines. But a similar procedure can be followed to enroll keys using the UEFI settings of a physical machine. Refer to your hardware vendor documentation for how to do this.

1. Create a key pair to sign the binaries and enroll into the UEFI firmware db key database

$ cat << EOF > configuration_file.config
[ req ]
default_bits = 4096
distinguished_name = req_distinguished_name
prompt = no
string_mask = utf8only
x509_extensions = myexts

[ req_distinguished_name ]
O = Organization
CN = Organization signing key
emailAddress = E-mail address

[ myexts ]
basicConstraints=critical,CA:FALSE
keyUsage=digitalSignature
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid
EOF

$ openssl req -x509 -new -nodes -utf8 -sha256 -days 36500 -batch -config \
configuration_file.config -outform DER -out public_key.der -keyout private_key.priv

$ openssl x509 -in public_key.der -inform DER -outform PEM -out public_key.pem

2. Enroll the public cert in the UEFI firmware db database

$ sudo cp public_key.der /boot/efi/

On reboot, enter into the UEFI firmware settings and go to:

Device Manager
    Secure Boot Configuration
        Secure Boot Mode 

That will allow to enroll a key by choosing:

Custom Secure Boot Options
    DB options
        Enroll Signature
            Enroll Signature Using File

After that, when booting the enrolled key should be present in db and added to the Linux kernel .platform keyring:

$ mokutil --test-key public_key.der 
public_key.der is already in db

$ keyctl list %:.platform
3 keys in keyring:
577585465: ---lswrv     0     0 asymmetric: Microsoft Windows Production PCA 2011: a92902398e16c49778cd90f99e4f9ae17c55af53
137367868: ---lswrv     0     0 asymmetric: Organization signing key: ae6f49657c238754eedc503e98cb97afdb064706
984007323: ---lswrv     0     0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4

3. Install sd-boot and remove shim and grub2 packages

$ sudo bootctl install
$ sudo mkdir /boot/efi/$(cat /etc/machine-id)
$ sudo rm /etc/dnf/protected.d/{shim,grub2}*.conf
$ sudo dnf remove shim* grub2* -y
$ sudo rm -r /boot/loader
$ sudo dnf reinstall kernel-core -y

4. Sign the sd-boot and Linux kernel binaries

$ sudo dnf install -y sbsigntools pesign

$ sudo sbsign --key private_key.priv --cert public_key.pem  /usr/lib/systemd/boot/efi/systemd-bootx64.efi \
--output /boot/efi/EFI/systemd/systemd-bootx64.efi

$ sudo sbsign --key private_key.priv --cert public_key.pem /lib/modules/$(uname -r)/vmlinuz \
--output /boot/efi/$(cat /etc/machine-id)/$(uname -r)/linux
Image was already signed; adding additional signature

Check that the binaries have been signed correctly:

$ sudo pesign -S -i /boot/efi/EFI/systemd/systemd-bootx64.efi
---------------------------------------------
certificate address is 0x7f348c12b1c8
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is Test signing key
The signer's email address is e-mail address
Signing time: Thu Sep 01, 2022
There were certs or crls included.
---------------------------------------------

$ sudo pesign -S -i /boot/efi/$(cat /etc/machine-id)/$(uname -r)/linux
---------------------------------------------
certificate address is 0x7fb81a4b11e8
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is Fedora Secure Boot Signer
No signer email address.
Signing time: Thu Apr 28, 2022
There were certs or crls included.
---------------------------------------------
certificate address is 0x7fb81a4b1b30
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is kernel-signer
No signer email address.
Signing time: Thu Apr 28, 2022
There were certs or crls included.
---------------------------------------------
certificate address is 0x7fb81a4b26f8
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is Organization signing key
The signer's email address is e-mail address
Signing time: Thu Sep 01, 2022
There were certs or crls included.
---------------------------------------------

On reboot the machine should be able to boot the signed sd-boot and Linux kernel with Secure Boot enabled:

$ mokutil --sb-state
SecureBoot enabled

5. Sign an out-of-tree kernel module

This step is mentioned for completeness, since some users need to load out-of-tree kernel modules. And this is only possible for signed modules when Secure Boot is enabled.

If that’s needed, following is an example on how to do it:

$ sudo dnf install kernel-devel -y

$ git clone https://github.com/maK-/SimplestLKM.git

$ pushd SimplestLKM

$ make

$ /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 key-private_key.priv \
public_key.der hello.ko

$ modinfo hello.ko | grep signature
signature: 91:F7:20:16:1C:85:AC:DC:54:25:C2:B2:E9:ED:02:93:79:43:1D:7F:

$ insmod hello.ko

$ dmesg
[ 1653.929819] hello: loading out-of-tree module taints kernel.
[ 1653.930435] Hello world!

A pain point with this setup is that one need to manually sign the sd-boot and Linux kernel binaries each time that those are updated. This could be automated of course but that would require to have the private key in the filesystem and that would defeat the whole purpose of the Secure Boot protection.

Since in that case an attacker that gets root access could be able to sign any binaries they want and make their attack persistent across reboots.

For this reason what I’ve been done is keeping the private key in an external media and only attach it when there is a need to sign a new binary, which makes updating packages more inconvenient.

Hopefully this won’t be a problem anymore once the mentioned shim issue #472 gets resolved. Because then the sd-boot binary could get signed during package build with the Fedora key just like GRUB and there won’t be a need to sign neither sd-boot nor the Linux kernel binaries with a custom key.

And that’s all for today. Happy hacking!

Using an I2C SSD1306 OLED on Fedora with a Raspberry Pi

Linux 5.18 version landed in Fedora 36 and it includes a new ssd130x DRM driver for the Solomon OLED display controllers.

One of the supported devices is SSD1306, which seems to be a quite popular display controller for small and cheap (I bought a pack of 3 on Amazon for less than 15€) monochrome OLED panels.

I do a lot of development and testing on RPi boards and found that these small OLED panels are useful to get console output without the need to either connect a HDMI monitor or a serial console.

This blog post explains how to setup Fedora 36 to use a SSD1306 OLED connected through I2C.

First you need connect the SSD1306 display to the RPi, Adrafruit has an excellent article on how to do that.

Then you need to install Fedora 36, this can be done by downloading an aarch64 Fedora raw image (e.g: Workstation Edition) and executing the following command:

sudo arm-image-installer --image Fedora-Workstation-36-1.5.aarch64.raw.xz --target=rpi4 --media=$device --addconsole --addkey=id_rsa.pub --norootpass --resizefs

where $device is the block device for your uSD (e.g: /dev/sda).

Fedora 36 was released with Linux 5.17, but as mentioned the ssd130x DRM driver landed in 5.18, so you need to update the Fedora kernel:

sudo dnf update kernel -y

Finally you need to configure the RPi firmware to enable the I2C1 pins and load the ssd1306 Device Tree Blob Overlay (dtbo) to register a SSD1306 I2C device, e.g:

sudo cat << EOF >> /boot/efi/config.txt
dtparam=i2c1=on
dtoverlay=ssd1306,inverted
EOF

The ssd1306.dtbo supports many parameters in case your display doesn’t match the defaults. Take a look to the RPi overlays README for more details about all these parameters and their possible values.

The ssd130x DRM driver registers an emulated fbdev device that can be bound with fbcon and use the OLED display to have a framebuffer console. If you want to do that, it’s convenient to change the virtual terminal console font to a smaller one, e.g:

sudo sed -i 's/FONT=.*/FONT="drdos8x8"/' /etc/vconsole.conf
sudo dracut -f

And that’s it. If your SSD1306 uses SPI instead, this is supported since Linux 5.19 but that would be the topic for another post.

Happy hacking!

How to troubleshoot deferred probe issues in Linux

When working on the retro handheld console mentioned in a previous post, I had an issue where the LCD driver was not probed when booting a custom Linux kernel image built.

To understand the problem, first some knowledge is needed about how devices and drivers are registered in the Linux kernel, how these two sets are matched (bound) and what is a probe deferral.

If you are not familiar with these concepts, please read this post where are explained in detail.

The problem is that the st7735r driver (that’s needed for the SPI TFT LCD panel I was using) requires a GPIO based backlight device. To make it more clear, let’s look at the relevant bits in the adafruit-st7735r-overlay.dts that’s used as a Device Tree Blob (DTB) overlay to register the needed devices:

fragment@2 {
...
    af18_backlight: backlight {
        compatible = "gpio-backlight";
...
    };
};

fragment@3 {
...
    af18: adafruit18@0 {
        compatible = "jianda,jd-t18003-t01";
        backlight = <&af18_backlight>;
...
    };
};

We see that the adafruit18@0 node for the panel has a backlight property whose value is a phandle to the af18_backlight label used to refer to the backlight node.

The drivers/gpu/drm/tiny/st7735r.c probe callback then uses the information in the DTB to attempt getting a backlight device:

static int st7735r_probe(struct spi_device *spi)
{
...
	dbidev->backlight = devm_of_find_backlight(dev);
	if (IS_ERR(dbidev->backlight))
		return PTR_ERR(dbidev->backlight);
...
}

The devm_of_find_backlight() function returns a pointer to the backlight device if this could be found or a -EPROBE_DEFER error pointer if there is a backlight property defined in the DTB but this could not be found.

For example, this can happen if the driver that registers that expected backlight device was not yet probed.

If the probe callback returns -EPROBE_DEFER, the kernel then will put the device that matched the driver but failed to probe in a deferred probe list. The list is iterated each time that a new driver is probed (since it could be that the newly probed driver registered the missing devices that forced the probe deferral).

My problem then was that the needed driver (CONFIG_BACKLIGHT_GPIO since the backlight node has compatible = gpio-backlight) was not enabled in my kernel, leading to the panel device to remain in the deferred probe list indefinitely due a missing backlight device that was never registered.

This is quite a common issue on Device Tree based systems and something that it used to take me a lot of time to root cause when I started working on Linux embedded platforms.

A few years ago I added a /sys/kernel/debug/devices_deferred debugfs entry that would expose the list of devices deferred to user-space, which makes much more easier to figure out what devices couldn’t be bound due their driver probe being deferred.

Later, Andrzej Hajda improved that and added to the devices_deferred debugfs entry support to print the reason of the deferral.

So checking what devices were deferred and the reason is now quite trivial, i.e:

$ cat /sys/kernel/debug/devices_deferred 
spi0.0  spi: supplier backlight not ready

Much better than spending a lot of time looking at kernel logs and adding debug printous to figure out what’s going on.

Happy hacking!

Linux drivers and devices registration, matching, aliases and modules autoloading

Every once in a while I have to explain these concepts to someone so I thought that it could be something worth to write about.

Device drivers and devices registration

The Linux kernel documentation covers quite well how device drivers and devices are registered and how these two are bound. But the summary is that drivers and devices are registered independently and each of these specify their given bus type. The Linux kernel device model then uses that information to bind drivers with devices of the same bus type.

Drivers and devices are registered using the driver_register() function which is usually called from either the drivers’ module_init() function or platform code.

Devices are registered using the register_device() function which is usually called by subsystems that parses a list of devices from some hardware topology description, some enumerable bus or platform code that hardcodes the devices to be registered.

Drivers and device matching (binding)

When a driver is registered for a given bus, the list of devices registered for that bus is iterated to find a match.

In the same manner, when a device is registered, the list of drivers registered for the same bus is iterated to find a match.

That way, it doesn’t matter the order in which drivers and devices are registered. They a device will be bound to a driver regardless of which one was registered first.

Drivers’ probe callback

If a match is found, the driver’s probe callback is executed. This function handler contains the driver-specific logic to bind the driver with a device and any setup needed for this. The probe function returns 0 if the driver could be bound to the device successfully or a negative errno code if the driver was not able to bound the device.

Probe deferral

A special errno code -EPROBE_DEFER is used to indicate that the bound failed because a driver could not provide all the resources needed by the device. When this happens, the device is put into a deferred probe list and the probe is retried again at a later time.

That later time is when a new driver probes successfully. When this happens, the device deferred probe list is iterated again and all devices are tried to bind again with their matched driver.

If the newly probed driver provides a resource that was missing by drivers whose probe was deferred, then their probe will succeed this time and their bound devices will be removed from the deferred list.

If all required resources are provided at some point, then all drivers should probe correctly and the deferred list should become empty.

It is a simple and elegant (albeit inefficient) solution to the fact that drivers and devices registration are non-deterministic. This leads to drivers not having a way to know if a resource won’t ever be available or is just that the driver that would provide the resource has just not probed yet.

But even if the kernel probed the drivers in a deterministic order (i.e: by using device dependency information), the driver would have no way to know if for example the missing resource would be provided by a driver that was built as a kernel module and would be loaded much later by user-space or even manually by an operator.

Module device tables

Each driver provides information about what devices can be matched against and usually this information is provided on a per firmware basis. For example, a driver that supports devices registered using both ACPI and Device Tree hardware descriptions, will contain separate ID tables for the ACPI and OpenFirmware (OF) devices that can be matched.

To illustrate this, the drivers/input/touchscreen/hideep.c has the following device ID tables:

static const struct i2c_device_id hideep_i2c_id[] = {
	{ HIDEEP_I2C_NAME, 0 },
	{ }
};
MODULE_DEVICE_TABLE(i2c, hideep_i2c_id);

#ifdef CONFIG_ACPI
static const struct acpi_device_id hideep_acpi_id[] = {
	{ "HIDP0001", 0 },
	{ }
};
MODULE_DEVICE_TABLE(acpi, hideep_acpi_id);
#endif

#ifdef CONFIG_OF
static const struct of_device_id hideep_match_table[] = {
	{ .compatible = "hideep,hideep-ts" },
	{ }
};
MODULE_DEVICE_TABLE(of, hideep_match_table);
#endif

Module aliases

If information defined in these tables are exported using the MODULE_DEVICE_TABLE() macro, then these will be in the drivers kernel modules as alias entries in the module information.

For example, one can check the module aliases for a given module using the modinfo command, i.e:

$ modinfo drivers/input/touchscreen/hideep.ko | grep alias
alias:          i2c:hideep_ts
alias:          acpi*:HIDP0001:*
alias:          of:N*T*Chideep,hideep-tsC*
alias:          of:N*T*Chideep,hideep-ts

Here are listed the legacy I2C platform, ACPI and OF devices that are exported by MODULE_DEVICE_TABLE(i2c, hideep_i2c_id), MODULE_DEVICE_TABLE(acpi, hideep_acpi_id) and MODULE_DEVICE_TABLE(of, hideep_match_table) respectively.

Module autoloading

The module aliases information is only used by user-space, the kernel uses the actual device tables to match the driver with the registered devices. In fact, the MODULE_DEVICE_TABLE() is a no-op if the driver is built-in the kernel image and not built as a module.

The way this work is that when a device is registered for a bus type, the struct bus_type.uevent callback is executed and the bus driver reports a uevent to udev to take some actions. The uevent contains key-value pairs and one of them is the device MODALIAS.

For example, on my laptop when the PCI bus is enumerated and my GPU registered the following uevent MODALIAS will be sent (as shown by udevadm monitor -p):

KERNEL[189823.929341] add   /devices/pci0000:00/0000:00:02.0 (pci)                                                  
ACTION=add                                              
DEVPATH=/devices/pci0000:00/0000:00:02.0                                                                               
SUBSYSTEM=pci                                                                                                          
...
MODALIAS=pci:v00008086d00003EA0sv000017AAsd00002292bc03sc00i00                                                         
...

This information is then used by udev and pass to kmod to load the module if needed. It will do something like:

$ modprobe pci:v00008086d00003EA0sv000017AAsd00002292bc03sc00i00

Since mod{probe,info} can also take a module alias besides the module name. For exapmle, the following should tell the module that matches this alias:

$ modinfo pci:v00008086d00003EA0sv000017AAsd00002292bc03sc00i00 | grep ^name
name:           i915

This information is also present in sysfs, i.e:

$ cat /sys/devices/pci0000\:00/0000\:00\:02.0/uevent 
DRIVER=i915
PCI_CLASS=30000
PCI_ID=8086:3EA0
PCI_SUBSYS_ID=17AA:2292
PCI_SLOT_NAME=0000:00:02.0
MODALIAS=pci:v00008086d00003EA0sv000017AAsd00002292bc03sc00i00

$ /sys/devices/pci0000\:00/0000\:00\:02.0/modalias 
pci:v00008086d00003EA0sv000017AAsd00002292bc03sc00i00

In theory, drivers should only define device ID tables for the firmware interfaces that they support. That is, a driver that supports devices registered through let’s say ACPI should only need a struct acpi_device_id. And also the same table should be used to match the driver with a device and send the MODALIAS information. For example if a device was registered through OF, only the struct of_device_id should be used for both matching and module alias reporting.

But in practice things are more complicated and there are exceptions in some subsystems, although that’s a topic for another time since this post got already too long.

If you are curious about the possible pitfalls though, I wrote about a bug chased some time ago in Fedora where the cause was a driver not reporting the MODALIAS that one would expect.

Happy hacking!

Building a retro handheld console with Fedora and a RPi zero

I built a retro console for my kids some time ago and they asked me if they could have one but that was portable.

Now that I finished the retro handheld console, thought that could be useful to share how it was done in case others wanted to replicate.

Hardware

I used a Waveshare 128×128, 1.44inch LCD display HAT which is a good fit because it contains both a LCD display and GPIO keys that can be used as a gamepad.

The board is a HAT for the Raspberry Pi Zero 2W board, HATs are expansion boards whose connectors are compatible with the RPi Zero 2W pinout.

And that is all the hardware needed if the console will just be powered with a micro USB cable. But if the goal is to power it through a battery, then more components are needed. Instead of listing them here and explaining how to do that part, I will refer to this excellent guide that I followed.

Software

I just used a stock Fedora Server image for this project, no additional software was needed than what is already packaged in the distro.

The image can be flashed using the arm-image-installer tool. There is no support for the RPi Zero 2W but since is quite similar to the RPi3, that can just be used as the target instead, i.e:

sudo arm-image-installer --image=Fedora-Server-36-1.5.aarch64.raw.xz \
--target=rpi3 --media=/dev/$device --addkey=id_rsa.pub --norootpass --resizefs

Where $device is the block device for the uSD card used to install the OS.

Then I followed these steps:

  1. Boot the uSD card, go through Fedora initial setup, create a retroarch user and make it member of the video and input groups.

  2. Install a Libretro emulator (i.e: mGBA for Game Boy Advance) and the Retroarch frontend.

sudo dnf install libretro-mgba retroarch
  1. Create a user service to run the game.
$ mkdir -p ~/.config/systemd/user/

$ cat &lt; ~/.config/systemd/user/retroarch.service
&gt; [Unit]
Description=Start Doom

[Service]
ExecStart=retroarch -L /lib64/libretro/mgba_libretro.so /home/retroarch/roms/doom.zip
Restart=always

[Install]
WantedBy=default.target
EOF
  1. Enable the service and lingering for the retroarch user, the latter is needed to allow the service to start at boot even when the user was not logged in.
$ systemctl --user enable retroarch.service
$ sudo loginctl enable-linger retroarch
  1. Add RPi config snippets to support the LCD and GPIO keys.

Two Device Tree Blob Overlays (DTBO) are used to drive the SPI controller of the LCD panel and the GPIO keys. These are adafruit-st7735r.dtbo and gpio-key.dtbo.

The overlays support options that can be configured in the RPi config.txt file such as the pins used, display resolution, if the display has to be rotated, the input event code that has to be reported for each GPIO key and so on.

For the HAT mentioned above, the following has to be added to the /boot/efi/config.txt file:

# Enable SPI
dtparam=spi=on

# TFT LCD Hat
dtoverlay=adafruit-st7735r,128x128,dc_pin=25,reset_pin=27,led_pin=24,rotate=90

# GPIO keys configuration

# Directional pad (KEY_UP, KEY_LEFT, KEY_RIGHT, KEY_DOWN, KEY_ENTER)
dtoverlay=gpio-key,gpio=6,active_low=1,gpio_pull=up,label=UP,keycode=103
dtoverlay=gpio-key,gpio=5,active_low=1,gpio_pull=up,label=LEFT,keycode=105
dtoverlay=gpio-key,gpio=26,active_low=1,gpio_pull=up,label=RIGHT,keycode=106
dtoverlay=gpio-key,gpio=19,active_low=1,gpio_pull=up,label=DOWN,keycode=108
dtoverlay=gpio-key,gpio=13,active_low=1,gpio_pull=up,label=PRESS,keycode=28

# Buttons (KEY_X, KEY_C)
dtoverlay=gpio-key,gpio=21,active_low=1,gpio_pull=up,label=KEY_1,keycode=45
dtoverlay=gpio-key,gpio=20,active_low=1,gpio_pull=up,label=KEY_2,keycode=44

# Power (KEY_POWER), useful to power off the console
dtoverlay=gpio-key,gpio=16,active_low=1,gpio_pull=up,label=KEY_3,keycode=116
  1. Workaround a bug in the st7735r driver that prevents the module to be auto loaded.

There is a bug in the st7735r driver which causes the module to not be loading automatically. To workaround this issue, create a modules-load.d config snippet to force the module to be loaded:

$ echo st7735r | sudo tee /etc/modules-load.d/st7735r.conf

Unfortunately this is a quite common bug in SPI drivers. But for this particular driver the workaround should not be needed in the future since it was already fixed by this commit. But at the time of this writing, the Fedora version used (36) still does not contain the fix.

  1. Prevent the simpledrm driver to be initialized.

The firmware seems to add a "simple-framebuffer" Device Tree node even when there is no monitor connected in its mini HDM port. This leads to the simpledrm driver to be probed, so it needs to be denied listed using a kernel command line parameter:

$ sudo grubby --update-kernel=DEFAULT \
--args=initcall_blacklist=simpledrm_platform_driver_init
  1. Update the adafruit-st7735r.dtbo to the latest version.

This was the only change I needed in Fedora 36. The issue is that the adafruit-st7735r.dtbo overlay is for the legacy fb_st7735r driver instead of the DRM st7735r driver. The latter has a different Device Tree node property to specify the display rotation and so the rotate=90 option specified in the config.txt file will be ignored.

$ wget https://github.com/raspberrypi/firmware/raw/master/boot/overlays/adafruit-st7735r.dtbo
$ sudo mv adafruit-st7735r.dtbo /boot/efi/overlays/

Happy gaming!

Fedora 36: A brave new (DRM/KMS only) world

The upcoming Fedora release includes a change proposed by Peter Robinson and yours truly, that disables all the Frame Buffer Device (fbdev) drivers in the Linux kernel package.

This means that starting from Fedora 36, only Direct Rendering Manager (DRM) / Kernel Mode Settings (KMS) drivers will be used. No more legacy stuff.

Many Linux developers have been arguing for over a decade now that fbdev is deprecated and it has been almost 7 years since the previous fbdev maintainer said that no new fbdev drivers would be added to the kernel. So this feels like a change long overdue.

This was possible thanks to the work of Thomas Zimmermann @ SUSE, who added the simpledrm driver, that exposes the system framebuffer set-up by the firmware as a DRM device.

Using the simpledrm driver, a framebuffer would be available even if the platform does not have a DRM driver or the user disables it using the nomodeset kernel command-line parameter.

Before this change, the video drivers used when DRM drivers were disabled with the nomodeset parameter, where either efifb (for systems booted with EFI) or vesafb (for systems booted with legacy BIOS and a VESA video mode selected on boot using the vga kernel command line parameter).

The need to rely on the system framebuffer initialized by the firmware was the primary reason why these fbdev drivers were still enabled in the Fedora kernel.

But the simpledrm replaces both, since is able to expose any system framebuffer, as long as platform code pass it as a proper I/O memory region for the driver to use it.

That includes not only the EFI system framebuffer provided through the Graphics Output Protocol (GOP) and the VESA framebuffer configured with the vga parameter, but also framebuffers whose information is passed to the kernel through other hardware description mechanisms. For example, as a "simple-framebuffer" Device Tree node, which is common on aarch64 devices.

Even when no fbdev drivers are used anymore, the fbdev subsystem is still enabled. This is needed because the The Framebuffer Console (fbcon) needs an fbdev device to bind. But fortunately the DRM subsystem provides a generic fbdev emulation layer, so DRM drivers can also provide a fbdev interface for fbcon (or even to user-space that still expects an fbdev device to work).

While working on this, we found that a lot of assumptions made all over the stack were no longer true once simpledrm was used instead of {efi,vesa}fb. So changes were needed in the Linux kernel, Plymouth, GDM, Xorg, etc.

But that will be a topic for another post since this one got too long already!

A framebuffer hidden in plain sight

Soon after I set up my Rockpro64 board, Peter Robinson told me about an annoying bug that happened on machines with a Rockchip SoC.

The problem was that the framebuffer console just went away after GRUB booted the Linux kernel. We started looking at this and Peter mentioned the following data points:

  • Enabling early console output on the framebuffer registered by the efifb driver (earlycon=efifb efi=debug) would get some output but at some point everything would just go blank.
  • The display worked when passing fbcon=map:1 and people were using that as a workaround.
  • Preventing the efifb driver to be loaded (modprobe.blacklist=efifb) would also make things to work.

So the issue seemed to be related to the efifb driver somehow but wasn’t clear what was happening.

What this driver does is to register a framebuffer device that relies on the video output configured by the firmware/bootloader (using the EFI Graphics Output Protocol) until a real driver takes over an re-initializes the display controller and other IP blocks needed for video output.

I read The Framebuffer device and The Framebuffer Console sections in the Linux documentation to get more familiar about how this is supposed to work.

What happens is that the framebuffer console is bound by default to the first framebuffer registered, which is the one registered by the efifb driver.

Later, the rockchipdrm driver is probed and a second framebuffer registered by the DRM fbdev emulation layer but the frame buffer console is still bound to the first frame buffer, that’s using the EFI GOP but this gets destroyed when the kernel re-initializes the display controller and related IP blocks (IOMMU, clocks, power domains, etc).

So why are users left with a blank framebuffer? It’s because the framebuffer is registered but it’s not attached to the console.

Once the problem was understood, it was easy to solve it. The DRM subsystem provides a drm_aperture_remove_framebuffers() helper function to remove any existing drivers that may own the framebuffer memory, but the rockchipdrm driver was not using this helper.

The proposed fix (that landed in v5.14-rc1) then is for the rockchipdrm driver to call the helper to detach any existing early framebuffer before registering its own.

After doing that, the early framebuffer is unbound from the framebuffer console and the one registered by the rockchipdrm driver takes over:

[   40.752420] fb0: switching to rockchip-drm-fb from EFI VGA

The curious case of the ghostly modalias

I was finishing my morning coffee at the Fedora ARM mystery department when a user report came into my attention: the tpm_tis_spi driver was not working on a board that had a TPM device connected through SPI.

There was no /dev/tpm0 character device present in the system, even when the driver was built as a module and the Device Tree (DT) passed to the kernel had a node with a "infineon,slb9670" compatible string.

Peter Robinson chimed in and mentioned that he had briefly looked at this case before. The problem, he explained, is that the module isn’t auto-loaded but that manually loading it make things to work.

At the beginning he thought that this was just a common issue of a driver not having module alias information. This would lead to kmod not knowing that the module has to be loaded, when the kernel reported a MODALIAS uevent as a consequence of the SPI device being registered.

But when checking the module to confirm that theory, he found that there were alias entries:

$ modinfo drivers/char/tpm/tpm_tis_spi.ko | grep alias
alias:          of:N*T*Cgoogle,cr50C*
alias:          of:N*T*Cgoogle,cr50
alias:          of:N*T*Ctcg,tpm_tis-spiC*
alias:          of:N*T*Ctcg,tpm_tis-spi
alias:          of:N*T*Cinfineon,slb9670C*
alias:          of:N*T*Cinfineon,slb9670
alias:          of:N*T*Cst,st33htpm-spiC*
alias:          of:N*T*Cst,st33htpm-spi
alias:          spi:cr50
alias:          spi:tpm_tis_spi
alias:          acpi*:SMO0768:*

Since the board uses DT to describe the hardware topology, the TPM device should had been registered by the Open Firmware (OF) subsystem. And should cause the kernel to report a "MODALIAS=of:NspiTCinfineon,slb9670", which should had matched the "of:N*T*Cinfineon,slb9670" module alias entry.

But when digging more on this issue, things started to get more strange. Looking at the uevent sysfs entry for this SPI device, he found that the kernel was not reporting an OF modalias but instead a legacy SPI modalias: "MODALIAS=spi:slb9670".

But how come? a user asked, the device is registered using DT, not platform code! Where is this modalias coming from? Is this legacy SPI device a ghost?

Peter said that he didn’t believe in paranormal events and that there should be a reasonable explanation. So armed with grep, he wanted to get to the bottom of this but got preempted by more urgent things to do.

Coincidentally, I had chased down that same ghost before many moons ago. And it’s indeed not a spirit from the board files dimension but only an incorrect behavior in the uevent logic of the SPI subsystem.

The reason is that the SPI uevent handler always reports a MODALIAS of the form "spi:foobar" even for devices that are registered through DT. This leads to the situation described above and it’s better explained by looking at the SPI subsystem code:

static int spi_uevent(struct device *dev, struct kobj_uevent_env *env)
{
	const struct spi_device		*spi = to_spi_device(dev);
	int rc;

	rc = acpi_device_uevent_modalias(dev, env);
	if (rc != -ENODEV)
		return rc;

	return add_uevent_var(env, "MODALIAS=%s%s", SPI_MODULE_PREFIX, spi->modalias);
}

Conversely, this is what the platform subsystem uevent handler does (which properly reports OF module aliases):

static int platform_uevent(struct device *dev, struct kobj_uevent_env *env)
{
	struct platform_device	*pdev = to_platform_device(dev);
	int rc;

	/* Some devices have extra OF data and an OF-style MODALIAS */
	rc = of_device_uevent_modalias(dev, env);
	if (rc != -ENODEV)
		return rc;

	rc = acpi_device_uevent_modalias(dev, env);
	if (rc != -ENODEV)
		return rc;

	add_uevent_var(env, "MODALIAS=%s%s", PLATFORM_MODULE_PREFIX,
			pdev->name);
	return 0;
}

Fixing the SPI core would be trivial, but the problem is that there are just too many drivers and Device Trees descriptions that are relying on the current behavior.

It should be possible to change the core, but first all these drivers and DTs have to be fixed. For example, the I2C subsystem had the same issue but has already been resolved.

A workaround then in the meantime could be to add to the legacy SPI device ID table all the entries that are found in the OF device ID table. That way, a platform using for example a DT node with compatible "infineon,slb9670" will match against an alias "spi:slb9670", that will be present in the module.

And that’s exactly what the proposed fix for the tpm_tis_spi driver does.

$ modinfo drivers/char/tpm/tpm_tis_spi.ko | grep alias
alias:          of:N*T*Cgoogle,cr50C*
alias:          of:N*T*Cgoogle,cr50
alias:          of:N*T*Ctcg,tpm_tis-spiC*
alias:          of:N*T*Ctcg,tpm_tis-spi
alias:          of:N*T*Cinfineon,slb9670C*
alias:          of:N*T*Cinfineon,slb9670
alias:          of:N*T*Cst,st33htpm-spiC*
alias:          of:N*T*Cst,st33htpm-spi
alias:          spi:cr50
alias:          spi:tpm_tis_spi
alias:          spi:slb9670
alias:          spi:st33htpm-spi
alias:          acpi*:SMO0768:*

Until the next mystery!

A lethal spurious interrupt

A big part of my work on Fedora/RHEL is to troubleshoot and do root cause analysis across the software stack. Because many of these projects are decades old, this usually feels like being stuck somewhere between being an archaeologist and a detective.

Many bugs are boring but some are interesting, either because the investigation made me learn something new or due to the amount of effort that was sunk into figuring out the problem. So I thought that it would be a nice experiment to share a little about the ones that are worth mentioning. This is the first of such posts, I may write more in the future if have time and remember to do it:

It was a dark and stormy night when Peter Robinson mentioned a crime to me, a Rockpro64 board was found to not boot when the CONFIG_PCIE_ROCKCHIP_HOST option was enabled. He also already had found the criminal, it was the CONFIG_DEBUG_SHIRQ option.

I have to admit that I only knew CONFIG_DEBUG_SHIRQ by name and that it was a debug option for shared interrupts, but didn’t even know what this option was about. So the first step was to read the help text of the Kconfig symbol to learn more on this option.

Enable this to generate a spurious interrupt just before a shared interrupt handler is deregistered (generating one when registering is currently disabled). Drivers need to handle this correctly. Some don’t and need to be caught.

This was the smoking gun: a spurious interrupt!

We now knew what was the weapon used but we still had questions, why would triggering an interrupt lead to the board being hung? The next step then was to figure out where exactly this was happening, it certainly would have to be somewhere in the driver’s IRQ handler code path.

By looking at the pcie-rockchip-host driver code, we see two IRQ handlers registered: rockchip_pcie_subsys_irq_handler() for the "pcie-sys" IRQ and rockchip_pcie_client_irq_handler() for the "pcie-client" IRQ.

Adding some debug printouts to both would show us that the issue was happening in the latter, when calling to rockchip_pcie_read(). This function just hangs indefinitely and never returns.

Peter wrote in the filled bug that this issue was reported before and there was even an RFC patch posted by a Rockchip engineer, who mentioned the assessment of the problem:

With CONFIG_DEBUG_SHIRQ enabled, the irq tear down routine
would still access the irq handler register as a shared irq.
Per the comment within the function of __free_irq, it says
"It’s a shared IRQ — the driver ought to be prepared for
an IRQ event to happen even now it’s being freed". However
when failing to probe the driver, it may disable the clock
for accessing the register and the following check for shared
irq state would call the irq handler which accesses the register
w/o the clk enabled. That will hang the system forever.

The proposed solution was to check in the rockchip_pcie_read() function if a rockchip->hclk_pcie clock was enabled before trying to access the PCIe registers’ address space. But that wasn’t accepted because it was solving the symptom and not the cause.

But it did confirm our findings, that the problem was an IRQ handler being called before it was expected and that the PCIe register access hangs due to a clock not being enabled.

With all of that information and reading once more the pcie-rockchip-host driver code, we could finally reconstruct the crime scene:

  1. "pcie-sys" IRQ is requested and its handler registered.
  2. "pcie-client" IRQ is requested and its handler registered.
  3. probe later fails due to readl_poll_timeout() returning a timeout.
  4. the "pcie-sys" IRQ handler is unregistered.
  5. CONFIG_DEBUG_SHIRQ triggers a spurious interrupt.
  6. "pcie-client" IRQ handler is called for this spurious interrupt.
  7. IRQ handler tries to read PCIE_CLIENT_INT_STATUS with clocks gated.
  8. the machine hangs because rockchip_pcie_read() call never returns.

The root cause of the problem then is that the IRQ handlers are registered too early, before all the required resources have been properly set up.

Our proposed solution then is to move all the IRQ initialization into a later stage of the probe function. That makes it safe for the IRQ handlers to be called as soon as they are registered.

Until the next mystery!