How to troubleshoot deferred probe issues in Linux

When working on the retro handheld console mentioned in a previous post, I had an issue where the LCD driver was not probed when booting a custom Linux kernel image built.

To understand the problem, first some knowledge is needed about how devices and drivers are registered in the Linux kernel, how these two sets are matched (bound) and what is a probe deferral.

If you are not familiar with these concepts, please read this post where are explained in detail.

The problem is that the st7735r driver (that’s needed for the SPI TFT LCD panel I was using) requires a GPIO based backlight device. To make it more clear, let’s look at the relevant bits in the adafruit-st7735r-overlay.dts that’s used as a Device Tree Blob (DTB) overlay to register the needed devices:

fragment@2 {
    af18_backlight: backlight {
        compatible = "gpio-backlight";

fragment@3 {
    af18: adafruit18@0 {
        compatible = "jianda,jd-t18003-t01";
        backlight = <&af18_backlight>;

We see that the adafruit18@0 node for the panel has a backlight property whose value is a phandle to the af18_backlight label used to refer to the backlight node.

The drivers/gpu/drm/tiny/st7735r.c probe callback then uses the information in the DTB to attempt getting a backlight device:

static int st7735r_probe(struct spi_device *spi)
	dbidev->backlight = devm_of_find_backlight(dev);
	if (IS_ERR(dbidev->backlight))
		return PTR_ERR(dbidev->backlight);

The devm_of_find_backlight() function returns a pointer to the backlight device if this could be found or a -EPROBE_DEFER error pointer if there is a backlight property defined in the DTB but this could not be found.

For example, this can happen if the driver that registers that expected backlight device was not yet probed.

If the probe callback returns -EPROBE_DEFER, the kernel then will put the device that matched the driver but failed to probe in a deferred probe list. The list is iterated each time that a new driver is probed (since it could be that the newly probed driver registered the missing devices that forced the probe deferral).

My problem then was that the needed driver (CONFIG_BACKLIGHT_GPIO since the backlight node has compatible = gpio-backlight) was not enabled in my kernel, leading to the panel device to remain in the deferred probe list indefinitely due a missing backlight device that was never registered.

This is quite a common issue on Device Tree based systems and something that it used to take me a lot of time to root cause when I started working on Linux embedded platforms.

A few years ago I added a /sys/kernel/debug/devices_deferred debugfs entry that would expose the list of devices deferred to user-space, which makes much more easier to figure out what devices couldn’t be bound due their driver probe being deferred.

Later, Andrzej Hajda improved that and added to the devices_deferred debugfs entry support to print the reason of the deferral.

So checking what devices were deferred and the reason is now quite trivial, i.e:

$ cat /sys/kernel/debug/devices_deferred 
spi0.0  spi: supplier backlight not ready

Much better than spending a lot of time looking at kernel logs and adding debug printous to figure out what’s going on.

Happy hacking!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s