Writing eeprom with secure boot on RPi5 make the device unbootable

[dandrej]

Describe the bug

The recovery procedure is applied to the Raspberry Pi 5 8GB. It is described in secure-boot-recovery5/readme.md and ends with success (green HDMI display, green LED flashes rapidly). After the power is off and on again the board LED flashes 1 long 2 short. The hardware doesn't boot images but is still in nRPIBOOT mode. Applying the same eeprom image again produces the same results (eeprom flashing with success indication). Applying the recovery5 without the secure boot makes the device bootable again.
This behavior repeats on three different RPi5 boards.

Steps to reproduce the behaviour

git clone --recurse-submodules --shallow-submodules --depth=1 https://github.com/raspberrypi/usbboot
cd usbboot
make
cd secure-boot-recovery5
_# fix the wrong symlink recovery.original.bin -> ../recovery5/recovery.bin
rm recovery.original.bin
ln -s ../firmware/2712/recovery.bin recovery.original.bin
../tools/update-pieeprom.sh -f -k ../secure-boot-example/example-private.pem
../rpiboot -d .

Device(s)

Other

Compute Module IO board.

No response

RPIBOOT logs

No response

Kernel logs

No response

Device UART logs

29.05 RPi: RECOVERY release VERSION:951e1cc9 DATE: 2024/10/21 TIME: 15:27:49
29.09 BOOTMODE: 0x07 partition 0 build-ts BUILD_TIMESTAMP=1729520869 serial 0088dae0 boardrev d04170 stc 29109977
29.20 AON_RESET: 00000003 PM_RSTS 00001000
29.72 BOOT_ORDER: 0xf3
29.72 USB-OTG disconnect
31.01 Read config.txt bytes 1385 hnd 0x0
31.04 pieeprom.sig
31.04 hash: e2b3acef58f800097906a37f4a920c6c18e5c508f309f4298e1c6ec0a85d7c56
31.11 ts: 1731947243
32.19 Reading EEPROM: 2097152 bytes 0xc1160000
34.04 2483ms
35.81 Writing EEPROM
+................+........................................................................................................................................................+++..................................................................................................................................................................................................................................................................................................................................................+ 36.50
36.50 768ms

36.50 Verify BOOT EEPROM
36.52 Reading EEPROM: 2097152 bytes 0xc1160000
38.06 2550ms
39.84 BOOT-EEPROM: UPDATED
39.84 USB-OTG disconnect
39.86 RECOVERY: COMPLETE: REBOOT 0

[timg236]

Not sure what you are asking for here.
The original bootcode5.bin must NOT be signed with the customer key but enabling secure boot it MUST be signed with the customer key.
The bootrom will not boot a non-secure system with a signed image and will also not boot a secure-system with an unsigned image.

[pelwell]

It might be helpful for you to explain:

1. what you are trying to achieve,
2. how what you have described fits into that, and
3. what you would have expected to happen?

[dandrej]

1. what you are trying to achieve
   I try to switch on the secure boot. So the bootloader may check the boot.img sign.

2. how what you have described fits into that
   I have RPi4 and RPi5 system.
   When I apply instructions in secure-boot-recovery/readme.md to RPi4 I have the system that (after power off/on) tries to boot from SD card, SSD, etc. And it can boot from the SD card with boot.img signed properly.
   When I apply instructions in secure-boot-recovery5/readme.md to RPi5 I have the system that (after power off/on) still in nRPIBOOT mode. It doesn't try to boot from anything.

3. what you would have expected to happen?
   I would expected be able to boot the properly signed image on RPi5 after activation procedure described in secure-boot-recovery5/readme.md like as secure-boot-recovery/readme.md works in the case of RPi4.

[timg236]

If you are able to revert the change and use recovery5 then it suggests that secure-boot was never enabled.

Please can you post the full sequence of rpiboot commands and corresponding UART logs from Pi5 for each stage.

[dandrej]

The full sequence of rpiboot command is in Steps to reproduce the behavior. There is no UART log of this process. The log is empty. I connect the same USB TTL converter on the same RPi5 pins as on RPi4. And RPi4 produces the debug log and RPi5 produces no log. The uart_2ndstage=1 exists in the both config.txt. BOOT_UART=1 exists in the both boot.conf.

[pelwell]

Pi 5 debug output is on the 3-pin connector labelled UART (between the two HDMI connectors).

[timg236]

I forgot to add, when designing a secure-boot system we recommend using the higher level sb-provisioner tool because it supports provisioning of LUKS encrypted file-systems. It's built on top of these low level tools and is open-source shell script if a bespoke solution is required.

https://github.com/raspberrypi/rpi-sb-provisioner

[dandrej]

Add debug UART log to initial report.

1. Having RPi5 with secure boot disabled.
2. Applying secure boot and getting the log stored in the initial message

cd secure-boot-recovery5
../tools/update-pieeprom.sh -f -k ../secure-boot-example/example-private.pem
../rpiboot -d .

3. Result. The RPi5 is non-bootable. It enters nRPIBOOT mode just after powering on without holding the power button.

[timg236]

What OS is being booted at step (3) and how was it signed and provisioned?

Please can you also check that you have uncommented in secure-boot-recovery5/config.txt
#program_pubkey=1

EDIT: Please could you re-run step (2) and paste the contents of the UART log

[dandrej]

I have the OS that successfully boots on RPi4 with a secure boot. I plan to boot it on RPi5 using the same key for secure boot. But no OS can boot on step 3. The system is constantly in nRPIBOOT after power is on. If I connect the system to my laptop by USB cable I see the next lines in dmesg:
[31154.123788] usb 1-6.2.3: new high-speed USB device number 53 using xhci_hcd
[31154.212621] usb 1-6.2.3: New USB device found, idVendor=0a5c, idProduct=2712, bcdDevice= 0.00
[31154.212626] usb 1-6.2.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[31154.212628] usb 1-6.2.3: Product: BCM2712 Boot
[31154.212629] usb 1-6.2.3: Manufacturer: Broadcom
[31154.212630] usb 1-6.2.3: SerialNumber: 0088dae0
I press no power button during power on.

[dandrej]

Here is the log after the re-run of step (2)

1031.75 RPi: RECOVERY release VERSION:951e1cc9 DATE: 2024/10/21 TIME: 15:27:49
1031.79 BOOTMODE: 0x07 partition 0 build-ts BUILD_TIMESTAMP=1729520869 serial 0088dae0 boardrev d04170 stc 1031479638
1031.90 AON_RESET: 00000003 PM_RSTS 00001000
1031.42 BOOT_ORDER: 0xf3
1031.42 USB-OTG disconnect
1033.31 Read config.txt bytes 1385 hnd 0x0
1033.34 pieeprom.sig
1033.34 hash: e2b3acef58f800097906a37f4a920c6c18e5c508f309f4298e1c6ec0a85d7c56
1033.41 ts: 1731947243
1034.55 Reading EEPROM: 2097152 bytes 0xc1160000
1037.39 2483ms
1037.17 Bootloader EEPROM is up to date
1037.18 USB-OTG disconnect
1037.20 RECOVERY: COMPLETE: REBOOT 0

[timg236]

It doesn't look at though it's enabling secure-boot.

Please can you confirm the contents of secure-boot-recovery5/config.txt

[dandrej]

No difference from the github repo.
config.txt

uart_2ndstage=1

# Mark the EEPROM as write protected when the EEPROM /WIP pin is pulled low.
# See https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#eeprom-write-protect
#eeprom_write_protect=1

# Uncomment to write to enable secure-boot by writing. This
# locks the device to the public key in the EEPROM by storing the
# sha256 hash of the public key in OTP.
#
# This option also prevents the ROM from loading recovery.bin from SD/EMMC
# which means that the bootloader can only be updated via RPIBOOT or self-update.
#
# Uncomment program_pubkey=1 to enable this
# WARNING: THIS OPTION MODIFIES THE BCM2712 CHIP AND IS IRREVERSIBLE.
#
# N.B The revoke_keydey option is not required and not currently implemented
# because on 2712 the bootloader firmware must also be signed by the customer
# key if secure-boot is enabled.
#program_pubkey=1

# Permanently disable VideoCore JTAG access.
# Warning: This option limits the ability to do failure analysis on
# boards returned to resellers or Raspberry Pi.
#program_jtag_lock=1

# If recovery_reboot is set then the Pi will disconnect USB and
# reboot after the flashing the firmware.
#recovery_reboot=1

# Uncomment to instruct recovery.bin to send metadata including OTP fields
# Specify -j dirname on the command line to specify the directory where
# metadata should be stored (JSON format)
#recovery_metadata=1

boot.conf

[all]
BOOT_UART=1
POWER_OFF_ON_HALT=1

# Boot Order Codes, from https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#BOOT_ORDER
# Try SD first (1), followed by, USB (RP1), NVMe PCIe, then network
BOOT_ORDER=0xf2641

# Disable self-update mode - to prevent automatic updates of unsigned bootloader images
ENABLE_SELF_UPDATE=0

# Select signed-boot mode in the EEPROM. This can be used to during development
# to test the signed boot image. Once secure boot is enabled via OTP this setting
# has no effect i.e. it is always 1.
SIGNED_BOOT=1

[timg236]

I think the problem is that this line is still commented out in your config.txt file so it won't enable secure-boot but will flash a signed binary which will fail to boot because bootrom is still expected unsigned.

You have to explicitly enable this in this config to ensure that the docs have been read first before potentially bricking the device (sb-provisioner wraps these operations)
#program_pubkey=1

From readme

WARNING: Modifications to OTP are irreversible. Once revoke_devkey has been set it is not possible to unlock secure-boot mode or use a different private key.
To enable this edit the config.txt file in this directory and set program_pubkey=1
program_pubkey - If 1, write the hash of the customer's public key to OTP.

There's an additional commit that I've just pushed to fixup a stale link which might cause a problem
commit e333f58 (HEAD -> master, origin/master)
Author: Tim Gover [email protected]
Date: Tue Nov 19 18:38:52 2024 +0000

secure-boot-recovery5: Fixup recovery.bin link

For reference success should look like this

  3.87 RPi: RECOVERY release VERSION:951e1cc9 DATE: 2024/10/21 TIME: 15:27:49
  3.91 BOOTMODE: 0x07 partition 0 build-ts BUILD_TIMESTAMP=1729520869 serial a7eb274c boardrev b04170 stc 3691240
  3.01 AON_RESET: 00000003 PM_RSTS 00001000
  3.41 BOOT_ORDER: 0xf3
  3.41 USB-OTG disconnect
  6.60 Read config.txt bytes     1383 hnd 0x0
  6.65 pieeprom.sig
  6.65 hash: e9137fc41fa45afab622886b4c85197f8d2b8f22666b0c6abc74358c3630156d
  6.71 ts: 1732042410
  6.38 Reading EEPROM: 2097152 bytes 0xc1160000
  9.22 2483ms
 11.03 Writing EEPROM
+................+........................................................................................................................................................+++..................................................................................................................................................................................................................................................................................................................................................+ 13.41
 13.41 2037ms

 13.41 Verify BOOT EEPROM
 13.43 Reading EEPROM: 2097152 bytes 0xc1160000
 15.30 2483ms
 17.11 BOOT-EEPROM: UPDATED
 17.11 secure_boot_provision program_pubkey 1
 17.17 bootconf.sig
 17.17 hash: ab9d6453836fba2f2922c7e7c448da4e600b857ff37f977da094b36e3c461325
 17.24 rsa2048: dbe5cd0f6bd068f71d876a7c1b9cd3d942f23baf0baf0bd1f706bcf786f8714ce78e407ab4ff8e143982a1934156524ce3fd1e5b3c7b12ffd2c4813c8cf0cc1f23d9be089e6866cbecef90058615f6fe234c36306f17397163720c878bf3770e1001041971db821b45f3e775b97dc97088d7f39f82e5947731d4c3040dcb089c868f8d6bcf8262a0915d0519bccc4f9d5686a3f3a49eed07fcb0e1f0f0cc7d3b48e8b6e03725cf3ca576cb0185a78a7a48d12c6ac9627ed1f330b1af56635d4e5f3d3676629abfa49e3393eda6e84cdd26e319518abd2224e93fdcc2ad054d6ed401f1b2805180d194210369883a43b02cf22299c485dee965a38c753bf05284
 17.71 RSA verify
 17.06 rsa-verify pass (0x0)
 17.07 Public key hash 8251a63a2edee9d8f710d63e9da5d639064929ce15a2238986a189ac6fcd3cee
 17.14 Write OTP key
 17.21 OTP updated for key 8251a63a2edee9d8f710d63e9da5d639064929ce15a2238986a189ac6fcd3cee
 17.27 USB-OTG disconnect
 17.29 RECOVERY: COMPLETE: REBOOT 1
 17.32 RESET

[dandrej]

According to the docs, the secure boot is switched on in two steps:

1. Enabling stage SIGNED_BOOT=1
2. Locking stage program_pubkey=1

From readme

After verifying that the signed OS image boots successfully the system can be locked into secure-boot mode.

So on RPi4 I can enable signed boot, test it by booting the signed images, clear it if something goes wrong, or after everything is OK lock it.

Can I test the secure boot images before the lock on RPi5? The documentation https://github.com/raspberrypi/usbboot/tree/master/secure-boot-recovery5#locking-secure-boot-mode said I could, but you just told me I can't.

[timg236]

On Pi5 it’s possible to verify unsigned ramdisk boot and offline rpi-eeprom-digest can be used to check the signature of the ramdisk. However, unlike Pi4 it’s not possible to program the key digest to OTP without enabling secure boot. This is because on Pi5 the chip Rom now verifies the customer key digest if non zero.
The docs probably need a tweak to clarify SIGNED_BOOT=1 behaviour.
For additional debug the rpiboot_metadata feature can be used to verify the key digest in otp during the provisioning process

[dandrej]

I confirm that with program_pubkey=1 the bootloader works properly, and the signed boot image is detected and executed.

So, in summary, my problems were:

1. I expected that after running the instructions with the default configs in secure-boot-recovery5 on rpi5, I would get the system in the same state as after running the instructions with the default configs in secure-boot-recovery on rpi4. This expectation did not correspond to reality.
2. I could not find the ACT code of one long and two short flashes in the documentation, which probably means a bootloader signature verification error or entering nRPIBOOT mode.

Now these questions are clearer for me. Thank you for your support.