Can't boot Pi 5 behind PEX 8619 16-lane 16-port PCIe Switch

[geerlingguy]

Describe the bug

I'm running a Pi 5 / BCM2712 with an external Broadcom PEX 8619 PCIe switch. It is on the latest firmware (Mon 23 Sep 13:02:56 UTC 2024 (1727096576)), with all the latest Pi OS updates applied.

After #1833 was fixed, many PCIe switches worked, but it seems this one does not — I was wondering if that's an expected behavior (e.g. each switch would need to be supported independently), or if it's a bug.

I booted the Pi directly of an NVMe SSD, but behind this switch it will not boot off it.

To reproduce

Boot with a valid boot NVMe SSD behind a PEX 8619 PCIe switch.

Expected behaviour

The Pi boots off the NVMe SSD, assuming your BOOT_ORDER is correct.

Actual behaviour

The Pi does not boot off the NVMe SSD.

System

$ uname -a
Linux pi 6.6.51+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.51-1+rpt3 (2024-10-08) aarch64 GNU/Linux

$ sudo rpi-eeprom-update -a
BOOTLOADER: up to date
   CURRENT: Mon 23 Sep 13:02:56 UTC 2024 (1727096576)
    LATEST: Mon 23 Sep 13:02:56 UTC 2024 (1727096576)
   RELEASE: default (/lib/firmware/raspberrypi/bootloader-2712/default)
            Use raspi-config to change the release.

Logs

$ lspci
...
0000:01:00.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:01:00.1 System peripheral: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:01.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:02.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:03.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:04.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:05.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:06.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:07.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:08.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:09.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:0a.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:0b.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:0c.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:0d.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:0e.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:02:0f.0 PCI bridge: PLX Technology, Inc. PEX 8619 16-lane, 16-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
0000:09:00.0 Non-Volatile memory controller: Phison Electronics Corporation PS5013 E13 NVMe Controller (rev 01)
...

$ dmesg | grep nvm
[    2.561905] nvme nvme0: pci function 0000:09:00.0
[    2.566641] nvme 0000:09:00.0: enabling device (0000 -> 0002)
[    2.608247] nvme nvme0: allocated 128 MiB host memory buffer.
[    2.615247] nvme nvme0: 1/0/0 default/read/poll queues
[    2.625060]  nvme0n1: p1 p2

Additional context

This is a kind of ridiculous switch, but one that seems to work reliably for plugging in a LOT of PCIe Gen 2 devices to a Pi.

And I also tried adding dtoverlay=pciex1-compat-pi5 to my /boot/firmware/config.txt but that made no difference.

[peterharperuk]

each switch would need to be supported independently

No. We've not added anything "special" to handle specific switches. We have however chosen a simple solution to dealing with them by carving out the memory space. From memory I think there's 1GB available and we give each 64MB which means you're probably hitting the limit. Does the bootloader complain at all?

[geerlingguy]

@peterharperuk - That... I didn't check. I will boot it back up and check next week, I'm already out of the office today :)