bf.cfg files are limited to 16 KB in size and if they are larger bfb-install fails silently

[lgarbarini]

Wanted to post this in case others get tripped up and in the hope that the BSP docs are updated:

We can end up producing decently large bf.cfg files on the fly to do customization. Most of this involves injecting cloud-init mime-multipart user-data for things like systemd units/config files. This means that occasionally our bf.cfg files exceeded 16 KB which resulted in sporadically failing deployments.

This limit appears to be only documented in the code here:

bfb-build/ubuntu/22.04/install.env/common

Lines 615 to 619 in 8132268

	boot_fifo_path="/sys/bus/platform/devices/MLNXBF04:00/bootfifo"
	if [ -e "${boot_fifo_path}" ]; then
	cfg_file=$(mktemp)
	# Get 16KB assuming it's big enough to hold the config file.
	dd if=${boot_fifo_path} of=${cfg_file} bs=4096 count=4 > /dev/null 2>&1

It would be great if the BSP docs covered this and/or bfb-install warned that this will likely fail.

EDIT: also cross-filed as nvidia bug 4704764