Intel GPU PassThrough

[simurf]

Just a quick query, can an inbuilt Intel gpu be passed through to a jail and then to docker in the jail?
Or do I need a seperate gpu to pass through a gpu.
I've got gpu_passthrough_intel=1 in the config file, but when I test in the jail it tells me no device is available.
Below is the result of the test in the jail.

Is there something I'm missing to enable it or will I need a second gpu in the machine to pass through?

root@docker:~# ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i bunny.mp4 -f null - && echo 'SUCCESS!'
Warning: using insecure memory!
ffmpeg version 5.1.4-0+deb12u1 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 12 (Debian 12.2.0-14)
configuration: --prefix=/usr --extra-version=0+deb12u1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-sndio --enable-libjxl --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-librav1e --enable-shared
libavutil 57. 28.100 / 57. 28.100
libavcodec 59. 37.100 / 59. 37.100
libavformat 59. 27.100 / 59. 27.100
libavdevice 59. 7.100 / 59. 7.100
libavfilter 8. 44.100 / 8. 44.100
libswscale 6. 7.100 / 6. 7.100
libswresample 4. 7.100 / 4. 7.100
libpostproc 56. 6.100 / 56. 6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bunny.mp4':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp42isomavc1
creation_time : 2012-03-13T08:58:06.000000Z
encoder : HandBrake 0.9.6 2012022800
Duration: 00:00:10.03, start: 0.000000, bitrate: 629 kb/s
Chapters:
Chapter #0:0: start 0.000000, end 10.000000
Metadata:
title : Chapter 1
Stream #0:00x1: Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, smpte170m/smpte170m/bt709, progressive), 320x176, 300 kb/s, 25 fps, 25 tbr, 90k tbn (default)
Metadata:
creation_time : 2012-03-13T08:58:06.000000Z
vendor_id : [0][0][0][0]
encoder : JVT/AVC Coding
Stream #0:10x2: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 160 kb/s (default)
Metadata:
creation_time : 2012-03-13T08:58:06.000000Z
vendor_id : [0][0][0][0]
Stream #0:20x3: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 160 kb/s
Metadata:
creation_time : 2012-03-13T08:58:06.000000Z
vendor_id : [0][0][0][0]
Stream #0:30x4: Data: bin_data (text / 0x74786574), 0 kb/s
Metadata:
creation_time : 2012-03-13T08:58:06.000000Z
[AVHWDeviceContext @ 0x56371109da00] No VA display found for device /dev/dri/renderD128.
Device creation failed: -22.
[h264 @ 0x5637110be8c0] No device available for decoder: device type vaapi needed for codec h264.
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))
Device setup failed for decoder on input stream #0:0 : Invalid argument

[Jip-Hop]

can an inbuilt Intel gpu be passed through to a jail and then to docker in the jail?

This is possible. No need for additional GPU. I see you found the command to test if Intel GPU pass through is working in the code. Good job.

Too bad it doesn't work according to your logging. I don't know why and you can't easily test if Intel GPU works on the host... Does your CPU actually have a GPU built in? Some CPUs are without integrated graphics (F and KF versions).

[simurf]

Yep, it's definately got an integrated gpu.
I was using it to pass through to apps in Scale, but am trying to move them all to jails and docker.
I've also got an Nvidea card in the machine, but that's isolated to a VM.

The show up as the below in TrueScales interface.
Intel Corporation Alder Lake-S GT1 [UHD Graphics 770]
NVIDIA Corporation AD103 [GeForce RTX 4080]

Looks like I'll have to keep googling and trying things to see if I can make it work.
It's the last two things I need to get working in Jails.
The other being passthrough of a usb Coral.
I was hoping starting with the gpu was going to be the easier one.

Any other suggestions greatfully received.

[simurf]

Ok, so looks like it was an error in my config from when I was trying to get usb pass through working.
I stripped my jailmaker config right back and just put back what was immediately needed.
And the gpu is now working.
Now on to working out how to pass in my usb coral, to get my Frigate stack up and running in a jail.

[b-neufeld]

Ok, so looks like it was an error in my config from when I was trying to get usb pass through working. I stripped my jailmaker config right back and just put back what was immediately needed. And the gpu is now working. Now on to working out how to pass in my usb coral, to get my Frigate stack up and running in a jail.

@GooeyLizard9000 I'd love to know if you can share details about getting Frigate up and running. I'm almost done migrating a Ubuntu VM (on SCALE) packed of Docker containers over to Jailmaker. Just started looking at Frigate and it doesn't seem to be intuitive (nor was the original setup). I will start a thread if I get it figured out.

[simurf]

@b-neufeld Finally got back to nutting out getting Frigate running.
I hadn't had time to touch it for a bit, but when I started frigate again using the last docker compose it finally came up and recognised the usb coral.
Below is what I can remember I've changed.

In the jlmkr config I made sure gpu_passthrough_intel=1 was on.
It didn't work initially, but then on one reboot, it started working and hasn't stopped since.
I've also added perf_event_open to --system-call-filter='add_key keyctl bpf perf_event_open' so I can see the gpu usage in frigate.
For the usb coral I've also added --bind='/dev/bus/usb/'
I may have changed something else along the way to get it working, but that was a couple of weeks ago and can't remember.

As for my frigate config file, it's the same as it was when running under TrueNas Scale apps.

services:
frigate:
container_name: frigate
privileged: true # this may not be necessary for all setups
image: ghcr.io/blakeblackshear/frigate:stable
shm_size: 128mb # update for your cameras based on calculation above
devices:
- /dev/bus/usb:/dev/bus/usb # passes the USB Coral
- /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
volumes:
- /etc/localtime:/etc/localtime:ro
- /mnt/data/frigate/config/config.yml:/config/config.yml
- /mnt/data/frigate/media:/media/frigate
- type: tmpfs
target: /tmp/cache
tmpfs:
size: 1000000000
networks:
- traefik
ports:
- 5000:5000
- 8554:8554 # RTSP feeds
- 8555:8555/tcp # WebRTC over tcp
- 8555:8555/udp # WebRTC over udp
environment:
FRIGATE_RTSP_PASSWORD: PASSWORD
labels:
- traefik.enable=true
- traefik.http.routers.frigate.entrypoints=http
- traefik.http.routers.frigate.rule=Host(frigate.example.net)
- traefik.http.middlewares.frigate-https-redirect.redirectscheme.scheme=https
- traefik.http.routers.frigate.middlewares=frigate-https-redirect
- traefik.http.routers.frigate-secure.entrypoints=https
- traefik.http.routers.frigate-secure.rule=Host(frigate.example.net)
- traefik.http.routers.frigate-secure.tls=true
- traefik.http.routers.frigate-secure.service=frigate
- traefik.http.services.frigate.loadbalancer.server.port=5000
- traefik.docker.network=traefik
restart: unless-stopped
networks:
traefik:
external: true