Input does not work in Fedora VNC installer environment with accelerated video with tigervnc 1.14.0+

[AdamWill]

Describe the bug
Fedora's installer - anaconda - uses tigervnc to provide a VNC install option. If you boot a Fedora installer (not live) image with the inst.vnc parameter, it will boot up and start a tigervnc Xvnc environment with the installer running in it, and prompt you to connect to it. This affects Fedora 41 (we are currently trying to release Fedora 41, and this is a potential release blocker) and updated Fedora 40 installer images (though this isn't a big deal as we don't officially ship any of those).

With tigervnc 1.14.0 and 1.14.1, interaction with the installer does not work if the installer is running on a system with support for hardware graphics acceleration (tested on at least three AMD and one Intel system, so far). Neither mouse nor keyboard input to the installer is recognized; you're effectively stuck on the first screen. If hardware acceleration is not available (e.g. if running in a VM without 3D passthrough, or booting with nomodeset to effectively disable it), things work fine.

With tigervnc 1.13.1, everything works OK. I verified this by building two Fedora 40 installer images from the current updated Fedora 40 repositories; one with all standard packages (including tigervnc 1.14.0), one with a rebuilt 1.13.1 with the version hacked to appear newer than 1.14.0 so it would be pulled into the image. The image with 1.14.0 shows the bug, the image with 1.13.1 does not.

I could not reproduce this bug running a tigervnc server 'normally' from an installed Fedora 41 system - that is, according to https://src.fedoraproject.org/rpms/tigervnc/blob/rawhide/f/HOWTO.md . I tried with both LXQt and GNOME desktops, and remoting in worked fine, including interaction with GTK 3 apps (the installer is a GTK 3 app). So there's something special about the installer environment, but I'm not sure what.

The code the installer uses to launch tigervnc can be seen at https://github.com/rhinstaller/anaconda/blob/fedora-41/pyanaconda/vnc.py .

To Reproduce
Steps to reproduce the behavior:

1. Boot a Fedora 41 installer image (e.g. https://dl.fedoraproject.org/pub/alt/stage/41_RC-1.3/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-41-1.3.iso ) on a system with out-of-the-box hardware acceleration support (most AMD, Intel or NVIDIA adapters should do), with the inst.vnc kernel arg
2. Connect in on the IP and port shown at the console
3. Try and click on anything, or type anything
4. See error

Expected behavior
Clicking and typing should work.

Client (please complete the following information):

• OS: Fedora 41
• VNC client: vinagre
• VNC client version: 3.22.0
• Client downloaded from: distro package

Server (please complete the following information):

• OS: Fedora 41 or updated Fedora 40
• VNC server: TigerVNC
• VNC server version: 1.14.0 or 1.14.1
• Server downloaded from: distro package
• Server was started using: anaconda code (see link above)

[grulja]

Might be a blind shot, but I've observed some issues here with the new hardware acceleration support in the new Tigervnc. Can we try to add -renderNode=foobar option to Xvnc config?

[AdamWill]

does that disable it? yeah, we could try patching anaconda for that. I had come to the same conclusion and am currently running a build which reverts #1771 (or tries to).

[CendioOssman]

We are not aware of any such issue with AMD and Intel, though, so we are very keen in debugging this. I fully understand disabling it for now to get Fedora 41 out the door, but we'd like to work to a state where this can be left on for users.

Do you have any idea what might be different in the installer environment compared to a normal one? Is it the same TigerVNC and Mesa binaries?

[AdamWill]

Same binaries. Difference is either in something about the installer app running on gnome-kiosk (that's basically what the installer env is), or exactly how it runs tigervnc, I guess. We only found this this morning so it's a bit of a rush, and I'm sitting on a plane about to go on vacation for a week, so great. I'll try and get someone else to follow up.

[grulja]

does that disable it? yeah, we could try patching anaconda for that. I had come to the same conclusion and am currently running a build which reverts #1771 (or tries to).

I think it does or @CendioOssman can confirm whether I'm right. It's definitely easier thing to try than reverting this change completely.

[CendioOssman]

Yes, that should completely disable it. And the "foobar" thing is only needed because of a bug in the vncserver script. So if you call Xvnc directly, you can specify -rendernode "" instead. The vncserver script is also fixed in 1.14.1.

[kparal]

I can confirm that with Xvnc -rendernode foobar the bug is avoided and input works as expected.

[AdamWill]

So the anaconda code sets all these other options for Xvnc:

        xvnccommand = [XVNC_BINARY_NAME, ":%s" % constants.X_DISPLAY_NUMBER,
                       "-depth", "24", "-br",
                       "IdleTimeout=0", "-auth", "/dev/null", "-once",
                       "DisconnectClients=false", "desktop=%s" % (self.desktop,),
                       "SecurityTypes=%s" % SecurityTypes, "rfbauth=%s" % rfbauth]

most of them don't look relevant, but I wonder if -depth 24 might be involved...I don't know what -br does...

[CendioOssman]

I assume you got a workaround fixed to get the release out. But did anyone dig further in to discover what the triggering condition is?

[AdamWill]

Yeah, for the F41 release we just went with your recommendation of having anaconda pass -rendernode foobar to work around this. F42 is actually removing VNC support from the installer entirely (replaced by RDP), so it won't have this issue (it'll probably have lots of exciting different issues instead!) For that reason I hadn't got back to trying to figure out the exact trigger for this yet, sorry :( Too many more urgent things on fire. I will keep it in my giant backlog pile but I honestly don't know when I'd have the cycles to poke at it more.