Add sound with pulseaudio

[polhaghverdian]

Hi,

Is it possible to add sound with pulseaudio, and stream it to the clients such as novnc or vncviewer?

Any help will be appreciated!

[dcommander]

Not currently. I am trying to find funding to implement this feature. Probably it would involve setting up a pulseaudio daemon within the Xvnc process, using that daemon to intercept audio from applications, forwarding the audio through the VNC connection using an RFB extension, and playing the audio on the client using Java methods. It is likely to be straightforward to implement but also labor-intensive (i.e. much testing and debugging will be required.)

[jave]

I made a small experiment with sound.

• wireguard vpn between server and client. You need this to stream udp to the client from the server
• the Roc audio server for streaming audio from the vnc server to the client

It works, but is a bit specialized.

[dcommander]

I think others have succeeded in setting up a PulseAudio daemon and client to stream audio from the TurboVNC session to the client machine, but the idea behind this proposed enhancement is to do that automatically and to stream the audio through the existing RFB connection, so no additional setup would be required.

[jave]

yes i did this same thing a couple of years ago, as did others, so its not original in any way.

My point though, which i managed to obscure, is that you can compare x2go:s sound support, which works with pulseaudio over ssh, with that of sound servers using rtsp, like roc, or webrtc.

pulseaudio over ssh demands a flawless connection, otherwise the sound degrades quickly.

sound over udp using a sound codec like opus, which adapts to the bandwidth, are much more usable.

Perhaps the opus codec could be used over rfb, im not sure.

anyway, these were just datapoints.

[eGit]

Sound would be nice. I thought it would work automatically but maybe I was mistaken.

[dcommander]

@eGit VNC doesn't inherently have audio capabilities. The RFB protocol stands for "Remote Framebuffer Update", reflecting its original intent as a protocol for delivering screen updates from server to client. Audio can only be implemented via an RFB extension, and there are likely to be numerous issues associated with that (how to capture the audio on the server, how to synchronize audio and video, etc.) I made a formal estimate to a potential sponsor in early 2019 proposing 100-150 hours of funding to fully implement this feature. That is based on the labor that was required to implement remote drawing tablet support, which has a similar scope and similar touch points on both client and server.

[hildred]

Here is a idea. Assume rfb is running on tcp5901, lets use udp5901, so the vnc client will connect to :1 will use 5901 and login as normal. It will then indicate that it will support second channel tokens. the server will send a second channel token and continue as normal. the client on receiving a second channel token will then fire off a sound client which will connect to udp5900 and authenticate using the second channel token received by the vnc client, at that point the connection continues using some common sound streaming protocol. the main drawback of this is that there is no synchronization of audio which would make most videos unwatchable, but probably would not be an issue for most desktop sound, and probably be acceptable for those games which don't need a mouse. the main two advantages of this is that one we don't have to design sound support, just run two instances of ffmpeg or the like and it works, and two second channel tokens could also be used for other side-band communication such as the oft requested file transfer and chat and could also be used for video streaming or other things we haven't thought of.

If you like the basic Idea, Let me know and I will write a proposed extension specification.

[dcommander]

The usefulness of this feature without A/V sync or the ability to straightforwardly tunnel through SSH would be so limited as to not be worth pursuing. I have only a limited budget every year to use for community support, and that budget is usually exhausted after six months, leaving me to support the community on my own dime for the rest of the year. Thus, I have a vested interest in not introducing esoteric features that would be hard to support. What you're proposing would take a lot of time to get right, and I would rather spend that time implementing the feature the right way. Creating a UDP sideband is problematic for a variety of reasons.

A stereo CD-quality audio stream requires 1.4 Mbps, so modern networks will be able to accommodate that with no problems. However, it should also be possible, using the RFB flow control extensions, to detect situations in which the bandwidth is too constrained to allow CD-quality audio and to dial down the audio quality to accommodate the available bandwidth. That is only possible if the audio is being sent as part of the RFB stream.

As far as ancillary uses for the sideband channel, more people explicitly don't want RFB file transfer in TurboVNC than want it, because file transfer is a security risk. The only people who have requested that feature are using our viewer with a Windows VNC server, which is out of scope for our project. I have explicitly refused to implement file transfer for the TurboVNC Server. Too many people don't want it, and I don't see the point, since there are plenty of graphical SCP clients available out there. The limit of my willingness to address that feature is to modify the Windows TurboVNC Viewer so that it supports the more recent RFB file transfer protocol used by v2.x of the TightVNC Server, but that would have to be a funded development project.

[jave]

some more datapoints.

Recently support for the roc streaming server was implemented in pipewire. Pipewire is a new audio daemon which replaces both jackd and pulseaudio.

I use this a lot with turbovnc and its very stable and convenient, much more so than using the corresponding pulseaudio streaming.

the roc streaming protocol has latency guarantees, which makes it usable for running something like ardour under turbovnc.

Anyway, I'm not saying this is the way to implement an integrated solution for turbovnc, I'm just saying that for certain usecases this is a perfectly acceptable solution.

[Ithinkit]

Now I've been using TurboVNC with VirtualGL for many years and am thankful for it being available. Thank you dcommander. Several years ago I've tried to get sound working without success.

Today I tested the GUI configurator paprefs for pulseaudio (Debian11). It will send and receive sound over a local network, but the quality is very poor with a wireless connection. However, here is a method that works really well. It is also very simple to get it running:

# Set ffmpeg to stream the local desktop sound
# Make sure pulseaudio "default" is correctly configured.
# Make sure port 5004 is not blocked on your server side when requested from the client (listener).
# IP-numbers is for the target machine. Using localhost fails here.
# RTP uses UDP, so an SSH tunnel will not work, but VPN should.
ffmpeg -f pulse -i default -acodec libmp3lame -ab 128k -ac 2 -ar 44100 -f rtp rtp://10.0.0.3:5004?pkt_size=64

# Nftables example
# Server: /etc/nftables.conf
                # allow streaming rtp to 10.0.0.3
                ip saddr 10.0.0.3 udp dport 5004 accept

# client (10.0.0.3)

# A) ffplay
# run ffplay without buffer and display
# IP number is server
ffplay -infbuff -nodisp rtp://10.0.0.2:5004

# B) VLC
# open network stream =>
# To minimize sound lag show options and set VLC caching to 0 ms
# IP number is server
rtp://10.0.0.2:5004

The quality is excellent on a wireless network and there is no noticable lag when playing and scrubbing music. Enjoy!

Edit: In some cases there can be delay to the sound. Probably the settings on both the server and client side could be tweaked to keep things in sync at all times.

[dcommander]

A more formal proposal for integrating audio capabilities into TurboVNC:

Major components:

• The QEMU RFB audio extension:
  https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#qemu-audio-client-message
  https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#qemu-audio-server-message
  This would be used to transfer the audio packets over the network. It would need to be implemented in both the TurboVNC Server and the TurboVNC Viewer. The implementation is straightforward but would likely require a lot of testing.

• Server side: a custom session-specific PulseAudio daemon that interfaces with the RFB server for a particular TurboVNC session. The general consensus from the community is that PulseAudio would cover the lion's share of Linux audio applications. GStreamer, libcanberra, SDL, Qt, and Safe ALSA can use PulseAudio as a back end, and I get the impression that those APIs cover almost all "casual" audio applications. High-end/low-level audio applications that use JACK or Full ALSA would not work. The PulseAudio daemon might be a separate application, launched by the vncserver script along with Xvnc, or it might be embedded into Xvnc. I would need to do additional research in order to
  figure out which approach makes the most sense. Once the session-specific PulseAudio daemon is started, then it should be a simple matter of manipulating environment variables in order to direct applications to the Unix domain socket on which the daemon is listening.

• Client side: an interface between the QEMU RFB audio extension and the O/S-specific audio API. Fortunately, this seems straightforward to implement using existing Java Swing classes.

I imagine this project being of similar scope to the remote X Input and Wacom tablet feature that was implemented in TurboVNC 2.1, since that feature had similar touch points on the client and server and a similar need to implement an esoteric RFB extension. That feature took over 100 hours, so I would anticipate the audio feature requiring in the
neighborhood of 100-150 hours with a development timeframe of two months to get to production quality.

[dcommander]

Another note regarding security. I'm not sure how current this document is:
https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/Developer/AccessControl
I first encountered it over six years ago, but it has at least been updated within the past two years.

Because the TurboVNC Server (like any X server) is single-threaded, I would have to be really careful to ensure that malicious PulseAudio applications can't crash or freeze Xvnc or kill the hypothetical embedded PulseAudio server (thus necessitating a TurboVNC session restart.) Design-wise, it may make more sense to implement a PulseAudio sink in Xvnc rather than having Xvnc act as a full-blown PulseAudio server. These are just naive ideas. I would have to dig into the problem to figure out what makes the most sense.

[marioroy]

I tried the pulseaudio path with success.

On the Mac (client side), I installed pulseaudio via brew install pulseaudio. An entry is needed in /usr/local/etc/pulse/default.pa to enable native protocol over tcp. Your network acl may differ, so change the acl accordingly.

load-module module-native-protocol-tcp auth-ip-acl=192.168.1.0/24 auth-anonymous=1

Reload the pulseaudio daemon.

brew services reload pulseaudio

On the server side, I added a line to /opt/TurboVNC/bin/xstartup.turbovnc, at the top of the file (around line 22).

export PULSE_SERVER=127.0.0.1:4714

Launch vncserver as usual. That completes the server side.

/opt/TurboVNC/bin/vncserver 

Back on the client side, tunnel the audio connection using OpenSSH.

ssh -N -L 59001:127.0.0.1:5901 -R 4714:127.0.0.1:4713 192.168.1.34

TurboVNC connects to 127.0.0.1:59001. Having audio is wonderful :)

[dcommander]

I appreciate the tips and tricks regarding using PulseAudio along with TurboVNC via a separate SSH tunnel, but this issue is about integrating PulseAudio into the TurboVNC Server so that audio is automatic when using TurboVNC.