MCU server implementation audio mix and relay

[MestreDosMagros]

Hey, i'm trying to implement an MCU server for a POC, it only have to be able to get up to 5 people on call, but following the standards of an MCU, each participant will only have one peer connection open with the server, and basically i have two problems.

First problem

I was able to connect 2 people on the server and relay the audio using the SendRtpRaw(SDPMediaTypesEnum.audio, pkt.Payload, pkt.Header.Timestamp, pkt.Header.MarkerBit, pkt.Header.PayloadType) method. But when the third connection enters the call, the packets are lost and the call goes wild.

I know that i have to take all these audio streams that i receive on the OnRtpPacketReceived method, merge into just one stream and then send this stream to all the persons in the call. But i dont have much of an idea of how doing this.

Second problem

I made an client using Angular to test my server while developing, but the actual objective is to create an native android app that connect to the server to comunicate with others, but the android client always crashes a few moments after connecting, and the native android library dont give me any clue of whats happening :(.

Also, on the few seconds while the android app still connected, the audio quality is very poor with a lot of echoing, while the web client receive an almost perfect sound.

If you can help me giving me an ideia of why this is happening and where i can start doing the audio mixing, i would be much obliged.

My code till this moment

Web client (Angular)
Web API

The web API that the mobile app is using

• The mobile app is using a different API, but i used the same methods in the same way as the one that connect to the web client, just not using websockets, the signaling is made via http

[sipsorcery]

I know that i have to take all these audio streams that i receive on the OnRtpPacketReceived method, merge into just one stream and then send this stream to all the persons in the call. But i dont have much of an idea of how doing this.

You need an audio mixer. I've never done it myself but a quick search on Naudio and mixing looks like it should be possible.

Also, on the few seconds while the android app still connected, the audio quality is very poor with a lot of echoing, while the web client receive an almost perfect sound.

Are you using a different WebRTC SDK for your Android client? I'm guessing yes since I am yet to hear of anyone having success attempting a .NET based iOS or Android client that can handle audio.

The problems with the audio could be occurring due to the way you're fowarding the audio. I'd recommend checking the audio issues with only two peers. If that's OK and the issues occur when you add subsequent peers then it could be your mixing (or lack of).

[MestreDosMagros]

Are you using a different WebRTC SDK for your Android client?

No, i'm using the oficial google library of WebRTC for Android, but this problem has been solved. I needed to define a name for the media objects as the library was attempting to use the same thread to set the mic feedback.

For the audio mixer, i'm trying to mix all the audios together, but i'm using OPUS codec for receiving and sending audio data, the difficult part for me is how to interpret the packets received on the OnRtpPacketReceived method, im receiving raw packets? the header is striped from the bytes array to the RTPPacketHeaderObject?

I'm "sucessifully" converted the packets to .opus files using the Concentus .Net library like this:

public class Opusenc
    {
        public Opusenc()
        {
        }

        public void EncodeRawAudioPacket()
        {
            var encoder = OpusEncoder.Create(48000, 2, OpusApplication.OPUS_APPLICATION_VOIP);
            encoder.Bitrate = 96000;

            var bytes = File.ReadAllBytes(Path.Combine(Directory.GetCurrentDirectory(), $"1.raw"));

            using var fileOut = new FileStream(Path.Combine(Directory.GetCurrentDirectory(), $"result.opus"), FileMode.Create);

            var tags = new OpusTags();

            tags.Fields[OpusTagName.Title] = "Title";
            tags.Fields[OpusTagName.Artist] = "Artist";

            var oggOut = new OpusOggWriteStream(encoder, fileOut, tags); // instead of 44100 pass the actual input sample rate of your audio here

            short[] samples = BytesToShorts(bytes);

            oggOut.WriteSamples(samples, 0, samples.Length);
            oggOut.Finish();
        }

        public static short[] BytesToShorts(byte[] input)
        {
            return BytesToShorts(input, 0, input.Length);
        }

        public static short[] BytesToShorts(byte[] input, int offset, int length)
        {
            short[] processedValues = new short[length / 2];
            for (int c = 0; c < processedValues.Length; c++)
            {
                processedValues[c] = (short)(((int)input[(c * 2) + offset]) << 0);
                processedValues[c] += (short)(((int)input[(c * 2) + 1 + offset]) << 8);
            }

            return processedValues;
        }
    }

Using opusinfo tool i get this result after the conversion:

opusdec:

But this packets are so small that i need to smush them together to see if this really is working, so i transform like 800 packets from .opus to .wav using opusenc tool and them tried do use ffmpeg to concat all the files, as the audio duration from this files are 0.02 seconds, i should have and audio file after concat with 16s. but i still with the same 0.02s audio file at the end :/

I tried to do with the FFmpeg.Net library too, like this:

public class FFmpegAudioMixer
    {
        private readonly FFOptions _options;

        public FFmpegAudioMixer()
        {
            _options = new FFOptions
            {
                BinaryFolder = @"C:\ProgramData\chocolatey\lib\ffmpeg\tools\ffmpeg\bin",
                TemporaryFilesFolder = @"C:\temp"
            };
        }

        public void MixAudioData()
        {
            try
            {
                var file = File.ReadAllBytes(@"C:\temp\wavs\1.wav");
                var file2 = File.ReadAllBytes(@"C:\temp\wavs\2.wav");

                var audioStream1 = new MemoryStream(file);
                var audioStream2 = new MemoryStream(file2);

                var mediaAnalisys = FFProbe.Analyse(audioStream1, int.MaxValue, _options);
                var mediaAnalisys2 = FFProbe.Analyse(audioStream2, int.MaxValue, _options);

                var durationAudio1 = mediaAnalisys.Duration;
                var durationAudio2 = mediaAnalisys2.Duration;

                audioStream1.Position = 0;
                audioStream2.Position = 0;

                var outStream = new MemoryStream();

                FFMpegArguments
                    .FromPipeInput(new StreamPipeSource(audioStream1), options =>
                    {
                        options.WithDuration(durationAudio1);
                    })
                    .AddPipeInput(new StreamPipeSource(audioStream2), options =>
                    {
                        options.WithDuration(durationAudio2);
                    })
                    .OutputToPipe(new StreamPipeSink(outStream), options =>
                    {
                        options.ForceFormat("mp3");
                        options.WithCustomArgument(@"-filter_complex amerge=inputs=2 -ac 2");
                    })
                    .NotifyOnOutput((str, dt) =>
                    {
                        Console.WriteLine(str);
                    })
                    .ProcessSynchronously(true, _options);

                using FileStream fl = File.Create(@"C:\temp\result.mp3");
                fl.Write(outStream.GetBuffer(), 0, (int)outStream.Length);
                fl.Close();
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
        }
    }

That i know works because i did it succesfully with two .mp3 files.

I'm i bit lost on what to do right now

[sipsorcery]

One suggestion would be to skip OPUS for now and instead use PCM. That will simplify your decode/encode stage a lot. You get your calls to use PCM by using it as the only advertised audio codec in your SDP offer or answer.

To employ an audio mixer the high level steps are:

• Use an event handler to get all incoming PCM audio packets.
• Hopefully you'll be able to find an audio mixer that takes PCM inputs. See here for how I feed PCM straight into the NAudio Wave Provider. Maybe there is an equivalent AudioMixerWaveProvider. Note I don't know if there is and have never come across it.
• Take the results from the audio mixer, hopefully still in PCM format, and send them directly in an RTP packet to each of the call participants.

You should also be synchronising the RTP timestamps across the participants before feeding into the mixer. To start with I would skip this stage. It should be good enough to get usable audio without such a synchronisation. You could look at this once you've managed to get the mixer working.