Hey, all! Just found this project and have a couple questions.

[bitbound]

Hello! I just discovered SipSorcery and have some questions. 😄

I've been working on an open-source remote control tool for a while now, and I'm using Microsoft's Mixed Reality WebRTC library for streaming the desktop.

It works great, but it doesn't support Linux or Mac. I currently have Windows and Linux implementations for my project, and will hopefully have time to do Mac some day. Currently, the Linux app is relaying diffed frames through the server via SignalR with MessagePack protocol. It would be nice to have a single, cross-platform library for WebRTC.

One of the nice things with the MixedReality-WebRTC library is that it has an ExternalVideoTrackSource class that lets you supply Argb32 frames for the source, making it easy to feed it Bitmap screen grabs.

So my questions are:

• Does SipSorcery have something similar to ExternalVideoTrackSource?
• Are DataChannels fully implemented?
• Are there plans to create media libraries for other platforms (like SIPSorceryMedia.Windows)?

It seems like it might be kind of offensive asking "Is this better than XYZ" type of questions. I hope it doesn't come across like that. It's great to see a fully-managed implementation of WebRTC! 🎈

Thanks!

[sipsorcery]

Does SipSorcery have something similar to ExternalVideoTrackSource?

Yep. The idea is you implement IVideoSource for whatever video you want to stream.

Within the video source you'll need to get the frames into an I420 pixel format, since that's what the VP8 and H264 encoders generally require. If your source is supplying ARGB frames the conversion to I420 is trivial.

Are DataChannels fully implemented?

Yes... But to be honest it needs a bit of work as can be seen in issues like #380 and similar. The SCTP implementation (which is the protocol that implements data channels on top of the rest WebRTC stack) was ported from a Java library and it's not as clean as it could be. Hopefully at some point it will be rewritten properly in C#.

Are there plans to create media libraries for other platforms (like SIPSorceryMedia.Windows)?

Not really, at least not by me. As I'm sure you've found already it's a lot of work to support cross-platform apps. In theory .NET Core is meant to allow a single code base to work everywhere but in practice there are still a lot of quirks that have to be worked around. And then for features that aren't supported by .NET Core such as audio and video devices it's a huuuuge job.

Instead the plan is to try and port the VP8 video codec to C#, see VP8.Net. That would mean this library could then be used in .NET Core applications for audio/video WebRTC streams + data channels WITHOUT any native libraries required. The audio and video sinks and sources would need to be implemented separately for each platform. The benefit would be the sinks and source components would only have to deal with the audio and/or video devices and not anything to do with WebRTC.

If you do give this library a spin I'd love to hear how you go!

[bitbound]

Thank you for the quick reply! When I have some free time, I'll try it out and report back. Take care!

[bitbound]

I tried swapping out MixedReality WebRTC with SIP Sorcery this weekend. I ran into a few roadblocks, though.

The first issue I had was addIceCandidate failing on the browser side. I eventually realized it was because "candidate:" wasn't being prepended to the candidate string. I don't know if that's normal, but after manually prepending, it started working.

After that, I had some problems with DataChannels. I need to push a lot of data through (file, screenshots, etc.), and I kept getting deadlocks in ThreadedAssociation.sendAndBlock. It looks like something's not working correctly with the congestion control.

I then realized that RTCDataChannel.bufferedAmount never changes, so my own means of controlling backpressure wasn't working.

[sipsorcery]

The first issue I had was addIceCandidate failing on the browser side. I eventually realized it was because "candidate:" wasn't being prepended to the candidate string. I don't know if that's normal, but after manually prepending, it started working.

Best bet is to stick with JSON serialisation. The sipsorcery library should work fine with the browser's JSON serialise and deserialise calls. The candidate: prefix is part of SDP rather than being an ICE setting. But again you don't have to get into those details if you use JSON.

The Getting Started WebRTC example has the code you need. Here's how to deserialise ICE candidates in the browser:

ws = new WebSocket(document.querySelector('#websockurl').value, []);
            ws.onmessage = async function (evt) {
                var obj = JSON.parse(evt.data);
                if (obj?.candidate) {
                    pc.addIceCandidate(obj);
                }
                else if (obj?.sdp) {
                    await pc.setRemoteDescription(new RTCSessionDescription(obj));
                    pc.createAnswer()
                        .then((answer) => pc.setLocalDescription(answer))
                        .then(() => ws.send(JSON.stringify(pc.localDescription)));
                }
            };

After that, I had some problems with DataChannels.

Yes, as alluded to in my post above more cracks have appeared in the data channels implementation. I've reluctantly come to the decision it will require a rewrite of the SCTP implementation from scratch. I'll hopefully be starting on it this week and hopefully it will only take a matter of weeks.

If you've got some test code for you app and the sipsorcery library written then maybe come back and check again in a month or so when hopefully data channels will be in a better state.

[bitbound]

Thanks for the reply. :) I'd be happy to test this again after the update. I'll keep the branch up-to-date in my repo.

As for the candidate: prefix, I see now that the RTCIceCandidate.toJson method prepends it. However, I'm using SignalR Core with MessagePack serialization instead of JSON, so I'm getting the properties directly from the RTCIceCandidate.

On the SignalR Hub, it's received and sent to the browser like so:

        public async Task SendIceCandidateToBrowser(string candidate, string sdpMid, ushort sdpMLineIndex, string usernameFragment, string viewerConnectionID)
        {
            await Connection.SendAsync("SendIceCandidateToBrowser", candidate, sdpMid, sdpMLineIndex, usernameFragment, viewerConnectionID);
        }

Edit: Just to be clear, the candidate prefix thing isn't an issue or problem. It just confused me at first. :)

[bitbound]

DataChannels are working great now! :) Thanks!