Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Corrupt packages and visual artefacts in SRT Server #855

Open
backface opened this issue Nov 27, 2024 · 6 comments
Open

Corrupt packages and visual artefacts in SRT Server #855

backface opened this issue Nov 27, 2024 · 6 comments
Assignees
@ioppermann
Labels
bug Program error or software error or software anomaly.

Comments

@backface
Copy link

backface commented Nov 27, 2024 •
edited
Loading

Describe the bug

Corrupt packages in the SRT Servers output stream that correlate with visual artifacts in the video

Process log entries show up like:

2024-11-25T10:30:28.000Z [in#0/mpegts @ 0x7e349dfb6a80] [warning] corrupt input packet in stream 0

or sometimes also as:

2024-11-27T11:26:32.000Z [mpegts    ] [warning] changing packet size to 204
2024-11-27T11:26:32.000Z [mpegts    ] [warning] changing packet size to 192
2024-11-27T11:26:32.000Z [mpegts    ] [warning] changing packet size to 188
2024-11-27T11:26:32.000Z [mpegts    ] [warning] changing packet size to 204
2024-11-27T11:26:32.000Z [mpegts    ] [warning] changing packet size to 192
2024-11-27T11:26:32.000Z [mpegts    ] [warning] changing packet size to 204
2024-11-27T11:26:32.000Z [mpegts    ] [warning] changing packet size to 192
2024-11-27T11:26:32.000Z [mpegts    ] [warning] changing packet size to 188
2024-11-27T11:26:32.000Z [mpegts    ] [warning] Packet corrupt (stream = 0, dts = NOPTS).

this is also a re-filing - or sum up - of existing issues that seem to be related or show the same problem such as #839 and #755

After a series of tests, I can confirm that this happens regardless of the input video, whether it is a still image or not, regardless of the encoder (OBS on Mac and ffmpeg on Linux), whether the upstream is delivered from a remote machine via network or with ffmpeg from localhost or with a latency setting of latency=1000000 or more. I have tried running Restreamer on 2 different machines (different versions of Ubuntu and Debian) and with network_mode=host.
Sometimes, it shows up more frequently than at other times or is more or less obvious (it might certainly be enhanced by network issues!), but it shows nonetheless, in best cases, only every 4 or 5 minutes.

Crucially this problem does not appear when using the RTMP server or when bypassing restreamer sending the upstream directly to a different SRT server. So far I can only conclude that this must be a bug in the SRT server implementation. If I can do anything reasonable to track this down let me know.

To Reproduce

  • Start a docker instance of restreamer latest (v2.12.0)
  • Create a channel with an SRT server and send an upstream, for instance with
ffmpeg -re \
  -stream_loop -1 \
  -i BigBuckBunny.mp4 \
  -vcodec libx264 \
  -g 60 \
  -b:v 5M \
  -ar 44100 -acodec aac \
  -b:a 128K \
  -f mpegts \
  "srt://localhost:6000?mode=caller&transtype=live&streamid=[....]&latency=1000000"
  • Watch video/logs for up to 4 or 5 minutes.

Expected behavior

Clear, uncorrupted video stream

Screenshots

Bildschirmfoto_2024-11-27_00-05-54

Desktop (please complete the following information):

  • OS: Several (Ubuntu, Debian,, Mac OS)
  • Browser: Several (Firefox and others, as well as none/or headless in another srt or rtmp publication)

Additional context

  • Server OS: Ubuntu 24.04.1 LTS, Debian GNU/Linux 12, Mac OS
  • Docker version 27.3.1, Docker version 23.0.4
  • restreamer-v2.12.0 / datarhei-core v16.16.0 (linux/amd64)
@backface backface added the bug Program error or software error or software anomaly. label Nov 27, 2024
@ioppermann ioppermann self-assigned this Nov 27, 2024
@ioppermann
Copy link
Member

Thanks for this detailed bug report. Unfortunately, I couldn't reproduce yet the described bug (I tested on MacOS with ffmpeg and Restreamer on the same machine), I have a suspicion where the problem might be.

You send the stream with ffmpeg and with a certain latency to the Restreamer SRT server. Restreamer is then pulling the stream via SRT from it's own SRT server with a certain latency in order to convert it into HLS. This internal SRT latency is by default 20ms and cannot be changed currently. This might cause the issues when there are some packet drops internally.

The solution would be to make the internal latency configurable.

To exclude that the SRT server itself is the problem, try to pull the stream from the SRT server (e.g. with ffplay) with a different latency. Use the same URL as for sending, but remove the ,mode:publish part from the streamid parameter.

ffplay -f mpegts -i "srt://localhost:6000?mode=caller&transtype=live&streamid=[...]&latency=100000"

@backface
Copy link
Author

yes, I am also seeing these artifacts and corrupt packages if directly pulled by ffplay and even higher latencies. Most surprisingly I am seeing even more of these artifacts (and corrupt packages logs) in the stream received with ffplay - despite the latency settings - than in the restreamer preview when watched side by side!

@ioppermann
Copy link
Member

I tried to reproduce this. My setup is a MacBook Pro that is sending the stream to the Restreamer SRT server running on a Raspberry Pi 4. And also pulling from the SRT server back to the MacBook. All via wifi.

Sending the stream:

ffmpeg -re -stream_loop -1 -i big_buck_bunny.mp4 -codec copy -f mpegts "srt://192.168.1.154:6000?mode=caller&transtype=live&streamid=e0016116-a18d-42fc-b542-3570a371cd41.stream,mode:publish&latency=120000"
ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
  built with Apple clang version 16.0.0 (clang-1600.0.26.3)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/7.1 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.100 / 61. 19.100
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'big_buck_bunny.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf61.7.100
  Duration: 00:09:56.46, start: 0.000000, bitrate: 5589 kb/s
  Stream #0:0[0x1](eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1280x720, 5146 kb/s, 24 fps, 24 tbr, 19200 tbn (default)
      Metadata:
        handler_name    : Apple Video Media Handler
        vendor_id       : [0][0][0][0]
        encoder         : H.264
        timecode        : 00:00:00:00
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 437 kb/s (default)
      Metadata:
        handler_name    : Apple Sound Media Handler
        vendor_id       : [0][0][0][0]
  Stream #0:2[0x3](eng): Data: none (tmcd / 0x64636D74)
      Metadata:
        handler_name    : Apple Video Media Handler
        timecode        : 00:00:00:00
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Output #0, mpegts, to 'srt://192.168.1.154:6000?mode=caller&transtype=live&streamid=e0016116-a18d-42fc-b542-3570a371cd41.stream,mode:publish&latency=120000':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf61.7.100
  Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1280x720, q=2-31, 5146 kb/s, 24 fps, 24 tbr, 90k tbn (default)
      Metadata:
        handler_name    : Apple Video Media Handler
        vendor_id       : [0][0][0][0]
        encoder         : H.264
        timecode        : 00:00:00:00
  Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 437 kb/s (default)
      Metadata:
        handler_name    : Apple Sound Media Handler
        vendor_id       : [0][0][0][0]
Press [q] to stop, [?] for help
frame=206779 fps= 24 q=-1.0 size= 6056052KiB time=02:23:35.76 bitrate=5758.2kbits/s speed=   1x    

[q] command received. Exiting.

[out#0/mpegts @ 0x600001f9c3c0] video:5392518KiB audio:460323KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 3.475855%
frame=206791 fps= 24 q=-1.0 Lsize= 6056277KiB time=02:23:36.27 bitrate=5758.1kbits/s speed=   1x

Pulling the data

ffmpeg -f mpegts -i "srt://192.168.1.154:6000?mode=caller&transtype=live&streamid=e0016116-a18d-42fc-b542-3570a371cd41.stream&latency=120000" -codec copy -f null -
ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
  built with Apple clang version 16.0.0 (clang-1600.0.26.3)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/7.1 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.100 / 61. 19.100
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
...
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] no frame!
Input #0, mpegts, from 'srt://192.168.1.154:6000?mode=caller&transtype=live&streamid=e0016116-a18d-42fc-b542-3570a371cd41.stream&latency=120000':
  Duration: N/A, start: 8.737667, bitrate: N/A
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
  Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, progressive), 1280x720, 24 fps, 24 tbr, 90k tbn
  Stream #0:1[0x101](eng): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, 5.1, fltp, 445 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf61.7.100
  Stream #0:0: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, progressive), 1280x720, q=2-31, 24 fps, 24 tbr, 90k tbn
  Stream #0:1(eng): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, 5.1, fltp, 445 kb/s
Press [q] to stop, [?] for help
14:22:40.936679/!W:SRT.br: @410272501: RCV-DROPPED 6 packet(s). Packet seqno %1907416130 delayed for 60.178 ms
[mpegts @ 0x1240050e0] Packet corrupt (stream = 0, dts = 4543500).
[in#0/mpegts @ 0x6000032a8300] corrupt input packet in stream 0
    Last message repeated 1 times
15:28:30.965826/!W:SRT.br: @410272501: RCV-DROPPED 123 packet(s). Packet seqno %1909626488 delayed for 1726.802 ms
[mpegts @ 0x1240050e0] Packet corrupt (stream = 0, dts = 359839050).
[in#0/mpegts @ 0x6000032a8300] corrupt input packet in stream 0
    Last message repeated 1 times
[mpegts @ 0x1240050e0] Packet corrupt (stream = 0, dts = 361174050).ed=   1x    
[in#0/mpegts @ 0x6000032a8300] corrupt input packet in stream 0
[mpegts @ 0x1240050e0] Packet corrupt (stream = 0, dts = 361211550).
[in#0/mpegts @ 0x6000032a8300] corrupt input packet in stream 0
[mpegts @ 0x1240050e0] Packet corrupt (stream = 0, dts = 762737700).eed=   1x    
[in#0/mpegts @ 0x6000032a8300] corrupt input packet in stream 0
[mpegts @ 0x1240050e0] Packet corrupt (stream = 0, dts = 762917700).eed=   1x    
[in#0/mpegts @ 0x6000032a8300] corrupt input packet in stream 0
frame=206499 fps= 24 q=-1.0 size=N/A time=02:23:25.45 bitrate=N/A speed=   1x    

[q] command received. Exiting.

[out#0/null @ 0x600003bac000] video:5386027KiB audio:462487KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame=206511 fps= 24 q=-1.0 Lsize=N/A time=02:23:25.95 bitrate=N/A speed=   1x

There were some errors, but only occasionally, not in the extreme extend you are describing.

@backface
Copy link
Author

backface commented Dec 4, 2024

hmm. I compared it with other SRT implementations, and interestingly I run into similar issues if I try a self-compiled version of https://github.com/Edward-Wu/srt-live-server (with Ubuntu's libsrt) over a local network and even localhost (on my most powerful machine). But I can distribute a clear, uncorrupted stream even over long-distance networks with ovenmediaengine - so there must be some key difference in srt server implementations...

@ioppermann
Copy link
Member

srt-live-server and OME are using the Haivision reference SRT implementation. Restreamer is using an implementation of the SRT protocol in Go. The SRT specification leaves some room for variation in implementation.

SRT is based on UDP, which might not be treated with highest priority by network devices on the way from you to the server and back and packets will get dropped. SRT tries to recover from packet loss, but it depends on the configured latency how good it manages to.

I updated the Restreamer's SRT server to the latest version. Please try out the datarhei/restreamer:dev image.

@backface
Copy link
Author

Just gave it a quick run: running datarhei/restreamer:dev image with default settings does not solve that issue.
Sorry but we have now moved to using OME.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ioppermann ioppermann

Labels
bug Program error or software error or software anomaly.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ioppermann @backface