Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WebRTC stream stops working #139

Open
mnorrsken opened this issue Jul 8, 2023 · 30 comments
Open

WebRTC stream stops working #139

mnorrsken opened this issue Jul 8, 2023 · 30 comments
Labels
bug Something isn't working

Comments

@mnorrsken
Copy link

mnorrsken commented Jul 8, 2023 •
edited
Loading

What happened

Installed the latest version with all the compiling etc, got it running with WebRTC it was very nice for a while, but after a few minutes/hours it just stops working. /stream and /snapshot still works but not /webrtc

It starts working again if I restart crowsnest service from within mainsail.

Ive reverted and disabled RTSP and only running MJPG again. Not a big deal, I just wanted to report this.

What did you expect to happen

WebRTC streaming working indefinitely

How to reproduce

This is on my MainsailOS 1.0.1 RPi 3 b+ fitted with a Logitech C270 USB camera. Reproduced by running WebRTC for a while in Mainsail (with a web client open for approx an hour or so).

Additional information

I didn't find any errors, and cpu/mem usage of host was ok, but I found a few suspect websocket lines in journal.
crowsnest.txt
suspect.txt
@mnorrsken mnorrsken added the bug Something isn't working label Jul 8, 2023
@mnorrsken
Copy link
Author

Woops this seems to be a camera-streamer bug. Will test some things and report back here before closing.

@mnorrsken
Copy link
Author

I increased my RPi video ram from 160 to 256mb and this particular issue seems to be gone, or at least havnt happened for 12+ hours.

@PSandro
Copy link

PSandro commented Jul 20, 2023

increasing the GPU memory also resolved this issue for me. Should we take this up into documentation/readme?

@mdvorak
Copy link

mdvorak commented Jul 25, 2023

Increasing video ram did not fix the issue for me. It still fails after a while, streaming 800x600 resolution from USB webcam.

@stonemandave
Copy link

Still an issue. I have 2 printers same setup and one works while the other just stops.

@4rianton
Copy link

still an issue..my rtc feed keeps stopping with both of my cameras. GPU memory set to 256..

@SaltyPaws
Copy link

I can confirm I have the same issue on both my Voron zero and 2.4 mainly with raspicam v1. The voron 2.4 is also running a logitech USB webcam, which keeps running significantly longer compared to the raspicam but eventually also goes black. a reboot solves the issue.
hardware:
2.4
/dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_26E11DCF-
Raspberry Pi 4 Model B Rev 1.1

both systems:
Operating System: Raspbian GNU/Linux 11 (bullseye)
Kernel: Linux 6.1.47-v8+
Architecture: arm64

Raspicam v1: /base/soc/i2c0mux/i2c@1/ov5647@36
Computer: RPI4
Software version (latest as of oct 25)
crowsnest v4.0.4-6-g767c53aa
fluidd v1.26.1
klipper v0.11.0-299-gb1f597c5
Klipper-Adaptive-Meshing-Purging v1.1.2-4-g2389a994
moonraker v0.8.0-186-g2641fc54

gpumem: 128

@SaltyPaws
Copy link

Even with gpumem 256, the WebRTC for the raspicam still stops streaming.

@mryel00 mryel00 reopened this Oct 29, 2023
@mryel00
Copy link
Member

mryel00 commented Oct 29, 2023 •
edited
Loading

Forgot to reopen this issue. Can you guys please try to use --camera-force_active=1 as a custom_flag?

@recrudesce
Copy link

recrudesce commented Oct 29, 2023 •
edited
Loading

Forgot to reopen this issue. Can you guys please try to use --camera-force_active=1 as a custom_flag?

Where do we set this ?
Never mind, worked it out :)

@mryel00
Copy link
Member

mryel00 commented Oct 29, 2023 •
edited
Loading

Where do we set this ?

Inside the crowsnest.conf under custom_flags

@recrudesce
Copy link

Literally just worked it out as you replied :P Thanks !

@mryel00
Copy link
Member

mryel00 commented Oct 29, 2023

Also just as an overall information for this. We couldn't reproduce this. So please attach a debug log next time with the issue reproduced, that we can maybe try to make some assumptions based on the output.

@SaltyPaws
Copy link

crowsnest.log

With --camera-force_active=1 flag in combination with gpumem=256 I have had no more issues. After 48hr+ both cameras are still running. I have attached my log file. Perhaps we just need to update the documentation. I will report back if I encounter any more issues.

@mryel00
Copy link
Member

mryel00 commented Nov 1, 2023

@SaltyPaws thank you for the confirmation, that this works for you too. Next time a debug log would be nice. To create one, just set log_level: debug inside your config. Someone on the Discord found out that this flag might help and there was a line about restarting the camera inside the debug log. So if something like that also appears for others, it's most likely the cam itself that shuts down.
Also I would need some confirmation, if the gpumem=256 is needed too or if --camera-force_active=1 is enough. In the case of the latter, I would just add it to the default start parameters.

mryel00 added a commit to mryel00/crowsnest-dev that referenced this issue Nov 1, 2023
@mryel00
fix(camera-streamer.sh): force camera to be always active
a8b1ea6 
Potentially fixes mainsail-crew#139
@mryel00 mryel00 mentioned this issue Nov 3, 2023
Merged
@meteyou meteyou closed this as completed in 293768f Nov 3, 2023
@SaltyPaws
Copy link

SaltyPaws commented Nov 3, 2023 •
edited
Loading

To further confirm this is the right fix, after 5 days, my USB camera shutdown, it did not have --camera-force_active=1 in the config. The Raspicam V1 was still up and running (it has the flag). Wow, wait 5 days to confirm a bug, I would call that hard to find! I will update the flag for both cams, and change gpumem back to 128 to test.

@SaltyPaws
Copy link

@SaltyPaws thank you for the confirmation, that this works for you too. Next time a debug log would be nice. To create one, just set log_level: debug inside your config. Someone on the Discord found out that this flag might help and there was a line about restarting the camera inside the debug log. So if something like that also appears for others, it's most likely the cam itself that shuts down. Also I would need some confirmation, if the gpumem=256 is needed too or if --camera-force_active=1 is enough. In the case of the latter, I would just add it to the default start parameters.

Looks like gpumem=256 is not needed. Running for 2 days now with gpumem=128 and --camera-force_active=1 on both cameras, everything is still up and running.

@mryel00
Copy link
Member

mryel00 commented Nov 5, 2023

@SaltyPaws Just for your information. --camera-force_active=1 is now set by default, since Crowsnest v4.0.5 released 2 days ago.

@mryel00 mryel00 mentioned this issue Nov 8, 2023
Closed
@mryel00
Copy link
Member

mryel00 commented Nov 15, 2023 •
edited
Loading

We had to remove this parameter from default values as it causes problems with some printers #202
So please set --camera-force_active=1 again manually inside the custom_flags. We are already in touch with the developer of camera-streamer to find a solution for this.

v4.0.5 has it set by default and v4.1.0 removed that default again.
v4.1.0 only added one new feature #203 that is most likely only needed for SpeederPads and changed some things on the installer. So you can stay on v4.0.5 for now, if you like, without missing out on cool new features.

@mryel00 mryel00 reopened this Nov 15, 2023
@Gyrohammer
Copy link

Gyrohammer commented Nov 16, 2023 •
edited
Loading

I've just had this happen as well with a generic USB cam. Logs indicate no issue, crowsnest looks normal. A restart fixes it but it goes offline within three to four days.

I just restarted the Pi, crowsnest has been updated to v4.1.0 and --camera-force_active=1 has been applied. I'll update this when it goes offline next to get some kind of timeline.

Edit: The camera has just gone offline again, I've restarted crowsnest and its blank. Unsure what happened, crowsnest logs look fine. Going to try this solution and see how long it holds.

@mryel00
Copy link
Member

mryel00 commented Nov 16, 2023

Edit: The camera has just gone offline again, I've restarted crowsnest and its blank. Unsure what happened, crowsnest logs look fine. Going to try this solution and see how long it holds.

This shouldn't help in any way for this. It seems like there was a bug with ustreamer in the beginning, that one is already fixed for quite some time. Also this problem is about camera-streamer and not ustreamer.

Within 3-4 days I would consider fine. camera-streamer WebRTC isn't perfect, but this issue is something that is pretty hard to debug as it seems like appearing at random. --camera-force_active=1 seems to help but there are still some user reporting issues with a "fast" disconnect, like you for example.

I also recommend everyone to always attach a log as there are some good and maybe helpful informations about your setup.

@hardKOrr
Copy link

hardKOrr commented Nov 22, 2023 •
edited
Loading

I'm running into this issue as well, I have a debug
crowsnest(12).log.

I am on a Raspberry Pi 3b+ using pi camera v1.2, I've set gpu memory to 256 and also added --camera-force_active=1 the pi is on WiFi.

@recrudesce
Copy link

recrudesce commented Dec 9, 2023 •
edited
Loading

I get this issue with RTSP streams too - after a while they just stop working.
Pi4, MainsailOS, USB camera.

Usually restarting crowsnest fixes the issue, unless it kernel panics the Pi, of course

image
crowsnest.log

Log attached.

@mryel00
Copy link
Member

mryel00 commented Dec 9, 2023

@recrudesce Ofc the rtsp will break down too. Both are provided by camera-streamer.
You don't have uploaded your log yet?
Does this kernel panic happens as soon as it breaks down or is it related to something else? Do you use the --camera-force_active=1 parameter?

@recrudesce
Copy link

log uploaded, no idea why github didn't like it before.

The kernel panic happens sometimes if I restart the service after it stops working. Not all the time, but sometimes. Then the SSH connection dies and the only way I can get back into the Pi is to physically unplug it to power it down.

I do use the parameter, yeah.

@mryel00
Copy link
Member

mryel00 commented Dec 9, 2023

I recommend to remove that parameter again. Your camera-streamer is restarting every 20s.
So you are one of the unlucky ones where this doesn't help.

@recrudesce
Copy link

I think this is a camera-streamer issue, because running camera-streamer separately (without crowsnest), I still get the crashes on service restart, and webrtc just stopping. I'll raise an issue over there.

@mryel00
Copy link
Member

mryel00 commented Dec 27, 2023

Crowsnest is just starting camera-streamer for you with maybe some extra parameters, like we tried with --camera-force_active=1.
I thought I already posted it here, but it doesn't seem like it. There is already a post about this on the camera-streamer GitHub: ayufan/camera-streamer#94

@g14nluc4

This comment was marked as off-topic.

Sign in to view
@meteyou
Copy link
Member

meteyou commented Jan 14, 2024

@g14nluc4 please do not write your problem in any issue. If you need help/support, you are generally in the wrong place in the GitHub issue tracker. there is more information here: https://docs.mainsail.xyz/faq/getting-help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests