Random issues in Sip Profile (suspect squlite DB corruption) #2296
Comments
|
One more note: affected extensions with BLF sticking, if we move the extension to a different sip profile, the problem clears itself on that extension immediately (but does NOT clear any other extensions of the BLF issue on that domain). |
|
See if that PR helps freeswitch/sofia-sip#233 |
|
I do not believe the referenced bug aligns with my case. My SIP Profile has "wss-binding" disabled and I have never used it. Also, Freeswitch does not crash, and when the bug occurs, its effect is limited to a single sip profile; all other sip profiles are functioning as usual. So far, this bug has only occurred on the sip profile "internal" (5060/5061 TLS), but has happened multiple times. |
|
Sorry, read more details on the PR, and it is unclear if its fixing just file corruption, or other corruption that could be limited to a single profile. We do use TLS for almost all of our SIP registrations. That said, we had about 340 devices each on two different profiles, and only one profile was affected. At no point did freeswitch in general go down/crash/etc; everything was contained to a single profile. |
|
The bug seems to be within libsofia's tport code. I am unsure if it affects TLS connections as well as WSS but the symptoms would include random file overwrites but also random sockets being closed when they should not be. I suspect that could display as a wide variety of symptoms but I'm not sure if the ones you describe fit the bill. I am suspecting not but am no expert on this, just someone who has that issue to the point where our service was unusable until the recent PR 233 patches. |
|
This issue was NOT resolved by the above referenced PR. As of Freeswitch 1.10.11, the issue remains. |
|
Can report same issues with internal profile, random. Stopping freeswitch and deleting I've actually now moved the db into postgres instead of sqlite - and the problem remains. Edit: Can't intercept either... 2024-10-15 15:51:49.862145 96.07% [ERR] switch_core_sqldb.c:1322 SQL ERR: [SELECT uuid, call_uuid, hostname FROM channels WHERE callstate IN ('RINGING', 'EARLY') AND (1 <> 1 OR direction = 'outbound' ) AND (1<>1 OR presence_id = '[email protected]' ) ORDER BY created_epoch DESC LIMIT 1 ] no such table: channelsfreeswitch 1.10.12-release-10222002881-a88d069d6f~bookworm amd64 |
|
What version of sofia-sip is in use? |
|
Just an FYI: I continue to experience the same problem after several
freeSwitch updates. I haven't found a way to determine what version of
Sofia is in use, can you provide a means of retrieving that from a running
instance?
…On Wed, Oct 16, 2024 at 8:01 AM themsley-voiceflex ***@***.***> wrote:
What version of sofia-sip is in use?
—
Reply to this email directly, view it on GitHub
<#2296 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGMKYV5PHMZALWNBTN4YBFTZ3Z5VVAVCNFSM6AAAAABKRTINX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJXGA4TQMJQGE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
I don't run f/s on Debian but a quick google for There were fixes to that package between 1.13.15 and 1.13.17 that stopped it from randomly writing to open filehandles that could have been pointing at anything. I saw it attempt (and fail) to write to /etc/passwd and to its own core.db file among many others. |
|
Another weird thing since upgrading to 1.10.12, this core.db with a space at the end: ls -lh /var/lib/freeswitch/db/|grep core
-rw-r----- 1 www-data www-data 488K Oct 17 09:52 core.db
-rw-r----- 1 www-data www-data 16K Oct 15 23:45 'core.db ' |
ii libsofia-sip-ua0 1.13.17-11077684608-9e5c40ed11~bookworm amd64 Sofia-SIP library runtime |
You can just find what version is installed on the system. dpkg -l |grep sofiaii freeswitch-mod-sofia 1.10.12-release-10222002881-a88d069d6f~bookworm amd64 mod_sofia for FreeSWITCH
ii freeswitch-mod-sofia-dbg 1.10.12-release-10222002881-a88d069d6f~bookworm amd64 mod_sofia for FreeSWITCH (debug)
ii libsofia-sip-ua0 1.13.17-11077684608-9e5c40ed11~bookworm amd64 Sofia-SIP library runtimeTry |
|
In my case, it is compiled from source from a "curated" source repository
(distributed with my PBX platform -- FusionPBX). So I'm still left without
anyway of knowing.
In my opinion, sofia should identify during startup or possibly with a CLI
command what its version is.
In the past, trying to address this, I had to recompile ONLY sofia with an
updated version of sofia that was not yet incorporated into the main source
tree. Once complete, I had no way of verifying that the new code was in
fact in use. A version identifier would fix this.
On Wed, Oct 16, 2024 at 4:00 PM Jad Seifeddine ***@***.***>
wrote:
… Just an FYI: I continue to experience the same problem after several
freeSwitch updates. I haven't found a way to determine what version of
Sofia is in use, can you provide a means of retrieving that from a running
instance?
… <#m_7273073828887614620_>
On Wed, Oct 16, 2024 at 8:01 AM themsley-voiceflex *@*.*> wrote: What
version of sofia-sip is in use? — Reply to this email directly, view it on
GitHub <#2296 (comment)
<#2296 (comment)>>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGMKYV5PHMZALWNBTN4YBFTZ3Z5VVAVCNFSM6AAAAABKRTINX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJXGA4TQMJQGE
<https://github.com/notifications/unsubscribe-auth/AGMKYV5PHMZALWNBTN4YBFTZ3Z5VVAVCNFSM6AAAAABKRTINX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJXGA4TQMJQGE>
. You are receiving this because you were mentioned.Message ID: @.*>
You can just find what version is installed on the system.
On debian based systems you could do
dpkg -l |grep sofia
ii freeswitch-mod-sofia 1.10.12-release-10222002881-a88d069d6f~bookworm amd64 mod_sofia for FreeSWITCHii freeswitch-mod-sofia-dbg 1.10.12-release-10222002881-a88d069d6f~bookworm amd64 mod_sofia for FreeSWITCH (debug)ii libsofia-sip-ua0 1.13.17-8116788228-50a509b72c~bookworm amd64 Sofia-SIP library runtime
—
Reply to this email directly, view it on GitHub
<#2296 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGMKYV7PPRJWFEZS5H2FMBTZ33V2NAVCNFSM6AAAAABKRTINX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJYGEYDSMRVGA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@kusznir you could do the following: Find the lib file like with libsofia-sip-ua.so.0 (libc6,x86-64) => /lib/libsofia-sip-ua.so.0Then find the version inside the lib file with libsofia-sip-ua.so.0
sofia-sip-1.13.17 |
|
see #2619 Seem's like it's fixed but not yet released - I built from source now running I will report back - but we were having exact same issues, stuck BLF, unable to page etc. I think the root cause is that the call channels stay open even after a call is finished. Lets see... |
Any update on your test? |
Yes - it did fix it. No longer having the problem. |
|
Is there a plan for when this will be incorporated into a full release?
…On Sun, Oct 27, 2024, 2:36 AM Jad Seifeddine ***@***.***> wrote:
@kusznir <https://github.com/kusznir>
see #2619 <#2619>
Seem's like it's fixed but not yet released - I built from source now
running 1.10.13-dev git 97cb672
I will report back - but we were having exact same issues, stuck BLF,
unable to page etc.
I think the root cause is that the call channels stay open even after a
call is finished.
Lets see...
Any update on your test?
Yes - it did fix it.
No longer having the problem.
—
Reply to this email directly, view it on GitHub
<#2296 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGMKYVYQ7VU7YBUQHBTKKKTZ5SXZ3AVCNFSM6AAAAABKRTINX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZZHEZDOMRRHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
With 1.10.10, after operating for a few days, one of my SIP profile's will develop corruption. Initially it shows itself as SOME (and only SOME, not all) domains on that profile will experience BLF "sticking". BLF will correctly update to a ringing state, with caller ID update, and it will stop blinking (ringing) correctly, but it will be stuck on in use indefinitely. The problem will escalate until eventually that profile will suddenly stop responding to SIP commands (INVITES, OPTIONS, etc), causing all devices on that profile to eventually show unregistered, and immediately stops call processing. This only effects one profile (I have 4 running on this server).
So far, it appears to be random. It has occurred on two servers (both in a BDC cluster), and does not show up immediately. Stopping freeswitch, deleting the sqlite dbs, and restarting it will fix it for a while, but the problem will eventually re-occur. No specific triggers found thus far.
I am not a programmer, so I have no idea how to do any kind of debug traces. The servers in question are in production with a total of about 700 extensions subscribed, so any issues affecting performance have to be corrected immediately (restart, clear, etc); time taken to further troubleshoot is unacceptable, as it prolongs the service outage.
We have not been able to reproduce on lab server, only production server with production traffic volumes are affected.
The text was updated successfully, but these errors were encountered: