Libuv/evt-tls segfaults when trying to write to a socket which is closed by the other party #20

Bendr0id · 2018-02-16T21:16:25Z

Libuv/evt-tls segfaults when trying to write to a socket which is closed by the other party

Libuv in this case just returns -1.

0  0x00007ffb845a1b54 in SSL_write () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
1  0x000000000057f710 in evt__tls__op (conn=0x25a46c0, op=EVT_TLS_OP_WRITE, buf=0x253bd52, sz=236) at /home/vidi/dev/src/c/xmrigCC/src/3rdparty/evt-tls/src/evt_tls.c:260
2  0x000000000057fa85 in evt_tls_write (c=0x25a46c0, msg=0x253bd52, str_len=236, on_write=0x57eca0 <on_evt_write>) at /home/vidi/dev/src/c/xmrigCC/src/3rdparty/evt-tls/src/evt_tls.c:354
3  0x000000000057edae in uv_tls_write (stream=0x25a4680, buf=0x7ffc09a6d0a0, cb=0x5426d6 <Client::onTlsWrite(uv_tls_s*, int)>) at /home/vidi/dev/src/c/xmrigCC/src/3rdparty/evt-tls/src/uv_tls.c:187

If you need more info, please let me know.

The text was updated successfully, but these errors were encountered:

deleisha · 2018-02-17T02:46:25Z

Thanks for letting us know this.

Would you please share output of gdb bt full for the dumped core.

Bendr0id · 2018-02-17T11:04:23Z

There you go:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007ffb845a1b54 in SSL_write () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
[Current thread is 1 (Thread 0x7ffb84dfe740 (LWP 18450))]
(gdb) bt full
#0  0x00007ffb845a1b54 in SSL_write () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
No symbol table info available.
#1  0x000000000057f710 in evt__tls__op (conn=0x25a46c0, op=EVT_TLS_OP_WRITE, buf=0x253bd52, sz=236) at /home/vidi/dev/src/c/xmrigCC/src/3rdparty/evt-tls/src/evt_tls.c:260
        r = 0
        bytes = 0
        tbuf = '\000' <repeats 16383 times>
        __PRETTY_FUNCTION__ = "evt__tls__op"
#2  0x000000000057fa85 in evt_tls_write (c=0x25a46c0, msg=0x253bd52, str_len=236, on_write=0x57eca0 <on_evt_write>) at /home/vidi/dev/src/c/xmrigCC/src/3rdparty/evt-tls/src/evt_tls.c:354
No locals.
#3  0x000000000057edae in uv_tls_write (stream=0x25a4680, buf=0x7ffc09a6d0a0, cb=0x5426d6 <Client::onTlsWrite(uv_tls_s*, int)>) at /home/vidi/dev/src/c/xmrigCC/src/3rdparty/evt-tls/src/uv_tls.c:187
        __PRETTY_FUNCTION__ = "uv_tls_write"
        evt = 0x25a46c0
#4  0x0000000000540c4a in Client::send (this=0x253b4d0, size=236) at /home/vidi/dev/src/c/xmrigCC/src/net/Client.cpp:301
        buf = {base = 0x253bd52 "{\"id\":2,\"jsonrpc\":\"2.0\",\"method\":\"submit\",\"params\":{\"id\":\"3488843e-244a-408c-9527-bf454fd564d0\",\"job_id\":\"yUAEPcuTAZ3LvjX3+wMvdDJsRS3a150\",\"nonce\":\"70ad2a15\",\"result\":\"d749883bef0a86f009b5d100bf711b7e"..., 
          len = 236}
#5  0x0000000000540657 in Client::submit (this=0x253b4d0, result=...) at /home/vidi/dev/src/c/xmrigCC/src/net/Client.cpp:187
        nonce = "70ad2a15"
        data = "d749883bef0a86f009b5d100bf711b7e3ecc8722b4b66fe960a05eaf89870600"
        size = 236
#6  0x000000000054cca5 in SinglePoolStrategy::submit (this=0x253b4a0, result=...) at /home/vidi/dev/src/c/xmrigCC/src/net/strategies/SinglePoolStrategy.cpp:43
No locals.
#7  0x000000000054a974 in Network::onJobResult (this=0x253b230, result=...) at /home/vidi/dev/src/c/xmrigCC/src/net/Network.cpp:128
No locals.
#8  0x0000000000550425 in Workers::onResult (handle=0x8a5180 <Workers::m_async>) at /home/vidi/dev/src/c/xmrigCC/src/workers/Workers.cpp:171
        result = {poolId = 0, jobId = {m_data = "yUAEPcuTAZ3LvjX3+wMvdDJsRS3a150", '\000' <repeats 32 times>}, diff = 5000, nonce = 355118448, result = "\327I\210;\357\n\206\360\t\265\321\000\277q\033~>̇\"\264\266o\351`\240^\257\211\207\006"}
        __for_range = @0x7ffc09a6d210: {<std::__cxx11::_List_base<JobResult, std::allocator<JobResult> >> = {_M_impl = {<std::allocator<std::_List_node<JobResult> >> = {<__gnu_cxx::new_allocator<std::_List_node<JobResult> >> = {<No data fields>}, <No data fields>}, 
              _M_node = {<std::__detail::_List_node_base> = {_M_next = 0x25b66f0, _M_prev = 0x25b66f0}, _M_data = 1}}}, <No data fields>}
        __for_begin = {_M_node = 0x25b66f0}
        __for_end = {_M_node = 0x7ffc09a6d210}
        results = {<std::__cxx11::_List_base<JobResult, std::allocator<JobResult> >> = {_M_impl = {<std::allocator<std::_List_node<JobResult> >> = {<__gnu_cxx::new_allocator<std::_List_node<JobResult> >> = {<No data fields>}, <No data fields>}, 
              _M_node = {<std::__detail::_List_node_base> = {_M_next = 0x25b66f0, _M_prev = 0x25b66f0}, _M_data = 1}}}, <No data fields>}
#9  0x00007ffb849f4553 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
No symbol table info available.
#10 0x00007ffb849f4636 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
No symbol table info available.
#11 0x00007ffb84a03055 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
No symbol table info available.
#12 0x00007ffb849f4efc in uv_run () from /usr/lib/x86_64-linux-gnu/libuv.so.1
No symbol table info available.
#13 0x000000000050c410 in App::start (this=0x253a4e0) at /home/vidi/dev/src/c/xmrigCC/src/App.cpp:179
        r = 0
#14 0x0000000000551c06 in main (argc=8, argv=0x7ffc09a70a28) at /home/vidi/dev/src/c/xmrigCC/src/xmrig.cpp:29
        app = 0x253a4e0
        res = 0

deleisha · 2018-02-20T07:10:36Z

Looks like, SSL_write is failing due to corrupted structures before reaching the network, libuv.

Are you on the latest code? there was fixes handling this type of issue.

Again, libuv-tls is just a sample and will need more work to be production ready

Bendr0id · 2018-02-20T07:23:08Z

Seems to be working quite well, except this. I'm using a snapshot from last week. Am 20.02.2018 08:10 schrieb dlmeetei <[email protected]>: Looks like, SSL_write is failing due to corrupted structures before reaching the network, libuv. Are you on the latest code? there was fixes handling this type of issue. Again, libuv-tls is just a sample and will need more work to be production ready — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#20 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAXZgYeU--27LjAqc1DY5__hC8DRg51Qks5tWm_sgaJpZM4SI74y>.

Bendr0id · 2018-02-20T10:57:49Z

It's not closed.. Problem still exists. Am 20.02.2018 11:56 schrieb dlmeetei <[email protected]>: Thanks for confirming. Closing this then — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#20 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAXZgTz176reqYoLuBPqcbfcleWOgKULks5tWqThgaJpZM4SI74y>.

deleisha · 2018-02-20T10:59:21Z

Sorry, closed by misreading lines.
Working on to reproduce it

deleisha · 2018-02-20T11:15:24Z

@Bendr0id
Before trying to reproduce it, Could please help me with this
Unlike other calls from bt full , the call to SSL_write does not show args, I tend to believe, those might be corrupted.
This calls writes to BIO_pair and occurs before NIO layer(libuv)
0x00007ffb845a1b54 in SSL_write () from /lib/x86_64-linux-gnu/libssl.so.1.0.0

Can you get bt full after installing sudo apt install libssl1.0.0-dbg( assuming you are on ubuntu)

Bendr0id · 2018-02-20T11:20:12Z

Yes, will create another core dump later.

Btw. here are the steps to reproduce:

Open a connection to a TLS server, keep the connection open.
Then shutdown the server and try to write to the socket from the client.

It should recognize that the server is gone and return an error instead of segfaulting.

Bendr0id · 2018-02-21T21:46:34Z

Sorry for late response..

Program terminated with signal SIGSEGV, Segmentation fault.
#0  SSL_write (s=0x0, buf=0x253bd52, num=236) at ssl_lib.c:1038
1038	ssl_lib.c: No such file or directory.
[Current thread is 1 (Thread 0x7ffb84dfe740 (LWP 18450))]
(gdb) bt full
#0  SSL_write (s=0x0, buf=0x253bd52, num=236) at ssl_lib.c:1038
No locals.
#1  0x000000000057f710 in evt__tls__op (conn=0x25a46c0, op=EVT_TLS_OP_WRITE, buf=0x253bd52, sz=236) at /home/vidi/dev/src/c/xmrigCC/src/3rdparty/evt-tls/src/evt_tls.c:251
        r = 0
        bytes = 0
        tbuf = '\000' <repeats 16383 times>
        __PRETTY_FUNCTION__ = "evt__tls__op"
#2  0x000000000057fa85 in evt_tls_connect (con=0x25a46c0, cb=0x253bd52) at /home/vidi/dev/src/c/xmrigCC/src/3rdparty/evt-tls/src/evt_tls.c:335
No locals.
#3  0x000000000057edae in on_evt_write (tls=0x25a4680, status=32764) at /home/vidi/dev/src/c/xmrigCC/src/3rdparty/evt-tls/src/uv_tls.c:176
        __PRETTY_FUNCTION__ = "on_evt_write"
        ut = 0x25a46c0
#4  0x0000000000540c4a in Client::send (this=0x253b4d0, size=236) at /home/vidi/dev/src/c/xmrigCC/src/net/Client.cpp:301
        buf = {base = 0x253bd52 "{\"id\":2,\"jsonrpc\":\"2.0\",\"method\":\"submit\",\"params\":{\"id\":\"3488843e-244a-408c-9527-bf454fd564d0\",\"job_id\":\"yUAEPcuTAZ3LvjX3+wMvdDJsRS3a150\",\"nonce\":\"70ad2a15\",\"result\":\"d749883bef0a86f009b5d100bf711b7e"..., 
          len = 236}
#5  0x0000000000540657 in Client::submit (this=0x253b4d0, result=...) at /home/vidi/dev/src/c/xmrigCC/src/net/Client.cpp:187
        nonce = "70ad2a15"
        data = "d749883bef0a86f009b5d100bf711b7e3ecc8722b4b66fe960a05eaf89870600"
        size = 236
#6  0x000000000054cca5 in SinglePoolStrategy::submit (this=0x253b4a0, result=...) at /home/vidi/dev/src/c/xmrigCC/src/net/strategies/SinglePoolStrategy.cpp:43
No locals.
#7  0x000000000054a974 in Network::onJobResult (this=0x253b230, result=...) at /home/vidi/dev/src/c/xmrigCC/src/net/Network.cpp:128
No locals.
#8  0x0000000000550425 in Workers::onResult (handle=0x8a5180 <Workers::m_async>) at /home/vidi/dev/src/c/xmrigCC/src/workers/Workers.cpp:171
        result = {poolId = 0, jobId = {m_data = "yUAEPcuTAZ3LvjX3+wMvdDJsRS3a150", '\000' <repeats 32 times>}, diff = 5000, nonce = 355118448, result = "\327I\210;\357\n\206\360\t\265\321\000\277q\033~>̇\"\264\266o\351`\240^\257\211\207\006"}
        __for_range = @0x7ffc09a6d210: {<std::__cxx11::_List_base<JobResult, std::allocator<JobResult> >> = {_M_impl = {<std::allocator<std::_List_node<JobResult> >> = {<__gnu_cxx::new_allocator<std::_List_node<JobResult> >> = {<No data fields>}, <No data fields>}, 
              _M_node = {<std::__detail::_List_node_base> = {_M_next = 0x25b66f0, _M_prev = 0x25b66f0}, _M_data = 1}}}, <No data fields>}
        __for_begin = {_M_node = 0x25b66f0}
        __for_end = {_M_node = 0x7ffc09a6d210}
        results = {<std::__cxx11::_List_base<JobResult, std::allocator<JobResult> >> = {_M_impl = {<std::allocator<std::_List_node<JobResult> >> = {<__gnu_cxx::new_allocator<std::_List_node<JobResult> >> = {<No data fields>}, <No data fields>}, 
              _M_node = {<std::__detail::_List_node_base> = {_M_next = 0x25b66f0, _M_prev = 0x25b66f0}, _M_data = 1}}}, <No data fields>}
#9  0x00007ffb849f4553 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
No symbol table info available.
#10 0x00007ffb849f4636 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
No symbol table info available.
#11 0x00007ffb84a03055 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
No symbol table info available.
#12 0x00007ffb849f4efc in uv_run () from /usr/lib/x86_64-linux-gnu/libuv.so.1
No symbol table info available.
#13 0x000000000050c410 in App::start (this=0x253a4e0) at /home/vidi/dev/src/c/xmrigCC/src/App.cpp:179
        r = 0
#14 0x0000000000551c06 in main (argc=8, argv=0x7ffc09a70a28) at /home/vidi/dev/src/c/xmrigCC/src/xmrig.cpp:29
        app = 0x253a4e0
        res = 0

But it seems that it has issues to look up source ssl_lib.c file :/, hope this helps.

deleisha · 2018-02-23T01:42:24Z

Thanks, this is useful.
Just as I thought, SSL_write is being passed 0x0, NULL pointer.
In frame 0,
What I don't understand yet how it got NULL, Are you using the SSL context after close CB is executed.
I am unable to reproduce it. I tried modifying the accompanying client code.
Working on it

Bendr0id · 2018-02-23T20:30:31Z

Digged some more into it. It seems that read_cb cb is not triggered with -1 when the TLS connection is dropped. Stock uv_read_start does exactly this. Howto detect that the connection is dropped and has to be reestablished?

deleisha · 2018-03-01T11:19:50Z

@Bendr0id
Please check if latest checkin fixes the issue.

We are also working on a redesign to make evt-tls production ready

sleep was added earlier to debug issue #20

deleisha closed this as completed Feb 20, 2018

deleisha reopened this Feb 20, 2018

deleisha self-assigned this Feb 20, 2018

deleisha added a commit that referenced this issue Mar 5, 2018

Remove spurious sleep added earlier for debugging

ba53a3b

sleep was added earlier to debug issue #20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Libuv/evt-tls segfaults when trying to write to a socket which is closed by the other party #20

Libuv/evt-tls segfaults when trying to write to a socket which is closed by the other party #20

Bendr0id commented Feb 16, 2018 •

edited by deleisha

Loading

deleisha commented Feb 17, 2018

Bendr0id commented Feb 17, 2018

deleisha commented Feb 20, 2018

Bendr0id commented Feb 20, 2018 via email

Bendr0id commented Feb 20, 2018 via email

deleisha commented Feb 20, 2018

deleisha commented Feb 20, 2018

Bendr0id commented Feb 20, 2018

Bendr0id commented Feb 21, 2018

deleisha commented Feb 23, 2018

Bendr0id commented Feb 23, 2018

deleisha commented Mar 1, 2018

Libuv/evt-tls segfaults when trying to write to a socket which is closed by the other party #20

Libuv/evt-tls segfaults when trying to write to a socket which is closed by the other party #20

Comments

Bendr0id commented Feb 16, 2018 • edited by deleisha Loading

deleisha commented Feb 17, 2018

Bendr0id commented Feb 17, 2018

deleisha commented Feb 20, 2018

Bendr0id commented Feb 20, 2018 via email

Bendr0id commented Feb 20, 2018 via email

deleisha commented Feb 20, 2018

deleisha commented Feb 20, 2018

Bendr0id commented Feb 20, 2018

Bendr0id commented Feb 21, 2018

deleisha commented Feb 23, 2018

Bendr0id commented Feb 23, 2018

deleisha commented Mar 1, 2018

Bendr0id commented Feb 16, 2018 •

edited by deleisha

Loading