Extremly high memory footprint of newer sslh version

[fdellwing]

We used sslh-fork head-2014-07-15 for a really long time and recently switched to a recent version sslh-fork v1.21c-30-gb72baa0 and noticed a massive difference in memory usage between the two.

With around 5000 incoming connections the old version uses around 2GB of memory, but the newer version uses around 11GB. We want to switch to sslh-select, but first need to check if the process hanging (#258) is resolved. But we would also like to know, if there is some general memory problem in newer versions of sslh-fork?

[yrutschle]

2014, that was a long time ago :-)

I don't see a valid reason for the memory usage to be that different. Once probing is done, sslh-fork is extremely simple: it's an infinite loop in sslh-fork.c:shovel(), which waits for activity, and upon activity copies one BUFSIZ (8k) in common.c:fd2fd(). None of that has changed in years. Everything else should be shared between all processes (fork normally copies the minimum possible, with pages shared between processes, until one writes on the page, at which point it gets copied).
I'm not too sure how to investigate this, I'll think about it.

[yrutschle]

Ok, after some investigation into how to... investigate... I find smem and pmap to be interesting tools.

In particular:

# smem -k | grep sslh-fork
  781 nobody   ./sslh-fork -F hood.cfg       144.0K        0    12.0K    40.0K 
  782 nobody   ./sslh-fork -F hood.cfg       116.0K    28.0K    48.0K   496.0K 
23868 nobody   ./sslh-fork -F hood.cfg        12.0K   128.0K   288.0K     2.0M 
24573 nobody   ./sslh-fork -F hood.cfg        12.0K   128.0K   288.0K     2.0M 
24617 root     grep sslh-fork                     0   344.0K   456.0K     2.2M 

The last two sslh-fork processes are actual connections, while the first two are listeners.
if I understand correctly, that means each process uses 128K of "USS", unique set size; the rest is 2MB which is shared between all processes.

So I would expect 5000 connections to use up 640MB...

[fdellwing]

I will see if I can reproduce this on a test system, so far it only was a problem on our productive system.

First quick scan shows, that the newer version indeed uses a lot more RAM than the old one does, but it does not explain the 9GB more used.

Old:

 5222 root     /usr/bin/sslh -t 5 --transp    64.0K     4.0K     4.0K     1.1M 
 5224 root     /usr/bin/sslh -t 5 --transp    44.0K    20.0K    20.0K   680.0K 
163731 root     /usr/bin/sslh -t 5 --transp    24.0K    44.0K    44.0K  1020.0K 
214885 root     /usr/bin/sslh -t 5 --transp    24.0K    44.0K    44.0K  1020.0K 
199767 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.3M 
202231 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.3M 
206193 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.1M 
209510 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.3M 
209702 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.3M 
210001 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.1M 
211543 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.1M 
212397 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.1M 
214737 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.3M 
215760 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.1M 
216812 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.3M 
220334 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.3M 
220501 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.3M 
220520 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.1M 
220823 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.1M 
222360 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.1M 
223575 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.3M 
223686 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.3M 
225692 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.1M 
229913 root     /usr/bin/sslh -t 5 --transp    20.0K    52.0K    52.0K     1.3M

New:

 3871 root     /usr/bin/sslh-fork -t 5 --t        0    40.0K    58.0K   924.0K 
 6471 root     /usr/bin/sslh-fork -t 5 --t        0    44.0K    60.0K   924.0K 
 3869 root     /usr/bin/sslh-fork -t 5 --t        0    44.0K    64.0K     1.4M 
 1454 root     /usr/bin/sslh-fork -t 5 --t        0   196.0K   209.0K     1.6M 
 2106 root     /usr/bin/sslh-fork -t 5 --t        0   196.0K   209.0K     1.6M 
 3295 root     /usr/bin/sslh-fork -t 5 --t        0   196.0K   209.0K     1.6M 
 4475 root     /usr/bin/sslh-fork -t 5 --t        0   196.0K   209.0K     1.6M 
 4968 root     /usr/bin/sslh-fork -t 5 --t        0   196.0K   209.0K     1.6M 
 5042 root     /usr/bin/sslh-fork -t 5 --t        0   196.0K   209.0K     1.6M 
 5046 root     /usr/bin/sslh-fork -t 5 --t        0   196.0K   209.0K     1.6M 
 5047 root     /usr/bin/sslh-fork -t 5 --t        0   196.0K   209.0K     1.6M 
 5053 root     /usr/bin/sslh-fork -t 5 --t        0   196.0K   209.0K     1.6M 
18539 root     /usr/bin/sslh-fork -t 5 --t        0   196.0K   209.0K     1.6M

[yrutschle]

Are you sure it's really 9GB and not the shared RSS that totals 9GB but is in fact 1.6M many times over?

The increase in shared memory isn't too surprising, the configuration system has become a lot more complex; it shouldn't matter much though.

The unique memory is still three times over what it used to be, which I find a bit surprising. I guess the two measures are done with the same compiler, options, libraries and so so on?
I'm not sure if I should compare current code with 10 year old code or just find out what's using memory today :-)

[fdellwing]

Well, the memory of the production system ran out, so yes, it used that much memory for real.

I do not think it helps much to compare them. I will try to reproduce the memory consumption, but it will take a while. I cannot use the production system for that.

[yrutschle]

Ok, another tool I'm finding useful is valgrind's massif:

valgrind --tool=massif ./sslh-fork -F server.cfg

Valgrind will write a massif.out.$PID for each forked process. Caveat: it needs to run in a process where it can write files, in case you're setuid nobody. The files are all about 80k for me, so not too big.

And then

massif-visualizer massif.out.123

shows pretty graphs of the heap usage, where you can see what uses what.

All my runs show pretty stable, minimal usage (expressed in tens of KB), so nothing to explain what you observe.

I know valgrind can instrument stack usage as well, I'll look into that.

[fdellwing]

We ran new tests with 1.22c and where not able to bring the server to crash. The problem might have been, that we ran the unstripped version of the binary that is more than twice the size of the stripped variant.

Release to production will restart soon, if it manifests again, I will report.

[fdellwing]

I was once again able to crash a server with this.

Running the stripped version of sslh-fork 1.22c, the server gave up at aroung 6000 child processes.

I really hope sslh-ev will drop soon and runs stable. It will be our saviour.

[yrutschle]

I still don't see how sslh-fork would use so much RAM, and still don't see how to investigate...
I am not sure what you mean with "sslh-ev will drop soon": I personally use it in production and I think it's pretty solid. It'd be very interesting to have a return of experience of a "big" setup like yours (mine does not have near as many concurrent connections...)

[fdellwing]

It would be nice not to have a release candidate tag on it before we throw it into production. It runs fine on a testserver so far, but that server has <10 concurrent connections.

[yrutschle]

like this? https://github.com/yrutschle/sslh/releases/tag/v2.0-rc1 :-)
(or did I miss some Github convention ro mark release candidates?)

[fdellwing]

not to have ;)

I would like to see a real release, but I can understand that it is not tested enough for releasing. In that case we need to talk internally if we can use it in RC state.

[yrutschle]

oooooh I see. Apparently I only read the words I want, sorry :-)

my argument "in favour" would be that the RC code will turn to "real release" anyway, so you may as well test it (I don't know if there are many people using "big setups" like yours testing release candidates...)

[fdellwing]

Well I guess we switched back mid patch two times to the old binary, so we might just do it a third time :D

[fdellwing]

Ok, I was now able to do a heap analyses and as I though, the newer version, 2.0RC in this case, uses a lot more RAM (x28 more) than the antike one. That is a very steep increase if you have a few thousand of these processes.

massif.out.846913.txt

loaded massif file: QUrl("file:///home/fade/massif/massif.out.846913")
description: "(none)"
command: "/usr/bin/sslh -t 5 --transparent -n -p 5.39.123.17:4443 --tls=172.16.10.17:443 --http=172.16.10.17:80 --openvpn=172.16.10.17:1194 --pidfile /var/run/sslh.pid"
time unit: "i"
snapshots: 47
peak: snapshot # 36 after 332485 "i"
peak cost: "11,7 KiB" heap "282 B" heap extra "0 B" stacks

massif.out.823229.txt

loaded massif file: QUrl("file:///home/fade/massif/new/massif.out.823229")
description: "(none)"
command: "/usr/bin/sslh-fork -t 5 --transparent -n -p 5.39.123.17:4443 --tls=172.16.10.17:443 --http=172.16.10.17:80 --openvpn=172.16.10.17:1194 --pidfile /var/run/sslh.pid"
time unit: "i"
snapshots: 76
peak: snapshot # 48 after 992346 "i"
peak cost: "336,9 KiB" heap "2,0 KiB" heap extra "0 B" stacks

In this case we are talking about ~330 MiB instead of ~11 MiB of needed RAM per 1000 processes.

[fdellwing]

@yrutschle Any news on this?

[yrutschle]

Afraid not... I am still unsure how to even investigate this, and it's not currently at the top of my... heap...