naemon reload does not release deleted objects.cache file

[pvdputte]

It appears that naemon keeps previous instances of the objects.cache file open when it reloads its config. As a result these cannot be freed in the filesystem and in no time the disk fills up. (our objects.cache is around 450 MiB)

When I restart naemon, everything is freed again.

Until recently we were running naemon 1.0.8, with that version this problem did not occur.

I've been running 1.4 for a few months now but until now I restarted naemon on each config change instead of reloading it (cfr. naemon/naemon-livestatus#117 (comment)). That's why I'm only noticing this now.

# ls -l /proc/2585040/fd|grep objects.cache
lrwx------ 1 naemon naemon 64 Apr 28 18:27 18 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 21 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 22 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 23 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 24 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 25 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 26 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 27 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 28 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 29 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 30 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 31 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 32 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 33 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 34 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 35 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 36 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 37 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 38 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 39 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 40 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 41 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 42 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 43 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 18:27 44 -> /var/lib/naemon/objects.cache

To reproduce:

# systemctl restart naemon
# ls -l /proc/2858224/fd|grep objects.cache
lrwx------ 1 naemon naemon 64 Apr 28 21:41 18 -> /var/lib/naemon/objects.cache

# systemctl reload naemon
# ls -l /proc/2858224/fd|grep objects.cache
lrwx------ 1 naemon naemon 64 Apr 28 21:41 18 -> /var/lib/naemon/objects.cache (deleted)
lrwx------ 1 naemon naemon 64 Apr 28 21:41 21 -> /var/lib/naemon/objects.cache

[sni]

Indeed, i can reproduce that. Haven't noticed that so far, because we disable the object cache by setting: object_cache_file=/dev/null
Which would be a workaround for you too. The object.cache is usually not required and just a
leftover from the time when guis need to parse that file to acutally display things instead of just using livestatus.

[pvdputte]

Perfect, thanks 👌

[sni]

Took a bit, but should be fixed with #483