Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wrong state views #872

Open
ggzengel opened this issue Dec 9, 2024 · 7 comments
Open

Wrong state views #872

ggzengel opened this issue Dec 9, 2024 · 7 comments

Comments

@ggzengel
Copy link

ggzengel commented Dec 9, 2024 •
edited
Loading

I don't know if this a error of icinga2/icingadb or the icingaweb2.

I use a filter /icingaweb2/icingadb/hosts?host.state.is_problem=y&sort=host.state.last_state_change%20desc&host.state.is_acknowledged=n.
Sometimes there are green hosts in this list.
If i click on this host i get a overview with 12 red services. But in details there are no red services.

image

image

Your Environment

Icinga Host:

icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.3-1)

System information:
  Platform: Debian GNU/Linux
  Platform version: 12 (bookworm)
  Kernel: Linux
  Kernel version: 6.1.0-27-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 12.2.0
  Build host: runner-hh8q3bz2-project-575-concurrent-0
  OpenSSL version: OpenSSL 3.0.15 3 Sep 2024

Icingaweb2 Host:

Icinga Web 2 Version 	2.12.2
Git commit 	7f9217b0521f19f770ad59ca17d7d4b3514e54e5
PHP Version 	8.2.24
Git commit date 	2024-11-13

icinga/icinga-php-library 	0.14.1
icinga/icinga-php-thirdparty 	0.12.1
Loaded Modules
grafana 		2.0.3 	[Configure](http://lx-icingaweb/icingaweb2/config/module/?name=grafana)
icingadb 		1.1.3 	[Configure](http://lx-icingaweb/icingaweb2/config/module/?name=icingadb)

ii  icinga-archive-keyring       1.0.0-1+debian12  all          GnuPG archive keys of the Icinga archive
ii  icinga-l10n                  1.3.0-1+debian12  all          Icinga localization
ii  icinga-php-library           0.14.1-1+debian12 all          Icinga PHP Library - IPL
ii  icinga-php-thirdparty        0.12.1-1+debian12 all          Icinga 3rd party PHP libraries
ii  icinga2                      2.14.3-1+debian12 amd64        host and network monitoring system
ii  icinga2-bin                  2.14.3-1+debian12 amd64        host and network monitoring system - daemon
ii  icinga2-common               2.14.3-1+debian12 all          host and network monitoring system - common files
un  icinga2-doc                  <none>            <none>       (no description available)
ii  icingacli                    2.12.2-1+debian12 all          Icinga CLI
ii  icingadb-web                 1.1.3-1+debian12  all          Icinga DB Web
ii  icingaweb2                   2.12.2-1+debian12 all          Icinga Web
ii  icingaweb2-common            2.12.2-1+debian12 all          Common files for Icinga Web and the Icinga CLI
un  icingaweb2-module-doc        <none>            <none>       (no description available)
un  icingaweb2-module-monitoring <none>            <none>       (no description available)

No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

Icingadb Host:

No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

ii  icinga-archive-keyring      1.0.0-1+debian12  all          GnuPG archive keys of the Icinga archive
ii  icinga2                     2.14.3-1+debian12 amd64        host and network monitoring system
ii  icinga2-bin                 2.14.3-1+debian12 amd64        host and network monitoring system - daemon
ii  icinga2-common              2.14.3-1+debian12 all          host and network monitoring system - common files
ii  icinga2-doc                 2.14.3-1+debian12 all          host and network monitoring system - documentation
ii  icingadb                    1.2.0-1+debian12  amd64        Icinga configuration and state database supporting multiple environments
ii  icingadb-redis              7.2.6-2+debian12  amd64        Icinga DB is a set of components for publishing,
un  icingadb-redis-sentinel     <none>            <none>       (no description available)
un  icingadb-redis-server       <none>            <none>       (no description available)
un  icingadb-redis-tools        <none>            <none>       (no description available)
un  icingadb-redis-tools-dbgsym <none>            <none>       (no description available)

Execution Details

check_source	"icinga.<domain>"
execution_end	2024-12-09T07:41:32.921+00:00
execution_start	2024-12-09T07:41:32.860+00:00
exit_status	0
performance_data	

[
    "rta=0.601ms;200.000;500.000;0;",
    "pl=0%;40;80;;",
    "rtmax=0.830ms;;;;",
    "rtmin=0.446ms;;;;"
]

previous_hard_state	2
schedule_end	2024-12-09T07:41:32.921+00:00
schedule_start	2024-12-09T07:41:32.860+00:00
scheduling_source	"icinga.<domain>"
state	up
ttl	0.00 s
vars_after	

{
    "attempt": 1,
    "reachable": true,
    "state": 0,
    "state_type": 1
}

vars_before	

{
    "attempt": 1,
    "reachable": true,
    "state": 0,
    "state_type": 1
}

Object Attributes
acknowledgement	0
acknowledgement_expiry	
n. a.
acknowledgement_last_change	
n. a.
action_url	
n. a.
address	"pfsense-ws.<domain>"
address6	
n. a.
check_attempt	1
check_command	"check_icmp-zmt"
check_interval	1.00 m
check_period	
n. a.
check_timeout	
n. a.
command_endpoint	
n. a.
downtime_depth	0
enable_active_checks	true
enable_event_handler	true
enable_flapping	false
enable_notifications	true
enable_passive_checks	true
enable_perfdata	true
event_command	
n. a.
executions	
n. a.
flapping	false
flapping_current	0
flapping_ignore_states	
n. a.
flapping_last_change	2023-08-29T08:30:32.015+00:00
flapping_threshold	0
flapping_threshold_high	30
flapping_threshold_low	25
force_next_check	false
force_next_notification	false
groups	

[
    "PFSENSE",
    "NOUPS",
    "NOICINGA"
]

handled	false
icon_image	
n. a.
icon_image_alt	
n. a.
last_check	2024-12-09T07:41:32.921+00:00
last_hard_state	up
last_hard_state_change	2024-12-08T01:03:49.100+00:00
last_reachable	true
last_state	up
last_state_change	2024-12-08T01:03:49.100+00:00
last_state_down	2024-12-08T01:02:48.683+00:00
last_state_type	1
last_state_unreachable	2024-11-15T14:58:39.207+00:00
last_state_up	2024-12-09T07:41:32.921+00:00
max_check_attempts	3
next_check	2024-12-09T07:42:32.860+00:00
next_update	2024-12-09T07:43:32.982+00:00
notes	
n. a.
notes_url	
n. a.
original_attributes	
n. a.
previous_state_change	2024-12-08T01:03:49.100+00:00
problem	false
retry_interval	30.00 s
severity	0
state	up
state_type	1
vars	

{
    "dns_query": {
        "Icinga Master": {
            "dns_lookup": "icinga.<domain>",
            "dns_server": "$address$"
        },
        "Local Icinga": {
            "dns_lookup": "$host.zone$",
            "dns_server": "$address$"
        },
        "myself": {
            "dns_lookup": "pfsense.work1.private",
            "dns_server": "$address$"
        }
    },
    "http": {
        "Web-Config": {
            "http_ssl": true,
            "http_string": "__csrf_magic"
        }
    },
    "nwc": {
        "interface-health": {
            "nwc_health_criticalx": [
                "discards_.*=40",
                "traffic_.*=:"
            ],
            "nwc_health_mode": "interface-health",
            "nwc_health_name": "_adminup_",
            "nwc_health_regexp": true,
            "nwc_health_warningx": [
                "discards_.*=30",
                "traffic_.*=:"
            ]
        },
        "interface-status": {
            "nwc_health_mode": "interface-status",
            "nwc_health_name": "_adminup_",
            "nwc_health_regexp": true
        },
        "uptime": {
            "nwc_health_mode": "uptime"
        }
    },
    "ssh_address": "pfsense-ws.<domain>",
    "ssl_cert": []
}

volatile	false
zone	"<hidden>"
Custom Variables
dns_query	3 items
Icinga Master	2 items
dns_lookup	icinga.<domain>
dns_server	$address$
Local Icinga	2 items
dns_lookup	$host.zone$
dns_server	$address$
myself	2 items
dns_lookup	pfsense.<hidden>
dns_server	$address$
http	1 item
Web-Config	2 items
http_ssl	1
http_string	__csrf_magic
nwc	3 items
interface-health	5 items
nwc_health_criticalx (Array)	2 items
[0]	discards_.*=40
[1]	traffic_.*=:
nwc_health_mode	interface-health
nwc_health_name	_adminup_
nwc_health_regexp	1
nwc_health_warningx (Array)	2 items
[0]	discards_.*=30
[1]	traffic_.*=:
interface-status	3 items
nwc_health_mode	interface-status
nwc_health_name	_adminup_
nwc_health_regexp	1
uptime	1 item
nwc_health_mode	uptime
ssh_address	pfsense-ws.<domain>
ssl_cert	0 items
Volatile State Details
check_attempt	1
check_commandline	"'sudo' '-n' '/usr/lib/nagios/plugins/check_icmp' '-H' 'pfsense-ws.<domain>'"
check_source	"icinga.<domain>"
check_timeout	1.00 m
execution_time	60.00 ms
hard_state	up
host_id	"8eb1ebe9764b4b4b503087455f1d4aa125638f93"
in_downtime	false
is_acknowledged	0
is_active	true
is_flapping	false
is_handled	false
is_problem	false
is_reachable	true
last_state_change	2024-12-08T01:03:49.100+00:00
last_update	2024-12-09T07:41:32.921+00:00
latency	0.00 s
next_check	2024-12-09T07:42:32.860+00:00
next_update	2024-12-09T07:43:32.982+00:00
normalized_performance_data	"rta=0.000601s;0.200000;0.500000;0 pl=0%;40;80 rtmax=0.000830s rtmin=0.000446s"
output	"OK - pfsense-ws.<domain>: rta 0.601ms, lost 0%"
performance_data	"rta=0.601ms;200.000;500.000;0; pl=0%;40;80;; rtmax=0.830ms;;;; rtmin=0.446ms;;;;"
previous_hard_state	down
previous_soft_state	down
scheduling_source	"icinga.<domain>"
severity	0
soft_state	0
state_type	1
@nilmerg nilmerg transferred this issue from Icinga/icingaweb2 Dec 9, 2024
@nilmerg
Copy link
Member

nilmerg commented Dec 9, 2024

Hi,

this is, I'm afraid, an expected limitation of the database + redis architecture.

It's very similar to Icinga/icingadb-web#741. Here, the counts for the statistics are collected based only on database rows. The list entries however, will be affected by redis.

So in Web, we cannot do anything (efficient) about this to prevent it. But the real problem for me, is that you experience this limitation to such an extent.

Do you have the icingadb check enabled and running? The backlog performance data should be near zero, which it probably isn't in your case. I have no other explanation right now, please have a look if that's correct.

@nilmerg nilmerg added the needs-feedback We'll only proceed once we hear from you again label Dec 9, 2024
@ggzengel
Copy link
Author

ggzengel commented Dec 9, 2024

Which value do you mean?

normalized_performance_data	"icingadb_heartbeat_age=0.965595s icinga2_heartbeat_age=1.966595s;6 icingadb_responsible_instances=1 clock_drift=0.006824s;5;30;0 icinga2_redis_query_backlog=0s;300;900;0 icingadb_history_backlog=0s;300;900;0 icingadb_runtime_update_backlog=0s;300;900;0 icinga2_last_full_dump_ago=507661.907856s icinga2_last_full_dump_duration=0.983856s icingadb_uptime=117597.720595s go_cgo_go_to_c_calls_calls=1 go_cpu_classes_gc_mark_assist_cpu_seconds=11.461857 go_cpu_classes_gc_mark_dedicated_cpu_seconds=123.111800 go_cpu_classes_gc_mark_idle_cpu_seconds=274.302136 go_cpu_classes_gc_pause_cpu_seconds=9.033400 go_cpu_classes_gc_total_cpu_seconds=417.909194 go_cpu_classes_idle_cpu_seconds=468930.785825 go_cpu_classes_scavenge_assist_cpu_seconds=0.000001 go_cpu_classes_scavenge_background_cpu_seconds=0.198200 go_cpu_classes_scavenge_total_cpu_seconds=0.198201 go_cpu_classes_total_cpu_seconds=470366.325371 go_cpu_classes_user_cpu_seconds=1017.432151 go_gc_cycles_automatic_gc_cycles=29179 go_gc_cycles_forced_gc_cycles=0 go_gc_cycles_total_gc_cycles=29179 go_gc_gogc_percent=100 go_gc_gomemlimit_bytes=9223372036854775808B go_gc_heap_allocs_bytes=502414842432B go_gc_heap_allocs_objects=176845839 go_gc_heap_frees_bytes=502381396112B go_gc_heap_frees_objects=176823793 go_gc_heap_goal_bytes=37788240B go_gc_heap_live_bytes=18386288B go_gc_heap_objects_objects=22046 go_gc_heap_tiny_allocs_objects=16740421 go_gc_limiter_last_enabled_gc_cycle=0 go_gc_scan_globals_bytes=470560B go_gc_scan_heap_bytes=32742000B go_gc_scan_stack_bytes=545104B go_gc_scan_total_bytes=33757664B go_gc_stack_starting_size_bytes=2048B go_godebug_non_default_behavior_execerrdot_events=0 go_godebug_non_default_behavior_gocachehash_events=0 go_godebug_non_default_behavior_gocachetest_events=0 go_godebug_non_default_behavior_gocacheverify_events=0 go_godebug_non_default_behavior_gotypesalias_events=0 go_godebug_non_default_behavior_http2client_events=0 go_godebug_non_default_behavior_http2server_events=0 go_godebug_non_default_behavior_httplaxcontentlength_events=0 go_godebug_non_default_behavior_httpmuxgo121_events=0 go_godebug_non_default_behavior_installgoroot_events=0 go_godebug_non_default_behavior_jstmpllitinterp_events=0 go_godebug_non_default_behavior_multipartmaxheaders_events=0 go_godebug_non_default_behavior_multipartmaxparts_events=0 go_godebug_non_default_behavior_multipathtcp_events=0 go_godebug_non_default_behavior_panicnil_events=0 go_godebug_non_default_behavior_randautoseed_events=0 go_godebug_non_default_behavior_tarinsecurepath_events=0 go_godebug_non_default_behavior_tls10server_events=0 go_godebug_non_default_behavior_tlsmaxrsasize_events=0 go_godebug_non_default_behavior_tlsrsakex_events=0 go_godebug_non_default_behavior_tlsunsafeekm_events=0 go_godebug_non_default_behavior_x509sha1_events=0 go_godebug_non_default_behavior_x509usefallbackroots_events=0 go_godebug_non_default_behavior_x509usepolicies_events=0 go_godebug_non_default_behavior_zipinsecurepath_events=0 go_memory_classes_heap_free_bytes=4915200B go_memory_classes_heap_objects_bytes=33446320B go_memory_classes_heap_released_bytes=75390976B go_memory_classes_heap_stacks_bytes=3080192B go_memory_classes_heap_unused_bytes=4802128B go_memory_classes_metadata_mcache_free_bytes=10800B go_memory_classes_metadata_mcache_inuse_bytes=4800B go_memory_classes_metadata_mspan_free_bytes=839040B go_memory_classes_metadata_mspan_inuse_bytes=238080B go_memory_classes_metadata_other_bytes=4621960B go_memory_classes_os_stacks_bytes=0B go_memory_classes_other_bytes=792685B go_memory_classes_profiling_buckets_bytes=7355B go_memory_classes_total_bytes=128149536B go_sched_gomaxprocs_threads=4 go_sched_goroutines_goroutines=739 go_sync_mutex_wait_total_seconds=5.899871s icingadb_responsible_for=117596.523595s icingadb_last_full_sync_ago=117594.584595s icingadb_last_full_sync_duration=1.938000s icingadb_config_sync_items_1min=0 icingadb_config_sync_items_5mins=0 icingadb_config_sync_items_15mins=0 icingadb_history_cleanup_items_1min=0 icingadb_history_cleanup_items_5mins=0 icingadb_history_cleanup_items_15mins=427 icingadb_history_sync_items_1min=7 icingadb_history_sync_items_5mins=51 icingadb_history_sync_items_15mins=156 icingadb_overdue_sync_items_1min=0 icingadb_overdue_sync_items_5mins=2 icingadb_overdue_sync_items_15mins=2 icingadb_state_sync_items_1min=7 icingadb_state_sync_items_5mins=51 icingadb_state_sync_items_15mins=154 icinga2_redis_queries_1min=12824 icinga2_redis_queries_5mins=65404 icinga2_redis_queries_15mins=196137 icinga2_redis_pending_queries=0 icinga2_config_dump_items_1min=0 icinga2_config_dump_items_5mins=0 icinga2_config_dump_items_15mins=0 icinga2_state_dump_items_1min=8 icinga2_state_dump_items_5mins=52 icinga2_state_dump_items_15mins=154 icinga2_history_dump_items_1min=8 icinga2_history_dump_items_5mins=52 icinga2_history_dump_items_15mins=154"

@ggzengel
Copy link
Author

ggzengel commented Dec 9, 2024

These hosts are VMs.
Web-VM and DB-VM are backed up with shutdown on every WE.
There is high load at the host because I use Proxmox with vzdump and the shutdown is in a serial order.
Icinga2 master is on a old V-Host and did not restart.

@nilmerg
Copy link
Member

nilmerg commented Dec 9, 2024

Which value do you mean?

This:

icinga2_redis_query_backlog=0s;300;900;0 icingadb_history_backlog=0s;300;900;0 icingadb_runtime_update_backlog=0s;300;900;0

All are zero. Which is fine…

I'll transfer this to icingadb, maybe the devs there are able to figure this out.

@nilmerg nilmerg transferred this issue from Icinga/icingadb-web Dec 9, 2024
@nilmerg nilmerg removed the needs-feedback We'll only proceed once we hear from you again label Dec 9, 2024
@ggzengel
Copy link
Author

ggzengel commented Dec 9, 2024

The state of this host and it's services changed while the icingadb server was rebooting:
last_hard_state_change 2024-12-08T01:03:49.100+00:00

Icinga2 master didn't reboot and was still running.

This is part of journalctl -S -2d | grep lx-icingadb.*icingadb:

Dec 08 01:02:01 lx-icingadb.<domain> icingadb[1016]: runtime-updates: Upserted 67 ServiceState items
Dec 08 01:02:01 lx-icingadb.<domain> icingadb[1016]: runtime-updates: Upserted 2 HostState items
Dec 08 01:02:08 lx-icingadb.<domain> systemd[1]: Stopping icingadb.service - Icinga DB...
Dec 08 01:02:08 lx-icingadb.<domain> icingadb[1016]: Exiting due to signal
Dec 08 01:02:08 lx-icingadb.<domain> systemd[1]: icingadb.service: Deactivated successfully.
Dec 08 01:02:08 lx-icingadb.<domain> systemd[1]: Stopped icingadb.service - Icinga DB.
Dec 08 01:02:08 lx-icingadb.<domain> systemd[1]: icingadb.service: Consumed 2h 15min 8.504s CPU time.
Dec 08 01:02:09 lx-icingadb.<domain> icingadb-redis-server[627]: 627:signal-handler (1733619729) Received SIGTERM scheduling shutdown...
Dec 08 01:02:09 lx-icingadb.<domain> systemd[1]: Stopping icingadb-redis.service - Redis database for Icinga DB...
Dec 08 01:02:09 lx-icingadb.<domain> icingadb-redis-server[627]: 627:M 08 Dec 2024 01:02:09.975 * User requested shutdown...
Dec 08 01:02:09 lx-icingadb.<domain> icingadb-redis-server[627]: 627:M 08 Dec 2024 01:02:09.975 * Saving the final RDB snapshot before exiting.
Dec 08 01:02:14 lx-icingadb.<domain> icingadb-redis-server[627]: 627:M 08 Dec 2024 01:02:14.474 * DB saved on disk
Dec 08 01:02:14 lx-icingadb.<domain> icingadb-redis-server[627]: 627:M 08 Dec 2024 01:02:14.474 * Removing the pid file.
Dec 08 01:02:14 lx-icingadb.<domain> icingadb-redis-server[627]: 627:M 08 Dec 2024 01:02:14.474 # Redis is now ready to exit, bye bye...
Dec 08 01:02:14 lx-icingadb.<domain> systemd[1]: icingadb-redis.service: Deactivated successfully.
Dec 08 01:02:14 lx-icingadb.<domain> systemd[1]: Stopped icingadb-redis.service - Redis database for Icinga DB.
Dec 08 01:02:14 lx-icingadb.<domain> systemd[1]: icingadb-redis.service: Consumed 2h 28min 37.334s CPU time.
Dec 08 01:05:18 lx-icingadb.<domain> systemd[1]: Hostname set to <lx-icingadb.<domain>>.
Dec 08 01:06:14 lx-icingadb.<domain> systemd[1]: Starting icingadb-redis.service - Redis database for Icinga DB...
Dec 08 01:06:20 lx-icingadb.<domain> icingadb-redis-server[593]: 593:C 08 Dec 2024 01:06:20.690 * Supervised by systemd. Please make sure you set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
Dec 08 01:06:20 lx-icingadb.<domain> icingadb-redis-server[593]: 593:C 08 Dec 2024 01:06:20.690 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Dec 08 01:06:20 lx-icingadb.<domain> icingadb-redis-server[593]: 593:C 08 Dec 2024 01:06:20.690 * Redis version=7.2.6, bits=64, commit=4e7416a9, modified=0, pid=593, just started
Dec 08 01:06:20 lx-icingadb.<domain> icingadb-redis-server[593]: 593:C 08 Dec 2024 01:06:20.690 * Configuration loaded
Dec 08 01:06:20 lx-icingadb.<domain> icingadb-redis-server[593]: 593:M 08 Dec 2024 01:06:20.690 * monotonic clock: POSIX clock_gettime
Dec 08 01:06:20 lx-icingadb.<domain> icingadb-redis-server[593]: 593:M 08 Dec 2024 01:06:20.691 * Running mode=standalone, port=6380.
Dec 08 01:06:20 lx-icingadb.<domain> icingadb-redis-server[593]: 593:M 08 Dec 2024 01:06:20.691 * Server initialized
Dec 08 01:06:20 lx-icingadb.<domain> icingadb-redis-server[593]: 593:M 08 Dec 2024 01:06:20.692 * Loading RDB produced by version 7.2.6
Dec 08 01:06:20 lx-icingadb.<domain> icingadb-redis-server[593]: 593:M 08 Dec 2024 01:06:20.692 * RDB age 251 seconds
Dec 08 01:06:20 lx-icingadb.<domain> icingadb-redis-server[593]: 593:M 08 Dec 2024 01:06:20.692 * RDB memory usage when created 51.79 Mb
Dec 08 01:06:22 lx-icingadb.<domain> icingadb-redis-server[593]: 593:M 08 Dec 2024 01:06:22.021 * Done loading RDB, keys loaded: 66, keys expired: 0.
Dec 08 01:06:22 lx-icingadb.<domain> icingadb-redis-server[593]: 593:M 08 Dec 2024 01:06:22.021 * DB loaded from disk: 1.330 seconds
Dec 08 01:06:22 lx-icingadb.<domain> icingadb-redis-server[593]: 593:M 08 Dec 2024 01:06:22.021 * Ready to accept connections tcp
Dec 08 01:06:22 lx-icingadb.<domain> systemd[1]: Started icingadb-redis.service - Redis database for Icinga DB.
Dec 08 01:06:44 lx-icingadb.<domain> systemd[1]: Starting icingadb.service - Icinga DB...
Dec 08 01:06:44 lx-icingadb.<domain> icingadb[1088]: Starting Icinga DB daemon (v1.2.0)
Dec 08 01:06:44 lx-icingadb.<domain> systemd[1]: Started icingadb.service - Icinga DB.
Dec 08 01:06:44 lx-icingadb.<domain> icingadb[1088]: Connecting to database at 'localhost:0'
Dec 08 01:06:44 lx-icingadb.<domain> icingadb[1088]: Connecting to Redis at 'localhost:6380'
Dec 08 01:06:44 lx-icingadb.<domain> icingadb[1088]: Starting history sync
Dec 08 01:06:45 lx-icingadb.<domain> icingadb[1088]: heartbeat: Received Icinga heartbeat
Dec 08 01:06:45 lx-icingadb.<domain> icingadb[1088]: Taking over
Dec 08 01:06:45 lx-icingadb.<domain> icingadb[1088]: Starting config sync
Dec 08 01:06:45 lx-icingadb.<domain> icingadb[1088]: Starting initial state sync
Dec 08 01:06:45 lx-icingadb.<domain> icingadb[1088]: Starting overdue sync
Dec 08 01:06:45 lx-icingadb.<domain> icingadb[1088]: config-sync: Updating 403 items of type host state
Dec 08 01:06:47 lx-icingadb.<domain> icingadb[1088]: Starting config runtime updates sync
Dec 08 01:06:47 lx-icingadb.<domain> icingadb[1088]: config-sync: Finished config sync in 1.938855906s
Dec 08 01:06:50 lx-icingadb.<domain> icingadb[1088]: config-sync: Updating 10216 items of type service state
Dec 08 01:06:54 lx-icingadb.<domain> icinga2[1156]: [2024-12-08 01:06:54 +0000] information/ApiListener: My API identity: lx-icingadb.<domain>
Dec 08 01:06:55 lx-icingadb.<domain> icingadb[1088]: database: Can't execute query. Retrying
Dec 08 01:06:56 lx-icingadb.<domain> icingadb[1088]: database: Can't execute query. Retrying
Dec 08 01:06:56 lx-icingadb.<domain> icingadb[1088]: database: Can't execute query. Retrying
Dec 08 01:06:57 lx-icingadb.<domain> icingadb[1088]: database: Query retried successfully after error
Dec 08 01:06:57 lx-icingadb.<domain> icingadb[1088]: database: Query retried successfully after error

@ggzengel
Copy link
Author

ggzengel commented Dec 9, 2024 

# journalctl -S -2d | grep lx-icingadb.*postgres
Dec 08 01:02:09 lx-icingadb.<domain> systemd[1]: postgresql.service: Deactivated successfully.
Dec 08 01:02:09 lx-icingadb.<domain> systemd[1]: Stopped postgresql.service - PostgreSQL RDBMS.
Dec 08 01:02:09 lx-icingadb.<domain> systemd[1]: Stopping [email protected] - PostgreSQL Cluster 15-main...
Dec 08 01:02:21 lx-icingadb.<domain> systemd[1]: [email protected]: Deactivated successfully.
Dec 08 01:02:21 lx-icingadb.<domain> systemd[1]: Stopped [email protected] - PostgreSQL Cluster 15-main.
Dec 08 01:02:21 lx-icingadb.<domain> systemd[1]: [email protected]: Consumed 5h 49.762s CPU time.
Dec 08 01:02:21 lx-icingadb.<domain> systemd[1]: Removed slice system-postgresql.slice - Slice /system/postgresql.
Dec 08 01:02:21 lx-icingadb.<domain> systemd[1]: system-postgresql.slice: Consumed 5h 49.762s CPU time.
Dec 08 01:05:18 lx-icingadb.<domain> systemd[1]: Created slice system-postgresql.slice - Slice /system/postgresql.
Dec 08 01:06:14 lx-icingadb.<domain> systemd[1]: Starting [email protected] - PostgreSQL Cluster 15-main...
Dec 08 01:06:43 lx-icingadb.<domain> systemd[1]: Started [email protected] - PostgreSQL Cluster 15-main.
Dec 08 01:06:43 lx-icingadb.<domain> systemd[1]: Starting postgresql.service - PostgreSQL RDBMS...
Dec 08 01:06:43 lx-icingadb.<domain> systemd[1]: Finished postgresql.service - PostgreSQL RDBMS.

@ggzengel
Copy link
Author

ggzengel commented Dec 9, 2024

Log of Icinga2 master while IcingaDB shutdown.

grep IcingaDB  icinga2.log.1
[2024-12-08 01:00:51 +0000] information/IcingaDB: Pending queries: 0 (Input: 209/s; Output: 209/s)
[2024-12-08 01:02:14 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:02:14 +0000] critical/IcingaDB: Error during receiving the response to a query which has been fired and forgotten: Connection reset by peer
[2024-12-08 01:02:14 +0000] critical/IcingaDB: Error during receiving the response to a query which has been fired and forgotten: 
[2024-12-08 01:02:14 +0000] critical/IcingaDB: Error during receiving the response to a query which has been fired and forgotten: 
[2024-12-08 01:02:14 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: Connection refused
[2024-12-08 01:02:19 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:02:23 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: Connection refused
[2024-12-08 01:02:28 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:02:43 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: Operation canceled
[2024-12-08 01:02:48 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:03:03 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: Operation canceled
[2024-12-08 01:03:08 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:03:13 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:03:18 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:03:25 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:03:30 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:03:34 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:03:39 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:03:45 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:03:50 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:03:54 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:03:59 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:04:05 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:04:10 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:04:14 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:04:19 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:04:23 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:04:28 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:04:35 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:04:40 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:04:43 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:04:48 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:04:57 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:05:02 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:05:07 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:05:12 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:05:18 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:05:23 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:05:26 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:05:31 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:05:36 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:05:41 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:05:44 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:05:49 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:05:53 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:05:58 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:06:07 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:06:12 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:06:16 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:06:21 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:06:26 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:06:31 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:06:35 +0000] critical/IcingaDB: Cannot connect to lx-icingadb.<domain>:6380: No route to host
[2024-12-08 01:06:40 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host 'lx-icingadb.<domain>:6380'
[2024-12-08 01:06:44 +0000] information/IcingaDB: Connected to Redis server
[2024-12-08 01:06:51 +0000] information/IcingaDB: Pending queries: 0 (Input: 277/s; Output: 227/s)
[2024-12-08 01:11:51 +0000] information/IcingaDB: Pending queries: 0 (Input: 180/s; Output: 180/s)
[2024-12-08 01:16:51 +0000] information/IcingaDB: Pending queries: 0 (Input: 196/s; Output: 196/s)
	(0) Processing check result for 'icinga.<domain>!IcingaDB'
	(0) Processing check result for 'icinga.<domain>!IcingaDB'
	(0) Processing check result for 'icinga.<domain>!IcingaDB'
[2024-12-08 01:21:51 +0000] information/IcingaDB: Pending queries: 0 (Input: 219/s; Output: 219/s)
[2024-12-08 01:26:51 +0000] information/IcingaDB: Pending queries: 0 (Input: 181/s; Output: 181/s)
[2024-12-08 01:31:51 +0000] information/IcingaDB: Pending queries: 0 (Input: 186/s; Output: 186/s)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ggzengel @nilmerg