feature(syslogng): move all default filters to client side #7041

fruch · 2023-12-31T22:06:26Z

since we have more and more filters on the log we apply by default trying to move them into client side, and not send them out at all, and not waste time apply those filter while reading the logs

Testing

provision test AWS/GCP/Azure
one longer longevity - https://jenkins.scylladb.com/job/scylla-staging/job/fruch/job/longevity-100gb-4h-test/41/

PR pre-checks (self review)

I added the relevant backport labels
I didn't leave commented-out/debugging code
k8s provision tests - at least the eks one passed
https://jenkins.scylladb.com/job/scylla-staging/job/fruch/job/longevity-mv-si-4days-test/15/

Reminders

Add New configuration option and document them (in sdcm/sct_config.py)
Add unit tests to cover my changes (under unit-test/ folder)
Update the Readme/doc folder relevent to this change (if needed)

soyacz · 2024-01-04T10:47:36Z

sdcm/sct_events/setup.py

@@ -59,38 +56,6 @@ def start_events_device(log_dir: Optional[Union[str, Path]] = None,
    time.sleep(EVENTS_SUBSCRIBERS_START_DELAY)

    # Default filters.


please add some reference here that we also do filtering in syslog-ng clients

I left some of the filter in place, since they are not db log or k8s specific (which works without syslog-ng, at least now)

soyacz

but how about K8s where we don't use syslog-ng?

sdcm/db_log_reader.py

sdcm/sct_events/database.py

sdcm/provision/common/utils.py

soyacz · 2024-01-24T06:51:37Z

how about K8s where we don't use syslog-ng?

fruch · 2024-01-24T14:54:40Z

how about K8s where we don't use syslog-ng?

there wouldn't be any filters. and we might get some of those issues surfacing

regardless we have issue with logging on k8s, which we might end using syslog-ng or any maybe vector for it.

let run some of the k8s provision test, and we'll see how bad it would be with those filters, and maybe we'll introduce them back only for k8s

fruch · 2024-01-24T15:06:34Z

the first run with it:

cpu usage is good (2 cpus), and still have a few peaks that I'll look at (probably long log lines), and cpu usage is ~1% compared to 25% we were seeing without the this PR

fruch · 2024-01-24T15:11:20Z

doing one more run as we are using in #7128 so it would be comparable

fruch · 2024-01-25T06:55:08Z

@vponomaryov

do you know why the k8s-local-kind-aws doesn't work ?


[2024-01-24T15:17:51.651Z] Unable to find image 'scylladb/scylla-operator:latest' locally

[2024-01-24T15:17:51.651Z] Error response from daemon: unauthorized: authentication required

[2024-01-24T15:17:51.651Z] must specify at least one container source

vponomaryov · 2024-01-26T13:36:09Z

@vponomaryov

do you know why the k8s-local-kind-aws doesn't work ?


[2024-01-24T15:17:51.651Z] Unable to find image 'scylladb/scylla-operator:latest' locally

[2024-01-24T15:17:51.651Z] Error response from daemon: unauthorized: authentication required

[2024-01-24T15:17:51.651Z] must specify at least one container source

The problem appeared using different image just trying to download it:

17:15:47  < t:2024-01-24 15:15:47,011 f:mini_k8s.py     l:500  c:sdcm.cluster_k8s     p:INFO  > local-dc-1: Start Kind cluster
17:17:28  < t:2024-01-24 15:17:26,591 f:mini_k8s.py     l:223  c:sdcm.cluster_k8s     p:INFO  > local-dc-1: Creating kubectl config
17:17:28  < t:2024-01-24 15:17:27,916 f:mini_k8s.py     l:263  c:sdcm.cluster_k8s     p:INFO  > local-dc-1: Pull `calico/kube-controllers:v3.24.5' to docker environment
17:17:28  < t:2024-01-24 15:17:28,805 f:base.py         l:146  c:LocalCmdRunner       p:ERROR > Error executing command: "docker pull -q calico/kube-controllers:v3.24.5"; Exit status: 1
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 2024-01-24 15:17:28.806: (TestFrameworkEvent Severity.ERROR) period_type=one-time event_id=a4dfbe6b-f241-4ba2-a85b-b6c45db85844, source=ScyllaOperatorFunctionalClusterTester.SetUp()
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > exception=Encountered a bad command exit code!
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Command: 'docker pull -q calico/kube-controllers:v3.24.5'
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Exit code: 1
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Stdout:
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Stderr:
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > 
17:17:28  < t:2024-01-24 15:17:28,807 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:ERROR > Error response from daemon: unauthorized: authentication required

It is docker problem.

Then, several days ago was released new docker version on ubuntu - 25.0.0-1.
Which was broken in some place leading to problems with various docker images.

It was around for a day then appeared new, fixed, version - 25.0.1-1.
So, I expect it to be the reason.

So, I re-triggered the CI jobs for this PR assuming that fixed docker version will be used.
And there is no such an error anymore, in the ongoing CI jobs for this PR docker images get downloaded successfully.

sdcm/provision/common/utils.py

sdcm/db_log_reader.py

sdcm/provision/common/utils.py

since we have more and more filters on the log we apply by default trying to move them into client side, and not send them out at all, and not waste time apply those filter while reading the logs

so far those weren't print out in SCT log, now they wouldn't be printed and would be handled by the event system, to stop wasting our time on those line we are not importent enough for us.

taking all old events that are generating warning/normal events and filter on the db nodes, while removing those events. this would help improve the performence at the test end, and lower the amount of regexes used on test runner, and lowe amount of warning event generated

A new severity level, that would help us stop generating some events we need those events for skipping some log line, so we never generate error or critical events from those. until now we generated an event for each one, and now we'll be able to skip them since for some cases there are too many of those warnings.

fruch · 2024-01-28T19:37:58Z

seems like this doesn't change much the CPU use, we still need %25 for reading the logs

soyacz · 2024-01-29T06:34:41Z

seems like this doesn't change much the CPU use, we still need %25 for reading the logs

So, is it worth merging?

fruch · 2024-06-02T15:41:47Z

seems like this doesn't change much the CPU use, we still need %25 for reading the logs

So, is it worth merging?

looks like it's not really helpful, closing this for now

fruch added the test-provision-aws Run provision test on AWS label Dec 31, 2023

github-actions bot assigned fruch Dec 31, 2023

fruch force-pushed the syslogng_client_filter branch 4 times, most recently from b0b71cd to df978f4 Compare January 2, 2024 21:47

soyacz reviewed Jan 4, 2024

View reviewed changes

fruch force-pushed the syslogng_client_filter branch from df978f4 to e76da56 Compare January 16, 2024 19:18

fruch added test-provision-gce Run provision test on GCE test-provision-azure labels Jan 16, 2024

fruch force-pushed the syslogng_client_filter branch 3 times, most recently from 00a329c to 65a2dbf Compare January 17, 2024 14:29

fruch mentioned this pull request Jan 18, 2024

Major lag in SCT logging handling #7069

Closed

fruch force-pushed the syslogng_client_filter branch 6 times, most recently from 5ecab51 to 7d482b8 Compare January 23, 2024 07:21

fruch marked this pull request as ready for review January 23, 2024 13:22

fruch requested review from soyacz, vponomaryov and roydahan January 23, 2024 15:04

fruch added backport/none Backport is not required and removed test-provision-aws Run provision test on AWS test-provision-gce Run provision test on GCE test-provision-azure labels Jan 23, 2024

soyacz reviewed Jan 23, 2024

View reviewed changes

sdcm/db_log_reader.py Show resolved Hide resolved

sdcm/sct_events/database.py Show resolved Hide resolved

roydahan reviewed Jan 23, 2024

View reviewed changes

sdcm/provision/common/utils.py Show resolved Hide resolved

roydahan reviewed Jan 23, 2024

View reviewed changes

sdcm/provision/common/utils.py Show resolved Hide resolved

roydahan reviewed Jan 23, 2024

View reviewed changes

sdcm/provision/common/utils.py Show resolved Hide resolved

fruch added test-provision-k8s-local-kind-aws Run provision test on k8s test-provision-k8s-eks labels Jan 24, 2024

fruch force-pushed the syslogng_client_filter branch from 7d482b8 to 2fc0117 Compare January 24, 2024 14:55

vponomaryov requested changes Jan 26, 2024

View reviewed changes

sdcm/provision/common/utils.py Show resolved Hide resolved

sdcm/db_log_reader.py Outdated Show resolved Hide resolved

sdcm/provision/common/utils.py Outdated Show resolved Hide resolved

fruch force-pushed the syslogng_client_filter branch from 2fc0117 to a2f2fe8 Compare January 28, 2024 12:42

fruch requested review from vponomaryov, roydahan and soyacz January 28, 2024 12:44

fruch added 4 commits January 28, 2024 21:25

feature(syslong): move all default filters to client side

886d4a1

since we have more and more filters on the log we apply by default trying to move them into client side, and not send them out at all, and not waste time apply those filter while reading the logs

fix(db_log_reader): stop EXCLUDE_FROM_LOGGING line from creating events

d0ea4c9

so far those weren't print out in SCT log, now they wouldn't be printed and would be handled by the event system, to stop wasting our time on those line we are not importent enough for us.

fruch force-pushed the syslogng_client_filter branch from a2f2fe8 to 11139d2 Compare January 28, 2024 19:25

fruch mentioned this pull request Jan 28, 2024

feature(issues): introduce SkipPerIssues #7100

Merged

3 tasks

fruch closed this Jun 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature(syslogng): move all default filters to client side #7041

feature(syslogng): move all default filters to client side #7041

fruch commented Dec 31, 2023 •

edited

Loading

soyacz Jan 4, 2024

fruch Jan 23, 2024

soyacz left a comment

soyacz commented Jan 24, 2024

fruch commented Jan 24, 2024

fruch commented Jan 24, 2024

fruch commented Jan 24, 2024

fruch commented Jan 25, 2024

vponomaryov commented Jan 26, 2024

fruch commented Jan 28, 2024

soyacz commented Jan 29, 2024

fruch commented Jun 2, 2024

		@@ -59,38 +56,6 @@ def start_events_device(log_dir: Optional[Union[str, Path]] = None,
		time.sleep(EVENTS_SUBSCRIBERS_START_DELAY)

		# Default filters.

feature(syslogng): move all default filters to client side #7041

feature(syslogng): move all default filters to client side #7041

Conversation

fruch commented Dec 31, 2023 • edited Loading

Testing

PR pre-checks (self review)

Reminders

soyacz Jan 4, 2024

Choose a reason for hiding this comment

fruch Jan 23, 2024

Choose a reason for hiding this comment

soyacz left a comment

Choose a reason for hiding this comment

soyacz commented Jan 24, 2024

fruch commented Jan 24, 2024

fruch commented Jan 24, 2024

fruch commented Jan 24, 2024

fruch commented Jan 25, 2024

vponomaryov commented Jan 26, 2024

fruch commented Jan 28, 2024

soyacz commented Jan 29, 2024

fruch commented Jun 2, 2024

fruch commented Dec 31, 2023 •

edited

Loading