[AMLII-2183] Logs From Integrations Rotation E2E Test and Yaml Template #31812

soberpeach · 2024-12-05T21:04:06Z

What does this PR do?

Ensures all logs are read for logs from integrations after there are file rotations. By setting the maximum log file size to 1 megabyte and each log to 256 kilobytes, we can ensure that after five log writes there will be at least one rotation. Each log has a counter variable included that is incremented by 1 in each iteration. This count is checked on each iteration to ensure no logs are dropped.

It also adds a method to generate YAML check configs programmatically in the e2e test.

Motivation

E2E test coverage for logs from integrations.

Describe how to test/QA your changes

Make sure e2e tests pass.

Possible Drawbacks / Trade-offs

N/A

Additional Notes

There will be more e2e tests written for this feature soon, including a test for multiple custom checks, and adding windows tests.

…liseth/integration-log-max-size-e2e-test

agent-platform-auto-pr · 2024-12-05T21:39:00Z

[Fast Unit Tests Report]

On pipeline 51737475 (CI Visibility). The following jobs did not run any unit tests:

Jobs:

tests_deb-arm64-py3
tests_deb-x64-py3
tests_flavor_dogstatsd_deb-x64
tests_flavor_heroku_deb-x64
tests_flavor_iot_deb-x64
tests_rpm-arm64-py3
tests_rpm-x64-py3
tests_windows-x64

If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help

agent-platform-auto-pr · 2024-12-05T21:44:11Z

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv aws.create-vm --pipeline-id=51737475 --os-family=ubuntu

Note: This applies to commit 39d0c08

cit-pr-commenter · 2024-12-05T22:09:57Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 909d7d8f-d36f-45b7-87c6-8923ba3cddc7

Baseline: 528fb2d
Comparison: 39d0c08
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	file_tree	memory utilization	+1.13	[+1.00, +1.25]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	+0.55	[-0.26, +1.36]	1	Logs
➖	otel_to_otel_logs	ingress throughput	+0.40	[-0.34, +1.14]	1	Logs
➖	file_to_blackhole_500ms_latency	egress throughput	+0.28	[-0.50, +1.06]	1	Logs
➖	file_to_blackhole_1000ms_latency_linear_load	egress throughput	+0.19	[-0.27, +0.65]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	+0.01	[-0.10, +0.11]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	+0.00	[-0.74, +0.74]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.01, +0.01]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	-0.01	[-0.88, +0.86]	1	Logs
➖	file_to_blackhole_0ms_latency_http1	egress throughput	-0.03	[-0.90, +0.83]	1	Logs
➖	file_to_blackhole_300ms_latency	egress throughput	-0.06	[-0.70, +0.59]	1	Logs
➖	file_to_blackhole_0ms_latency_http2	egress throughput	-0.08	[-0.98, +0.82]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-0.10	[-0.17, -0.03]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	-0.10	[-0.18, -0.02]	1	Logs bounds checks dashboard
➖	quality_gate_logs	% cpu utilization	-0.15	[-3.36, +3.07]	1	Logs
➖	quality_gate_idle	memory utilization	-0.19	[-0.22, -0.15]	1	Logs bounds checks dashboard
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-0.21	[-0.89, +0.46]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http1	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http1	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http2	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http2	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency_linear_load	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_300ms_latency	lost_bytes	10/10
✅	file_to_blackhole_300ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_logs	lost_bytes	10/10
✅	quality_gate_logs	memory_usage	10/10

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.

agent-platform-auto-pr · 2024-12-10T16:38:09Z

Package size comparison

Comparison with ancestor a2310cfe86c7bda2031672e1268cb749d631ec40

Diff per package

package	diff	status	size	ancestor	threshold
datadog-agent-amd64-deb	0.00MB	⚠️	1265.94MB	1265.94MB	140.00MB
datadog-iot-agent-amd64-deb	0.00MB	✅	113.28MB	113.28MB	10.00MB
datadog-dogstatsd-amd64-deb	0.00MB	✅	78.52MB	78.52MB	10.00MB
datadog-heroku-agent-amd64-deb	0.01MB	⚠️	502.50MB	502.50MB	70.00MB
datadog-agent-x86_64-rpm	0.00MB	⚠️	1275.18MB	1275.18MB	140.00MB
datadog-agent-x86_64-suse	0.00MB	⚠️	1275.18MB	1275.18MB	140.00MB
datadog-iot-agent-x86_64-rpm	0.00MB	✅	113.35MB	113.35MB	10.00MB
datadog-iot-agent-x86_64-suse	0.00MB	✅	113.35MB	113.35MB	10.00MB
datadog-dogstatsd-x86_64-rpm	0.00MB	✅	78.59MB	78.59MB	10.00MB
datadog-dogstatsd-x86_64-suse	0.00MB	✅	78.59MB	78.59MB	10.00MB
datadog-agent-arm64-deb	-0.01MB	✅	1001.02MB	1001.02MB	140.00MB
datadog-iot-agent-arm64-deb	0.00MB	✅	108.76MB	108.76MB	10.00MB
datadog-dogstatsd-arm64-deb	0.00MB	✅	55.74MB	55.74MB	10.00MB
datadog-agent-aarch64-rpm	-0.00MB	✅	1010.23MB	1010.24MB	140.00MB
datadog-iot-agent-aarch64-rpm	0.00MB	✅	108.83MB	108.83MB	10.00MB

Decision

⚠️ Warning

soberpeach · 2024-12-10T17:45:51Z

/merge

dd-devflow · 2024-12-10T17:46:11Z

Devflow running: `/merge`

View all feedbacks in Devflow UI.

2024-12-10 17:46:10 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2024-12-10 21:46:10 UTC ⚠️ MergeQueue: This merge request was unqueued

This merge request was unqueued

soberpeach · 2024-12-11T08:45:46Z

/merge

dd-devflow · 2024-12-11T08:45:56Z

Devflow running: `/merge`

View all feedbacks in Devflow UI.

2024-12-11 08:45:56 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2024-12-11 12:45:58 UTC ⚠️ MergeQueue: This merge request was unqueued

This merge request was unqueued

test/new-e2e/tests/agent-metrics-logs/log-agent/linux-log/integrations/logs_test.go

soberpeach · 2024-12-23T15:57:58Z

/merge

dd-devflow · 2024-12-23T15:58:09Z

Devflow running: `/merge`

View all feedbacks in Devflow UI.

2024-12-23 15:58:09 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2024-12-23 17:02:01 UTC ℹ️ MergeQueue: merge request added to the queue

The median merge time in main is 35m.

2024-12-23 17:35:04 UTC ℹ️ MergeQueue: This merge request was merged

agent-platform-auto-pr · 2024-12-23T16:35:29Z

Uncompressed package size comparison

Comparison with ancestor 528fb2d2b937037c0874f02d9512c364e697a6f8

Diff per package

package	diff	status	size	ancestor	threshold
datadog-agent-amd64-deb	1.41MB	⚠️	1189.61MB	1188.21MB	140.00MB
datadog-agent-x86_64-rpm	1.41MB	⚠️	1198.88MB	1197.47MB	140.00MB
datadog-agent-x86_64-suse	1.41MB	⚠️	1198.88MB	1197.47MB	140.00MB
datadog-agent-aarch64-rpm	1.41MB	⚠️	944.75MB	943.35MB	140.00MB
datadog-agent-arm64-deb	1.40MB	⚠️	935.51MB	934.10MB	140.00MB
datadog-dogstatsd-amd64-deb	0.00MB	✅	78.57MB	78.57MB	10.00MB
datadog-dogstatsd-x86_64-rpm	0.00MB	✅	78.65MB	78.65MB	10.00MB
datadog-dogstatsd-x86_64-suse	0.00MB	✅	78.65MB	78.65MB	10.00MB
datadog-dogstatsd-arm64-deb	0.00MB	✅	55.77MB	55.77MB	10.00MB
datadog-iot-agent-amd64-deb	0.00MB	✅	113.34MB	113.34MB	10.00MB
datadog-iot-agent-x86_64-rpm	0.00MB	✅	113.41MB	113.41MB	10.00MB
datadog-iot-agent-x86_64-suse	0.00MB	✅	113.41MB	113.41MB	10.00MB
datadog-iot-agent-arm64-deb	0.00MB	✅	108.81MB	108.81MB	10.00MB
datadog-iot-agent-aarch64-rpm	0.00MB	✅	108.88MB	108.88MB	10.00MB
datadog-heroku-agent-amd64-deb	-1.40MB	✅	505.09MB	506.49MB	70.00MB

Decision

⚠️ Warning

…l Template (#31812)" This reverts commit 5bd602d. Related to #incident-33564

…te (#31812)

soberpeach and others added 20 commits November 11, 2024 22:10

Test integrations logs writes 10 at a time

b17ac6c

Switch greater than or equal to

3bd2114

Rename files to comply with linter

38efced

Rename check files to comply with linter

70acb8e

Add a check for fake intake logs

3670985

Merge branch 'main' into lucas.liseth/integration-launcher-e2e-test

7529c56

Remove devmode

d19caa8

Check for logs tags

86d1004

Add max size check

fc36e97

Remove testing for implementation

375ddf8

Merge branch 'lucas.liseth/integration-launcher-e2e-test' into lucas.…

14c1fca

…liseth/integration-log-max-size-e2e-test

Update maxsize test to check fakeintake

8afa07f

Treat fakeintake as a black box

ad92114

Merge branch 'main' into lucas.liseth/integration-log-max-size-e2e-test

c949b19

Add log ID checking for log rotation test

758e3ae

Merge branch 'main' into lucas.liseth/integration-log-max-size-e2e-test

c17dec4

Prevent truncated logs by reducing log size

7430b2d

Updated tests to use yaml template

f44e61e

Merge branch 'main' into lucas.liseth/integration-log-rotation-e2e-test

f4ef995

Remove comment from python check

894174f

soberpeach requested a review from a team as a code owner December 5, 2024 21:04

github-actions bot added team/agent-metrics-logs medium review PR review might take time labels Dec 5, 2024

soberpeach added qa/no-code-change No code change in Agent code requiring validation changelog/no-changelog labels Dec 5, 2024

soberpeach added 2 commits December 6, 2024 10:06

Remove flaky monotonic counter

61ebd79

Combined python check files

e907737

soberpeach and others added 4 commits December 6, 2024 13:55

Renamed file

1b26684

Update fixtures file

536c064

Updated bounds for logs check

caba1fb

Merge branch 'main' into lucas.liseth/integration-log-rotation-e2e-test

f9d74d6

soberpeach and others added 4 commits December 12, 2024 18:05

Replace UUID for logs with increasing counts

ac80d1c

Merge branch 'main' into lucas.liseth/integration-log-rotation-e2e-test

e00b74f

Mofify test to remove flakiness

ca4ae9b

Remove print statement

5ca8269

carlosroman reviewed Dec 18, 2024

View reviewed changes

test/new-e2e/tests/agent-metrics-logs/log-agent/linux-log/integrations/logs_test.go Outdated Show resolved Hide resolved

carlosroman reviewed Dec 18, 2024

View reviewed changes

test/new-e2e/tests/agent-metrics-logs/log-agent/linux-log/integrations/logs_test.go Outdated Show resolved Hide resolved

carlosroman reviewed Dec 18, 2024

View reviewed changes

test/new-e2e/tests/agent-metrics-logs/log-agent/linux-log/integrations/logs_test.go Outdated Show resolved Hide resolved

carlosroman requested changes Dec 18, 2024

View reviewed changes

test/new-e2e/tests/agent-metrics-logs/log-agent/linux-log/integrations/logs_test.go Outdated Show resolved Hide resolved

addressed comments

a1c9f7e

carlosroman self-requested a review December 23, 2024 15:35

carlosroman approved these changes Dec 23, 2024

View reviewed changes

Merge branch 'main' into lucas.liseth/integration-log-rotation-e2e-test

39d0c08

dd-mergequeue bot merged commit 5bd602d into main Dec 23, 2024
210 of 211 checks passed

dd-mergequeue bot deleted the lucas.liseth/integration-log-rotation-e2e-test branch December 23, 2024 17:35

github-actions bot added this to the 7.62.0 milestone Dec 23, 2024

chouquette added a commit that referenced this pull request Dec 23, 2024

Revert "[AMLII-2183] Logs From Integrations Rotation E2E Test and Yam…

edf7ca1

…l Template (#31812)" This reverts commit 5bd602d. Related to #incident-33564

chouquette mentioned this pull request Dec 23, 2024

Revert "[AMLII-2183] Logs From Integrations Rotation E2E Test and Yam… #32492

Merged

louis-cqrl pushed a commit that referenced this pull request Dec 25, 2024

[AMLII-2183] Logs From Integrations Rotation E2E Test and Yaml Templa…

c5d3df3

…te (#31812)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AMLII-2183] Logs From Integrations Rotation E2E Test and Yaml Template #31812

[AMLII-2183] Logs From Integrations Rotation E2E Test and Yaml Template #31812

soberpeach commented Dec 5, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Dec 5, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Dec 5, 2024 •

edited

Loading

cit-pr-commenter bot commented Dec 5, 2024 •

edited

Loading

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

agent-platform-auto-pr bot commented Dec 10, 2024 •

edited

Loading

soberpeach commented Dec 10, 2024

dd-devflow bot commented Dec 10, 2024 •

edited

Loading

soberpeach commented Dec 11, 2024

dd-devflow bot commented Dec 11, 2024 •

edited

Loading

soberpeach commented Dec 23, 2024

dd-devflow bot commented Dec 23, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Dec 23, 2024

[AMLII-2183] Logs From Integrations Rotation E2E Test and Yaml Template #31812

[AMLII-2183] Logs From Integrations Rotation E2E Test and Yaml Template #31812

Conversation

soberpeach commented Dec 5, 2024 • edited Loading

What does this PR do?

Motivation

Describe how to test/QA your changes

Possible Drawbacks / Trade-offs

Additional Notes

agent-platform-auto-pr bot commented Dec 5, 2024 • edited Loading

agent-platform-auto-pr bot commented Dec 5, 2024 • edited Loading

Test changes on VM

cit-pr-commenter bot commented Dec 5, 2024 • edited Loading

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

agent-platform-auto-pr bot commented Dec 10, 2024 • edited Loading

Package size comparison

Decision

soberpeach commented Dec 10, 2024

dd-devflow bot commented Dec 10, 2024 • edited Loading

Devflow running: /merge

soberpeach commented Dec 11, 2024

dd-devflow bot commented Dec 11, 2024 • edited Loading

Devflow running: /merge

soberpeach commented Dec 23, 2024

dd-devflow bot commented Dec 23, 2024 • edited Loading

Devflow running: /merge

agent-platform-auto-pr bot commented Dec 23, 2024

Uncompressed package size comparison

Decision

soberpeach commented Dec 5, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Dec 5, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Dec 5, 2024 •

edited

Loading

cit-pr-commenter bot commented Dec 5, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Dec 10, 2024 •

edited

Loading

dd-devflow bot commented Dec 10, 2024 •

edited

Loading

Devflow running: `/merge`

dd-devflow bot commented Dec 11, 2024 •

edited

Loading

Devflow running: `/merge`

dd-devflow bot commented Dec 23, 2024 •

edited

Loading

Devflow running: `/merge`