Create different agent per each test execution #1724

mrodm · 2024-03-13T17:22:12Z

This PR adds support to create a new agent per each test run.

Created a new module agentdeployer that is in charge of creating agents separately from the servicedeployer module.

Added a new environment variable ELASTIC_PACKAGE_TEST_ENABLE_INDEPENDENT_AGENT to enable creating independent agents for tests (system tests). In order to enable this feature, this environment variable must be set to true:

ELASTIC_PACKAGE_TEST_ENABLE_INDEPENDENT=true elastic-package test system -v

ELASTIC_PACKAGE_TEST_ENABLE_INDEPENDENT=true elastic-package test system --config-file <path> --setup --test-independent-agent
ELASTIC_PACKAGE_TEST_ENABLE_INDEPENDENT=true elastic-package test system --test-independent-agent --no-provision
ELASTIC_PACKAGE_TEST_ENABLE_INDEPENDENT=true elastic-package test system --test-independent-agent --tear-down

Pending:

Relates #787

Added maxUnavailable and maxSurge keys into elastic-agent template to allow wait for the pods created by the daemonset.

mrodm · 2024-03-14T11:01:16Z

Containers are using the alias elastic-agent to run requests to the agent. That happens for instance in these test packages: hits_count_assertion and ti_anomali. Example:

Directly in the docker-compose scenario:

elastic-package/test/packages/parallel/ti_anomali/_dev/deploy/docker/docker-compose.yml

Line 21 in 12a8d0d

- STREAM_ADDR=http://elastic-agent:9080/

In configuration files (e.g. integrator_metadata.user_metadata.url field):

elastic-package/test/packages/parallel/ti_anomali/_dev/deploy/docker/files/test-intel.ndjson

Line 1 in 12a8d0d

    
           {"message_type":"start","integrator_metadata":{"user_metadata":"{\"url\":\"http://elastic-agent:9080/\",\"secret\":\"TheSecret\",\"request_timeout\":10}","ioc_field_names":["domain","itype","classification","lat","update_id","source_feed_id","id","confidence","severity","trusted_circle_ids","lon","date_first","source","state","import_session_id","value_type","srcip","org","date_last","country","detail2","resource_uri","added_at"]}}

Two different solutions here:

Update each package to use the corresponding new hostname in these commands
- Example: e35c1b5

Try to add a new environment variable that contains this information.

Example: e4d893e
In the example above where the hostname appears in the configuration file (ndjson) it does not work.

Set hostname as environment variable:

{"message_type":"start","integrator_metadata":{"user_metadata":"{\"url\":\"http://${AGENT_HOSTNAME}:9080/\",\"secret\":...}}

Error from the container log:

[2024-03-14 12:12:20,506] [ERROR] HTTPConnectionPool(host='$%7bagent_hostname%7d', port=9080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x786e2cfba7c0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

WDYT @jsoriano ? Should we go for the environment variables ?

I tried also to not define hostname in the compose scenario

elastic-package/internal/servicedeployer/_static/docker-custom-agent-base.yml

Line 9 in a3e3a80

hostname: docker-custom-agent

but doing so:

the elastic-agent enrolls just as a hash (container ID):

there is an error in hit_counts_assertion test package:

{"level":"debug","ts":"2024-03-14T17:58:32.839Z","caller":"output/util.go:23","msg":"Connecting...","address":"elastic-package-agent-hit_count_assertion-test-docker-custom-agent-1:9999"}
Error: dial tcp: lookup elastic-package-agent-hit_count_assertion-test-docker-custom-agent-1: no such host

So, I tried to set the hostname for the agents directly in the docker-compose scenario. That hostname is shared (using environment variables) with the service, so they could run queries against elastic-agent if needed (e.g. send some logs).

mrodm · 2024-04-04T17:11:14Z

internal/servicedeployer/terraform.go

@@ -113,15 +117,14 @@ func (tsd TerraformServiceDeployer) SetUp(ctx context.Context, svcInfo ServiceIn
 		env:             tfEnvironment,
 		shutdownTimeout: 300 * time.Second,
 	}
-	outCtxt := svcInfo


Avoid creating a temporal variable

mrodm · 2024-04-04T17:12:22Z

internal/testrunner/runners/system/runner.go

+	if r.options.RunTearDown || r.options.RunTestsOnly {
+		logger.Debug("Skip creating output directory")
+	} else {
+		outputDir, err := servicedeployer.CreateOutputDir(r.locationManager, svcInfo.Test.RunID)
+		if err != nil {
+			return servicedeployer.ServiceInfo{}, fmt.Errorf("could not create output dir for terraform deployer %w", err)
+		}
+		svcInfo.OutputDir = outputDir


Ensure that this folder is just created with --setup or regular testing commands.

The OutputDir and RunID values will be retrieved from ServiceState in case testing is run in stages (e.g. --no-provision)

mrodm · 2024-04-04T17:12:58Z

internal/testrunner/runners/system/runner.go

-	serviceDeployer, err := servicedeployer.Factory(serviceOptions)
-	if err != nil {
-		return nil, fmt.Errorf("could not create service runner: %w", err)
+	// Configure package (single data stream) via Fleet APIs.


Move creation of test Agent Policy as one of the first steps, so it can be used to enroll the new agents.

mrodm · 2024-04-05T11:12:44Z

internal/testrunner/runners/system/runner.go

+	// In case of custom agent (servicedeployer) enabling independent agents, update serviceOptions to include test policy too
+	if r.options.RunIndependentElasticAgent {
+		serviceOptions.PolicyName = agentInfo.Policy.Name
+	}


Currently, it is not set the Agent Policy name in all the scenarios, to avoid changing the current behaviour. Currently, in servicedeployer it is used the default policy Elastic-Agent (elastic-package)

internal/kibana/agents.go

elasticmachine · 2024-04-09T14:52:40Z

💚 Build Succeeded

Buildkite Build
Commit: ead4d7c

History

💚 Build #2750 succeeded bda3a2c
💚 Build #2733 succeeded 229b23f
💔 Build #2730 failed 8d5fce3
💚 Build #2726 succeeded b63b425
💚 Build #2725 succeeded 7bf8e90
💚 Build #2724 succeeded e697f34

cc @mrodm

mrodm added 21 commits March 8, 2024 13:49

Moved k8s elastic agent template to _static folder

a4fe7b1

Fix lint warnings

c505e52

Remove unused parameter

4e69a8a

WIP

33c4619

Add agent per test - PoC

5e88e06

Filter agents by status too

bb0f72e

Use different folder for logs in each agent

8ccfdcd

Remove unused elasticsearch API

1cec83f

Set different hostnames for agents

107d102

Add interface checks

0d04fe6

Remove unused function and add interface checks

35d3869

Add kubernetes agent deployer

d7fd5dd

Added maxUnavailable and maxSurge keys into elastic-agent template to allow wait for the pods created by the daemonset.

Do not install kubernetes agent in servicedeployer

11630da

Wait for agent hosts set in agentInfo

432ca62

Add checks to get the specific agent

49d296f

Set default agent deployer when docker folder is defined

2a96e93

Add note about possible error if tests are parallelized

0719a32

Consider terraform scenario as default one too (for agents)

61fbc4e

Add variants

c49a10f

Add note for running with stages

6888784

Keep managing previous custom-agent using servicedeployer

a7043d5

mrodm self-assigned this Mar 13, 2024

mrodm added 4 commits March 13, 2024 19:12

Skip temporarily some packages

fa5cca0

Update hostname in custom-agent (service deployer)

7623c61

Remove unused functions from servicedeployer

fb6f13e

Check datastream values for agent name

50ad13d

mrodm force-pushed the add_agent_deployer branch from 8064fe9 to 50ad13d Compare March 13, 2024 18:46

Add debug message

81b5023

Use docker custom names for elastic-agent in tests/configuration

e35c1b5

mrodm added 8 commits April 4, 2024 16:49

Rename policy fields in AgentInfo struct

0424d80

Move code to setupAgent and setupService

6ce4ed2

Remove AGENT_HOSTNAME from servicedeployer

e697f34

Remove unused field

595eccd

Ensure output dir is just created once - services

e35732e

Remove unnecessary field in agentInfo

fbb8f4c

Remove unused field

f4b6568

Avoid creating a temporal variable for svcInfo

7bf8e90

mrodm commented Apr 4, 2024

View reviewed changes

mrodm requested a review from jsoriano April 4, 2024 17:14

mrodm added 6 commits April 4, 2024 19:54

Use reference to add terraform outputs

b63b425

Remove map initialization in test

148668a

Merge remote-tracking branch 'upstream/main' into add_agent_deployer

f15b233

Replace some switch statements by ifs

8d5fce3

Rephrased comment

93fb976

Return default Agent policy if it is empty

229b23f

mrodm commented Apr 5, 2024

View reviewed changes

jsoriano approved these changes Apr 9, 2024

View reviewed changes

internal/kibana/agents.go Outdated Show resolved Hide resolved

Rename function to remove agent

bda3a2c

jsoriano approved these changes Apr 9, 2024

View reviewed changes

Add technical preview message

ead4d7c

jsoriano approved these changes Apr 9, 2024

View reviewed changes

mrodm merged commit 26e9a98 into elastic:main Apr 9, 2024
3 checks passed

mrodm deleted the add_agent_deployer branch April 9, 2024 15:37

This was referenced Apr 11, 2024

Allow to run multiple independent agents in kubernetes #1759

Merged

Remove variant data from agent deployer package #1762

Merged

mrodm mentioned this pull request May 16, 2024

Enable independent agents if root privileges are required at both package or data stream level #1840

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create different agent per each test execution #1724

Create different agent per each test execution #1724

mrodm commented Mar 13, 2024 •

edited

Loading

mrodm commented Mar 14, 2024 •

edited

Loading

mrodm Apr 4, 2024

mrodm Apr 4, 2024

mrodm Apr 4, 2024

mrodm Apr 5, 2024

elasticmachine commented Apr 9, 2024

Create different agent per each test execution #1724

Create different agent per each test execution #1724

Conversation

mrodm commented Mar 13, 2024 • edited Loading

mrodm commented Mar 14, 2024 • edited Loading

mrodm Apr 4, 2024

Choose a reason for hiding this comment

mrodm Apr 4, 2024

Choose a reason for hiding this comment

mrodm Apr 4, 2024

Choose a reason for hiding this comment

mrodm Apr 5, 2024

Choose a reason for hiding this comment

elasticmachine commented Apr 9, 2024

💚 Build Succeeded

History

mrodm commented Mar 13, 2024 •

edited

Loading

mrodm commented Mar 14, 2024 •

edited

Loading