Add config option to deploy custom elastic-agents as test services #786

marc-gr · 2022-04-11T07:41:23Z

For some integrations eg: auditbeat or winlogbeat dependant ones. Would be useful to define custom agents. This adds the ability to define them as a system test service and enroll them during the tests instead of the elastic-package stack ones.

Example test service definition:

version: '2.3'
services:
  auditd-agent:
    hostname: auditd-agent
    image: "docker.elastic.co/beats/elastic-agent-complete:8.2.0-SNAPSHOT"
    pid: host
    cap_add:
      - AUDIT_CONTROL
      - AUDIT_READ
    user: root
    healthcheck:
      test: "elastic-agent status"
      retries: 180
      interval: 1s
    environment:
      FLEET_ENROLL: "1"
      FLEET_INSECURE: "1"
      FLEET_URL: "http://fleet-server:8220"

Example service test config:

service: auditd-agent
custom_agent: true
data_stream:
  vars:
    audit_rules:
      - "-a always,exit -F arch=b64 -S execve,execveat -k exec"
    preserve_original_event: true

elasticmachine · 2022-04-11T07:46:03Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-05-17T09:42:06.702+0000
Duration: 27 min 54 sec

Test stats 🧪

Test	Results
Failed	0
Passed	706
Skipped	0
Total	706

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-05-04T09:31:14Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`30/30`)	💚
Files	64.22% (`70/109`)	👍
Classes	58.278% (`88/151`)	👍
Methods	47.333% (`284/600`)	👎 -0.221
Lines	32.207% (`2601/8076`)	👎 -0.076
Conditionals	100.0% (`0/0`)	💚

jsoriano

This approach looks cleaner to me than the previous one.

Could you add an example test package to test/packages that uses this approach?

internal/testrunner/runners/system/servicedeployer/factory.go

jsoriano

This is looking good, I added some comments, the only blocking one would be to decide how we want to configure the custom agent. Allowing any kind of docker-compose configuration may complicate maintainability of elastic-package, see my comment about this.

jsoriano · 2022-05-06T14:25:37Z

internal/testrunner/runners/system/servicedeployer/factory.go

+	case "agent":
+		dockerComposeYMLPath := filepath.Join(serviceDeployerPath, "docker-compose.yml")
+		if _, err := os.Stat(dockerComposeYMLPath); err != nil {
+			return nil, errors.Wrap(err, "can't find expected file docker-compose.yml")


Nit. Include the path where the file was expected to be found.

test/packages/with-custom-agent/apache/_dev/build/docs/README.md

Makefile

jsoriano · 2022-05-06T14:40:02Z

docs/howto/system_testing.md

+    environment:
+      FLEET_ENROLL: "1"
+      FLEET_INSECURE: "1"
+      FLEET_URL: "http://fleet-server:8220"


Umm, this makes me think that we might need some way to pass variables to these configs. These connection settings depend on the stack and how it is started. For example in #789 I am changing these options, and this would break scenarios with these environment variables.

An option can be to create a different configuration file, expected for example in _dev/deploy/agent/config.yml, that is used to "patch" a base agent configuration. So for example in this case, it could be something like this:

image: "docker.elastic.co/beats/elastic-agent-complete:8.2.0" pid: host cap_add: - AUDIT_CONTROL - AUDIT_READ user: root

This way every package using this deployer only needs to configure the minimal relevant set of settings. And we can control the settings needed for enrollment, or other settings that may be dependent of elastic-package stack.

Another middle-ground option could be to use env_file here to include an environment file that would be generated by the deployer, and that includes this kind of environment variables needed by agents to connect with the stack.

It looks like you're referring to this issue (Extend "profiles" with local patches).

Do you think that we need a proposal to sketch the final look and then iterate on that? Maybe we should focus on this specific issue. It might be tricky if we want "patches" to be backward compatible with older stacks.

I would consider this deployer a separate thing to possible general patches for profiles or the stack subcommand. I think that something like this "agent" deployer has value on itself. Even if later we consider more advanced configurations for profiles or the stack subcommand.

The problem I see with general local patches for elastic-package stack is that they may become a source of many reproducibility problems. You may patch the stack to work on one package, and later you start working on a different package and something unexpectedly doesn't work. Or it may be difficult to know or remember that a certain package needs a patched stack.
This could cause a new set of problems similar to the ones with the packages contained in the registry depending on where elastic-package stack up was executed (#599).
And as you mention it may be difficult to support patches with multiple versions of the stack in a general way.

I think it is fine to look for a way to start/patch specialized agents on test time as is being done in this PR. These agents are disposable and developers have more awareness on when they are being started. Packages could also select the version of the agent to use, limiting the problems of supporting multiple versions. Variants could help to test with multiple versions or configurations if there are differences.

And if some day we also have stack-level patches, I think it would be ok if the patches are different to the ones used for this "agent" deployer, at the end they are different things.

I think it is fine to look for a way to start/patch specialized agents on test time as is being done in this PR.

What about unpatching? Did you plan for this too or is it like the Kubernetes agent, once installed it stays there.

I think it is fine to look for a way to start/patch specialized agents on test time as is being done in this PR.

What about unpatching? Did you plan for this too or is it like the Kubernetes agent, once installed it stays there.

No need to unpatch. Current implementation starts this patched agent as a docker compose service, and destroys it on tear down. So it doesn't stay. I think this is a good approach.

(If I understand it correctly, please @marc-gr correct me if I am wrong 🙂 )

It is correct 👍

jsoriano

I like this approach :)

Added a couple of comments about implementation details.

internal/testrunner/runners/system/servicedeployer/custom_agent.go

mtojek

Good job! I left a few comments, but maybe it's better to sync over zoom to explain the technical decisions. Up to you, Marc :)

Makefile

mtojek · 2022-05-16T10:17:26Z

internal/testrunner/runners/system/servicedeployer/custom-agent-base-config.yml

+    environment:
+      - FLEET_ENROLL=1
+      - FLEET_INSECURE=1
+      - FLEET_URL=http://fleet-server:8220


Based on this file, isn't FLEET_TOKEN_POLICY_NAME not required?

Would only be needed if we want it to be enrolled to the current stack policy, but we are just interested in it during the test, so the policy will change right after for the test one, running the config useless.

Ah, you're right! Could you please drop a comment there in case somebody else will come here?

internal/testrunner/runners/system/servicedeployer/custom-agent-base-config.yml

mtojek · 2022-05-16T10:25:23Z

internal/testrunner/runners/system/servicedeployer/compose.go

 		ExtraArgs: []string{"--build", "-d"},
 	}
 	err = p.Up(opts)
 	if err != nil {
 		return nil, errors.Wrap(err, "could not boot up service using Docker Compose")
 	}

+	// Connect service network with stack network (for the purpose of metrics collection)


Is it safe to move docker.ConnectToNetwork before checking if the service is healthy? Do you think that it won't break if the service doesn't boot up in time?

From my understanding it makes no difference for the other cases, and in this one is a requirement (healthcheck only passes once enrolled, and needs to be in the same network to be enrolled). Do you think of any specific scenarios where this might be troublesome?

Yeah, but the compose.go code is generic, so it may affect also other services like kind or docker based.

You could verify the behavior by simulating a broken Docker service, for instance: a container that immediately fails setup.

mtojek · 2022-05-16T10:29:49Z

internal/testrunner/runners/system/servicedeployer/custom_agent.go

+
+	cd, err := newDockerComposeServiceDeployer(
+		append(ymlPaths, path),
+		appConfig.StackImageRefs(install.DefaultStackVersion).AsEnv(),


Does it mean that it won't be possible to use other stack versions?

Other stacks could be used by adding the image option in the custom-agent.yml file or using service variants.

Out of curiosity: does it mean that it's possible to boot up Elastic stack 8.1 and the agent 8.3? I'm wondering if we need a helper to select the proper stack. See kind deployer

internal/testrunner/runners/system/servicedeployer/custom_agent.go

internal/testrunner/runners/system/servicedeployer/compose.go

mtojek · 2022-05-16T10:38:02Z

internal/testrunner/runners/system/servicedeployer/custom_agent.go

+	}
+
+	for k, v := range cv {
+		bc.Services["custom-agent"][k] = v


Could you please clarify a bit what is the purpose of the processing here?

we take all the custom options from custom-agent.yml and use them to override the base config.

Is the base config one from ~/.elastic-agent?

mtojek · 2022-05-16T14:26:50Z

internal/testrunner/runners/system/servicedeployer/custom_agent.go

+
+	cd, err := newDockerComposeServiceDeployer(
+		append(ymlPaths, path),
+		appConfig.StackImageRefs(install.DefaultStackVersion).AsEnv(),


Out of curiosity: does it mean that it's possible to boot up Elastic stack 8.1 and the agent 8.3? I'm wondering if we need a helper to select the proper stack. See kind deployer

mtojek · 2022-05-16T14:40:30Z

internal/testrunner/runners/system/servicedeployer/custom_agent.go

+		return "", errors.Wrap(err, "marshal custom-agent config")
+	}
+
+	tf, err := os.CreateTemp("", dockerCustomAgentName)


nit: There is ~/.elastic-package/tmp directory and probably also a function to reach out to it.

mtojek · 2022-05-16T14:41:44Z

internal/testrunner/runners/system/servicedeployer/custom_agent.go

+}
+
+func createCustomAgentYaml(cfgPath string) (string, error) {
+	bc := struct {


nit: could you please replace shortcut-named vars with something meaningful? I admit that I got lost, cv, cb, bc, .. :)

mtojek · 2022-05-16T14:45:54Z

docs/howto/system_testing.md

+This is useful if you need different capabilities than the provided by the
+`elastic-agent` used by the `elastic-package stack` command. 
+
+`custom-agent.yml`


As the custom-agent.yml will be part of a package, we need to cover it with package-spec. You will need to open one more PR.

mtojek · 2022-05-16T14:55:18Z

internal/testrunner/runners/system/servicedeployer/custom_agent.go

+	if len(outCtxt.Ports) > 0 {
+		outCtxt.Port = outCtxt.Ports[0]
+	}


nit: I don't remember if this condition is still required.

if we keep it then the port can be referenced from the test config and that can be useful in some cases, not a hard requirement though so we can remove it if there is any concern

internal/testrunner/runners/system/servicedeployer/custom_agent.go

Add config option to deploy custom elastic-agents as test services

d515d21

marc-gr requested a review from mtojek April 11, 2022 07:41

marc-gr added 2 commits April 11, 2022 09:46

Merge remote-tracking branch 'upstream/main' into custom-agents

48f20d4

Tidy go.mod

fdf4b9f

marc-gr marked this pull request as draft April 11, 2022 08:06

marc-gr mentioned this pull request Apr 11, 2022

Add ability to deploy custom elastic-agents on different OS or runtimes #787

Open

marc-gr added 3 commits May 3, 2022 10:40

Merge remote-tracking branch 'upstream/main' into custom-agents

fbc5d52

Simplify runner changes

07820ad

Clean go.mod

af8e172

marc-gr force-pushed the custom-agents branch from 70369b0 to 05ce034 Compare May 3, 2022 12:00

Move logic to custom_agent deployer

31aec19

marc-gr force-pushed the custom-agents branch from 05ce034 to 31aec19 Compare May 4, 2022 09:04

marc-gr added 2 commits May 4, 2022 11:32

Use DockerComposeDeployedService code

15d7df4

Connect to network before healthcheck

9bcb382

jsoriano reviewed May 4, 2022

View reviewed changes

internal/testrunner/runners/system/servicedeployer/factory.go Outdated Show resolved Hide resolved

Add test, docs, and initialize the same as a composer deployer

5b649d9

marc-gr marked this pull request as ready for review May 4, 2022 15:17

marc-gr requested a review from jsoriano May 4, 2022 15:17

marc-gr added 7 commits May 5, 2022 11:08

Change Makefile

009e259

Format test package

d17bb7d

Add test to CI

99131b6

Bump stack versions

632a0f5

Ignore errors in test pipeline

2bf8399

Use apache as custom-agent test and revert stack version changes

ca9fb47

Fail if docker-compose.yml is not found

7b10af7

jsoriano reviewed May 6, 2022

View reviewed changes

marc-gr added 2 commits May 12, 2022 16:37

Change custom agent deployer to use a base agent config

6e108f9

Replace apache custom agent example with auditd_manager

a1cc748

Merge remote-tracking branch 'upstream/main' into custom-agents

aa5cb6e

jsoriano reviewed May 12, 2022

View reviewed changes

internal/testrunner/runners/system/servicedeployer/custom_agent.go Outdated Show resolved Hide resolved

internal/testrunner/runners/system/servicedeployer/custom_agent.go Outdated Show resolved Hide resolved

marc-gr added 2 commits May 16, 2022 11:43

Wrap compose service deployer

4b63068

Merge remote-tracking branch 'upstream/main' into custom-agents

f89d55f

marc-gr requested a review from jsoriano May 16, 2022 09:48

Format

c586f71

mtojek reviewed May 16, 2022

View reviewed changes

marc-gr added 4 commits May 16, 2022 16:08

Overwrite setup and teardown to avoid changing compose deployer

b158777

Use service variant to avoid overriding teardown entirely

241a1b4

Update docs

4af44a4

Revert Makefile unrelated change

b0ae2ad

mtojek reviewed May 16, 2022

View reviewed changes

Use installed resources instead of a temp file

5c6f901

mtojek approved these changes May 17, 2022

View reviewed changes

marc-gr added 2 commits May 17, 2022 11:25

add version to custom-agent.yml

f6ab76b

quote version field

e37de08

marc-gr mentioned this pull request May 17, 2022

Add custom agent deployer spec elastic/package-spec#335

Merged

2 tasks

marc-gr merged commit cfaadec into elastic:main May 17, 2022

marc-gr deleted the custom-agents branch May 17, 2022 10:10

jsoriano mentioned this pull request May 31, 2022

Enable SSL in the Elastic Stack #789

Merged

24 tasks

This was referenced Jun 8, 2022

Update elastic-agent-managed.yaml.tmpl clusterrole #844

Merged

Add mechanism to customize agent deployed by kubernetes test runner #845

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add config option to deploy custom elastic-agents as test services #786

Add config option to deploy custom elastic-agents as test services #786

marc-gr commented Apr 11, 2022 •

edited

Loading

elasticmachine commented Apr 11, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented May 4, 2022 •

edited

Loading

jsoriano left a comment

jsoriano left a comment

jsoriano May 6, 2022

jsoriano May 6, 2022

mtojek May 9, 2022

jsoriano May 9, 2022

mtojek May 9, 2022

jsoriano May 9, 2022

jsoriano May 9, 2022

marc-gr May 16, 2022

jsoriano left a comment

mtojek left a comment

mtojek May 16, 2022

marc-gr May 16, 2022

mtojek May 16, 2022 •

edited

Loading

mtojek May 16, 2022

marc-gr May 16, 2022

mtojek May 16, 2022

mtojek May 16, 2022

marc-gr May 16, 2022

mtojek May 16, 2022

mtojek May 16, 2022

marc-gr May 16, 2022

mtojek May 16, 2022

mtojek May 16, 2022

mtojek May 16, 2022

mtojek May 16, 2022

mtojek May 16, 2022

mtojek May 16, 2022

marc-gr May 17, 2022

Add config option to deploy custom elastic-agents as test services #786

Add config option to deploy custom elastic-agents as test services #786

Conversation

marc-gr commented Apr 11, 2022 • edited Loading

elasticmachine commented Apr 11, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented May 4, 2022 • edited Loading

🌐 Coverage report

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

mtojek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek May 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marc-gr commented Apr 11, 2022 •

edited

Loading

elasticmachine commented Apr 11, 2022 •

edited

Loading

elasticmachine commented May 4, 2022 •

edited

Loading

mtojek May 16, 2022 •

edited

Loading