The dynamic configuration items of Java apm agent cannot take effect dynamically

[arTG0D]

We kindly ask to post all questions and issues on the Discuss forum first. In addition to awesome, knowledgeable community contributors, core APM developers are on the forums every single day to help you out as well, so your questions will reach a wider audience there.

Version

Elastic Stack 8.13.3
Java apm agent : apm-agent-attach-cli-1.52.0.jar

BUG

I noticed in the official documentation that server_url, server_urls, and log_level are dynamic, when I comment out the server_url, when I enable server_urls, it doesn't take effect, the apm data is still sent to the old apm-server, and when I use log_level=DEBUG to view the logs, it doesn't take effect.
Until then, APM is working fine

Config

recording=true
service_name=xxxxx
environment=prod
#server_url=http://xxxxxx:8200
server_urls=http://xxxxxx:8200,http://xxxxxx:8200
service_node_name=xxxxx
log_level=INFO
log_file=/opt/apm/apm-agent.log
log_ecs_reformatting=SHADE
log_ecs_reformatting_dir=/opt/apm
capture_body=off
agent_reporter_health_metrics=true
agent_background_overhead_metrics=true
transaction_max_spans=500
circuit_breaker_enabled=true

[jackshirazi]

It's not clear what you are doing because you listed the attach-cli as your agent. If you are using the attach cli to attach the agent with one config, then using a different config and using the attach-cli again against the same JVM, that won't work. If you look at the output from the attach-cli, only the first attach works, the subsequent attaches don't, because you can only attach the agent once.

The dynamic capability works for the options that the agent can see, so you would need to change those in something it has visibility on. for example with using the attach-cli it would still see central config

[arTG0D]

It's not clear what you are doing because you listed the attach-cli as your agent. If you are using the attach cli to attach the agent with one config, then using a different config and using the attach-cli again against the same JVM, that won't work. If you look at the output from the attach-cli, only the first attach works, the subsequent attaches don't, because you can only attach the agent once.

The dynamic capability works for the options that the agent can see, so you would need to change those in something it has visibility on. for example with using the attach-cli it would still see central config

I started a spring application on a virtual machine using java-jar and attached it to the spring application using attach-cli for monitoring. When I changed the server_url in the apm.properties file, it did not take effect dynamically. I tried setting recording=false again, but it still didn't take effect dynamically.

[jackshirazi]

So the config file /opt/apm/elasticapm.properties is on the host/VM/container that the target JVM is running on, and that's the file that you changed properties in?

[arTG0D]

So the config file /opt/apm/elasticapm.properties is on the host/VM/container that the target JVM is running on, and that's the file that you changed properties in?

Yes. I'm sorry that I didn't describe it clearly before.

[jackshirazi]

log_level is changed dynamically all the time with our agents, so this suggests to me this is a configuration error somewhere. My suggestion is to start with log_level=DEBUG in the properties so you can then look at the agent log output and confirm it is using the config file you expect, and seeing the values you set. You should also see it attempting to reload the config every 30 seconds, eg

2024-12-24 11:41:53,664 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Beginning scheduled configuration reload (interval is 30 sec)...

[arTG0D]

log_level is changed dynamically all the time with our agents, so this suggests to me this is a configuration error somewhere. My suggestion is to start with log_level=DEBUG in the properties so you can then look at the agent log output and confirm it is using the config file you expect, and seeing the values you set. You should also see it attempting to reload the config every 30 seconds, eglog_level 一直随着我们的代理动态更改，所以这对我来说表明这是某个地方的配置错误。我的建议是，在属性中从 log_level=DEBUG 开始，这样您就可以查看代理日志输出，确认它使用的是您期望的配置文件，并看到您设置的值。您还应该看到它每 30 秒尝试重新加载一次配置，例如

2024-12-24 11:41:53,664 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Beginning scheduled configuration reload (interval is 30 sec)...

I set log_level=DEBUG, but I don't see the log information as you said, and when the agent log file reaches 51M, there is no log rotation, resulting in the log not continuing to output, is there a problem with my configuration or startup mode?
At present, the APM data of the spring application is still being updated

[jackshirazi]

You need to look for errors from the top of the log

[arTG0D]

I found these logs with ERROR
2024-12-20 15:34:13,819 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Read timed out, response code is -1
2024-12-25 09:56:11,750 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving ERROR event (sequence 319445026)
There are agent logs in this repository
https://github.com/arTG0D/apm-agent.log

[jackshirazi]

It's definitely reloading config from the central config. I can see that your server is running too slow so lots of spans are being dropped by the agent. I can't see any attempt to change any option in this log