centos8 artifact test fails on assert node_info_service.get_memory_available

[fruch]

2023-12-17 22:03:05.171: (TestFrameworkEvent Severity.ERROR) period_type=one-time event_id=fbfc594c-2ca5-42a7-9deb-7051664e0ca5, source=ArtifactsTest.test_scylla_service (artifacts_test.ArtifactsTest) [check node_exporter liveness]() message=Traceback (most recent call last):
File "/tmp/jenkins/workspace/scylla-master/artifacts-offline-install/artifacts-centos8-arm-test/scylla-cluster-tests/artifacts_test.py", line 345, in test_scylla_service
assert node_info_service.get_memory_available()
AssertionError

< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > Failed to get node info for timeout: 
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > Encountered a bad command exit code!
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > 
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > Command: 'curl -s localhost:9180/metrics'
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > 
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > Exit code: 7
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > 
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > Stdout:
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > 
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > 
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > 
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > Stderr:
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > 
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > 
< t:2023-12-31 23:54:24,082 f:__init__.py     l:31   c:sdcm.utils.adaptive_timeouts p:WARNING > 
< t:2023-12-31 23:54:24,082 f:remote_base.py  l:521  c:RemoteLibSSH2CmdRunner p:DEBUG > Running command "sudo systemctl start scylla-server.service"...

• https://jenkins.scylladb.com/job/scylla-enterprise/job/artifacts/job/artifacts-centos8-arm-test/196/
• https://jenkins.scylladb.com/job/scylla-enterprise/job/artifacts/job/artifacts-centos8-arm-test/182/
• https://jenkins.scylladb.com/job/scylla-master/job/artifacts-offline-install/job/artifacts-centos8-arm-test/270/

[roydahan]

@fruch why is it an SCT issue?
I saw the same failures in some of the ami tests of 2024.2.2

[fruch]

@fruch why is it an SCT issue? I saw the same failures in some of the ami tests of 2024.2.2

cause it was new introduced check at the time.

anyhow where did you seen those ? I don't see any email with those failure...

[fruch]

@roydahan

I can only assume you we're looking at jenkins output, and the the same warning, i.e. not this failure.

the actually failure I see on artifacts of 2024.2 is #9444

[roydahan]

I indeed see it only in output.log, but I don't see anything else: https://argus.scylladb.com/tests/scylla-cluster-tests/f60cc98b-b99f-4335-bb2e-a6cb2fe6c9ec
or
https://argus.scylladb.com/tests/scylla-cluster-tests/b0b63acc-4a65-4c56-93b8-69ed5fcd6688

[fruch]

I indeed see it only in output.log, but I don't see anything else: https://argus.scylladb.com/tests/scylla-cluster-tests/f60cc98b-b99f-4335-bb2e-a6cb2fe6c9ec
or
https://argus.scylladb.com/tests/scylla-cluster-tests/b0b63acc-4a65-4c56-93b8-69ed5fcd6688

It's a failure that is visible only on Jenkins output, and it can kill a builder mid run.

[fruch]

this isn't happening anymore