Fix issues with the Bulk API Scale Tests

[khansaad]

Description

This PR fixes issues with the Bulk API scale tests

Fixes # (#1399 )

Type of change

• Bug fix
• New feature
• Docs update
• Breaking change (What changes might users need to make in their application due to this PR?)
• Requires DB changes

How has this been tested?

Please describe the tests that were run to verify your changes and steps to reproduce. Please specify any test configuration required.

• New Test X
• Functional testsuite

Test Configuration

• Kubernetes clusters tested on: Openshift

Checklist 🎯

• Followed coding guidelines
• Comments added
• Dependent changes merged
• Documentation updated
• Tests added or updated

Additional information

Include any additional information such as links, test results, screenshots here

[chandrams]

@khansaad - Have you tested it with the scale test and thanos setup?

[khansaad]

@khansaad - Have you tested it with the scale test and thanos setup?

With prometheus in the resource hub cluster, scale test worked fine. But with thanos in the kruize-scalelab cluster, I'm getting 503 after sometime.

However, in both cases the ConstraintViolation exception is not seen now.

[khansaad]

scale-test-thanos-kruize-scalelab.zip

[khansaad]

scale-test-prometheus-resourceHub.zip

[chandrams]

@khansaad - I think 503 is with 1k containers, can you share the build you used.

[khansaad]

@khansaad - I think 503 is with 1k containers, can you share the build you used.

quay.io/khansaad/autotune_operator:time-range-fix

[chandrams]

@khansaad - I still see the issue with the above image

./bulk_scale_test.sh -i quay.io/khansaad/autotune_operator:time-range-fix -w 5 -r /tmp/res

{
    "status": "COMPLETED",
    "total_experiments": 1,
    "processed_experiments": 1,
    "notifications": null,
    "experiments": {
        "thanos|default|msc-0|tfb-qrh-sample-0(deployment)|tfb-0": {
            "notification": {
                "type": "ERROR",
                "message": "{\"documentationLink\":\"\",\"message\":\"Converting `org.hibernate.exception.ConstraintViolationException` to JPA `PersistenceException` : could not execute statement\",\"httpcode\":400,\"status\":\"ERROR\"}",
                "code": 400
            },
            "recommendations": {
                "status": "UNPROCESSED",
                "notifications": null
            }
        }
    },
    "webhook": null,
    "job_id": "84e9236d-eab8-4f4b-916e-92894404e156",
    "job_start_time": "2024-12-06T11:25:06.836Z",
    "job_end_time": "2024-12-06T11:25:07.182Z"
}

[chandrams]

I have re-tested this again, but haven't seen this issue. Could be intermittent though, for now we can merge this and will raise another issue if seen again

[chandrams]

LGTM