Allow data stream overrides in system tests for specific integration packages

[ShourieG]

Some integrations like amazon security lake use routing rules to route results to a different data stream from a central data stream. Having such a mechanism the standard system tests fail since it always searches for hits in the central data stream which is generally where the system tests reside. We need a way to dynamically instruct elastic-package to look for hits in a specific data stream during system tests. We could use a new package-spec variable to do this potentially.

This issue has been created following conversations in this PR

[ShourieG]

Hi @jsoriano, any updates as to when or if this will get implemented ?

[jsoriano]

This task is not planned yet, ccing @kpollich for prioritization.

[kpollich]

We have a quality sprint in August that would probably be a good time to look into improving the system tests to support this kind of option, but that's probably the soonest we can look into this with other competing priorities considered.

[ShourieG]

Hi @kpollich, I noticed one more thing while locally making some hacks to get this working, it won't be enough to just detect the hits in the re-routed data stream but we also need to validate the tests against the field mappings of the re-routed data stream.

[ShourieG]

Hi @kpollich, any updates on this ?

[kpollich]

This unfortunately didn't make it into our quality sprint, and ecosystem bandwidth is pretty low right now with LogsDB and OTel priorities taking up the vast majority of our time. We likely won't get to this for 2-3 months.

[ShourieG]

@kpollich, Could you share some ideas on how to proceed with this implementation from your end or the philosophies in design that should be considered in implementing this feature, maybe we can take up the development on our end and have the PR reviewed by your team ?

[kpollich]

maybe we can take up the development on our end and have the PR reviewed by your team ?

This is perfectly acceptable to me, and we'd be happy to help support the implementation. @jsoriano could probably help more with nailing down a proposal for the implementation details here.

[jsoriano]

On a first thought now, I think a good starting point could be @ShourieG's original PR #1913, but without hard-coding the list of packages.

There are a couple of options I think we could explore for that:

• If this is needed for all data streams using routing rules, use the override if the data stream has routing rules.
• If we cannot reliabily auto-detect when the override is needed, we can add a flag to system test configuration files that indicates that.

@ShourieG could you give a try to the first point? It would be to replace dataStreamOverrides[scenario.pkgManifest.Name] with some function that detects if there are routing rules in the data stream being tested.

[ShourieG]

On a first thought now, I think a good starting point could be @ShourieG's original PR #1913, but without hard-coding the list of packages.

There are a couple of options I think we could explore for that:

• If this is needed for all data streams using routing rules, use the override if the data stream has routing rules.
• If we cannot reliabily auto-detect when the override is needed, we can add a flag to system test configuration files that indicates that.

@ShourieG could you give a try to the first point? It would be to replace dataStreamOverrides[scenario.pkgManifest.Name] with some function that detects if there are routing rules in the data stream being tested.

@jsoriano, Absolutely, I'll plan a task to take this up in our next sprint and work through the suggestions.

[taylor-swanson]

I'll be interested in leveraging this feature as well for my work for the Syslog Smart Router project. It also uses routing rules to send logs to other data streams which I will need to verify.