Update Tester instances to trigger just one test #1898
Conversation
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> |
There was a problem hiding this comment.
There is one failure per pipeline test.
This error is the result of installing the pipelines of cisco_asa. Previously, pipelines were installed just onace, now those pipelines are installed once per test (Run()).
There was a problem hiding this comment.
Will these failures be predictable if we parallelize?
At some point we may consider installing the pipelines by installing the packages, then we could install the package only once. The pipelines we test now are not exactly the same as what is executed at the end, see #1743.
There was a problem hiding this comment.
Maybe another mid-ground option is to install all pipelines included in the package in the Setup step, and remove them on TearDown, but then we would be installing all pipelines, even the ones that are not tested. Not sure I like this approach.
There was a problem hiding this comment.
Will these failures be predictable if we parallelize?
I guess this would be a source of flakiness. A pipeline test running in parallel with another one could report the errors from the other pipeline test and vice versa. So I guess the number of errors could vary.
It's also true that for now there would be support to run in parallel just in the system tests. The other type of tests would be running sequentially.
Maybe another mid-ground option is to install all pipelines included in the package in the Setup step, and remove them on TearDown, but then we would be installing all pipelines, even the ones that are not tested.
Do you mean installing and removing them in the SetupRunner and TearDownRunner methods? so those are just performed once
Currently, those methods just return error in the interface, if we want to do that probably they should also return []testrunner.Result , to return errors if any in those steps.
mrodm
changed the title
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> |
There was a problem hiding this comment.
Will these failures be predictable if we parallelize?
At some point we may consider installing the pipelines by installing the packages, then we could install the package only once. The pipelines we test now are not exactly the same as what is executed at the end, see #1743.
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> | ||
| <failure>detected ingest pipeline warnings: 2: character class has '-' without escape
regular expression has redundant nested repeat operator.*</failure> |
There was a problem hiding this comment.
Maybe another mid-ground option is to install all pipelines included in the package in the Setup step, and remove them on TearDown, but then we would be installing all pipelines, even the ones that are not tested. Not sure I like this approach.
💛 Build succeeded, but was flaky
Failed CI StepsHistory
cc @mrodm |
Part of #787
Depends on #1895
Changed how Tester instances are created in each runner. In this PR, now it is created a Tester per each configuration file and variant (if required), so each Tester instance should be in charged of just one test.
This PR affects the following testers:
Static tests keep creating one Tester per data stream folder, and Asset tests just create one test per package.
Note: