Add karate.scenarioOutline metadata #2636
Conversation
| * match karate.scenario.exampleIndex == <num> | ||
|
|
||
| # Confirm scenarioOutline result | ||
| * match karate.scenarioOutline == |
There was a problem hiding this comment.
the whole PR looks good and I will definitely merge.
one question - is it really necessary to have a new scenarioOutline API, won't the existing scenario suffice ?
There was a problem hiding this comment.
Could put it in with karate.scenario but I feel like it's weird to store information about other scenarios in there. I think I prefer karate.scenarioOutline but I can change it to whatever you think is best.
There was a problem hiding this comment.
the way I look at it is at run time, only scenarios exist. some scenarios happen to originate from scenario outlines. does that make sense ?
There was a problem hiding this comment.
Yeah that makes sense, I definitely learned that the code is structured that way as well. Maybe the karate.scenario object can have an outline section when it originates from a Scenario Outline and it can include some/all of the same data I currently put into karate.scenarioOutline?
There was a problem hiding this comment.
since exampleTableIndex is added to ScenarioResult.java won't that be sufficient ? if exampleIndex is not -1 it means it originated from an outline. and you can check exampleTableIndex for your requirement. I'm still not sure how you can identify if something is the last example
There was a problem hiding this comment.
Yeah I like the idea of adding lastExampleInOutline to karate.scenario. The only other piece of the puzzle is how to get the results of the other scenarios in the outline.
Also when running in parallel will lastExampleInOutline still hold true? No right? Or at least we can't guarantee that when the lastExampleInOutline has finished, that the others in the outline also have.
Maybe I could have that variable only populate when not running in parallel? Or since you are more familiar with how parallel execution works maybe there's a good way to track the completion of parallel scenario?
There was a problem hiding this comment.
just checked, parallel should be okay because it depends on the final "selected" list of scenarios to be resolved: https://github.com/karatelabs/karate/blob/v1.5.0/karate-core/src/main/java/com/intuit/karate/core/ScenarioIterator.java#L74
which means you will have one scenario (per outline) with lastExampleInOutline set to true.
regarding timing - yeah that's a great question. can I suggest something - instead of using the afterScenario hook suppose you use afterFeature, then you iterate over the scenario-results in FeatureResult and there may not be any other complexity. come to think of it - it may not require this PR after all :|
guessing that you may want the scenario execution recorded in "real time" - but you seem to have been ok to wait until the last example completes, so I hope that's not a must-have. even if it is - I would focus on making sure the time-stamps on the scenario execution are accurate and robust enough for your system of record, if not - that would be a high-priority enhancement
let me know what you think and apologies again for the back and forth, it is hard to switch context and remember all the nuances of code I haven't looked at for a while.
There was a problem hiding this comment.
No worries totally understand the context switching. I don't think there is any way to iterate over the scenario results in karate.feature. I think karate.feature just contains high level metadata like description, title, etc...
Yeah there is no absolute need to update the system of record immediately. But it is kind of fun to watch it update realtime :) Our old method was to write files with the results and combine them at the very end of the suite. This is fine but it leads to lots of files and if we want to cancel the suite the system of record is not updated. So this really is just an enhancement. I can see value in providing the karate.scnarioOutline metadata but ultimately this is your project so I will respect whatever your final decision is.
Also one thing I forgot to mention is that in order to update the correct test in the system of record we have to keep track of an API ID. Currently this API ID is just defined as a karate variable in each Scenario, so we would lose these IDs in the afterFeature if there were multiple scenarios. We have a pretty large test suite so it would be infeasible for us make any large scale change to switch to afterFeature
There was a problem hiding this comment.
acknowledged. let me think over this a bit
There was a problem hiding this comment.
@OwenK2 I'm thinking that we should introduce a new afterScenarioOutline hook that may be much cleaner. I can see this also being useful for the @setup tag that has proven very useful - and being adopted more by teams. I would consider adding this to the Java hook - RuntimeHook
that said - I'm not yet able to figure when to call this hook and how to detect that the last scenario in an outline-batch has completed - and have this work even under parallel execution.
my current guess is here: https://github.com/karatelabs/karate/blob/v1.5.0/karate-core/src/main/java/com/intuit/karate/core/FeatureRuntime.java#L197 - where you can see a synchronize on the feature result. so perhaps we can add intelligence within that synchronize to a) see if we are part of an outline and b) see if all other scenarios in that outline are already in the feature-result.
let me know what you think. if we go down this path - I agree that we need a new data-structure like you did already that returns an array of results for a given scenario-outline.
| @@ -101,13 +104,13 @@ public List<Scenario> getScenarios(FeatureRuntime fr) { | |||
| if (selectedForExecution) { | |||
| Table table = examples.getTable(); | |||
| if (table.isDynamic()) { | |||
| Scenario scenario = toScenario(table.getDynamicExpression(), -1, table.getLineNumberForRow(0), examples.getTags()); | |||
| Scenario scenario = toScenario(table.getDynamicExpression(), examples.getIndex(), -1, table.getLineNumberForRow(0), examples.getTags()); | |||
There was a problem hiding this comment.
so following up from my previous comment - suggest since in this loop we know how many examples and example tables are there - we will be able to emit an extra boolean if this is the last scenario selected for execution within an outline
if that solves your current ask - I would just do that, as I really don't see a need for tracking counts of examples and tables :|
There was a problem hiding this comment.
Sounds good I can definitely make that simplification. What about the scenarioResults though? What do you feel would be the best way to get that information from an API perspective.
There was a problem hiding this comment.
For now I just removed the table index code in 6e84b5d. Didn't want to push too far ahead before we finalized something.
Description
This PR adds a
karate.scenarioOutlinevariable that is accessable from within allScenario Outlines. It contains metadata about the outline which may be useful.Untested:
@setupor other karate features that I am less familiar with