--preset=merge for moz-webgpu-cts process-reports

[ErichDonGubler]

Depends on #13 for concept and implementation.

---

I, a developer of Firefox's implementation of WebGPU, often encounter distinct phases of expectation adjustment when I make changes to Firefox. Usually, I first discover all expected permanent outcomes for a revision of Firefox's code (perhaps with the help of moz-webgpu-cts process-reports … from #13!). Afterwards, there is often another phase in which I have not yet discovered all intermittent test cases. This activity tends to require a high number of test executions to adjust new expectations adequately.

Fortunately, to an even extent greater than reset-* heuristics, this task is automatable. The inputs and flow will be similar. I want moz-webgpu-cts process-reports to adjusts expectations to be the union of outcomes in current metadata and in processed reports.

Q: Why can we not use ./mach wpt-update --update-intermittent …?

./mach wpt-update --update-intermittent … appears to not implement behavior documented at the Updating Expectations section of wptrunner (emphasis mine):

When this option is used, the expected key stores expected intermittent statuses in addition to the primary expected status. If there is more than one status, it appears as a list. The default behaviour of this option is to retain any existing intermittent statuses in the list unless --remove-intermittent is specified.

Once upon a time, WebGPU CTS had expectations like this for tests that had a large number of small subtests:

[example_test]
  [subtest1]
    expected: TIMEOUT

  [subtest2]
    expected: NOTRUN

  [subtest3]
    expected: NOTRUN

  # ...

  [subtestN]
    expected: NOTRUN

Because the ./mach wpt-update … script only emits observed TIMEOUT and NOTRUN outcomes to test metadata, I wanted to manually make an update, so that these tests would always have TIMEOUT and NOTRUN together as accepted intermittent subtest outcomes, namely:

[example_test]
  [subtest1]
    expected: [TIMEOUT, NOTRUN]

  [subtest2]
    expected: [NOTRUN, TIMEOUT]

  [subtest3]
    expected: [NOTRUN, TIMEOUT]

  # ...

  [subtestN]
    expected: [NOTRUN, TIMEOUT]

I expected ./mach wpt-update … to maintain these intermittent statuses unless --remove-intermittent was specified. However, running ./mach wpt-update --update-intermittent … instead removed the TIMEOUT and NOTRUN statuses that I had added, despite not specifying --remove-intermittent. That is, wpt-update acted as I would expect if --remove-intermittent were also specified.

[ErichDonGubler]

CC @jmaher, in case it's interesting for discussing later.

[ErichDonGubler]

This is being implemented as --preset=merge in #14. Will change vocabulary in OP to match.

[ErichDonGubler]

Closed with #14.