Define the scope of web-platform-tests #215
Conversation
gsnedders
changed the title
| Additionally, other features can be added as | ||
| [tentative](https://web-platform-tests.org/writing-tests/file-names.html#:~:text=.tentative,-%3A%20) | ||
| tests: | ||
|
|
||
| * Web browser behavior currently being explored via an | ||
| [explainer](https://tag.w3.org/explainers/), | ||
| but without a specification yet written. | ||
| (XXX: require prototyping to have started?) | ||
|
|
||
| * Historic but unspecified features in web browsers, | ||
| especially where major browsers are interoperable, | ||
| where there exists a long-term intention to specify them. | ||
| (Note: this excludes features which have been deliberately removed from specifications. | ||
| These are explicitly out of scope.) |
There was a problem hiding this comment.
This arguably doesn't actually match up with what the documentation currently says:
Indicates that a test makes assertions not yet required by any specification, or in contradiction to some specification. This is useful when implementation experience is needed to inform the specification. It should be apparent in context why the test is tentative and what needs to be resolved to make it non-tentative.
The "in contradiction to some specification" is kinda troubling — this implies that if some spec change is made, with no expectation of implementations changing behaviour any time soon, you could change the existing tests to be tentative and add new non-tentative tests for the new behaviour. IMO, that should be against policy — anything that is tentative should be on a path to becoming non-tentative. We should probably be explicit in the scope of what can be tentative that everything which is tentative should be on a path to becoming non-tentative.
I don't think the two things listed here are a complete list of cases; we also have cases:
- where there are WG resolutions not yet applied to the spec making tests tentative (e.g.,
/css/css-overflow/line-clamp/line-clamp-005.tentative.htmlis an example of this, with [css-overflow] Is continue: discard working in the fragment tree useful? w3c/csswg-drafts#7708 (comment)), - where there is a proposal to change the spec to some other behaviour (e.g.,
svg-aam/role/role-img.tentative.html, with SVG image role should align with HTML on missing versus empty alt w3c/svg-aam#32), - where there is a proposed addition to the spec (e.g.,
/html/dom/elements/global-attributes/the-anchor-attribute-001.tentative.html, with Add anchor attribute whatwg/html#9144), - where the spec is unclear (e.g,
webdriver/tests/bidi/input/release_actions/sequence_tentative.py, with Should a navigation clean-up thebrowsing context input state map? w3c/webdriver#1859).
The other thing that stands out is:
% rg --glob '*tentative*' 'https?://((bugs\.webkit\.org|bugs\.chromium\.org|crbug\.com|issues\.chromium\.org)/|github\.com/.*/(issue|pull))' --files-without-match | wc -l
2078
Which shows it's very often not obvious "why the test is tentative and what needs to be resolved to make it non-tentative", though that is really a separate issue. (And looking at a few random tests, the commit history often doesn't make it obvious either.)
No description provided.