You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The integration tests currently run in a fixed order, see https://github.com/mozilla/pdf.js/blob/master/test/integration-boot.mjs#L24, which is in contrast with e.g. the unit test that deliberately run in a random order to bring any isolation issues (dependencies between tests) to the surface. Most likely this wasn't done for the integration tests yet because we used to have quite a lot of intermittent issues and dependencies between tests when we started with the integration tests, but over time we have fixed most of them so hopefully running the integration tests in a random order is more feasible nowadays.
We should try this out and make a list of any issues we encounter. This could involve closing the document between tests in the same describe block to avoid state leaking through to subsequent tests; see e.g. PR #19064 for which an alternative solution was implemented.
The objective is to reduce the number of test failures (currently if there is a failure in one test it'll currently cause all subsequent dependent tests to fail too) and to improve debugging of the remaining intermittents (currently we sometimes need to run a full describe block only to debug one test which is time-consuming).
The text was updated successfully, but these errors were encountered:
I have tested this locally today and that has resulted in a relatively good overview of what needs to be done to achieve the goal of independent tests that can be run in a random order.
Some spec files already work in random mode, but others are problematic because they contains tests that build upon the result of previous tests and therefore fail immediately if the document is closed between tests. Fortunately this process does bring dependencies between subsequent tests to the surface so we can fix them, and from what I can tell enabling random order should directly be possible then because I haven't found dependencies between non-subsequent tests.
This work roughly involves closing the document between all tests, fixing every failing test and enabling random mode. This is the result of the test:
File
Passes if document is closed
Passes (three times) if random mode is enabled
accessibility_spec.mjs
✅
✅
annotation_spec.mjs
✅
✅
caret_browsing_spec.mjs
✅
✅
copy_paste_spec.mjs
✅
✅
find_spec.mjs
✅
✅
freetext_editor_spec.mjs
❌ (8 of 53 failed)
❌
highlight_editor_spec.mjs
✅
✅
ink_editor_spec.mjs
❌ (1 of 19 failed)
❌
scripting_spec.mjs
❌ (3 of 63 failed)
❌
stamp_editor_spec.mjs
✅
✅
text_field_spec.mjs
✅
✅
text_layer_spec.mjs
✅
✅
viewer_spec.mjs
✅
✅
I have a local branch with fixes and refactorings, and once the work is complete I'll create PRs for each self-contained chunk.
The integration tests currently run in a fixed order, see https://github.com/mozilla/pdf.js/blob/master/test/integration-boot.mjs#L24, which is in contrast with e.g. the unit test that deliberately run in a random order to bring any isolation issues (dependencies between tests) to the surface. Most likely this wasn't done for the integration tests yet because we used to have quite a lot of intermittent issues and dependencies between tests when we started with the integration tests, but over time we have fixed most of them so hopefully running the integration tests in a random order is more feasible nowadays.
We should try this out and make a list of any issues we encounter. This could involve closing the document between tests in the same
describe
block to avoid state leaking through to subsequent tests; see e.g. PR #19064 for which an alternative solution was implemented.The objective is to reduce the number of test failures (currently if there is a failure in one test it'll currently cause all subsequent dependent tests to fail too) and to improve debugging of the remaining intermittents (currently we sometimes need to run a full
describe
block only to debug one test which is time-consuming).The text was updated successfully, but these errors were encountered: