switch off tests that are flaky under simulation #46
Conversation
lsf37
force-pushed
the
flaky-tests
branch
2 times, most recently
from
ad4695f to
e9cd00b
Compare
| test_debug_set_data_breakpoint, config_set(CONFIG_HARDWARE_DEBUG_API)) | ||
| test_debug_set_data_breakpoint, | ||
| config_set(CONFIG_HARDWARE_DEBUG_API) && | ||
| !config_set(CONFIG_SIMULATION)) |
There was a problem hiding this comment.
Can you add a comment here why this test does not work ion the simulation? If this mainly due to #43 then we should just mention this. But then it is not just an issue for the simulation.
There was a problem hiding this comment.
Good point, will add a comment. I've only observed the problem in #43 under simulation so far. If it really is a race condition, it seems to be only triggered under simulation conditions.
If we see it elsewhere as well then we should definitely change the flag, but I haven't yet so far. In fact, the only flaky test I've seen so far on hardware is MULTICORE0004, which does say (flaky) in its description.
What is flaky in the hardware tests is mostly boards not coming up for boot or not recognising correctly that they have booted and then timing out on the boot process. Problems ranging from "network cable fell out", to dodgy earth connection messing up serial, to old cable etc. Will hopefully become stable again over time.
| @@ -1625,7 +1625,7 @@ static int test_simple_preempt(struct env *env) | |||
| return sel4test_get_result(); | |||
| } | |||
| DEFINE_TEST(SCHED0021, "Test for pre-emption during running of many threads with equal prio", test_simple_preempt, | |||
| true); | |||
| !config_set(CONFIG_SIMULATION)); | |||
There was a problem hiding this comment.
Same here, should have a comment why this does not work in the simulation. We could mention #42 here.
These are currently BREAKPOINT_002 and SCHED0021. The corresponding issues #43 and #42 remain open, and when resolved positively, these tests should be enabled again for simulation runs. Signed-off-by: Gerwin Klein <[email protected]>
| @@ -263,8 +264,11 @@ test_debug_set_data_breakpoint(struct env *env) | |||
| } | |||
| return sel4test_get_result(); | |||
| } | |||
| /* This test is flaky under simulation. See also #43 */ | |||
There was a problem hiding this comment.
should better day "GitHub Issue #43" to give a bit more context that this number refers to.
| @@ -1624,8 +1624,10 @@ static int test_simple_preempt(struct env *env) | |||
|
|
|||
| return sel4test_get_result(); | |||
| } | |||
| /* This test is flaky under simulation. Probably a race condition that only | |||
| comes out under simulator timing conditions. See also #42 */ | |||
There was a problem hiding this comment.
Same here, better say "GitHub Issue #42" to give a bit more context that this number refers to.
These are currently BREAKPOINT_002 and SCHED0021. The corresponding issues #43 and #42 remain open, and when resolved positively, these tests should be enabled again for simulation runs.