Android tests - Wrong test result when a RuntimeException is thrown

[alexandre-lefranc]

I'm submitting a ...

• bug report
• feature request
• support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

When using Android tests and throwing a RuntimeException in the app, the test result is Success instead of Broken or Failed.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

class CrashActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        throw IllegalStateException("Crashing!")
   }
}

Please tell us about your environment:

Allure kotlin -> 2.4.0
Allure version (to generate reports) -> 2.27.0

Other information

I suggest to set a default test result status in fun testStarted()

    override fun testStarted(description: Description) {
        val uuid = testCases.get()
        val result = createTestResult(uuid, description).apply {
             status = Status.BROKEN
        }
        lifecycle.scheduleTestCase(result)
        lifecycle.startTestCase(uuid)
    }

[Yasmidrog]

Getting the same problem, but instead of Success results there are no results at all and no JSONs in the reports directory.

[MisterSawyer007]

Getting the same problem
I think it is critical, since you can skip the crash and not notice that the test has not been added to the report
no results if there was a bug in the developer's code: which caused the crash (reproduced with: NullPointerExeception, IllegalStateException etc)

[JoelWhitney]

I've also experienced this. It seems like depending on where the app crashes from, it may bypass the rule lifecycle methods, and instead jump to the runners onException() -- this results in none of the Allure test handling methods getting called. I tried to manually add into a custom runner but haven't had look so far