Understanding the output of git regression tests

[zrrbite]

Hi!

New to git-sdk and I'm running regression tests just to get an idea. There are a few things I don't quite understand that you can probably clear up in a jiff!

I'm seeing some conflicting ways of reporting results, and I'm also curious to know how seriously these results are to be taken. In the first snippet there are "known breakages" where each failing test reports "not ok". E.g.:

not ok 14 - clone --local detects misnamed objects # TODO known breakage
# still have 1 known breakage(s)
# passed all remaining 70 test(s)

But here are some tests that, despite reporting "ok", have the word "fail" in them, and the whole suite of tests is marked as passed. Here "fail" probably has something to do with the nature of the test itself but tests 71 and 72 appear identical based on the description, for instance, while 73 has the keyword "fail" attached to it.

*** t1000-read-tree-m-3way.sh ***
...
ok 69 - 11 (fail) - must match and be up-to-date in O && A && B && O!=A && O!=B && A!=B case
ok 70 - 11 (fail) - must match and be up-to-date in O && A && B && O!=A && O!=B && A!=B case
ok 71 - 12 - must match A in O && A && B && O!=A && A==B case
ok 72 - 12 - must match A in O && A && B && O!=A && A==B case
ok 73 - 12 (fail) - must match A in O && A && B && O!=A && A==B case
ok 74 - 13 - must match A in O && A && B && O!=A && O==B case
ok 75 - 13 - must match A in O && A && B && O!=A && O==B case
ok 76 - 14 - must match and be up-to-date in O && A && B && O==A && O!=B case
ok 77 - 14 - may match B in O && A && B && O==A && O!=B case
ok 78 - 14 (fail) - must match and be up-to-date in O && A && B && O==A && O!=B case
ok 79 - 14 (fail) - must match and be up-to-date in O && A && B && O==A && O!=B case
ok 80 - 15 - must match A in O && A && B && O==A && O==B case
ok 81 - 15 - must match A in O && A && B && O==A && O==B case
ok 82 - 15 (fail) - must match A in O && A && B && O==A && O==B case
ok 83 - 16 - A matches in one and B matches in another.
# passed all 83 test(s)

Can someone help me understand this?

Thanks for your time!

Best regards,
Martin

[zrrbite]

Okay, looking at the test-code it appears that "(fail)" tests for this suite ( t1000-read-tree-m-3way.sh) are named as such because they use:

read_tree_must_fail

However, other test in the suite that also use read_tree_must_fail do not have "(fail)" in their description.

To summarize

This seems right:

test_expect_success '3 - matching A alone is OK in !O && A && !B case.' '
	rm -f .git/index AN &&
	cp .orig-A/AN AN &&
	git update-index --add AN &&
	echo extra >>AN &&
	read_tree_must_succeed -m $tree_O $tree_A $tree_B
'

test_expect_success '3 (fail) - must match A in !O && A && !B case.' '
	rm -f .git/index AN &&
	cp .orig-A/AN AN &&
	echo extra >>AN &&
	git update-index --add AN &&
	read_tree_must_fail -m $tree_O $tree_A $tree_B

While this seems to not follow the same naming convention. But I'm just rambling and my observations are either wrong or extremely minor.

test_expect_success '1 - must not have an entry not in A.' '
	rm -f .git/index XX &&
	echo XX >XX &&
	git update-index --add XX &&
	read_tree_must_fail -m $tree_O $tree_A $tree_B
'

[dscho]

When encountering such puzzling messages, it is usually a good idea is to look at the history of them (the Git project puts a lot of emphasis on thorough and helpful commit messages). Sadly, the Git annotate/blame output is not very helpful here because it points to a clean-up patch. You'll need to manually navigate to the annotate/blame of the parent commit to eventually be pointed to the commit that added those test cases: f225b21. Unfortunately, that commit's message is not quite up to today's standards. We have to interpret it a bit (and notice that the test cases were marked as test_expect_failure) and realize that the Git project had a revelation about test_expect_failure: It is too unspecific, marking a test case as successful even when one of it commands fails for unexpected reasons. Basically, the test case is about verifying that a specific Git command fails in a very specific way, for expected reasons. That's why it is marked as fail.

And yes, I hear your pain. It is often very hard to figure out why a regression test fails, and even harder to fix the regression. That's the outcome on the Git project focusing for a long time on adding regression tests that do not really help fix regressions, only to detect regressions. Another contributing factor is that there are many test cases that are still marked as test_expect_failure: Officially, these are meant as reminders that there are known regressions/bugs that still need to be fixed, but in practice those failures are mostly ignored.

[zrrbite]

Thanks for taking the time to reply and clarify, Johannes. I know how follow the breadcrumbs now. Also, I don't expect following up on regressions and bugs is the main focus of this particular project.

Thanks for all your hard work. I'd love to help any way I can.