Testing API followup

[firelizzard18]

There are a variety of features I want to implement and changes I want to make, following the completion of the initial test controller implementation, #1579. Instead of opening a bunch of issues, I'm going to list my thoughts here, so the team can decide what does and does not merit an issue.

• Refactoring - Refactor testing support #1702 - consolidate testing, refactor prior test infrastructure (prior to CL 330809) to be a layer on top of runTests.
• Settings - respect files.exclude etc - Respecting vscode exclude directives. #1739 - CL
• Subtests - Make subtests runnable - CL
• Subtests - Discover subtests programatically¹ Allow "subtest at cursor" to work with table-driven test setup #1602
• Build tags - Improve handling of build-tag-specific tests: src/goTestExplorer: improve handling of tests in files with build tags #1673
• Build tags - Testing API tag support: Support test tags microsoft/vscode#129456
• Profiling - I want to run a test using the profiler to collect CPU and/or Memory data Profiling CPU and Memory of Go programs #1685
• Profiling - I want to see the results for the profiler in a graphical way - CL
• Profiling - The same for all the tests/benchmarks in a package
• Profiling - I want to be able to compare two profiler results
• Profiling - Retain profiles across VSCode sessions - that is, don't store profiles in a session-specific temp directory
• Test suites - Add support for test suites such as github.com/stretchr/testify/suite - CL

Test Output

go test outputs events. Those events are (hopefully) attached to a specific test, but otherwise have no structure.

• Determining what file and line the output comes from
  • I attach output to a given file:line by looking for file.go:123: .... This is less than ideal, but I think it's about all we can do
• Differentiating between errors and generic log messages
  • go test does not have any notion of error messages, so this has to be heuristic
  • I was using logic I stole from my extension to recognize when output is an error message from stretchr/testify
  • Testing: non-error output microsoft/vscode#129201

Sub-tests

statue: done

Currently, I discover sub tests by analyzing the test output. If I see a test event where the Test field is TestFoo/Bar, I create Bar as a sub-test of TestFoo.

• If a file is edited and the edit range overlaps a test, I delete all sub-tests of that test.
  • Given that sufficiently compelx/obfuscated sub-tests will not be semantically recognizable¹, I don't see a way around this.
  • If I don't do this, edits to a test with semantically unrecognizable sub-tests will result in stale sub-test definitions, which IMO is a worse UX.
• When a test is run, I delete all dynamically discovered sub-tests of that test.

¹ There will always be sub-tests that cannot be identified without resorting to code execution of some sort. Sufficiently complex/obfuscated code will defeat any attempt to identify 1) locations where (*testing.T).Run is called, 2) whether and how many times the call occurred, and 3) the name passed to each call.

Debugging

status: done for a single test

#1590 does not support debugging tests. Debugging a test is very different than running a test, in that tests are run by spawning go test as a Node child process, whereas tests are debugged by starting a VSCode debugging session. Proper support for debugging tests with a test controller would require updating the test state (running, failed, passed, etc), would require getting output from the debug session. I opted not to add all this complexity to the already rather large initial implementation.

[firelizzard18]

/cc @hyangah

[firelizzard18]

Wrong button...

[gopherbot]

Change https://golang.org/cl/343433 mentions this issue: src/goTestExplorer: support running sub-tests

[gopherbot]

Change https://golang.org/cl/343489 mentions this issue: src/goTestExplorer: handle stretchr suites

[firelizzard18]

CL 345470 contributes a view to the testing pane that lists captured profiles:

Currently, the view does not include any information about what package or module a test is in. Clicking on a test shows the test (opens the containing document and scrolls to the appropriate line). Clicking on a profile opens that profile (currently text only).

[gopherbot]

Change https://golang.org/cl/345470 mentions this issue: src/goTest: add view for profiles

[firelizzard18]

@hyangah VSCode insiders has support for tagging tests and run profiles microsoft/vscode#129456. If a test is tagged, that test can only be run by run profiles that also have that tag. Tests can have multiple tags, run profiles can only have one tag. Now that running subtests works, I don't think we have any use for tags.

Originally, I planned to use this to improve the UX around running subtests - it would remove the option to run them at all, instead of trying to run them and then seeing an error message. But that's not relevant any more. We could also use this to handle build tags, but once the document symbol provider is switched to gopls, that will be moot. Once gopls has better support for build tags, this could be more relevant.

[SuddenGunter]

@firelizzard18 is there an issue for tracking state of the "debugging unit tests" feature?

[firelizzard18]

@SuddenGunter This issue is the closest thing to that. One-at-a-time debugging was merged earlier today: CL 348571. That should be in the nightly build tonight. I do have plans to remove the one-at-a-time restriction, but I'm planning on working on refactoring first (#1702).

You can see all the testing API CLs under the vscode-testing-api topic. I haven't started on it in earnest, but the refactoring work will be under the vscode-testing-refactor topic.

[firelizzard18]

For my own issue tracking:

Issue	Work to be done
#2445	Statically detect simple table-driven tests
#1602	Support "run at cursor" for entries in a table
#1734	Let the user manually enter a test or subtest name to execute

[firelizzard18]

Remaining items are or should be tracked separately