Create a "simple" interop dashboard

[laurentsenta]

As a libp2p/test-plans maintainer, I want the tooling required to define interop test matrixes and retrieve data in a simple form (a flat table of interop results).

the dashboard was discussed in #44 (comment))
This will support the epic in #53

Tasks

• Create a poc to demonstrate the feature
  • create tooling to generate pairs of tests,
  • generate a "simple" output, like a markdown table or a csv output,
• Implement native support in Testground
  • feat: support interop matrix testground/testground#1516
• Run this code and CI and figure out a "good" output
  • output to a workflow?
  • output to the README or a status file in the repo?
  • other?
• use this tooling in CI to build & run "every" interop combinations we know about,
  • Implement the "definitive" version for test generation & output generation
  • Implement CI actions
• In a follow-up issue: create a nicer output (Canonical interop tests & dashboard #62) that can be shared on a website (see EPIC: support multi-dimensional testing #53)

[laurentsenta]

example run:
https://github.com/libp2p/test-plans/actions/runs/3247284158

• Builds 12 instances (25 minutes)
• Run 66 ping test with a SINGLE iteration (8 minutes)

outcome:

run_index;run_id;status;
0;rust-master x rust-v0.47.0;pass;
1;rust-master x rust-v0.46.0;pass;
2;rust-master x rust-v0.45.1;pass;
3;rust-master x rust-v0.44.0;pass;
4;rust-master x go-master;pass;
5;rust-master x go-v0.22;pass;
6;rust-master x go-v0.21;pass;
7;rust-master x go-v0.20;pass;
8;rust-master x go-v0.19;pass;
9;rust-master x go-v0.17;pass;
10;rust-master x go-v0.11;pass;
11;rust-v0.47.0 x rust-v0.46.0;pass;
...
60;go-v0.20 x go-v0.19;pass;
61;go-v0.20 x go-v0.17;pass;
62;go-v0.20 x go-v0.11;pass;
63;go-v0.19 x go-v0.17;pass;
64;go-v0.19 x go-v0.11;pass;
65;go-v0.17 x go-v0.11;pass;

markdown:

(each test is clickable, in theory we could link to the corresponding action + step, which will be very useful for debugging)

	rust-master	rust-v0.47.0	rust-v0.46.0	rust-v0.45.1	rust-v0.44.0	go-master	go-v0.22	go-v0.21	go-v0.20	go-v0.19	go-v0.17	go-v0.11
rust-master	⚪	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢
rust-v0.47.0	🟢	⚪	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢
rust-v0.46.0	🟢	🟢	⚪	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢
rust-v0.45.1	🟢	🟢	🟢	⚪	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢
rust-v0.44.0	🟢	🟢	🟢	🟢	⚪	🟢	🟢	🟢	🟢	🟢	🟢	🟢
go-master	🟢	🟢	🟢	🟢	🟢	⚪	🟢	🟢	🟢	🟢	🟢	🟢
go-v0.22	🟢	🟢	🟢	🟢	🟢	🟢	⚪	🟢	🟢	🟢	🟢	🟢
go-v0.21	🟢	🟢	🟢	🟢	🟢	🟢	🟢	⚪	🟢	🟢	🟢	🟢
go-v0.20	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	⚪	🟢	🟢	🟢
go-v0.19	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	⚪	🟢	🟢
go-v0.17	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	⚪	🟢
go-v0.11	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	🟢	⚪

[laurentsenta]

@mxinden @marten-seemann @julian88110

I got a working proof of concept that emulates new testground features & shows how we can use them in this repo.

"Laurent is writing bash scripts again"

• Not so fast, It's typescript! :)
• I believe the hard problem here is making interop matrix maintainable: I'd like to validate usage with you and make sure it checks all your requirements,
• Once we validate the solution, I'll take time to do a clean implementation in Testground, thoroughly tested. In the meantime, you'll have enough to start writing test generators. This may help prepare tables in EPIC: support multi-dimensional testing #53.

This is the branch: https://github.com/libp2p/test-plans/tree/feat/interop-dashboard/composer

I'd like to have a sync to discuss:

• The new composition format proposed,
• How we'll refactor the interop tests & whether this covers everything in EPIC: support multi-dimensional testing #53,
• Which features will be testground vs which features will stay in libp2p/test-plans.

[BigLep]

I'm not close to the details here.

The other thing I'd love to get visibility into is what "dimensions" are covered when we say "rust-v0.47.0" is green with "go-v0.11". It's a lot of information but I think we want a way to expose/show the transports/muxers/etc. that passed, failed, or were skipped (because they don't make sense).

Idea 1:
Maybe one way to do this is to add a lot more of the dimensions into the keys:

"rust-v0.47.0 tcp tls yamux" is green with "go-v0.11 tcp tls yamux"
"rust-v0.47.0 tcp tls yamux" is skipped with "go-v0.11 tcp tls mplex" because it doesn't make sense

Idea 2:
or maybe we have a matrix for each "transport / security / muxer" combination:

Something like a matrix for "tcp tls yamux"
and then we see all the implementation/version combinations and their status (like you have above)

Side thought:
I want to make sure the dashboard can render everything that should be tested that isn't yet. For example, if we don't have any nodejs tests, I'd ideally want that surfaced so it's very clear that there is a whole gap of coverage. Or if for example rust is the only implementation to have added "tcp tls yamux" interop tests, then it should be clear that go, js, etc. are behind.

[laurentsenta]

@BigLep I agree, there is a discussion and a proto-table shared in the EPIC related to multi-dimensional testing: #53 (comment). I linked your comment in the EPIC's description.

I clarified the description and title for this issue:

As a libp2p/test-plans maintainer, I want the tooling required to define interop test matrixes and retrieve data in a simple form (a flat table of interop results).

[laurentsenta]

Quick update,
With the feature now in Testground (testground/testground#1516) I am testing it out in CI with a simple dashboard output.

This outputs the generated markdown table as a test summary:
https://github.com/libp2p/test-plans/actions/runs/3533136038/attempts/1#summary-9668033712

[BigLep]

Awesome @laurentsenta - thanks.

Looking at https://github.com/libp2p/test-plans/actions/runs/3533136038/attempts/1#summary-9668033712 specifically, should I be able to click a green dot and learn more as to what ran to make it green?

It's totally fine if this is a work in progress, but my hope is one can drill down to get details. This is really important for the failure case and to get a better understanding of "what was asserted for a given cell in the table".

[tinytb]

@laurentsenta -- to comment on next steps re: #75

[laurentsenta]

We had a discussion async with @jxs and @MarcoPolo

This is the plan:

• We use the result.csv file as a coordination point; it'll assume"A x B" as a test id means libp2p A dialing to libp2p B.
• Using this, the libp2p team can work on composer: add initial version #75 and follow up at their rhythm, and we can complete the work in this ticket.

Next steps for this issue:

• Generate a reference results.csv that might be used by the team,
• Implement a simple script to generate the markdown (simplify the work in the composer branch)
• Generate that table during CI and use it as run output
• Generate a link to the README.md nightly (@galargh shared https://github.com/libp2p/test-plans/pull/86/files which solves this)

[laurentsenta]

Code and Dashboard ready in PR #88

@BigLep I made sure to take into account your comment. We don't have the final composer ready yet, this should be a useful base we can improve on when we move forward with #53 #62, and #75.

• We render a dashboard directly in the github job summary so the reason why a cell is red lives right beside the dashboard.
• Each cell in the markdown links to the job itself, which means we can embed this table somewhere else. When we implement a more advanced dashboard, we can add granularity to this linking.
• We link to that dashboard from the README, using a badge:

(failing on purpose for the sake of demo)

[BigLep]

@MarcoPolo : what is remaining here?

[MarcoPolo]

The only thing not done here is an easy permalink to the latest results (a small change that I just hadn’t prioritized). Everything else is done.

I’ll close this issue with the pr that runs these tests on pushes to master, then we can use that to generate a permalink for the latest results.

[galargh]

The only thing not done here is an easy permalink to the latest results (a small change that I just hadn’t prioritized). Everything else is done.

You could reuse parts of
#88 for this.