Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add etcd test analyzer build and integrate into measure-test-flakiness workflow #15513

Merged
merged 1 commit into from
Mar 29, 2023

Conversation

chaochn47
Copy link
Member

@chaochn47 chaochn47 commented Mar 20, 2023

The PR is to address the ask from #13167 (comment). @serathius

The workflow stays running every Sunday midnight.

Measure-test-flakiness workflow output will additionally show the top contributor of the failed Tests name in Test/Test-arm workflow (only covers integration tests right now) in the past 1 week.

Example output https://github.com/chaochn47/etcd/actions/runs/4546435089/jobs/8015061879


Running against E2E workflows does not produce meaningful output, need to investigate more in the tool

dev-dsk-chaochn-2c-a26acd76 % bin/etcd-test-analyzer run -token $GITHUB_TOKEN -max-age=168h -workflow E2E -branch main
Fetching workflow runs from GitHub
Analyzing workflow "E2E"
    [E2E]: Finding workflow runs
    [E2E]: Downloading artifacts
    [E2E]: Downloading artifacts for E2E to "/tmp/test-results-2184436320"
    ...
Runs: 0, Pass: 0, Fail: 0, Pcnt: NaN
Analyzing workflow "E2E-arm64"
    [E2E-arm64]: Finding workflow runs
    [E2E-arm64]: Downloading artifacts
    [E2E-arm64]: Downloading artifacts for E2E-arm64 to "/tmp/test-results-2423396168"
    [E2E-arm64]: no artifacts found
    ...

Maybe @endocrimes can help shed lights on the issue.

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

@chaochn47 chaochn47 force-pushed the add_etcd_test_analyzer branch from 7b1069a to c1c850f Compare March 20, 2023 06:43
Makefile Outdated Show resolved Hide resolved
@serathius
Copy link
Member

Running against E2E workflows does not produce meaningful output, need to investigate more in the tool

E2e tests don't upload Junit reports. #13152
Definetly something worth doing now, with the new tool.

@chaochn47
Copy link
Member Author

chaochn47 commented Mar 20, 2023

E2e tests don't upload Junit reports. #13152

Appreciate the insights!! I can take a look at this.

@chaochn47 chaochn47 force-pushed the add_etcd_test_analyzer branch 2 times, most recently from 055cbdf to c9dad59 Compare March 20, 2023 23:48
scripts/build_lib.sh Outdated Show resolved Hide resolved
scripts/build_lib.sh Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
@chaochn47 chaochn47 force-pushed the add_etcd_test_analyzer branch 2 times, most recently from 1d60bb6 to b8bbd9e Compare March 28, 2023 03:58
Makefile Show resolved Hide resolved
Copy link
Member

@fuweid fuweid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM(non-binding)

Thanks

Just share some ideas about makefile~

Makefile Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
@chaochn47 chaochn47 force-pushed the add_etcd_test_analyzer branch from b8bbd9e to a8c20f9 Compare March 28, 2023 18:30
@chaochn47
Copy link
Member Author

chaochn47 commented Mar 28, 2023

Addressed all the comments.

The example output in run on push workflow. https://github.com/chaochn47/etcd/actions/runs/4546435089/jobs/8015061879. The second one is tracked in #15545

Runs: 15, Pass: 13, Fail: 2, Pcnt: 0.866667
│───────────────────────────────────│────────────────────────────────────────────────────────│───────────────│─────────────────────────────────────────────────────────│
│ TEST NAME                         │ PACKAGE                                                │ FAILURE COUNT │ RECENT FAILURE                                          │
│───────────────────────────────────│────────────────────────────────────────────────────────│───────────────│─────────────────────────────────────────────────────────│
│ ExampleCluster_memberAddAsLearner │ go.etcd.io/etcd/tests/v3/integration/clientv3/examples │             1 │ https://github.com/etcd-io/etcd/actions/runs/4495063935 │
│ TestV3WatchRestoreSnapshotUnsync  │ go.etcd.io/etcd/tests/v3/integration                   │             1 │ https://github.com/etcd-io/etcd/actions/runs/4516121595 │
│───────────────────────────────────│────────────────────────────────────────────────────────│───────────────│─────────────────────────────────────────────────────────│

Copy link
Member

@ahrtr ahrtr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thank you @chaochn47

@ahrtr ahrtr merged commit 8aeef56 into etcd-io:main Mar 29, 2023
@chaochn47 chaochn47 deleted the add_etcd_test_analyzer branch March 29, 2023 00:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants