ci: opt-in steps with build.env rather than pr labels

[xiangjinwu]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Similar to #12870, but without the need to create a PR (e.g. #13457).

Example usage:
https://buildkite.com/risingwavelabs/pull-request#new
CI_STEPS=build,build-other,docslt,e2e-test

Notes about regular expression:
Buildkite conditionals (doc src) uses golang regexp package:

Metacharacters: *+?()|
The operator precedence, from weakest to strongest binding, is first alternation, then concatenation, and finally the repetition operators.

• (^|,) and (,|$$) are used to prevent suffix and prefix match respectively. For example misc-check shall not match /check/ and build-other shall not match /build/. Parentheses are required to evaluate alternation before concatenation.
• tests? is used to accept both singular test and plural tests. Parentheses are not required because repetition already evaluates before concatenation. Similarly we allow benchmarks? and backwards?, but no other typos are tolerated.
• /(^|,)(cpu-flamegraph|heap-flamegraph)(,|$$)/ is used to match either cpu-flamegraph or heap-flamegraph. This is the only example where user do not need to manually specify the dependency step flamegraph-env-build. Note the use of one more pair of parentheses to evaluate alternation before concatenation.

Tests runs:

• Without CI_STEPS, pipeline pull-request shall include the same set of steps before this PR. (link)
• Without CI_STEPS, pipeline main-cron shall include the same set of steps before this PR. (link)
• With CI_STEPS=, pipeline pull-request runs the selected steps.
  • build,build-other,docslt,e2e-test (link)
  • cpu-flamegraph (link)
  • regress-test,build,unit-tests,misc-check (link)
• With CI_STEPS=, pipeline main-cron runs the selected steps.
  • build,s3-source-test (link)

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

[kwannoel]

I think this is useful to use as part of a bisect tool for main-cron. That is long overdue. It will not be able to bisect before this PR. But any commits after can be bisected with the help of this.

I will create a bisect pipeline which can be triggered adhoc.
Then using the functionality introduced from this PR, we can run specified tests via buildkite between specified commits.

It should not be automatically triggered, in case many tests fail due to some known reason.

[kwannoel]

+1 for trailing / leading comma to simplify regex.

[lmatz]

LGTM

[xiangjinwu]

content updated to cover most steps in pull-request and main-cron

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (4833477) 68.47% compared to head (cd09932) 68.41%.
Report is 8 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13464      +/-   ##
==========================================
- Coverage   68.47%   68.41%   -0.07%     
==========================================
  Files        1505     1505              
  Lines      259155   259155              
==========================================
- Hits       177458   177289     -169     
- Misses      81697    81866     +169     

Flag	Coverage Δ
rust	68.41% <ø> (-0.07%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kwannoel]

Rest LGTM, thanks for this!