Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] add sv cbo_max_reorder_node_use_greedy and skip DP and greedy reorder without unknown stats (backport #54593) (backport #54605) #54612

Merged
merged 1 commit into from
Jan 2, 2025

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Jan 2, 2025

Why I'm doing:

Add Session Variable cbo_max_reorder_node_use_greedy

When there are approximately 20 join nodes, the greedy join reorder can also consume significant time. For instance, in TPC-DS Q64, JoinReorderGreedy takes 600ms. Additionally, cbo_push_down_aggregate_on_broadcast_join triggers an extra join reorder during the RBO stage, increasing query latency by 600ms.

Therefore, introduce a session variable cbo_max_reorder_node_use_greedy (16 by default) .

skip DP and greedy reorder without unknown stats for cbo_push_down_aggregate_on_broadcast_join

The ReorderJoinRule::rewrite method lacks a critical piece of logic compared to ReorderJoinRule::transform: skipping DP and greedy reorder when certain column statistics are unknown. This omission causes hive_tpcds.query17.sql to regress.

What I'm doing:

Fixes https://github.com/StarRocks/StarRocksTest/issues/8933

Test

Environment

  • 4 BE, each 16 vCPUs and 64GB Memory.
  • TPC-DS 1T

There is not query degression, and Q64 from 9911ms to 6188ms.

  Baseline PR Baseline/PR
  452674 446008 1.01
QUERY01 846 846 1.00
QUERY02 1291 1321 0.98
QUERY03 1680 1693 0.99
QUERY04 19946 20027 1.00
QUERY05 792 772 1.03
QUERY06 240 266 0.90
QUERY07 1590 1565 1.02
QUERY08 352 433 0.81
QUERY09 10990 10977 1.00
QUERY10 439 439 1.00
QUERY11 11494 11781 0.98
QUERY12 126 132 0.95
QUERY13 975 1007 0.97
QUERY14-1 10994 10906 1.01
QUERY14-2 10263 10194 1.01
QUERY15 1272 1295 0.98
QUERY16 886 885 1.00
QUERY17 1432 1449 0.99
QUERY18 1332 1366 0.98
QUERY19 345 352 0.98
QUERY20 141 142 0.99
QUERY21 120 125 0.96
QUERY22 2873 2998 0.96
QUERY23-1 67854 66400 1.02
QUERY23-2 66668 65186 1.02
QUERY24-1 4204 4230 0.99
QUERY24-2 4207 4220 1.00
QUERY25 1271 1209 1.05
QUERY26 821 825 1.00
QUERY27 1101 1079 1.02
QUERY28 10806 10892 0.99
QUERY29 2197 2205 1.00
QUERY30 440 455 0.97
QUERY31 3117 3163 0.99
QUERY32 170 173 0.98
QUERY33 594 648 0.92
QUERY34 1015 1042 0.97
QUERY35 2102 2081 1.01
QUERY36 1007 1017 0.99
QUERY37 548 553 0.99
QUERY38 6481 6286 1.03
QUERY39-1 497 511 0.97
QUERY39-2 298 298 1.00
QUERY40 200 217 0.92
QUERY41 119 118 1.01
QUERY42 141 148 0.95
QUERY43 708 705 1.00
QUERY44 3373 3388 1.00
QUERY45 633 639 0.99
QUERY46 1899 1872 1.01
QUERY47 4828 4876 0.99
QUERY48 825 850 0.97
QUERY49 907 904 1.00
QUERY50 4885 4890 1.00
QUERY51 6878 6942 0.99
QUERY52 145 147 0.99
QUERY53 1050 1055 1.00
QUERY54 729 788 0.93
QUERY55 139 147 0.95
QUERY56 350 409 0.86
QUERY57 3592 3577 1.00
QUERY58 341 369 0.92
QUERY59 4842 4817 1.01
QUERY60 777 801 0.97
QUERY61 521 510 1.02
QUERY62 1074 1078 1.00
QUERY63 1062 1050 1.01
QUERY64 9911 6188 1.60
QUERY65 8223 8304 0.99
QUERY66 951 961 0.99
QUERY67 49266 48772 1.01
QUERY68 690 680 1.01
QUERY69 443 417 1.06
QUERY70 4447 4412 1.01
QUERY71 2028 2193 0.92
QUERY72 2843 2795 1.02
QUERY73 451 467 0.97
QUERY74 10406 10033 1.04
QUERY75 11434 11394 1.00
QUERY76 2277 2271 1.00
QUERY77 449 406 1.11
QUERY78      
QUERY79 2837 2802 1.01
QUERY80 1061 1058 1.00
QUERY81 710 718 0.99
QUERY82 1102 1083 1.02
QUERY83 260 256 1.02
QUERY84 283 286 0.99
QUERY85 849 855 0.99
QUERY86 1165 1155 1.01
QUERY87 6250 6154 1.02
QUERY88 15462 15512 1.00
QUERY89 1148 1148 1.00
QUERY90 1012 1052 0.96
QUERY91 137 134 1.02
QUERY92 119 121 0.98
QUERY93 3882 3850 1.01
QUERY94 1193 1222 0.98
QUERY95 2647 2640 1.00
QUERY96 2209 2215 1.00
QUERY97 7697 7611 1.01
QUERY98 653 678 0.96
QUERY99 2344 2424 0.97

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

This is an automatic backport of pull request #54605 done by [Mergify](https://mergify.com).

…greedy reorder without unknown stats (backport #54593) (#54605)

Co-authored-by: zihe.liu <[email protected]>
(cherry picked from commit 2b8c522)
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@wanpengfei-git wanpengfei-git enabled auto-merge (squash) January 2, 2025 14:08
Copy link

sonarqubecloud bot commented Jan 2, 2025

@andyziye andyziye disabled auto-merge January 2, 2025 14:57
@andyziye andyziye merged commit ce41469 into branch-3.4.0-rc01 Jan 2, 2025
33 of 36 checks passed
@andyziye andyziye deleted the mergify/bp/branch-3.4.0-rc01/pr-54605 branch January 2, 2025 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants