feat(frontend): output explain result as graphviz dot format

[lyang24]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

add graphviz support for explain

Please do not leave this empty!
Add dot format output for explain, the output could be visualized as

Please explain IN DETAIL what the changes are in this PR and why they are needed:

• Summarize your change (mandatory)
• How does this PR work? Need a brief introduction for the changed logic (optional)
• Describe clearly one logical change and avoid lazy messages (optional)
• Describe any limitations of the current code (optional)
• Refer to a related PR or issue link (optional)
  close feat: explain a query and output a Graphviz representation of the query plan #18803

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Release note
Public Preview: Do note that the structure and format of EXPLAIN FORMAT=DOT is considered unstable, and may change overtime.

After this PR, one new explain option is supported
FORMAT DOT. it will output dot format for logical and physical plans.

examples:

    CREATE TABLE t (v1 int);
    explain (logical, format dot) SELECT approx_percentile(0.5) WITHIN GROUP (order by v1) from t;

result

    digraph {
        0 [ label = "LogicalAgg\laggs: [\"approx_percentile($expr1)\"]\l" ]
        1 [ label = "LogicalProject\lexprs: [\"t.v1::Float64 as $expr1\"]\l" ]
        2 [ label = "LogicalScan\ltable: \"t\"\lcolumns: [\"v1\"]\l" ]
        0 -> 1 [ ]
        1 -> 2 [ ]
    }

[kwannoel]

We should not add graphviz as an option, it should be an ExplainFormat.
Further the format is DOT, graphviz is not a format.
So it should be explain (logical, format dot).

[kwannoel]

Thanks for your work in this PR.

The output spawns new graph nodes for properties of plan nodes. I think it can be improved, these properties should be a part of their plan nodes' graph node.

Perhaps it's hard to implement with the petgraph::Graph interface. You may do some further investigation to see if petgraph::Graph can support it, or other more suitable approaches. I'll take a look into it as well.

[lyang24]

Thanks for your work in this PR.

The output spawns new graph nodes for properties of plan nodes. I think it can be improved, these properties should be a part of their plan nodes' graph node.

Perhaps it's hard to implement with the petgraph::Graph interface. You may do some further investigation to see if petgraph::Graph can support it, or other more suitable approaches. I'll take a look into it as well.

Thank you I have updated the pr. I had this approach to serialize every graph node as label, very simple but very inefficient. let me know if you found better options.

[kwannoel]

Hmm still looks a little off from the provided screenshot. Let me provide an example to make it clearer

Consider the following query plan:

    LogicalProject { exprs: [min(t.v1), sum($expr1)] }
    └─LogicalAgg { group_key: [$expr1], aggs: [min(t.v1), sum($expr1)] }
      └─LogicalProject { exprs: [(t.v1 + t.v2) as $expr1, t.v1] }
        └─LogicalScan { table: t, columns: [t.v1, t.v2, t._row_id] }

We'd expect the output to be something like this:

digraph G {
    1 [label="LogicalProject\nexprs: [min(t.v1), sum($expr1)]"];
    2 [label="LogicalAgg\ngroup_key: [$expr1]\naggs: [min(t.v1), sum($expr1)]"];
    3 [label="LogicalProject\nexprs: [(t.v1 + t.v2) as $expr1, t.v1]"];
    4 [label="LogicalScan\ntable: t\ncolumns: [t.v1, t.v2, t._row_id]"];

    1 -> 2;
    2 -> 3;
    3 -> 4;
}

A "plan node" is something like Logical___ or Stream___ or Batch__.
"properties" are things like exprs, stream_key, aggs, all these shouldn't have their own plan node.

[kwannoel]

Typically, our plan-nodes are structured with childless_record:

risingwave/src/frontend/src/optimizer/plan_node/logical_now.rs

Line 59 in 5b9cf24

childless_record("LogicalNow", vec)

That's just a XmlNode:

risingwave/src/frontend/src/optimizer/plan_node/utils.rs

Line 219 in 5b9cf24

pub(super) fn childless_record<'a>(

Next, if we look at the explain step:

risingwave/src/frontend/src/optimizer/plan_node/mod.rs

Lines 660 to 666 in 5b9cf24

	fn explain<'a>(&self) -> Pretty<'a> {
	let mut node = self.distill();
	let inputs = self.inputs();
	for input in inputs.iter().peekable() {
	node.children.push(input.explain());
	}
	Pretty::Record(node)

We can see that we construct a Pretty::Record for this node.

So we can just assign graph nodes to Pretty::Record, and avoid assigning it to other Pretty variants.

Next, an XmlNode has the following structure:

#[derive(Clone)]
pub struct XmlNode<'a> {
    pub name: Str<'a>,
    pub fields: CowAssocArr<'a>,
    /// Currently, if fields have `XmlNode` with children,
    /// they will not be considered during linearization.
    pub(crate) fields_is_linear: bool,
    pub children: Vec<Pretty<'a>>,
}

We can append the fields to the Graph Node's label.
Children will just have an edge from this Graph Node, and have their own GraphNode, if they are Pretty::Record.

[kwannoel]

Feel free to let me know if you encounter any difficulties.

[lyang24]

Typically, our plan-nodes are structured with childless_record:

risingwave/src/frontend/src/optimizer/plan_node/logical_now.rs

Line 59 in 5b9cf24

childless_record("LogicalNow", vec)

That's just a XmlNode:

risingwave/src/frontend/src/optimizer/plan_node/utils.rs

Line 219 in 5b9cf24

pub(super) fn childless_record<'a>(

Next, if we look at the explain step:

risingwave/src/frontend/src/optimizer/plan_node/mod.rs

Lines 660 to 666 in 5b9cf24

	fn explain<'a>(&self) -> Pretty<'a> {
	let mut node = self.distill();
	let inputs = self.inputs();
	for input in inputs.iter().peekable() {
	node.children.push(input.explain());
	}
	Pretty::Record(node)

We can see that we construct a Pretty::Record for this node.
So we can just assign graph nodes to Pretty::Record, and avoid assigning it to other Pretty variants.

Next, an XmlNode has the following structure:

#[derive(Clone)]
pub struct XmlNode<'a> {
    pub name: Str<'a>,
    pub fields: CowAssocArr<'a>,
    /// Currently, if fields have `XmlNode` with children,
    /// they will not be considered during linearization.
    pub(crate) fields_is_linear: bool,
    pub children: Vec<Pretty<'a>>,
}

We can append the fields to the Graph Node's label. Children will just have an edge from this Graph Node, and have their own GraphNode, if they are Pretty::Record.

Thank you for the pointers I learned a lot, the code is actually simpler and the results are much nicer as well.

[lyang24]

The alternative is to add each field to a new line, this is much closer to the desired pictures provided

        for (k, v) in r.fields.iter() {
            label.push_str("\n");
            label.push_str(&k);
            label.push_str(": ");
            label.push_str(&serde_json::to_string(&PrettySerde(v.clone())).expect("failed to serialize plan to dot"));
        }

However, the visualized output does not look great in terms of alignment, not sure if this is an dot output to png issue, or a rust string fmt issue.

[kwannoel]

see: https://github.com/risingwavelabs/risingwave/pull/19446/files#r1849893022

[kwannoel]

Could we add more test cases? Like in #19430.

Some further improvement would be to test large labels (stream_key has 200 columns for instance), and improve its DOT output, for instance doing truncation.
This can be tracked and worked on in a separate issue and PR. We have a good starting point from this PR.

Rest LGTM. Thanks for your efforts!

[lyang24]

Could we add more test cases? Like in #19430.

Some further improvement would be to test large labels (stream_key has 200 columns for instance), and improve its DOT output, for instance doing truncation. This can be tracked and worked on in a separate issue and PR. We have a good starting point from this PR.

Rest LGTM. Thanks for your efforts!

Yep added the very long test case but i realize the output is not readable after convert it into graph because the serialization of the (nested) children nodes.
so I added 1 more optimization to skip serialization of chidren on dot output because directed graph can captures that with edges.

[kwannoel]

In the large testcase that was just added, here's the original explain in text format:

risingwave/src/frontend/planner_test/tests/testdata/output/subquery_expr_correlated.yaml

Line 1308 in ad1d929

stream_plan: |-

In our DOT output, the StreamHashJoin nodes are treated as identical, since their labels are the same. However, they are actually distinct plan nodes.
We can use plan node id to distinguish them.

You can git cherry-pick this commit, it adds the id to the explain output: 5f29e9c. (main changes here: 5f29e9c#r149336748)

Rendered:

[lyang24]

In our DOT output, the StreamHashJoin nodes are treated as identical, since their labels are the same. However, they are actually distinct plan nodes.
We can use plan node id to distinguish them.

Great catch, thanks for the commit. git cherry-pick 5f29e9c do not work out on my laptop I included the change manually in another commit. Thanks again.

[kwannoel]

LGTM, thanks!

[kwannoel]

If you're interested there's a further extension of this task: #19542.

We wish to use it in our kernel dashboard, to visualize distributed stream job fragments.