0.12.0 goals

[joocer]

Release 0.11.0 implemented most of the new logical planner and implemented considerable part of the binder.

This work however, was mostly done to not run in normal operation, and isn't used by Opteryx to serve requests.

That should change in version 0.12.0, the old planner will be removed and the new planner used in its place.

There is already a proof of concept translation step that translates new plans to old plans to feed the query execution engine. This will likely remain for a few versions whilst the planner migration is complete.

The new planner will make it easier to implement new features and capabilities:

• A new function catalogue; to auto generate documentation, validate input parameters and improve error messages when functions are called incorrectly.
• Full predicate and projection pushdowns, these currently work by trial and error or only when there's a single table.
• Cost-based optimisations; part of the function catalogue includes cost estimates for function execution

0.12.0 is going to be complex enough without adding new features. So these will follow in later releases.

[joocer]

Given the current progress 0.12 will likely be another month away.

There is quite a lot of refactoring to the execution part of the engine to keep it functionally equivalent with the new planner. The bulk of these changes relate to the new schema approach.

The current approach was using metadata in pyarrow to describe the tables, mostly dynamically at execution time; the new approach does most of the schema handling at plan time and doesn't push the schemas around as/with payloads, they're baked into the execution steps by the binder.

[joocer]

A significant milestone was reached today - the regression tests now run again, they still have a significant level of failure but issues that prevented them from running have been resolved. Two connectors have been disabled to get to this milestone - those involving HadroDB (this engine is subject to change and so needed reworking of these tests anyway) and those involving MongoDB (it is expected this will be resolved by the release of 0.12.0).

[joocer]

With the regression tests now running (not passing) 0.12 can move from the heavy Dev mode back to a more controlled CI mode where the focus is back to getting tests to pass rather than the refactoring of code.

This looks like more frequent pushes to main with incremental improvements to test pass rates.

The primary focus is initially to restore core functionality (SQL92 and exemplar client uses) this will likely be followed by a brief period of performance tuning and extended functionality.

0.13 will likely focus on continuing to restore (sometimes reimagine) pre 0.12 functionality, bug fix and work on the V2 query optimizer. Work on the new execution engine is unlikely to be until 0.15 or later to help ensure bugs from the planner are addressed before work on another major component begins. Optimizer refactor is likely to be 0.14, although projection and predicate pushdowns before that.