Test DataFusion 44.0.0 with Comet

[alamb]

Is your feature request related to a problem or challenge?

• Part of [DISCUSS] More extensive pre-release testing #13661

We are trying to make sure the upgrade experience for DataFusion is better and to do so we are hoping to improve the testsing of major downstream projects

For version 44.0.0 we would like to test with comet prior to release

Describe the solution you'd like

I would like to create a PR to test DataFusion comet with a pre-released version of DataFusion (pinned to the main branch)

Describe alternatives you've considered

No response

Additional context

@andygrove has done this in apache/datafusion-comet#1154

[alamb]

Testing looks good from what I can tell ✅

[andygrove]

Reopening ... I upgraded to latest and disabled default features, enabled our miri checks again but they are still failing.

[andygrove]

error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux`
    --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/psm-0.1.24/src/lib.rs:319:14
     |
319  |     unsafe { rust_psm_stack_pointer() }
     |              ^^^^^^^^^^^^^^^^^^^^^^^^ can't call foreign function `rust_psm_stack_pointer` on OS `linux`
     ```

[alamb]

I found another issue when testing today:

• swap_hash_join is no longer public so comet doesn't compile #13898

[alamb]

Looks like we are good to go now: apache/datafusion-comet#1154

I am claiming success (again) 🤞