chore: Upgrade to DataFusion 44.0.0

[andygrove]

Which issue does this PR close?

N/A

Rationale for this change

• Part of Test DataFusion 44.0.0 with Comet datafusion#13835

Start updating in preparation for DF 44.0.0

What changes are included in this PR?

• Manually implement Hash & PartialEq for many expressions (due to changes in Implement Eq, PartialEq, Hash for dyn PhysicalExpr datafusion#13005)
• Other minor changes

How are these changes tested?

Existing tests

[kazuyukitanimura]

Stop running the test suite with miri

Hmmm
Since this is not an official release yet, should we wait for the miri fix?

[andygrove]

Stop running the test suite with miri

Hmmm Since this is not an official release yet, should we wait for the miri fix?

Yes, I think we should. There is a PR to fix in DF: apache/datafusion#13778

[andygrove]

I have upgraded to latest DF and reinstated the miri checks. I'll keep this in draft until the RC is available later this week.

[alamb]

I have upgraded to latest DF and reinstated the miri checks. I'll keep this in draft until the RC is available later this week.

I see the MIRI CI check fails:

https://github.com/apache/datafusion-comet/actions/runs/12455101824/job/34767108969?pr=1154

I can reproduce the failure locally on this branch with this command (needs rust nightly)

MIRIFLAGS="-Zmiri-disable-isolation" cargo +nightly miri test -p datafusion-comet -- test_unpack_dictionary_primitive

I have an idea of what is wrong -- I am looking into this

[alamb]

I found the issue:

• [minor] make recursive package dependency optional  datafusion#13778 (comment)

Working on a fix