Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Spark] Fix compile error on Spark master due to ParserInterface changes #3294

Merged
merged 1 commit into from
Jun 24, 2024

Conversation

zedtang
Copy link
Collaborator

@zedtang zedtang commented Jun 22, 2024

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

apache/spark#46665 introduced a new parseScript method in ParserInterface, which broke compilation against Spark master:

[error] /home/runner/work/delta/delta/spark/src/main/scala/io/delta/sql/parser/DeltaSqlParser.scala:75:7: class DeltaSqlParser needs to be abstract.
[error] Missing implementation for member of trait ParserInterface:
[error]   def parseScript(sqlScriptText: String): org.apache.spark.sql.catalyst.parser.CompoundBody = ???
[error] class DeltaSqlParser(val delegate: ParserInterface) extends ParserInterface {
[error]       ^
[warn] 100 warnings found
[error] one error found
[error] (spark / Compile / compileIncremental) Compilation failed

This PR fixes the issue by shimming the ParserInterface since parseScript is not available in Spark 3.5

How was this patch tested?

Existing DeltaSqlParserSuite

Does this PR introduce any user-facing changes?

No

@zedtang zedtang self-assigned this Jun 22, 2024
@vkorukanti
Copy link
Collaborator

@zedtang can you check the failure are related?

@zedtang
Copy link
Collaborator Author

zedtang commented Jun 24, 2024

@zedtang can you check the failure are related?

@vkorukanti I don't think it's related. I'm seeing the same error on my other PR's run.

### Executing cmd: pip3 install /home/runner/work/delta/delta/dist/delta_spark-3.3.0_SNAPSHOT-py3-none-any.whl
Traceback (most recent call last):
  File "/home/runner/work/delta/delta/python/run-tests.py", line 183, in <module>
    run_pypi_packaging_tests(root_dir)
  File "/home/runner/work/delta/delta/python/run-tests.py", line 156, in run_pypi_packaging_tests
    run_cmd(install_whl_cmd, stream_output=True)
  File "/home/runner/work/delta/delta/python/run-tests.py", line 84, in run_cmd
    raise Exception("Non-zero exitcode: %s" % (exit_code))
Exception: Non-zero exitcode: 1
##### Running SBT tests #####
Running command: ['/home/runner/work/delta/delta/build/sbt', 'clean', '++ 2.12.18', 'sparkGroup/test', '-v', '-J-XX:+UseG1GC', '-J-Xmx6G']
##### Running Python tests #####
Calling script %s /home/runner/work/delta/delta/python/run-tests.py
Running command: ['python3', '/home/runner/work/delta/delta/python/run-tests.py']
Traceback (most recent call last):
  File "run-tests.py", line 233, in <module>
    run_python_tests(root_dir)
  File "run-tests.py", line 86, in run_python_tests
    run_cmd(["python3", python_test_script], stream_output=True)
  File "run-tests.py", line 102, in run_cmd
    raise Exception("Non-zero exitcode: %s" % (exit_code))
Exception: Non-zero exitcode: 1
Error: Process completed with exit code 1.

@zedtang
Copy link
Collaborator Author

zedtang commented Jun 24, 2024

This made the failed Delta Spark Master Tests / test (2.13.13) job pass

@vkorukanti vkorukanti merged commit 956b950 into delta-io:master Jun 24, 2024
9 of 10 checks passed
@zedtang zedtang deleted the fix-spark-master branch June 24, 2024 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants