Support Java8 time objects

[yhosny]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

Added support for Java8 time objects when Java8Api is enabled.

The following query fails when Java8Api is enabled:

INSERT INTO $tableName REPLACE
       where (DATE IN (DATE('2024-03-11'), DATE('2024-03-12')))
       VALUES ('1', DATE('2024-03-13'))

How was this patch tested?

Unit Test.

Does this PR introduce any user-facing changes?

The potential public surface area for this change is limited to only external DateType, TimestampType. By default, Spark converts DateType values to java.sql.Date, TimestampType -> java.sql.Timestamp but with the SQL conf enabled, Spark uses java.time.LocalDate as the external type for DateType and java.time.Instant for TimestampType.

Necessity of fix: If users use Spark 3.0 and enable Java 8, they will be unable to parse and use dates in Delta tables correctly, leading to matching errors.

[sandip-db]

test cannot have the same name

[tdas]

can you give me a reference for this in Spark code? I want to understand how this is tested in Spark.

[tdas]

I would like to see a bit more information about the Java8 APIs from the Spark stuff before expanding public API support.

I dont agree with your description that this does not change public API completely. Its quite possible that this SQL conf will be enable in Spark by default. That would implicitly allow new SQL operations in Delta that would have failed earlier. So I do want to understand the public API surface area before I approve this.

[yhosny]

I would like to see a bit more information about the Java8 APIs from the Spark stuff before expanding public API support.

I dont agree with your description that this does not change public API completely. Its quite possible that this SQL conf will be enable in Spark by default. That would implicitly allow new SQL operations in Delta that would have failed earlier. So I do want to understand the public API surface area before I approve this.

Summarizing offline conversation:

• The potential public-facing surface area of the fix: The potential public surface area for this change is limited to only one external DateType. By default, Spark converts DateType values to java.sql.Date, but with the SQL conf enabled, Spark uses java.time.LocalDate as the external type for DateType.
• The necessity of the fix: If users use Spark 3.0 and enable Java 8, they will be unable to parse and use dates in Delta tables correctly, leading to matching errors.
• The availability: This fix would be available from Spark 3.0 onwards.