-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Spark] Reject unsupported type changes with Uniform #3947
base: master
Are you sure you want to change the base?
Changes from 4 commits
6b2a0ee
ba65061
8cc235e
783af1b
51e83c7
3a48dc8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
/* | ||
* Copyright (2021) The Delta Lake Project Authors. | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package org.apache.spark.sql.delta.uniform | ||
|
||
import org.apache.spark.sql.delta.typewidening.TypeWideningUniformTests | ||
|
||
class TypeWideningUniformSuite extends TypeWideningUniformTests with WriteDeltaHMSReadIceberg |
Large diffs are not rendered by default.
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,50 @@ | ||||||
/* | ||||||
* Copyright (2021) The Delta Lake Project Authors. | ||||||
* | ||||||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||||||
* you may not use this file except in compliance with the License. | ||||||
* You may obtain a copy of the License at | ||||||
* | ||||||
* http://www.apache.org/licenses/LICENSE-2.0 | ||||||
* | ||||||
* Unless required by applicable law or agreed to in writing, software | ||||||
* distributed under the License is distributed on an "AS IS" BASIS, | ||||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||||||
* See the License for the specific language governing permissions and | ||||||
* limitations under the License. | ||||||
*/ | ||||||
package org.apache.spark.sql.delta | ||||||
|
||||||
import org.apache.spark.sql.types.AtomicType | ||||||
|
||||||
/** | ||||||
* A type widening mode captures a specific set of type changes that are allowed to be applied. | ||||||
* Currently: | ||||||
* - NoTypeWidening: No type change is allowed. | ||||||
* - TypeEvolution(uniformIcebergEnabled = true): Type changes that are eligible to be applied | ||||||
* automatically during schema evolution and that are supported by Iceberg are allowed. | ||||||
* - TypeEvolution(uniformIcebergEnabled = false): Type changes that are eligible to be applied | ||||||
* automatically during schema evolution are allowed, even if they are not supported by Iceberg. | ||||||
*/ | ||||||
trait TypeWideningMode { | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
I think it makes sense to enforce exhaustiveness here. There shouldn't be other places that need to specify customs rules, I think. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
||||||
def shouldWidenType(fromType: AtomicType, toType: AtomicType): Boolean | ||||||
} | ||||||
|
||||||
object TypeWideningMode { | ||||||
/** | ||||||
* No type change allowed. Typically because type widening and/or schema evolution isn't enabled. | ||||||
*/ | ||||||
object NoTypeWidening extends TypeWideningMode { | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Just so it looks like a nice enum-like thingy ;) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
||||||
override def shouldWidenType(fromType: AtomicType, toType: AtomicType): Boolean = false | ||||||
} | ||||||
|
||||||
/** | ||||||
* Type changes that are eligible to be applied automatically during schema evolution are allowed. | ||||||
* Can be restricted to only type changes supported by Iceberg. | ||||||
*/ | ||||||
case class TypeEvolution(uniformIcebergEnabled: Boolean) extends TypeWideningMode { | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why not call this There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agree, that fits better |
||||||
override def shouldWidenType(fromType: AtomicType, toType: AtomicType): Boolean = | ||||||
TypeWidening.isTypeChangeSupportedForSchemaEvolution( | ||||||
fromType = fromType, toType = toType, uniformIcebergEnabled) | ||||||
} | ||||||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
double version?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That got me at first too: version is used twice in the error message: once in the main error DELTA_ICEBERG_COMPAT_VIOLATION and once in the sub error class UNSUPPORTED_TYPE_WIDENING