dbt-spark v1.5.0b1

dbt-spark 1.5.0-b1 - February 22, 2023

Features

• Support for data types constraints in Spark following the dbt Core feature #6271 (#558)

Fixes

• add merge_exclude_columns tests (#00)

Under the Hood

• remove tox call to integration tests (#584)

Dependencies

• Allow thrift 0.16.0 (#605)

Contributors

• @b-per (#558)
• @dave-connors-3 (#00)

dbt-spark v1.4.1

dbt-spark Changelog

• This file provides a full account of all changes to dbt-spark.
• Changes are listed under the (pre)release in which they first appear. Subsequent releases include changes from previous releases.
• "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
• Do not edit this file directly. This file is auto-generated using changie. For details on how to document a change, see the contributing guide

dbt-spark 1.4.1 - January 27, 2023

Dependencies

• Allow thrift 0.16.0 (#606, #605)

dbt-spark 1.4.0 - January 25, 2023

Features

• incremental predicates (#435, #436)
• Migrate dbt-utils current_timestamp macros into core + adapters (#483, #480)

Fixes

• Password doesn't pass to server using LDAP connection via thrift (#310) (#310, #396)
• Updated the Spark-testing profile name to spark_testing (#514, #513)
• Support new error messages in the future Spark. (#515, #520)
• Update policy code and test config to support py3.11 and tox 4.0 (#542, #543)
• Added escape_single_quotes macro as Spark/Databricks uses a single backslash instead of a single quote (#572, #573)

Under the Hood

• updating python version in tox (#536, #534)
• Rename exceptions to match dbt-core. (#557, #585)

Contributors

• @VShkaberda (#396)
• @akashrn5 (#513)
• @dave-connors-3 (#436)
• @graciegoheen (#573)
• @ueshin (#520)

Previous Releases

For information on prior major and minor releases, see their changelogs:

• 1.3
• 1.2
• 1.1
• 1.0
• 0.21
• 0.20
• 0.19 and earlier

dbt-spark v1.4.0

dbt-spark Changelog

• This file provides a full account of all changes to dbt-spark.
• Changes are listed under the (pre)release in which they first appear. Subsequent releases include changes from previous releases.
• "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
• Do not edit this file directly. This file is auto-generated using changie. For details on how to document a change, see the contributing guide

dbt-spark 1.4.0 - January 25, 2023

Features

• incremental predicates (#435, #436)
• Migrate dbt-utils current_timestamp macros into core + adapters (#483, #480)

Fixes

• Password doesn't pass to server using LDAP connection via thrift (#310) (#310, #396)
• Updated the Spark-testing profile name to spark_testing (#514, #513)
• Support new error messages in the future Spark. (#515, #520)
• Update policy code and test config to support py3.11 and tox 4.0 (#542, #543)
• Added escape_single_quotes macro as Spark/Databricks uses a single backslash instead of a single quote (#572, #573)

Under the Hood

• updating python version in tox (#536, #534)
• Rename exceptions to match dbt-core. (#557, #585)

Contributors

• @VShkaberda (#396)
• @akashrn5 (#513)
• @dave-connors-3 (#436)
• @graciegoheen (#573)
• @ueshin (#520)

Previous Releases

For information on prior major and minor releases, see their changelogs:

• 1.3
• 1.2
• 1.1
• 1.0
• 0.21
• 0.20
• 0.19 and earlier

dbt-spark v1.3.1

dbt-spark Changelog

• This file provides a full account of all changes to dbt-spark.
• Changes are listed under the (pre)release in which they first appear. Subsequent releases include changes from previous releases.
• "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
• Do not edit this file directly. This file is auto-generated using changie. For details on how to document a change, see the contributing guide

dbt-spark 1.3.1 - January 20, 2023

Fixes

• Support new error messages in the future Spark. (#515, #520)

Under the Hood

• updating python version in tox (#536, #534)
• fix post new release tox issues around passenv and allowlist_externals (#544, #544)

Contributors

• @ueshin (#520)

dbt-spark 1.3.0 - October 12, 2022

Features

• Support python model through notebook, currently supported materializations are table and incremental (#417, #377)
• Add changie to dbt-spark (#416, #418)
• merge exclude columns for spark models (#5260, #390)
• Array macros (#453, #454)
• implement testing for type_boolean in spark (#470, #471)
• Support job cluster in notebook submission method, remove requirement for user for python model submission (#444, #467)
• Migrate dbt-utils current_timestamp macros into core + adapters (#483, #480)

Fixes

• Pin pyodbc to version 4.0.32 to prevent overwriting libodbc.so and libltdl.so on Linux (#397, #398)
• python incremental model tmp table using correct schema (#441, #445)
• change to get_columns_in_relation to fix cache inconsistencies to fix cache issues in incremental models causing failure on on_schema_change (#447, #451)

Under the Hood

• Support core incremental refactor (#4402, #394)
• specify supported_languages for materialization that support python models (#437, #440)
• Submit python model with Command API by default. Adjusted run name (#424, #442)
• Better interface for python submission (#452, #452)
• ignore mypy typing issues (#461, #462)
• Enable Pandas and Pandas-on-Spark DataFrames for dbt python models (#468, #469)
• Convert df to pyspark DataFrame if it is koalas before writing (#473, #474)

Dependency

• Bump pyodbc from 4.0.32 to 4.0.34 (#417, #459)
• Bump black from 22.3.0 to 22.8.0 (#417, #458)
• Update click requirement from ~=8.0.4 to ~=8.1.3 (#417, #457)
• Bump mypy from 0.950 to 0.971 (#417, #456)
• Bump thrift-sasl from 0.4.1 to 0.4.3 (#417, #455)

Contributors

• @barberscot (#398)
• @chamini2 (#469)
• @colin-rogers-dbt (#480, #462)
• @dave-connors-3 (#390)
• @dbeatty10 (#454, #469, #474)
• @graciegoheen (#454)
• @jpmmcneill (#471)
• @ueshin (#474)

Previous Releases

For information on prior major and minor releases, see their changelogs:

• 1.2
• 1.1
• 1.0
• 0.21
• 0.20
• 0.19 and earlier

dbt-spark v1.4.0rc1

dbt-spark Changelog

• This file provides a full account of all changes to dbt-spark.
• Changes are listed under the (pre)release in which they first appear. Subsequent releases include changes from previous releases.
• "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
• Do not edit this file directly. This file is auto-generated using changie. For details on how to document a change, see the contributing guide

dbt-spark 1.4.0-rc1 - January 11, 2023

Fixes

• Added escape_single_quotes macro as Spark/Databricks uses a single backslash instead of a single quote (#572, #573)

Under the Hood

• Rename exceptions to match dbt-core. (#557, #585)

Contributors

• @graciegoheen (#573)

dbt-spark 1.4.0-b1 - December 15, 2022

Features

• incremental predicates (#435, #436)
• Migrate dbt-utils current_timestamp macros into core + adapters (#483, #480)

Fixes

• Password doesn't pass to server using LDAP connection via thrift (#310) (#310, #396)
• Updated the Spark-testing profile name to spark_testing (#514, #513)
• Support new error messages in the future Spark. (#515, #520)
• Update policy code and test config to support py3.11 and tox 4.0 (#542, #543)

Under the Hood

• updating python version in tox (#536, #534)

Contributors

• @VShkaberda (#396)
• @akashrn5 (#513)
• @dave-connors-3 (#436)
• @ueshin (#520)

Previous Releases

For information on prior major and minor releases, see their changelogs:

• 1.3
• 1.2
• 1.1
• 1.0
• 0.21
• 0.20
• 0.19 and earlier

dbt-spark v1.1.1

dbt-spark Changelog

• This file provides a full account of all changes to dbt-spark.
• Changes are listed under the (pre)release in which they first appear. Subsequent releases include changes from previous releases.
• "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
• Do not edit this file directly. This file is auto-generated using changie. For details on how to document a change, see the contributing guide

dbt-spark 1.1.1 - December 20, 2022

Features

• backport changie to 1.1.latest (#417, #425)

Fixes

• Support new error messages in the future Spark. (#515, #520)

Under the Hood

• updating python version in tox (#536, #534)
• fix post new release tox issues around passenv and allowlist_externals (#546, #546)

Contributors

• @ueshin (#520)

dbt-spark 1.1.0 - April 28, 2022

Features

• Add session connection method (#272, #279)
• Adds new integration test to check against new ability to allow unique_key to be a list. (#282), #291)

Fixes

• Closes the connection properly (#280, #285)

Under the hood

• Use dbt.tests.adapter.basic in test suite (#298, #299)
• Make internal macros use macro dispatch to be overridable in child adapters (#319, #320)
• Override adapter method 'run_sql_for_tests' (#323, #324)
• when a table or view doesn't exist, 'adapter.get_columns_in_relation' will return empty list instead of fail ([#328]#328)
• get_response -> AdapterResponse (#265)
• Adding stale Actions workflow (#275)
• Update plugin author name (fishtown-analytics → dbt-labs) in ODBC user agent (#288)
• Configure insert_overwrite models to use parquet (#301)

Contributors

• @JCZuurmond ( #279)
• @ueshin (#320)
• @amychen1776 (#288)
• @ueshin (#285)

Previous Releases

For information on prior major and minor releases, see their changelogs:

• 1.0
• 0.21
• 0.20
• 0.19 and earlier

dbt-spark v1.0.2

dbt-spark Changelog

• This file provides a full account of all changes to dbt-spark.
• Changes are listed under the (pre)release in which they first appear. Subsequent releases include changes from previous releases.
• "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
• Do not edit this file directly. This file is auto-generated using changie. For details on how to document a change, see the contributing guide

dbt-spark 1.0.2 - December 20, 2022

Features

• backport changie to 1.0.latest (#417, #426)

Under the Hood

• fix post new release tox issues around passenv and allowlist_externals (#547, #547)

dbt-spark 1.0.1 - April 19, 2022

Fixes

• Closes the connection properly (#280, #285)
• Make internal macros use macro dispatch to be overridable in child adapters (#319, #320)

Under the hood

• Configure insert_overwrite models to use parquet (#301)

Contributors

• @ueshin (#285, #320)

dbt-spark 1.0.0 - December 3, 2021

Features

• Add support for Apache Hudi (hudi file format) which supports incremental merge strategies (#187, #210)

Fixes

• Incremental materialization corrected to respect full_refresh config, by using should_full_refresh() macro (#260, #262)

Under the hood

• Refactor seed macros: remove duplicated code from dbt-core, and provide clearer logging of SQL parameters that differ by connection method (#249, #250)
• Replace sample_profiles.yml with profile_template.yml, for use with new dbt init (#247)
• Remove official support for python 3.6, which is reaching end of life on December 23, 2021 (dbt-core#4134, #253)
• Add support for structured logging (#251)

Contributors

• @grindheim (#262)
• @vingov (#210)

Previous Releases

For information on prior releases of dbt-spark prior to 1.0.0 please see

• 0.21
• 0.20
• 0.19 and earlier

dbt-spark v1.2.1

dbt-spark Changelog

• This file provides a full account of all changes to dbt-spark.
• Changes are listed under the (pre)release in which they first appear. Subsequent releases include changes from previous releases.
• "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
• Do not edit this file directly. This file is auto-generated using changie. For details on how to document a change, see the contributing guide

dbt-spark 1.2.1 - December 20, 2022

Features

• backport changie to 1.2.latest (#417, #418)

Fixes

• Support new error messages in the future Spark. (#515, #520)

Under the Hood

• updating python version in tox (#536, #534)
• fix post new release tox issues around passenv and allowlist_externals (#545, #545)

Contributors

• @ueshin (#520)

dbt-spark 1.2.0 - July 26, 2022

Features

• Add grants to materializations (#366, #381)

Fixes

• Pin pyodbc to version 4.0.32 to prevent overwriting libodbc.so and libltdl.so on Linux (#397, #398)
• Incremental materialization updated to not drop table first if full refresh for delta lake format, as it already runs create or replace table (#286, #287)
• Apache Spark version upgraded to 3.1.1 (#348, #349)
• adapter.get_columns_in_relation (method) and get_columns_in_relation (macro) now return identical responses. The previous behavior of get_columns_in_relation (macro) is now represented by a new macro, get_columns_in_relation_raw (#354, #355)

Under the hood

• Update SparkColumn.numeric_type to return decimal instead of numeric, since SparkSQL exclusively supports the former (#380)
• Initialize lift + shift for cross-db macros (#359)
• Add invocation env to user agent string (#367)
• Use dispatch pattern for get_columns_in_relation_raw macro (#365)

Contributors

• @barberscott (#398)
• @grindheim (#287)
• @nssalian (#349)
• @ueshin (#365)
• @dbeatty10 (#359)

Previous Releases

For information on prior major and minor releases, see their changelogs:

• 1.1
• 1.0
• 0.21
• 0.20
• 0.19 and earlier

dbt-spark v1.4.0b1

dbt-spark 1.4.0-b1 - December 15, 2022

Features

• incremental predicates (#435, #436)
• Migrate dbt-utils current_timestamp macros into core + adapters (#483, #480)

Fixes

• Password doesn't pass to server using LDAP connection via thrift (#310) (#310, #396)
• Updated the Spark-testing profile name to spark_testing (#514, #513)
• Support new error messages in the future Spark. (#515, #520)
• Update policy code and test config to support py3.11 and tox 4.0 (#542, #543)

Under the Hood

• updating python version in tox (#536, #534)

Contributors

• @VShkaberda (#396)
• @akashrn5 (#513)
• @dave-connors-3 (#436)
• @ueshin (#520)

dbt-spark 1.3.0

dbt-spark 1.3.0 - October 12, 2022

Features

• Support python model through notebook, currently supported materializations are table and incremental (#417, #377)
• Add changie to dbt-spark (#416, #418)
• merge exclude columns for spark models (#5260, #390)
• Array macros (#453, #454)
• implement testing for type_boolean in spark (#470, #471)
• Support job cluster in notebook submission method, remove requirement for user for python model submission (#444, #467)
• Migrate dbt-utils current_timestamp macros into core + adapters (#483, #480)

Fixes

• Pin pyodbc to version 4.0.32 to prevent overwriting libodbc.so and libltdl.so on Linux (#397, #398)
• python incremental model tmp table using correct schema (#441, #445)
• change to get_columns_in_relation to fix cache inconsistencies to fix cache issues in incremental models causing failure on on_schema_change (#447, #451)

Under the Hood

• Support core incremental refactor (#4402, #394)
• specify supported_languages for materialization that support python models (#437, #440)
• Submit python model with Command API by default. Adjusted run name (#424, #442)
• Better interface for python submission (#452, #452)
• ignore mypy typing issues (#461, #462)
• Enable Pandas and Pandas-on-Spark DataFrames for dbt python models (#468, #469)
• Convert df to pyspark DataFrame if it is koalas before writing (#473, #474)

Dependency

• Bump pyodbc from 4.0.32 to 4.0.34 (#417, #459)
• Bump black from 22.3.0 to 22.8.0 (#417, #458)
• Update click requirement from ~=8.0.4 to ~=8.1.3 (#417, #457)
• Bump mypy from 0.950 to 0.971 (#417, #456)
• Bump thrift-sasl from 0.4.1 to 0.4.3 (#417, #455)

Contributors

• @barberscot (#398)
• @chamini2 (#469)
• @colin-rogers-dbt (#480, #462)
• @dave-connors-3 (#390)
• @dbeatty10 (#454, #469, #474)
• @graciegoheen (#454)
• @jpmmcneill (#471)
• @ueshin (#474)