Skip to content

Latest commit

 

History

History
303 lines (229 loc) · 9.69 KB

CHANGELOG.md

File metadata and controls

303 lines (229 loc) · 9.69 KB
Raw

Changes for Datasplash

0.7.25 - 2024-12-06

Added

  • without-validation option in read-bw-raw.

Changed

  • Bump beam to 2.61.0.

0.7.24 - 2024-10-30

Added

  • BQ: temp-dataset and temp-project options for reading.

Changed

  • Bump beam to 2.60.0.

0.7.23 - 2024-09-16

Changed

  • Bump beam to 2.59.0. Downgrade kafka-clients to match beam's version.

0.7.22 - 2024-07-01

Changed

  • Build compression-type map from Compression enum.
  • Bump beam to 2.57.0.

0.7.21 - 2024-05-07

Changed

  • Bump beam to 2.56.0
  • update of BQ schema spec types to now allow :json and :range
  • add support to set collation for STRING type

0.7.20 - 2024-03-16

Changed

  • Bump beam to 2.54.0
  • Bump charred to 1.034
  • Bump math.combinatronics
  • Bump clojure
  • Downgrade slf4j to 1.7.x as v2 is not compatible with beam > 2.53.0
  • Replace :dev profile logback dep with slj4j-simple

0.7.19 - 2023-12-15

  • Bump beam to 2.52.0
  • Add taoensso/encore required dependency for nippy.

v0.7.18 - 2023-09-29

  • Bump beam to 2.50.0
  • Bump charred to 1.033
  • Bump superstring to 3.2.0

v0.7.17 - 2023-04-07

  • add ds/distinct-by, ds/keep
  • normalize BQ schema
  • support for maxLength, precision, scale fields in BQ tables
  • new PAssert testing utilities
  • clean up aot files to reduce size
  • BQ specs for schema-validation
  • new exported clj-kondo config
  • bump charred to 1.028, beam sdk to 2.46.0

v0.7.16 - 2023-01-27

  • Bump charred to 1.019, beam to 2.44.0

v0.7.15 - 2022-12-15

  • Bump charred to 1.018

v0.7.14 - 2022-12-02

  • adapt charred options to avoid throwing on empty lines
  • Bump charred to 1.016, beam to 2.43.0

v0.7.13 - 2022-10-26

  • Add BQ raw read :query-location option.
  • Add support for Pcoll set operations.
  • Bump beam to 2.42.0
  • Bump charred to 1.0.14

v0.7.12 - 2022-09-16

  • handle collection for generating pipeline args
  • Bump beam to 2.41.0
  • Bump charred to 1.0.12

v0.7.11 - 2022-08-23

  • Add BQ clustering & optimized-write options in write-bq-table.
  • Bump beam to 2.40.0.
  • Bump charred to 1.0.11.
  • Bump nippy to 3.2.0.

v0.7.10 - 2022-06-24

  • Bump nippy to fix serializing of boolean values in map collections.
  • Bump Beam to 2.39.0.

v0.7.9 - 2022-05-24

  • Bump charred to fix more white-space issues. Enable options to match previous output.
  • Add write-json-file-test.

v0.7.8 - 2022-05-16

  • More complete tests, again.
  • Add ability to specify a teardown function for a DoFn
  • Invoke start-bundle, finish-bundle and teardown-fn when specified
  • Reuse no-op function in dofn
  • switch from cheshire to com.cnuernber/charred.

v0.7.7 - 2022-05-13

  • Enforce target jdk as 1.8
  • Bump clojure to 1.11.1
  • more complete tests

v0.7.6 - 2022-04-23

  • don't require bq, pubsub, datastore in api.clj - Issue #94
  • Bump beam to 2.38.0

v0.7.5 - 2022-04-09

  • replace cheshire usage with jsonista by @domparry
  • Bump beam to 2.37.0; bump other deps

v0.7.4 - 2021-12-15

  • Datastore fix by @domparry for breaking change introduced by nippy bump in 0.7.3
  • Bump beam dep (2.34.0)
  • Bump tools.logging (1.2.2)
  • Clean up

v0.7.3 - 2021-08-14

  • Resolve various java compiler warnings
  • Bump dependencies including beam (2.31.0)
  • Bump nippy to 3.1.1, adding alter-var-root of nippy/*thaw-serializable-allowlist* to allow serializable org.apache.beam.sdk.values.KV
  • Fix probems with Id keys in Google Datastore storage

v0.7.2 - 2021-01-07

  • fix locking problem with requires in safe-exec => Issue #97
  • Update Apache Beam to 2.26.0

v0.7.1 - 2020-10-06

  • Update Apache Beam to 2.24.0

v0.7.0 - 2020-09-07

  • Add Kafka IO support
  • Add delete-datastore-raw to datastore features
  • add :schema-update-options for BQ write
  • Update Apache Beam to 2.22.0
  • Performance improvements
  • Fixed require bug in partition-by

v0.6.6 - 2019-09-11

Fixes

  • Fix cogroup-by for numbers of pcoll > 10 => Issue #83
  • Fix :many-files option in read-text-file => Issue #75

v0.6.5 - 2019-08-23

  • Add FileIO Read to read a PCollection of files
  • Add ElasticSearchIO
  • Upgrade to Apache Beam 2.15.0
  • Upgrade dependencies

v0.6.4 - 2019-01-16

  • Upgrade to Apache Beam 2.9.0
  • Upgrade dependencies

v0.6.3 - 2018-11-19

  • Add json-schema in bq write
  • Combine-fn accepts now map as arguments
  • Remove deprecated write-by

v0.6.2 - 2018-11-15

  • Upgrade to Apache Beam 2.8.0
  • Support dynamic file writes
  • Support :deflate and :zip compressions
  • bq/->time-partitioning now accepts :field and :require-partition-filter
  • Improve docs

v0.6.1 - 2018-03-20

Changes

  • Upgrade to Dataflow 2.3.0
  • Support time-partitioned Big Query tables (thanks to @neuromantik33)

Fixes

  • Fix broken input from BigQuery
  • Fix missing safe-exec from filename policy
  • Fix broken serializing after group-by in some cases
  • Fix rare (hopefully) bug introduced by being lazy. with the vals after a group-by. Now we are greedy.

v0.6.0 - 2018-02-13

Changes

  • Compatibility with Dataflow 2.2.0
  • Compatibility with Clojure 1.9 and specs (some code had to be ported to Java shims because proxy is not Serializable anymore in Clojure 1.9)

Fixes

  • Removed superfluous logging, safe-exec call and type hints

v0.5.3 - 2017-12-18

Fixes

  • Fix options on PipelineWithOptions to have it work again on Dataflow (oops, sorry!)

v0.5.2 - 2017-12-14

Changes

  • Fixed defoptions again and made it useful for interop with Beam
  • Added :checkpoint option, will be made more useful later
  • Switched to new PubSub API
  • Add support for custom filename-policy

Fixes

  • Fixed a possible bug causing StackOverflows on combine

v0.5.1 - 2017-10-05

Changes

  • Added wait-pipeline-result

Fixes

  • Fix a bug with incorrect call to seq in combine
  • Add a missing safe-exec in write-bq-table-raw

[v0.5.0] - 2017-09-18

Changes

  • Update to v2.x Dataflow version (Apache Beam)
  • Add new streaming features (custom timestamp in PubSub #30 (thanks to @RoIT), BigQuery streaming writes)
  • Add compressed TextIO output #31 (thanks to @RoIT)
  • Add FileNamePolicy compatibility for TextIO

[v0.4.1] - 2017-03-09

Changes

  • Handle recursion in RECORD schema generation for Big Query #9 (thanks to @torbjornvatn)
  • Add windows and trigger support #13 (thanks to @RoIT)
  • Bump Dataflow SDK to 1.9.0
  • Add support for standard SQL syntax for Big Query #11 (thanks to @torbjornvatn)
  • Read data from pubsub topics #10 (thanks to @torbjornvatn)

v0.4.0 - 2016-11-12

Breaking Changes

  • Update Datastore API to v1, add wrappers for creating Entities from and converting to Clojure maps. Some breakage in the datasplash.datastore namespace ensues.

Changes

  • Add pubsub namespace #6 (Thanks to @MartinSahlen)
  • Put slf4j deps in :dev profiles #8 (Thanks to @torbjornvatn)