v1.0.2

This is a minor release to fix generation of SQLAlchemy connection strings using the python-oracledb driver. It also updates the CI to run tests against Python 3.12 and 3.13.

What's Changed

• Run tests on newer Python versions by @leorudczenko in #216
• Make OracleDbHelper.sqlalchemy_connection_string use oracle+oracledb instead of oracle by @leorudczenko in #217

Full Changelog: v1.0.1...v1.0.2

v1.0.1

This is a minor release to update the documentation to remove the placeholder references to v0.14.3 that were in place prior to v1 release.

v1.0.0

ETL Helper version 1.0 contains many breaking changes and new ways of working.

Breaking changes

• The cxOracle driver has been replaced with the python-oracledb driver. python-oracledb does not depend on Oracle Instant Client and all related code e.g., setup_oracle_client, has been removed as it is no longer required. This is a great sign of progress, as one of the initial purposes of ETL Helper was to give an easy way to jump through the hoops of installing Oracle Instant Client in CI pipelines. The read_lob flag has been deprecated as part of this change. Setting the encoding at connection time via encoding kwarg is also deprecated in the python-oracledb driver, which always uses utf8. See the Oracle driver documentation for details.
• The default row factory has been changed from namedtuple_row_factory, to dict_row_factory. This is because dictionaries are mutable and so easier to use in transform functions. They are also more widely understood. To recreate the previous behaviour in existing scripts, import the row factory with from etlhelper.row_factories import namedtuple_row_factory set row_factory=namedtuple_row_factory.
• get_rows is deprecated; use fetchall instead. This does the same thing, but is better aligned with the DB API 2.0.
• fetchmany is deprecated; use iter_chunks instead. The behaviour of ETL Helper's fetchmany was different to cursor.fetchmany as defined by the DB API 2.0 and this was confusing. It is also possible to use chunks = etl.iter_chunks(...); do_something(next(chunks)).
• dump_rows is deprecated; use for row in iter_rows(...); my_func(row) instead.
• The ETL Helper logger is no longer given a handler by default. This is the correct behaviour for a library. Activate logging with etl.log_to_console().
• The minimum supported Python version is 3.9. This allows us to benefit from security and typing improvements.

Non-breaking changes

• There is a new Sphinx-based documentation webpage at https://britishgeologicalsurvey.github.io/etlhelper.
• Documentation for the previous version (v0.14.3) can still be found at: https://github.com/BritishGeologicalSurvey/etlhelper/tree/v0.14.3
• There is a new preferred syntax, using import etlhelper as etl to provide easy access to the public functions.
• executemany and load functions can now take a transform function.
• Type hints have been added to the ETL functions.
• The build system uses pyproject.toml instead of setup.py and versioneer.
• The CI pipeline has moved from Travis to GitHub actions.
• The CI pipeline runs the PostgreSQL integration tests (which cover the bulk of the application logic).

What's Changed

• Deprecate get rows by @leorudczenko in #182
• Set default row factory to as dict by @volcan01010 in #183
• Run postgresql integration tests in ci by @volcan01010 in #184
• Read LOBs as string/bytes instead of streaming by @volcan01010 in #181
• Add README updates to index.rst by @KoalaGeo in #185
• More type hints by @volcan01010 in #186
• Etl post type hint docs by @ximenesuk in #187
• Overhaul docs by @volcan01010 in #190
• More docs updates by @volcan01010 in #192
• Merge long-running for_v1 branch. by @volcan01010 in #193
• Adjust annotations by @leorudczenko in #194
• Release doc updates by @volcan01010 in #195
• Run Sphinx deploy for changes to etlhelper or .github folders by @volcan01010 in #196
• Bump ubuntu version and use python 3.10 by @leorudczenko in #197
• Integration test fixes by @volcan01010 in #198
• Use sqlite connection kwarg example by @volcan01010 in #199
• Use PyPI list for published releases by @volcan01010 in #200
• Update docs code demos by @leorudczenko in #201
• Update documentation by @jbow0131 in #204
• Update csv_files.py imports by @sophiedorward in #206
• Add type hint to connection types in db_helper_factory by @rachelheaven in #207
• Pre release tweaks by @volcan01010 in #208
• Skip test that passes locally but fails in CI by @ximenesuk in #210
• Database to API / NoSQL comments by @spenny-liam in #211
• Add link to log_to_console() by @volcan01010 in #209

New Contributors

• @sophiedorward
• @spenny-liam
• @lyncop
• @rachelheaven
• @jbow0131

Full Changelog: v0.14.3...v1.0.0

v1.0.0rc3

Release candidate for Version 1.0.

Draft release notes for v1.0.0 are here: https://github.com/BritishGeologicalSurvey/etlhelper/releases/tag/untagged-7b87dd3ee68b8f887de6

v1.0.0rc2

Release candidate for Version 1.0.

Draft release notes for v1.0.0 are here: https://github.com/BritishGeologicalSurvey/etlhelper/releases/tag/untagged-7b87dd3ee68b8f887de6

v0.14.3

This is a minor release, with the main purpose of adding a deprecation warning to inform users that ETL Helper will soon move to version 1. This will include breaking changes. Users are advised to pin the version of their dependencies.

Other changes include swapping the CI system from Travis CI to GitHub Actions, documentation updates and minor tweaks to module importing and the reporting of SQLite column type.

Merge requests merged

• GitHub actions by @leorudczenko in #162
• Update README.md by @rbroth in #156
• 159 add deprecation warning by @leorudczenko in #165
• Add row_factories import to init by @leorudczenko in #173
• Change column type to lowercase by @volcan01010 in #178
• Docs fix by @volcan01010 in #179
• • Update pip to fix psycopg2 install by @volcan01010 in #180

New Contributors

• @rbroth made their first contribution in #156

Full Changelog: v0.14.2...v0.14.3

v0.14.2

This is a minor release to allow schema-qualified table names e.g. schema.table in the validate_identifiers() and to apply the validation to table names passed to the copy_table_rows() function.

What's Changed

• Allow schema-qualified names in validate_identifier by @volcan01010 in #141
• Validate table name in copy_table_rows by @volcan01010 in #142

Full Changelog: v0.14.1...v0.14.2

v0.14.1

This release is a minor fix to allow the test suite to pass in Python 3.6.

What's Changed

• Py36 fix by @volcan01010 in #138

Full Changelog: v0.14.0...v0.14.1

v0.14.0

This release adds the following new features:

• The generate_insert_sql function (used by load) validates the table and column names used. As these are user-provided they were a potential route for SQL injection attacks. An ETLHelperBadIdentifierError is raised on bad input.
• The abort_etlhelper_threads() function has been added. When called, this raises an ETLHelperAbort exception at the end of the current chunk. It can be used to cancel jobs that are running in threads, such as in GUI applications. See README.md.
• list_row_factory and tuple_row_factory have been added. Data can now be exported as all of the main Python container types e.g. namedtuple, dictionary, list and tuple. Lists are mutable so are convenient to use with transform functions. Tuples are fast to create and access. See README.md

Pull requests merged

• Validate table identifiers by @volcan01010 in #134
• Repo tweaks by @volcan01010 in #135
• Add list and tuple row factories by @volcan01010 in #136
• Abort function by @volcan01010 in #137

Full Changelog: V0.13.0...v0.14.0

v0.13.0

This release adds the table_info() function that provides basic metadata about the columns in a table. It is useful when browsing data and when auto-generating SQL queries.

What's Changed

• Make information on column types accessible by @ximenesuk and @volcan01010 in #128

Full Changelog: v0.12.2...v0.13.0