[TDL-25607] : API version upgrade #67

shantanu73 · 2024-05-15T11:02:44Z

Description of change

Klaviyo is moving it's legacy APIs v1/v2 to new APIs as per this document.
The following changes are made to include these API changes:

Changed the base url for all streams
Moved API key authorization from query params to request headers
Modified schema for all streams to include all relevant fields.
Modified transformation logic to flatten the API response and get all relevant fields.
Python version upgrade to be compatible with v3.11.7 and module version upgrades taken from this PR
Events stream mapping change taken from this PR by @jwhitneyinsurify

Manual QA steps

Ran syncs for customer connection

Risks

Old API implementation is getting deprecated and it'll sunset soon, will cause connections to fail.

Rollback steps

revert this branch

1. Added new APIs for metrics & events streams and removed metric stream which is deprecated. 2. Added API_VERSION which is going to part of headers in Klaviyo API calls. 3. Made code changes to include changes in response of new APIs. 4. Added API_KEY in headers which will be used for authorizing Klaviyo APIs in new implementation.

1. Added headers in authed_get to include authorization parameter in headers for new Klaviyo API implementation. 2. Added request params for GET API calls for Klaviyo. 3. Added logic to flatten the event response to include all relevant fields.

1. Modified base urls for all FULL table streams. 2. Modified schemas for all FULL table streams.

1. Fixed logic for transfrom_and_write_records method to include FULL table streams. 2. Added query params for FULL table streams to be included in API calls. 3. Removed get_all_pages method as all the streams use cursor based pagination now. 4. Fixed pagination logic in get_all_using_next method.

…ype for all relationship attributes from object to array.

1. Fixed pagination logic. 2. Fixed get_available_metrics method.

…lds of API response.

…tap-klaviyo into TDL-25607-API-version-upgrade

1. Added schemas for some streams and removed schemas for streams which don't exist in new API implementation. 2. Changed primary key for global_exclusions from email to id.

tap_klaviyo/__init__.py

config.sample.json

README.md

rdeshmukh15 · 2024-05-20T15:34:02Z

Klaviyo metrics API link should be updated to the latest in README.md

rdeshmukh15 · 2024-05-20T15:36:48Z

Have you compared the synced data for old and new both the versions with same start_date for all the streams?

…ason.

…exclusions.

shantanu73 · 2024-05-21T12:17:05Z

Have you compared the synced data for old and new both the versions with same start_date for all the streams?

Yes, and the results were same for old and new implementations.

* Changes: 1. Added and removed relevant streams to fix discovery test. 2. Removed unsubscribe stream as the test account does not have sufficient data to test it. * Fixed All fields & Automatic fields tests. * Fixed comments for all fields & automatic fields tests. * Changes: 1. Fixed bookmark test and added appropriate streams with insufficient data in untestable_streams. 2. Fixed start date test and changed assertions for specific streams as per data avilable. 3. Fixed pagination test by keeping different page sizes for different streams. * Modified Klaviyo API docs link in README. * Update config.yml - use stitch-tap-tester image (#69) * Update config.yml * assert changed * Updates for python 3.11 (#66) * Updates for python 3.11 * Update CHANGELOG.md [skip ci] * update circle config * pylint happy * install pylint ----------------------------- * typo ----------------------------- * removed bump version changes * changes config.yml * changes in config.yml to pass unit tests * fixes unit tests * changes in config.yml in setup step * linting command change in config.yml * added step to install pylint in config.yml --------- Co-authored-by: “rdeshmukh15” <[email protected]> * updated docker image in config.yml --------- Co-authored-by: Patrick Lynskey <[email protected]> Co-authored-by: “rdeshmukh15” <[email protected]> Co-authored-by: Leslie VanDeMark <[email protected]>

tap_klaviyo/utils.py

1. Changed included list to dict to optimize computations. This is a review comment resolution.

1. Created STREAM_PARAMS_MAP to get params_list for each FULL_TABLE stream to avoid if-else condition. This change is part of review comment resolution.

shantanu73 and others added 14 commits May 15, 2024 03:50

Added new config parameter api_version.

622b5e4

Modified the shared event schema to include new API response fields.

884e665

Changes:

3f7843f

1. Added headers in authed_get to include authorization parameter in headers for new Klaviyo API implementation. 2. Added request params for GET API calls for Klaviyo. 3. Added logic to flatten the event response to include all relevant fields.

Fixed pylint errors.

0192bb0

Changes:

f18fe2e

1. Modified base urls for all FULL table streams. 2. Modified schemas for all FULL table streams.

Modified schemas for campaigns, lists & event streams to change datat…

b97b1e6

…ype for all relationship attributes from object to array.

Changes:

1f5054a

1. Fixed pagination logic. 2. Fixed get_available_metrics method.

Fixed json schema for event campaings & lists to include relevant fie…

c506c75

…lds of API response.

fix unit tests

d2662d8

Fixed campaigns schema to get campaign-messages.

6554d8d

Merge branch 'TDL-25607-API-version-upgrade' of github.com:singer-io/…

2552fb0

…tap-klaviyo into TDL-25607-API-version-upgrade

Change:

1d5e6f5

1. Added schemas for some streams and removed schemas for streams which don't exist in new API implementation. 2. Changed primary key for global_exclusions from email to id.

rdeshmukh15 reviewed May 20, 2024

View reviewed changes

tap_klaviyo/__init__.py Outdated Show resolved Hide resolved

rdeshmukh15 reviewed May 20, 2024

View reviewed changes

tap_klaviyo/__init__.py Show resolved Hide resolved

rdeshmukh15 reviewed May 20, 2024

View reviewed changes

config.sample.json Outdated Show resolved Hide resolved

rdeshmukh15 reviewed May 20, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

shantanu73 added 3 commits May 20, 2024 18:18

Added params list to get all global_exclusions based on supression re…

6a8508d

…ason.

Added predictive_analytics to the additional fields param for global_…

78fb801

…exclusions.

Removed api_version from config file.

f426901

rdeshmukh15 approved these changes May 21, 2024

View reviewed changes

shantanu73 and others added 2 commits May 21, 2024 13:45

Added comment to explain change in base url for global_exclusions.

dbebeea

somethingmorerelevant reviewed May 23, 2024

View reviewed changes

tap_klaviyo/utils.py Outdated Show resolved Hide resolved

tap_klaviyo/utils.py Outdated Show resolved Hide resolved

shantanu73 added 2 commits May 23, 2024 09:06

Changes:

50f6e45

1. Changed included list to dict to optimize computations. This is a review comment resolution.

Changes:

424e7c1

1. Created STREAM_PARAMS_MAP to get params_list for each FULL_TABLE stream to avoid if-else condition. This change is part of review comment resolution.

shantanu73 requested a review from somethingmorerelevant May 23, 2024 09:24

somethingmorerelevant approved these changes May 23, 2024

View reviewed changes

shantanu73 merged commit 0af8fa5 into master May 23, 2024
3 checks passed

This was referenced May 24, 2024

Changing EVENT_MAPPINGS to reflect recent Klaviyo schema name changes #64

Closed

Bump requests from 2.20.0 to 2.31.0 #62

Closed

bobbrodie mentioned this pull request Jul 24, 2024

List profile count missing from schema #72

Open

rdeshmukh15 mentioned this pull request Dec 6, 2024

Update to new API #65

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TDL-25607] : API version upgrade #67

[TDL-25607] : API version upgrade #67

shantanu73 commented May 15, 2024 •

edited

Loading

rdeshmukh15 commented May 20, 2024

rdeshmukh15 commented May 20, 2024 •

edited

Loading

shantanu73 commented May 21, 2024

[TDL-25607] : API version upgrade #67

[TDL-25607] : API version upgrade #67

Conversation

shantanu73 commented May 15, 2024 • edited Loading

Description of change

Manual QA steps

Risks

Rollback steps

rdeshmukh15 commented May 20, 2024

rdeshmukh15 commented May 20, 2024 • edited Loading

shantanu73 commented May 21, 2024

shantanu73 commented May 15, 2024 •

edited

Loading

rdeshmukh15 commented May 20, 2024 •

edited

Loading