jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-08 12:03:02 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,570 Commits 5,225 Branches 518 Tags
516a8a59c4eccbfdbfde27cb9bca8b6647ea5036
Commit Graph

11570 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
maxi297
516a8a59c4 🤖 Bump minor version of Airbyte CDK 2023-06-22 17:01:00 +00:00
Maxime Carbonneau-Leclerc
a45a1e3341 Maxi297/refactoring declarative state management (#27445) 
* [ISSUE #26581] per partition cursor

* [ISSUE #26581] format

* [ISSUE #26581] clean up state management

* [ISSUE #26581] improving Hashabledict

* [ISSUE #26581] format cdk

* [ISSUE #26581] fix tests

* [ISSUE #26581] code review from girarda

* Retrigger pipeline

* Decouple cursor and stream slicer and pushing state management as far up cursor as possible

* Format cdk

* Small fixes/comments

* DatetimeBasedCursor should not update state based on slice (for now at least since it wasn't doing this before)

* [ISSUE #26581] code review

* Automated Commit - Formatting Changes

* [ISSUE #26581] validation overlapping keys

* [ISSUE #26581] add typing

* [ISSUE #26581] code review

* Remove SyncMode from stream_slices

* Removing SyncMode from stream_slices up until SimpleRetriever and fixing typing

* format cdk
 2023-06-22 12:54:36 -04:00
Artem Inzhyyants
aa9b4ee5e6 Source Google Ads: handle authorization error (#27608) 
* Source Google Ads: Fix pattern descriptor

* 🤖 Auto format source-google-ads code [skip ci]

* Source Google Ads: handle authorization error

* Source Google Ads: handle authorization error

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
 2023-06-22 18:34:39 +02:00
Alexandre Girard
d548587161 [ISSUE #27289] Document macros output in the manifest schema (#27600) 
* Add example for macros

* Update changelog

* Revert "Update changelog"

This reverts commit 2993e5820e.
 2023-06-22 09:14:17 -07:00
Brian Lai
02e4bd07f7 [26989] Add request filter for cloud and integration test fixtures for e2e sync testing (#27534) 
* add the request filters and integration test fixtures

* pr feedback and some tweaks to the testing framework

* optimize the cache for more hits

* formatting

* remove cache
 2023-06-22 12:14:07 -04:00
Serhii Lazebnyi
d96f6cbcdb Source Shopify: Added new streams (#27315) 
* Added new streams

* Delete blocked streams

* Delete unnecessary scopes

* Bump version

* Updated configured catalog

* Fix test_privileges_validation

* Fixed CAT for shopify

* Fix CAT for shopify

* Fix abnormal state

* Firmat code

* Updated doc
 2023-06-22 18:13:32 +02:00
Serhii Chvaliuk
019153f178 Source Google Analytics v4: - add pk and lookback window (#26283) 
* primary_key added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* bump 0.2.3

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* Automated Change

* test_parse_response fixed

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* Automated Change

* primary_key added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* metadata = None added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* LOOKBACK_WINDOW added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* google-analytics-data-api.md updated

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* threshold_days: 2 added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* remove uuid from acceptance-test-config.yml

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* uuid removed

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* Automated Change

* oss_registry.json - removed

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* date_range added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* date_range added for pivot_report pk

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* improve LOOKBACK_WINDOW

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* date_range -> (startDate, endDate)

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* bump 0.3.0

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* get_source_defined_primary_key added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* added comment about cohort get_json_schema

* Major Version Number Bump

---------

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: grubberr <grubberr@users.noreply.github.com>
Co-authored-by: Daryna Ishchenko <darina.ishchenko17@gmail.com>
 2023-06-22 19:00:38 +03:00
Catherine Noll
a8e99a46e6 File CDK: define streams via glob list (#27476) 2023-06-22 11:50:35 -04:00
Artem Inzhyyants
0c3d4499d6 Source S3: fix start date (#27611) 
* Source S3: fix start date

* Source S3: update docs

* Source S3: bump version
 2023-06-22 17:17:52 +02:00
Maxime Carbonneau-Leclerc
37f04f7ba0 ✨ Source Zenloop: improve error message on state discrepancy (#27610) 
* [ISSUE #26607] improve error message on state discrepancy

* trigger pipeline
 2023-06-22 11:12:26 -04:00
maxi297
49a60ef735 🤖 Bump patch version of Airbyte CDK 2023-06-22 14:04:07 +00:00
Maxime Carbonneau-Leclerc
d9a5e2d873 🐛 Source zenloop: update to state per partition (#27556) 
* [ISSUE #26581] per partition cursor

* [ISSUE #26581] format

* [ISSUE #26581] clean up state management

* [ISSUE #26581] improving Hashabledict

* [ISSUE #26581] format cdk

* [ISSUE #26581] fix tests

* [ISSUE #26581] code review from girarda

* Retrigger pipeline

* [ISSUE #26581] code review

* Automated Commit - Formatting Changes

* [ISSUE #26581] validation overlapping keys

* [ISSUE #26581] add typing

* [ISSUE #26581] code review

* [ISSUE #26607] zenloop migration (#27243)

* [ISSUE #26607] zenloop migration implementation without tests

* [ISSUE #26607] zenloop migration adding edge cases

* [ISSUE #26607] add cursor field for state

* [ISSUE #26607] update abnormal state

* [ISSUE #26607] ensure default state

* [ISSUE #26607] updating CATs state

* [ISSUE #26607] revert migrating cursor

* [ISSUE #26607] remove default cursor value

* [ISSUE #26607] improve error message

* [ISSUE #26607] changelog

---------

Co-authored-by: Augustin <augustin@airbyte.io>

* 🤖 Auto format source-zenloop code [skip ci]

* Automated Commit - Formatting Changes

* [ISSUE #26581] move partition serialization to JSON

* Revert "[ISSUE #26607] zenloop migration (#27243)"

This reverts commit 5c6f19b775.

* Revert "Revert "[ISSUE #26607] zenloop migration (#27243)""

This reverts commit e363fd6cb8.

* [ISSUE #26607] update zenloop version

* TMP specify cdk version

* [ISSUE #26607] do not lock zenloop airbyte_cdk version

* trigger pipeline

* Automated Commit - Formatting Changes

* trigger pipeline

---------

Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
 2023-06-22 08:42:20 -05:00
Daryna Ishchenko
dc7255bb6f 🐛 Source Google Directory: update version in cloud to 0.2.0 (#27236) 
* update version in cloud

* 🤖 Auto format source-google-directory code [skip ci]

* increased connectors version

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
 2023-06-22 13:30:53 +03:00
Joe Reuter
5f05971924 Connector builder: Add documentation for unified oauth authenticator (#27061) 
* add documentation

* adjust docs
 2023-06-22 12:20:32 +02:00
Artem Inzhyyants
eef872e9f3 Source S3: Add logging for file reading (#27604) 
* Source S3: Add logging for file reading

* Source S3: update docs
 2023-06-22 10:53:32 +02:00
Subodh Kant Chaturvedi
8c8f041a8c postgres-source: complete implementation for for ctid and xmin sync (#27302) 
* initial ctid for testing

* initial ctid for testing

* initial ctid for testing

* Automated Commit - Format and Process Resources Changes

* add version and state type to xmin status

* add logic to swtich between xmin and ctid sync

* npe fixes

* use enum

* refactor

* add relation node logic + validation for vacuuming + more refactor

* refine test + make PR ready for review

* remove un-wanted changes

* missed this one

* remove irrelevant comments

* add more assertions

* remove jdbc log

* address review comments

* 🤖 Auto format source-postgres code [skip ci]

* skip streams under vacuum

* 🤖 Auto format source-postgres code [skip ci]

* update log message

* 🤖 Auto format source-postgres code [skip ci]

* comment

* latest round of review comments

* missed this file

* 🤖 Auto format source-postgres code [skip ci]

* source-postgres : Add logic to handle xmin wraparound (#27466)

* Initial logic for xmin wraparound

* Add tests

* Address comments

* add xmin-wraparound check

* address review comments

* 🤖 Auto format source-postgres code [skip ci]

* missed this

* 🤖 Auto format source-postgres code [skip ci]

---------

Co-authored-by: Akash Kulkarni <akash@airbyte.io>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>

---------

Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
 2023-06-22 03:02:48 -05:00
Ben Church
3bd340a971 Update to use rev-parse (#27598) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-21 20:42:56 -05:00
Charles
bf52bcffce deprecate otel docs (#27589) 2023-06-21 16:44:05 -07:00
Lee Danilek
77ea559f33 🚨🚨 🐛 Convex source fix skipped records (#27226) 
* update cursor in next_page_token, add request_headers

* .

* fix unit tests

* also fix SyncMode.incremental so it doesn't return duplicate records in race conditions

* fix client version in json_schemas

* fix version number

* fix sync_mode

* cache json_schemas to pass typecheck

* never mind

* fix sync_mode

* update metadata and doc

* backward compatiblity bypass

---------

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
 2023-06-21 18:18:34 -03:00
Ryan Fu
4d76ac5809 Turn off java formatter (#27579) 2023-06-21 21:13:20 +00:00
Augustin
e742f8d5b5 connectors-ci: better connector test debugging experience (1/2) (#27550) 2023-06-21 17:28:04 +00:00
maxi297
59ff7b1c6c 🤖 Bump minor version of Airbyte CDK 2023-06-21 17:06:04 +00:00
Maxime Carbonneau-Leclerc
8926970c86 Issue 26581/per partition cursor (#27223) 
* [ISSUE #26581] per partition cursor

* [ISSUE #26581] format

* [ISSUE #26581] clean up state management

* [ISSUE #26581] improving Hashabledict

* [ISSUE #26581] format cdk

* [ISSUE #26581] fix tests

* [ISSUE #26581] code review from girarda

* Retrigger pipeline

* [ISSUE #26581] code review

* Automated Commit - Formatting Changes

* [ISSUE #26581] validation overlapping keys

* [ISSUE #26581] add typing

* [ISSUE #26581] code review

* [ISSUE #26607] zenloop migration (#27243)

* [ISSUE #26607] zenloop migration implementation without tests

* [ISSUE #26607] zenloop migration adding edge cases

* [ISSUE #26607] add cursor field for state

* [ISSUE #26607] update abnormal state

* [ISSUE #26607] ensure default state

* [ISSUE #26607] updating CATs state

* [ISSUE #26607] revert migrating cursor

* [ISSUE #26607] remove default cursor value

* [ISSUE #26607] improve error message

* [ISSUE #26607] changelog

---------

Co-authored-by: Augustin <augustin@airbyte.io>

* 🤖 Auto format source-zenloop code [skip ci]

* Automated Commit - Formatting Changes

* [ISSUE #26581] move partition serialization to JSON

* Revert "[ISSUE #26607] zenloop migration (#27243)"

This reverts commit 5c6f19b775.

* [ISSUE #26607] revert zenloop

---------

Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
 2023-06-21 12:59:11 -04:00
Augustin
3050c24ab4 fix typo in env var name (#27553) 2023-06-21 10:54:20 -05:00
Denys Davydov
076065cce4 🐛 Source Facebook marketing: reduce start date by one day (#27483) 
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* #1975 source fb marketing: reduce start date by one day

* source fb marketing: upd changelog

* 🤖 Auto format source-facebook-marketing code [skip ci]

* format code

* #1975 source fb marketing: remove cloud version override

* source facebook marketing: update CAT config

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
 2023-06-21 18:32:49 +03:00
Artem Inzhyyants
aaf8f34dbe Source Stripe: Fix formatting (#27522) 
* Source Stripe: Fix formatting

* 🤖 Auto format source-stripe code [skip ci]

* Source Stripe: Bump version

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
 2023-06-21 14:16:39 +02:00
Augustin
22480ac642 ref: broaden the use of the run-dagger-pipeline action (#27506) 2023-06-21 06:47:33 -05:00
Artem Inzhyyants
5f35afee11 Source Google Analytics Data API: Fix formatting (#27531) 
* Source Google Analytics Data API: Fix formatting

* Source Stripe: Bump version
 2023-06-21 12:54:53 +02:00
Octavia Squidington III
bad8157c44 🤖 Cloud Availability updater: new connectors to deploy [20230621] (#27542) 2023-06-21 10:26:00 +02:00
Denys Davydov
0f002d2362 🐛 Source Klaviyo: do not store state in the future (#27498) 
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* #2132 source klaviyo: do not store state in the future

* source klaviyo: upd changelog

* 🤖 Auto format source-klaviyo code [skip ci]

* reformat code

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
 2023-06-21 10:41:23 +03:00
Augustin
2e3ee2deed qa-check: add pydash deps (#27544) 2023-06-21 02:28:50 -05:00
Amalia Advincula-Roye
a0f8b3519a Update google-analytics-v4.inapp.md (#27536) 
Fixed custom report syntax
 2023-06-20 18:53:14 -04:00
Ben Church
943150a19b Add any glob to filter to ensure all steps are run in CI (#27525) 
* Ensure all steps are run

* Update comment

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-20 15:14:28 -07:00
Augustin
473112548e connectors-ci: fix CAT not run for java connectors (#27474) 2023-06-20 23:54:40 +02:00
girarda
3a8601eee8 🤖 Bump minor version of Airbyte CDK 2023-06-20 21:49:37 +00:00
Alexandre Girard
dee2e9a905 🐛 Use url encoding in oauth refresh request (#27523) 
* Revert "🐛 CDK: replace `data` with `json` when making OAuth calls (#27350)"

This reverts commit 780f4415d9.

* Revert "Set content-type header on oauth request (#27225)"

This reverts commit 2864f72ff4.
 2023-06-20 14:41:06 -07:00
Akash Kulkarni
2f807440da Fix issue in streaming JDBC database (#27212) 
* Fix issue in streaming JDBC database

* Bump + publish

* 🤖 Auto format source-db2 code [skip ci]

* 🤖 Auto format source-mssql code [skip ci]

* 🤖 Auto format source-postgres-strict-encrypt code [skip ci]

* 🤖 Auto format source-postgres code [skip ci]

* 🤖 Auto format source-mssql-strict-encrypt code [skip ci]

* 🤖 Auto format source-oracle code [skip ci]

* 🤖 Auto format source-tidb code [skip ci]

* 🤖 Auto format source-snowflake code [skip ci]

* 🤖 Auto format source-redshift code [skip ci]

* 🤖 Auto format source-mysql code [skip ci]

* 🤖 Auto format source-mysql-strict-encrypt code [skip ci]

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
 2023-06-20 12:08:47 -05:00
Ben Church
38a189e829 Breaking Changes: QA migration check (#27294) 
* Add migration guide check

* Add test todo

* migration_guide to -migrations

* Update tests

* Valid tests

* Add tests

* Update tools/ci_connector_ops/ci_connector_ops/utils.py

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* Ensure that the migration path is checked

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
 2023-06-20 09:30:23 -07:00
Roman Yermilov [GL]
80e6a2666b 🐛Source Amazon Ads: fix portfolioId type (#25874) 
* Source Amazon Ads: fix portfolioId type

* Source Amazon Ads: update metadata

* Source Amazon Ads: disable backward compatibility tests for 1.1.0

* 🤖 Auto format source-amazon-ads code [skip ci]

---------

Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
 2023-06-20 11:22:43 -05:00
Evan Tahler
9cbb7f267a Update postgres.md (#27325) 
* Update postgres.md

* Update postgres.md

* Update postgres.md
 2023-06-20 15:16:47 +00:00
Ryan Fu
3dccf638f0 Bumps Snowflake defaults to prep for async snowflake (#27487) 2023-06-20 00:42:40 -05:00
Artem Inzhyyants
1bc32f25c3 Source Salesloft: update expected records (#27484) 
* Source Salesloft: update expected records

* 🤖 Auto format source-salesloft code [skip ci]

* Source Salesloft: update test config

* Source Salesloft: bump version

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
 2023-06-19 23:22:48 +02:00
Artem Inzhyyants
3df819612f Source Stripe: Add new Streams (#27362) 
* Source Stripe: Add Stream: Cards

* Source Stripe: update docs; tests config; bump version

* Source Stripe: Add Stream: Payment Methods; Refactor

* Source Stripe: update docs

* Source Stripe: update docs

* Source Stripe: fix unit tests

* Source Stripe: update abnormal_state

* Source Stripe: update schemas

* Source Stripe: update acceptance tests

* Source Stripe: fix address type

* Source Stripe: fix schemas

* Source Stripe: Add stream: Reviews

* Source Stripe: update docs

* Source Stripe: update acceptance tests

* Source Stripe: update schema

* Source Stripe: add stream Cardholders

* Source Stripe: update expected records

* Source Stripe: add expected records for cardholder

* Source Stripe: fix unit test

* Source Stripe: add abnormal state

* Source Stripe: add stream: Authorizations

* Source Stripe: Add Stream: Setup Attempts; Usage Records

* Source Stripe: update docs

* Source Stripe: Add Stream: Transfer Reversals

* Source Stripe: bump versions + update docs

* Source Stripe: Add Stream: Transactions

* Source Stripe: Fix Unit test

* Source Stripe: Refactor

* Source Stripe: Refactor

* Source Stripe: Fix full refresh test

* Source Stripe: bump version

* Source Stripe: update source

* Source Stripe: remove cache file

* Source Stripe: update expected records

---------

Co-authored-by: Augustin <augustin@airbyte.io>
 2023-06-19 20:28:29 +02:00
Augustin
e9d7c54adc connectors ci: auto format connectors code (#27353) 2023-06-19 17:44:43 +00:00
Artem Inzhyyants
15a75b51d8 Source Sentry: Update expected records (#27472) 2023-06-19 18:49:32 +02:00
Catherine Noll
f464a330f8 File-based CDK module scaffolding (#27122) 
Includes CSV schema inference & record parser (#27176)

---------

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2023-06-19 11:01:11 -04:00
Artem Inzhyyants
5ded14c1cd Source Xero: Update Airbyte CDK version to 0.40 (#27471) 
* Source Xero: Update CDK version to 0.40

* Source Xero: Update CDK version to 0.40
 2023-06-19 16:43:51 +02:00
Artem Inzhyyants
d0db23ae51 Source Trello: Update expected records (#27470) 
* Source Trello: Update expected records

* Source Trello: Update expected records + schema

* Source Trello: Update docs
 2023-06-19 16:43:39 +02:00
Arsen Losenko
2e821bf1c1 🐛 Source Typeform: add undeclared columns to streams (#27455) 
* Source Typeform: add undeclared columns to streams

* Update changelog

* Add date-time format to datetime fields

* Change version of connector

* Update version in changelog

* Change version to 0.2.0
 2023-06-19 16:28:15 +03:00
Artem Inzhyyants
ebd738e4ac Source Pipedrive: Update expected records (#27467) 2023-06-19 15:06:26 +02:00