jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-07 09:05:45 -05:00
Code Issues Packages Projects Releases Wiki Activity
10,375 Commits 5,181 Branches 518 Tags
a1da244d3c313ef504ffdfd63056cff1bf90e081
Commit Graph

4449 Commits

Author SHA1 Message Date
Daryna Ishchenko
a1da244d3c Source Paystack: enable in cloud (#24085) 
* made needed changes for enabling in cloud

* added changelog

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-16 13:45:19 +02:00
Denys Davydov
cd16cc8cf5 #1526 source google search console: update docs (#24133) 2023-03-16 13:03:34 +02:00
Denys Davydov
d534751ce7 Source Salesloft: certify to beta (#23937) 
* #19217 Source Salesloft: certify to beta

* #19217 source salesloft: update docs

* #19217 source salesloft: upd changelog

* #19217 source salesloft: upd expected records

* #19217 source salesloft: upd expected records

* #19217 source salesloft: upd expected records

* #19217 source salesloft: upd CAT config

* #19217 source salesloft: fix CAT config

* #19217 source salesloft: remove advanced_oauth

* Automated Change

* auto-bump connector version

---------

Co-authored-by: davydov-d <davydov-d@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-16 12:03:28 +02:00
Adam Schmidt
98f0080b2a 🎉 New Source: OpsGenie [python cdk] (#16768) 
* feat: Opsgenie connector

* fix: PR feedback

* Update acceptance-test-config.yml

* chore: patch tests

* chore: added empty_streams

* Delete .dccache

* Update setup.py

* task: fix up unittests

* Update acceptance-test-config.yml

* Update acceptance-test-config.yml

* Update source_definitions.yaml

* Update source_specs.yaml

* run config seed

---------

Co-authored-by: Vincent Koc <koconder@users.noreply.github.com>
Co-authored-by: Vincent Koc <koconder@gmail.com>
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: Sunny <6833405+sh4sh@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
 2023-03-16 01:42:09 +00:00
Zaza Javakhishvili
1119e6dc8c 🎉 Source Amazon Seller Partner - add Belgium marketplace (#24098) 
* amazon spadd belgium marketplace

* amazon spadd belgium marketplace

* amazon spadd belgium marketplace

* fix unit test spec

* auto-bump connector version

---------

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-15 22:18:47 -03:00
Duy Nguyen
2e85c94c74 #23949 Remove check() from read() method for sources Cockroach DB and Postgres (#24000) 
* Removed extraneous read on the check() method for CockroachDB and Postgres
* Bumped dockerfile version for relevant conenctors
* Bumped versions for strick-encrypt tests
* Removed withoutPublication and withoutReplicationSlot tests for postgres cdc
* Automated Change
* Updated sources docs
* Update incorrect year in change logs doc for Cockroach and Postgres
* Removed read override method from cockroachDB
* Removed read method override for source-postgres
* auto-bump connector version

---------

Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-15 23:23:36 +00:00
John Aitchison
6bc96b514e Source Intercom: extend Contacts schema with opted_out_subscription_types (#22095) 
* new contacts prop, opted_out_subscription_types

* bump intercom version

* update changelog

* fix typo in changelog

* update connectors.md

---------

Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
 2023-03-15 17:44:20 -03:00
VitaliiMaltsev
9e94126fd6 MySQL/MsSQL sources fixed NPE during cursor values validation (#24082) 
* MySQL/MsSQL sources  fixed NPE during cursor values validation

* added logging

* bump versions

* auto-bump connector version

* manually update mssql definitions

* Automated Change

* Automated Commit - Formatting Changes

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
 2023-03-15 19:27:31 +00:00
Marcos Marx
133077797d Source Mailchimp - Add Reports stream #23464 (#24112) 
* init commit

* update doc

* add reports stream

* Bump mailchimps connector docker image

* fix tests

* update abnormal stream state for report mailchimp

* add mailchimp allowedHosts

* run config seed files

---------

Co-authored-by: Haithem Souala <haithem.souala@melting-point.fr>
Co-authored-by: Haithem Souala <haithem.souala@woopit.fr>
 2023-03-15 16:27:19 -03:00
Denys Davydov
3a63e66e42 Source Airtable: update spec and doc (#24093) 
* #1651 source airtable: update spec and doc

* #1651 source airtable: upd changelog

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-15 21:19:24 +02:00
Sophia Wiley
b19998eeb7 edited links in docs (#24108) 2023-03-15 19:57:54 +01:00
Edward Gao
294cdbcf4a Normalization Bigquery: Add more reserved words (#24077) 
* add current times for bigquery

* bump version + changelog
 2023-03-15 18:33:59 +00:00
Siddhant Singh
585fad6772 fixing the docs for surveycto (#20749) 
Co-authored-by: Sajarin <sajarindider@gmail.com>
 2023-03-15 15:23:41 -03:00
Alexandre Girard
9864e4bf19 Update links (#24100) 
* Update links

* Add missing files
 2023-03-15 10:20:44 -07:00
Denys Davydov
e9a21d621f Source Stripe: cache customers (#24051) 
* #1632 source stripe: cache customers

* #1632 source stripe: review fixes

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-15 18:54:27 +02:00
Marcos Marx
afb4a00bff Source Zoho CRM: Set airbyte type to string for zoho autonumbers when they include prefix or suffix #23818 (#24034) 
* feature: correctly set autonumber type if it contains prefix or suffix

Zoho connector assumes any autonumber data type maps to airbyte's
big_integer. But that is not true as in Zoho you can create incremental
fields with strings in them via the prefix/suffix config params. Taking
this into account we choose string format in case prefix and suffix are
not empty.

* fix unit test and bump connector version

* correct auto_numbe ref

* update docker python base and fix auth

* fix python version file

* fix tests and validate with integration test

* fix  flake

* auto-bump connector version

---------

Co-authored-by: Gerard Clos <clos.gerard@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-15 12:35:20 -03:00
Denys Davydov
7daf9223a4 Source Instagram: do not emit incomplete records (#24043) 
* #1640 source instagram: do not emit incomplete records

* #1640 source instagram: upd changelog

* #1640 source instagram: exit gracefully

* Update airbyte-integrations/connectors/source-instagram/source_instagram/streams.py

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* Update airbyte-integrations/connectors/source-instagram/source_instagram/streams.py

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* #1640 source instagram: review fixes

* #1640 source instagram: fix unit tests

* auto-bump connector version

---------

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-15 16:13:22 +02:00
Serhii Chvaliuk
a2c29e0716 Source Trello: Enable in Cloud (#24045) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2023-03-15 15:45:44 +02:00
Eugene
4d69d6eeb7 Destination-snowflake: added write attempt to internal staging Check method (#24058) 
* [22212] Destination-snowflake: added write attempt to internal staging Check method
 2023-03-15 10:01:52 +00:00
VitaliiMaltsev
ace229f971 MySQL/MsSQL Source: stop sync on null cursor value (#23908) 
* MySQL Source: stop sync on null cursor value

* updated MsSQL test

* Fixed testTableWithNullCursorValueShouldThrowException test for Mssql

* fixed MsSQL and MySQL tests

* modified MsSQL query

* fixed typo

* fixed typo

* simplified Mysql query logic

* simplified Mssql query logic

* add quotes to Mssql query

* add mysql view test

* MySQL and MsSQL source throw a warning if a cursor column contains null value

* disabled tests; log warn if null cursor values found

* removed logging in AbstractDbSource

* bump version

* bump version

* auto-bump connector version

* updated changelog

* Automated Change

* updated definitions

* Automated Change

---------

Co-authored-by: ievgeniit <etsybaev@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
 2023-03-14 22:34:58 +02:00
Roman Yermilov [GL]
a9e23bbeb0 Source Amazon Seller Partner: prepare for cloud (#23980) 
* Source Amazon Seller Partner: enable on cloud

- make app_id required
- fix acceptance tests
- fix documentation
- fix check_connection

* Source Amazon Seller Partner: increase end date gap before now

* Source Amazon Seller Partner: update unittests

* Source Amazon Seller Partner: update changelog

* Source Amazon Seller Partner: bump version, update changelog

* Source Amazon Seller Partner: update changelog

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-15 00:10:04 +04:00
Amruta Ranade
4f9350d3e8 Fixed docs nits (#24070) 2023-03-14 15:33:47 -04:00
Amruta Ranade
8c6bfffa3d Edited the GA4 doc (#23911) 2023-03-14 15:21:30 -04:00
Ganpat Agarwal
2b81405f2f 🎉 Source Amazon Ads : use optional config report_record_types (#18677) 
* use optional config report_record_types

* update spec

* update image version

* update

* correct indentation

* remove lint changes

* add tests for sponsored products report

* update spec

* add test for video report

* increment amazon-ads connector version

* add test_strictness_level as high

* gradle format update

* update

* update tests for new METRIC_RESPONSE

* align validation and transformation

* bump connector version

* auto-bump connector version

---------

Co-authored-by: Juozas V <monai@cure.lt>
Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: Nataly Merezhuk <65251165+natalyjazzviolin@users.noreply.github.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-14 15:43:04 -03:00
Amruta Ranade
a802fb4c4a Edited the Airtable doc (#24065) 2023-03-14 14:29:04 -04:00
Denys Davydov
3eecf5408c Source S3: infer schema of the first file only (#23189) 
* #1470 Source S3: infer schema of the first file

* #1470 source s3: upd changelog

* #1470 source s3: review fixes

* #1470 source s3: review fixes

* #1470 source s3: bump version

* #1470 source s3: review fixes

* auto-bump connector version

---------

Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-14 20:09:15 +02:00
Prateek Mukhedkar
f6e6ae6c95 Add allowedHosts to MySQL and MSSQL DB Sources, for network isolation (#23939) 
* Add allowedHosts to MySQL and MSSQL DB Sources, for network isolation.
 2023-03-14 11:04:55 -07:00
Tuan Nguyen
200b035ec5 🎉 New Source: GCS (#23186) 
* initial commit

* fix test error

* Update get_gcs_blobs logic

* add docs

* Update source_definitions.yaml

* Update airbyte-integrations/connectors/source-gcs/source_gcs/source.py

Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com>

* Update airbyte-config/init/src/main/resources/seed/source_definitions.yaml

Co-authored-by: Denys Davydov <davydov.den18@gmail.com>

* Update airbyte-integrations/connectors/source-gcs/source_gcs/helpers.py

Co-authored-by: Denys Davydov <davydov.den18@gmail.com>

* Update airbyte-integrations/connectors/source-gcs/source_gcs/helpers.py

Co-authored-by: Denys Davydov <davydov.den18@gmail.com>

* update docker file for pandas package

* reimplement read_csv file

* add logic to filter selected streams

* close file_obj after reading

* fix format and tests

* add another stream

* auto-bump connector version

---------

Co-authored-by: Sunny <6833405+sh4sh@users.noreply.github.com>
Co-authored-by: Denys Davydov <davydov.den18@gmail.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-14 15:00:18 -03:00
Ella Rohm-Ensing
09593d9e2b [source-instagram] fall back to start date if date cursor not available (#24042) 
* Cherry-pick instagram fix

* Bump version

* .parse takes string arguments

* auto-bump connector version

* revert changes, dummy changelog entry

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-14 08:20:51 +00:00
Alexandre Girard
5fecc7e626 add missing newline (#24035) 
* add missing newline

* Automated Commit - Formatting Changes

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-03-14 00:37:34 +00:00
Sherif A. Nada
1bc1ecfe6a Fix broken links in CDK speedrun tutorial (#24006) 2023-03-13 17:11:52 -07:00
Sherif A. Nada
1feda0817e Rearrange connector contribution docs (#24004) 2023-03-13 17:01:36 -07:00
Hans Lemm
71ec40299e 🐛 Source Google Ads: fix stream *Campaigns* primary keys for incremental load (#23999) 
* fix: Campaigns incremental key

* docs: update google-ads

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-13 17:42:18 -05:00
Mohamed Magdy
9257114733 🐛 Source Braintree: Update braintree Python library to the latest version (#23548) 
* Update `braintree` Python library to the latest version

To address the security fixes applied as recommended in an email sent from Paypal

* bump connector version

* add eof

* auto-bump connector version

---------

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-13 17:53:41 -03:00
Marcos Marx
39f084adac Source Cart: remove incremental from order statuses stream (#23317) 
* remove incremental from order statuses stream

* add changelog:

* auto-bump connector version

* Automated Change

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-13 20:15:11 +00:00
Sophia Wiley
76df2f240a Docs: Update broken links (#23940) 
* updated_links_in_docs

* updated more links

* typo

* removed backticks

* deleted links in deploying on cloud and local deployment
 2023-03-13 20:38:25 +01:00
Nataly Merezhuk
96f6f76833 [non-forked] 🐛 Source Stripe: Add missing field (cancel_at) to the Subscriptions stream (#23452) 
* Add missing field to the stream

* Update stripe.md

* Update connectors.md

* Update stripe.md

* Revert version bump.

* Enables high test strictness level.

* Automated Change

* auto-bump connector version

---------

Co-authored-by: Mohamad Hallak <16711801+mrhallak@users.noreply.github.com>
Co-authored-by: Sergio Ropero <42538006+sergio-ropero@users.noreply.github.com>
Co-authored-by: natalyjazzviolin <natalyjazzviolin@users.noreply.github.com>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-13 18:48:31 +00:00
Evan Tahler
4ee62d9dfb Update supported-data-types.md - fix link to well-known-type (#23961) 2023-03-13 08:49:07 -07:00
bartlomiejolma
dd5aeea2fc Update on-kubernetes-via-helm.md (#23726) 2023-03-13 11:01:29 -03:00
Sergio Ropero
771d21822c feat: Add CDC sync checkpointing based on time or records (#21727) 
* This commit adds new functionality that generates checkpoints when doing CDC synchronization.
For that purpose we encapsulate an AirbyteMessage Iterator on a new iterator that handles the
checkpoint messaging.

* Reformat code

* Reformat code

* Reformat code

* Reformat code

* Second attempt with ugly if statement

* Add `isRecordBehindOffset` function to make sure is safe to send the state.
Tests are failing as now it has more state messages:
expected: <1> but was: <3>

* Code formatting

* Add additional check if the record is part of the snapshot load to skip state message.

* Remove comments

* Fix imports

* Fix format

* Add check if the iterator has extra elements so we don't send state message twice (edge case)

* Add a new check to avoid sending multiple state messages with same offset.
Fix PR comments.
Not sending checkpoints... figuring out

* Modify MSSQL and MySQL implementations

* Adds better control on Maps and include a test for time checkpoint.
Also adds extra assert to verify there are no duplicate states

* Formatting

* Improve code documentation and use default for CdcStateHandler new functions

* Sort out missing `final` and types from comments

* Minor improve in checkpoint validation

* format files

* It's 2023!

* Import issues

* Changes after merging master

* Upgrade Debezium version in MySQL

* Bump Postgres and Alloydb

* auto-bump connector version

* Manually bumping version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-13 14:03:01 +01:00
Baz
8f7642f432 🐛 Source Iterable: improve retry on 500 - Generic Error (#23938) 2023-03-12 00:20:33 +02:00
Baz
00752aaa50 🐛 Source Google Analytics 4 (GA4): improve rate limits messages and retry logic (#23822) 2023-03-11 21:15:07 +02:00
zzztimbo
869c5b7e7f Bump Airbyte version from 0.41.0 to 0.42.0 2023-03-11 00:15:12 +00:00
Tim Chan
ddc5ae94cd ci: 🔧 revert version bumps (#23954) 2023-03-10 18:07:21 -06:00
zzztimbo
519a85492a Bump Airbyte version from 0.42.0 to 0.43.0 2023-03-10 23:52:03 +00:00
zzztimbo
2b80dd05cf Bump Airbyte version from 0.41.0 to 0.42.0 2023-03-10 23:28:56 +00:00
Ryan Fu
85391864f7 Ryan/periodic buffer flush (#23931) 
* Added support for periodic buffer flush with tests and uses env variable

* Improves code readability and encapulates testing logic

* Removed demo changes and created const for tests

* Updated constructor to reuse method signature

* Increases Snowflake parallel integration forks

* Bumps version number, fixes linting issues and constant format

* Generate seed spec
 2023-03-10 21:43:39 +00:00
Eugene
6672b9cb3a Destination-S3: Correctly generate int64 values (#23466) 
* [17564] Updated s3 avro to use long instead of int


---------

Co-authored-by: etsybaev <etsybaev@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-10 16:17:33 +00:00
Daryna Ishchenko
5f0d9f0f6f Source Files: decrease give up rate (#23723) 
* updated spec, docs, error messages

* added change log

* updated error message in unit tests

* added allowedHosts

* refactored error message and url descriprion

* updated example for url field in spec

* bump source-file verion for source-file-secure

* auto-bump connector version

* updated spec.json for source-file-secure

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-10 11:13:05 +02:00
Baz
bf33a5d063 🐛 Source Delighted: fix Date Since date-format bug in UI (#23909) 2023-03-09 21:04:54 +02:00