1
0
mirror of synced 2026-01-05 21:02:13 -05:00
Commit Graph

2802 Commits

Author SHA1 Message Date
Daryna Ishchenko
a1da244d3c Source Paystack: enable in cloud (#24085)
* made needed changes for enabling in cloud

* added changelog

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-16 13:45:19 +02:00
Denys Davydov
cd16cc8cf5 #1526 source google search console: update docs (#24133) 2023-03-16 13:03:34 +02:00
Denys Davydov
d534751ce7 Source Salesloft: certify to beta (#23937)
* #19217 Source Salesloft: certify to beta

* #19217 source salesloft: update docs

* #19217 source salesloft: upd changelog

* #19217 source salesloft: upd expected records

* #19217 source salesloft: upd expected records

* #19217 source salesloft: upd expected records

* #19217 source salesloft: upd CAT config

* #19217 source salesloft: fix CAT config

* #19217 source salesloft: remove advanced_oauth

* Automated Change

* auto-bump connector version

---------

Co-authored-by: davydov-d <davydov-d@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-16 12:03:28 +02:00
Adam Schmidt
98f0080b2a 🎉 New Source: OpsGenie [python cdk] (#16768)
* feat: Opsgenie connector

* fix: PR feedback

* Update acceptance-test-config.yml

* chore: patch tests

* chore: added empty_streams

* Delete .dccache

* Update setup.py

* task: fix up unittests

* Update acceptance-test-config.yml

* Update acceptance-test-config.yml

* Update source_definitions.yaml

* Update source_specs.yaml

* run config seed

---------

Co-authored-by: Vincent Koc <koconder@users.noreply.github.com>
Co-authored-by: Vincent Koc <koconder@gmail.com>
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: Sunny <6833405+sh4sh@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2023-03-16 01:42:09 +00:00
Zaza Javakhishvili
1119e6dc8c 🎉 Source Amazon Seller Partner - add Belgium marketplace (#24098)
* amazon spadd belgium marketplace

* amazon spadd belgium marketplace

* amazon spadd belgium marketplace

* fix unit test spec

* auto-bump connector version

---------

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-15 22:18:47 -03:00
Duy Nguyen
2e85c94c74 #23949 Remove check() from read() method for sources Cockroach DB and Postgres (#24000)
* Removed extraneous read on the check() method for CockroachDB and Postgres
* Bumped dockerfile version for relevant conenctors
* Bumped versions for strick-encrypt tests
* Removed withoutPublication and withoutReplicationSlot tests for postgres cdc
* Automated Change
* Updated sources docs
* Update incorrect year in change logs doc for Cockroach and Postgres
* Removed read override method from cockroachDB
* Removed read method override for source-postgres
* auto-bump connector version

---------

Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-15 23:23:36 +00:00
John Aitchison
6bc96b514e Source Intercom: extend Contacts schema with opted_out_subscription_types (#22095)
* new contacts prop, opted_out_subscription_types

* bump intercom version

* update changelog

* fix typo in changelog

* update connectors.md

---------

Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2023-03-15 17:44:20 -03:00
VitaliiMaltsev
9e94126fd6 MySQL/MsSQL sources fixed NPE during cursor values validation (#24082)
* MySQL/MsSQL sources  fixed NPE during cursor values validation

* added logging

* bump versions

* auto-bump connector version

* manually update mssql definitions

* Automated Change

* Automated Commit - Formatting Changes

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-15 19:27:31 +00:00
Marcos Marx
133077797d Source Mailchimp - Add Reports stream #23464 (#24112)
* init commit

* update doc

* add reports stream

* Bump mailchimps connector docker image

* fix tests

* update abnormal stream state for report mailchimp

* add mailchimp allowedHosts

* run config seed files

---------

Co-authored-by: Haithem Souala <haithem.souala@melting-point.fr>
Co-authored-by: Haithem Souala <haithem.souala@woopit.fr>
2023-03-15 16:27:19 -03:00
Denys Davydov
3a63e66e42 Source Airtable: update spec and doc (#24093)
* #1651 source airtable: update spec and doc

* #1651 source airtable: upd changelog

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-15 21:19:24 +02:00
Sophia Wiley
b19998eeb7 edited links in docs (#24108) 2023-03-15 19:57:54 +01:00
Siddhant Singh
585fad6772 fixing the docs for surveycto (#20749)
Co-authored-by: Sajarin <sajarindider@gmail.com>
2023-03-15 15:23:41 -03:00
Denys Davydov
e9a21d621f Source Stripe: cache customers (#24051)
* #1632 source stripe: cache customers

* #1632 source stripe: review fixes

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-15 18:54:27 +02:00
Marcos Marx
afb4a00bff Source Zoho CRM: Set airbyte type to string for zoho autonumbers when they include prefix or suffix #23818 (#24034)
* feature: correctly set autonumber type if it contains prefix or suffix

Zoho connector assumes any autonumber data type maps to airbyte's
big_integer. But that is not true as in Zoho you can create incremental
fields with strings in them via the prefix/suffix config params. Taking
this into account we choose string format in case prefix and suffix are
not empty.

* fix unit test and bump connector version

* correct auto_numbe ref

* update docker python base and fix auth

* fix python version file

* fix tests and validate with integration test

* fix  flake

* auto-bump connector version

---------

Co-authored-by: Gerard Clos <clos.gerard@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-15 12:35:20 -03:00
Denys Davydov
7daf9223a4 Source Instagram: do not emit incomplete records (#24043)
* #1640 source instagram: do not emit incomplete records

* #1640 source instagram: upd changelog

* #1640 source instagram: exit gracefully

* Update airbyte-integrations/connectors/source-instagram/source_instagram/streams.py

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* Update airbyte-integrations/connectors/source-instagram/source_instagram/streams.py

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* #1640 source instagram: review fixes

* #1640 source instagram: fix unit tests

* auto-bump connector version

---------

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-15 16:13:22 +02:00
Serhii Chvaliuk
a2c29e0716 Source Trello: Enable in Cloud (#24045)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2023-03-15 15:45:44 +02:00
Eugene
4d69d6eeb7 Destination-snowflake: added write attempt to internal staging Check method (#24058)
* [22212] Destination-snowflake: added write attempt to internal staging Check method
2023-03-15 10:01:52 +00:00
VitaliiMaltsev
ace229f971 MySQL/MsSQL Source: stop sync on null cursor value (#23908)
* MySQL Source: stop sync on null cursor value

* updated MsSQL test

* Fixed testTableWithNullCursorValueShouldThrowException test for Mssql

* fixed MsSQL and MySQL tests

* modified MsSQL query

* fixed typo

* fixed typo

* simplified Mysql query logic

* simplified Mssql query logic

* add quotes to Mssql query

* add mysql view test

* MySQL and MsSQL source throw a warning if a cursor column contains null value

* disabled tests; log warn if null cursor values found

* removed logging in AbstractDbSource

* bump version

* bump version

* auto-bump connector version

* updated changelog

* Automated Change

* updated definitions

* Automated Change

---------

Co-authored-by: ievgeniit <etsybaev@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-14 22:34:58 +02:00
Roman Yermilov [GL]
a9e23bbeb0 Source Amazon Seller Partner: prepare for cloud (#23980)
* Source Amazon Seller Partner: enable on cloud

- make app_id required
- fix acceptance tests
- fix documentation
- fix check_connection

* Source Amazon Seller Partner: increase end date gap before now

* Source Amazon Seller Partner: update unittests

* Source Amazon Seller Partner: update changelog

* Source Amazon Seller Partner: bump version, update changelog

* Source Amazon Seller Partner: update changelog

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-15 00:10:04 +04:00
Amruta Ranade
4f9350d3e8 Fixed docs nits (#24070) 2023-03-14 15:33:47 -04:00
Amruta Ranade
8c6bfffa3d Edited the GA4 doc (#23911) 2023-03-14 15:21:30 -04:00
Ganpat Agarwal
2b81405f2f 🎉 Source Amazon Ads : use optional config report_record_types (#18677)
* use optional config report_record_types

* update spec

* update image version

* update

* correct indentation

* remove lint changes

* add tests for sponsored products report

* update spec

* add test for video report

* increment amazon-ads connector version

* add test_strictness_level as high

* gradle format update

* update

* update tests for new METRIC_RESPONSE

* align validation and transformation

* bump connector version

* auto-bump connector version

---------

Co-authored-by: Juozas V <monai@cure.lt>
Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: Nataly Merezhuk <65251165+natalyjazzviolin@users.noreply.github.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-14 15:43:04 -03:00
Amruta Ranade
a802fb4c4a Edited the Airtable doc (#24065) 2023-03-14 14:29:04 -04:00
Denys Davydov
3eecf5408c Source S3: infer schema of the first file only (#23189)
* #1470 Source S3: infer schema of the first file

* #1470 source s3: upd changelog

* #1470 source s3: review fixes

* #1470 source s3: review fixes

* #1470 source s3: bump version

* #1470 source s3: review fixes

* auto-bump connector version

---------

Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-14 20:09:15 +02:00
Prateek Mukhedkar
f6e6ae6c95 Add allowedHosts to MySQL and MSSQL DB Sources, for network isolation (#23939)
* Add allowedHosts to MySQL and MSSQL DB Sources, for network isolation.
2023-03-14 11:04:55 -07:00
Tuan Nguyen
200b035ec5 🎉 New Source: GCS (#23186)
* initial commit

* fix test error

* Update get_gcs_blobs logic

* add docs

* Update source_definitions.yaml

* Update airbyte-integrations/connectors/source-gcs/source_gcs/source.py

Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com>

* Update airbyte-config/init/src/main/resources/seed/source_definitions.yaml

Co-authored-by: Denys Davydov <davydov.den18@gmail.com>

* Update airbyte-integrations/connectors/source-gcs/source_gcs/helpers.py

Co-authored-by: Denys Davydov <davydov.den18@gmail.com>

* Update airbyte-integrations/connectors/source-gcs/source_gcs/helpers.py

Co-authored-by: Denys Davydov <davydov.den18@gmail.com>

* update docker file for pandas package

* reimplement read_csv file

* add logic to filter selected streams

* close file_obj after reading

* fix format and tests

* add another stream

* auto-bump connector version

---------

Co-authored-by: Sunny <6833405+sh4sh@users.noreply.github.com>
Co-authored-by: Denys Davydov <davydov.den18@gmail.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-14 15:00:18 -03:00
Ella Rohm-Ensing
09593d9e2b [source-instagram] fall back to start date if date cursor not available (#24042)
* Cherry-pick instagram fix

* Bump version

* .parse takes string arguments

* auto-bump connector version

* revert changes, dummy changelog entry

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-14 08:20:51 +00:00
Hans Lemm
71ec40299e 🐛 Source Google Ads: fix stream *Campaigns* primary keys for incremental load (#23999)
* fix: Campaigns incremental key

* docs: update google-ads

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-13 17:42:18 -05:00
Mohamed Magdy
9257114733 🐛 Source Braintree: Update braintree Python library to the latest version (#23548)
* Update `braintree` Python library to the latest version

To address the security fixes applied as recommended in an email sent from Paypal

* bump connector version

* add eof

* auto-bump connector version

---------

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-13 17:53:41 -03:00
Marcos Marx
39f084adac Source Cart: remove incremental from order statuses stream (#23317)
* remove incremental from order statuses stream

* add changelog:

* auto-bump connector version

* Automated Change

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-13 20:15:11 +00:00
Sophia Wiley
76df2f240a Docs: Update broken links (#23940)
* updated_links_in_docs

* updated more links

* typo

* removed backticks

* deleted links in deploying on cloud and local deployment
2023-03-13 20:38:25 +01:00
Nataly Merezhuk
96f6f76833 [non-forked] 🐛 Source Stripe: Add missing field (cancel_at) to the Subscriptions stream (#23452)
* Add missing field to the stream

* Update stripe.md

* Update connectors.md

* Update stripe.md

* Revert version bump.

* Enables high test strictness level.

* Automated Change

* auto-bump connector version

---------

Co-authored-by: Mohamad Hallak <16711801+mrhallak@users.noreply.github.com>
Co-authored-by: Sergio Ropero <42538006+sergio-ropero@users.noreply.github.com>
Co-authored-by: natalyjazzviolin <natalyjazzviolin@users.noreply.github.com>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-13 18:48:31 +00:00
Sergio Ropero
771d21822c feat: Add CDC sync checkpointing based on time or records (#21727)
* This commit adds new functionality that generates checkpoints when doing CDC synchronization.
For that purpose we encapsulate an AirbyteMessage Iterator on a new iterator that handles the
checkpoint messaging.

* Reformat code

* Reformat code

* Reformat code

* Reformat code

* Second attempt with ugly if statement

* Add `isRecordBehindOffset` function to make sure is safe to send the state.
Tests are failing as now it has more state messages:
expected: <1> but was: <3>

* Code formatting

* Add additional check if the record is part of the snapshot load to skip state message.

* Remove comments

* Fix imports

* Fix format

* Add check if the iterator has extra elements so we don't send state message twice (edge case)

* Add a new check to avoid sending multiple state messages with same offset.
Fix PR comments.
Not sending checkpoints... figuring out

* Modify MSSQL and MySQL implementations

* Adds better control on Maps and include a test for time checkpoint.
Also adds extra assert to verify there are no duplicate states

* Formatting

* Improve code documentation and use default for CdcStateHandler new functions

* Sort out missing `final` and types from comments

* Minor improve in checkpoint validation

* format files

* It's 2023!

* Import issues

* Changes after merging master

* Upgrade Debezium version in MySQL

* Bump Postgres and Alloydb

* auto-bump connector version

* Manually bumping version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-13 14:03:01 +01:00
Baz
8f7642f432 🐛 Source Iterable: improve retry on 500 - Generic Error (#23938) 2023-03-12 00:20:33 +02:00
Baz
00752aaa50 🐛 Source Google Analytics 4 (GA4): improve rate limits messages and retry logic (#23822) 2023-03-11 21:15:07 +02:00
Ryan Fu
85391864f7 Ryan/periodic buffer flush (#23931)
* Added support for periodic buffer flush with tests and uses env variable

* Improves code readability and encapulates testing logic

* Removed demo changes and created const for tests

* Updated constructor to reuse method signature

* Increases Snowflake parallel integration forks

* Bumps version number, fixes linting issues and constant format

* Generate seed spec
2023-03-10 21:43:39 +00:00
Eugene
6672b9cb3a Destination-S3: Correctly generate int64 values (#23466)
* [17564] Updated s3 avro to use long instead of int


---------

Co-authored-by: etsybaev <etsybaev@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-10 16:17:33 +00:00
Daryna Ishchenko
5f0d9f0f6f Source Files: decrease give up rate (#23723)
* updated spec, docs, error messages

* added change log

* updated error message in unit tests

* added allowedHosts

* refactored error message and url descriprion

* updated example for url field in spec

* bump source-file verion for source-file-secure

* auto-bump connector version

* updated spec.json for source-file-secure

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-10 11:13:05 +02:00
Baz
bf33a5d063 🐛 Source Delighted: fix Date Since date-format bug in UI (#23909) 2023-03-09 21:04:54 +02:00
Daryna Ishchenko
08bba15dc9 Source Facebook Marketing: Decrease give up rate (#23625)
* Add userfriendly fields description in spe and docs. Extend error messsage for invalid Account ID case.

* added changelog

* updated spec and error message

* updated docs,

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-09 18:42:13 +02:00
Baz
9927b0ccd1 🐛 Source Iterable: add retry for 500 - Generic Error, increase reduce slice max attempts (#23821) 2023-03-09 17:52:40 +02:00
arnaudselva
8324098409 🎉 New Source: Unleash [low-code CDK] (#19923)
* 🎉 New Source: Unleash [low-code CDK]

* update unleash.md with pull request number

* add bootstrap.md

* remove "optional" mention in the spec descriptions

* fix formatting and update source-acceptance-test to connector-acceptance-test

* update to low-code framework update in airbyte-cdk 0.29.0

* Update airbyte-integrations/connectors/source-unleash/source_unleash/unleash.yaml

Co-authored-by: Sunny <6833405+sh4sh@users.noreply.github.com>

* Update airbyte-integrations/connectors/source-unleash/source_unleash/spec.yaml

Co-authored-by: Sunny <6833405+sh4sh@users.noreply.github.com>

* Update airbyte-integrations/connectors/source-unleash/source_unleash/unleash.yaml

Co-authored-by: Sunny <6833405+sh4sh@users.noreply.github.com>

* adjusting naming

* add source-unleash connector icon

* add icon to source_definitions.yaml

* auto-bump connector version

---------

Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: Sunny <6833405+sh4sh@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-09 13:45:21 +00:00
Serhii Chvaliuk
0036de4a3e Source Pipedrive: Add 11 streams: ActivityTypes, Currencies, Files, Filters, LeadLabels, Notes, PermissionSets, Products, ProductFields, Roles, DealProducts (#23789)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2023-03-09 14:54:17 +02:00
Daryna Ishchenko
c8a5533237 Source Confluence: Enable in cloud (#23775)
* Set additionalProperties: true, update docs and spec

* added changelog

* updated link to docs in spec

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-09 12:09:04 +02:00
Arsen Losenko
19368cc2d0 Source Drift: Prepare for Cloud (#23810)
* Update spec for CAT tests

* Add allowedHosts, update changelog

* Update Dockerfile

* Update changelog

* Update changelog message

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-09 11:32:07 +02:00
Prateek Mukhedkar
2d7765b156 Adding allowedHost for Postgres AlloyDB (#23596)
* For network isolation, source connector accepts a list of hosts it is allowed to connect to
2023-03-08 22:17:12 -08:00
Brian Lai
52f83d4cda [source-monday] [source-square] [source-greenhouse] [source-tempo] [source-zenloop] fix a few connectors with failing custom component unit tests + release some connectors w/ beta (#23231)
* fix a few connectors with failing custom component unit tests

* fix outdated expected records

* make square orders stream incremental

* Updated square SAT and delete custom extractor for monday.

* Added allowedHosts

* Delete DpathStringExtractor from init

* Updated abnormal records

* bump versions of greenhouse, tempo, woocommerce, and zenloop to use beta low code version

* remove woocommerce because it's already published with beta

* add allowedHosts for tempo and zenloop

* fix abnormal state file to use created_at for some streams

* auto-bump connector version

* bump versions for square and monday

* auto-bump connector version

* auto-bump connector version

* manually update square definition generation due to failure

* auto-bump connector version

---------

Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-08 14:44:27 -05:00
Artem Inzhyyants
ea0bb0829a Source Airtable: to GA (#23763)
* Source Airtable: docs update

* Source Airtable: http -> https
2023-03-08 18:40:10 +01:00
sh4sh
382e0aacfb fix zendesk chat changelog so 0.1.13 links to valid pr (#23876) 2023-03-08 12:11:14 -05:00
Prateek Mukhedkar
33ca7f3451 Adding allowHosts to all non-GA source database connectors (#23455)
Adding allowedHosts to clickhouse, cockroachdb, db2, oracle, snowflake, tidb source Connectors. 
---------

Co-authored-by: prateekmukhedkar <prateekmukhedkar@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-07 19:52:15 -08:00