1
0
mirror of synced 2026-02-01 04:01:50 -05:00
Commit Graph

2153 Commits

Author SHA1 Message Date
Octavia Squidington III
39d5cfa4e9 Bump Airbyte version from 0.35.32-alpha to 0.35.33-alpha (#10546)
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
2022-02-22 23:11:58 +05:30
Serhii Lazebnyi
ca1cf9bf39 🐛 Source Facebook Marketing: Fix backoff trigger with ConnectionError (#10348)
* Add AdAccount and Images stream implementation

* Update PR number

* Updated docker version

* Updated to linter

* Update to review

* Add comment to AdAccount read_records method

* Bumped version in seed, definitions and specs files

* Fix backoff errors code list

* Deleted empty file

* Updated to linter

* Updated PR number

* Added connection_reset_error to backoff

* Fix unittest

* Fix integration tests

* Bumped seed version
2022-02-22 17:26:03 +02:00
augan-rymkhan
9c9dfb25f1 Source Hubspot: add "format": "date-time" (#10492)
* add "format": "date-time"

* add "format": "date-time" to feedback_submissions

* add "format": "date-time" to deals

* bump version

* updated spec and def yaml

Co-authored-by: auganbay <auganenu@gmail.com>
2022-02-22 20:46:36 +06:00
Vadym Hevlich
5464b1c830 🐛 Normalization: Fix sync from HubSpot to MySQL fails with "Row size too large" on create table (#10485)
* Update mysql normalization to cast string as text.
Bump docker version.
Update basic-normalization.md docs.

* Update docs PR reference

* Update mysql normalization to cast string as for is_timestamp_with_time_zone type
2022-02-22 14:22:26 +02:00
andriikorotkov
1d66672fad 🐛 Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats (#10508)
* updated timestamp transformation with microseconds

* updated timestamp transformation with microseconds

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql tests

* removed extra logs

* fixed cursor for timestamp

* updated test

* updated type transformation

* updated type transformation

* updated type transformation

* updated tests

* updated oracle tests

* updated documentations and connectors versions

* updated documentations and connectors versions

* fix code style

* updated cursor for jdbc sources

* updated cursor for jdbc sources

* updated cursor for jdbc sources
2022-02-22 13:26:10 +02:00
oleh.zorenko
e23c29d3fa 🐛 Source Chargebee: add tests + fix broken coupon stream (#10366)
* 10192 Fix Chargebee source

Fix `coupon` stream.
Add unit tests.

* 10192 Update tests + update docs

* 10192 Bump connector's version
2022-02-22 10:20:21 +02:00
Harshith Mullapudi
382e2b7476 Publish all faros ai sources (#10526)
* Add more sources sponsored by Faros AI

* Fix doc link

* Add icons

* Fix phabricator links

* Remove references to phabricator source because we dont have stable source acceptance test for it yet

* fix: rebased with master

* fix: account name in docker image

* Delete config.json

* chore: added seed spec

* chore: fix formatting

Co-authored-by: Chris Wu <chris@faros.ai>
2022-02-22 11:34:47 +05:30
Harshith Mullapudi
d0a5547670 feat: clone sources and destinations API (#10396)
* feat: clone sources and destinations API

* fix: source is pulled without secrets
2022-02-22 10:06:18 +05:30
Vladimir remar
ccb3fe2a3d 🎉 Source TikTok Marketing: Add Audience reports (#10398)
* Update: add new streams(campaings by country, ads, ad_group, advertiser) for audience reports

* update: bump connector version
2022-02-21 18:56:47 -03:00
Baz
67b7fc96ad 🎉 Source Shopify: add tender_transactions stream (#10449) 2022-02-21 20:22:02 +02:00
Kshitij Grover
1e0ac30ebd 🎉 New Source: Orb (#9985)
* V1 of source_orb connector

* add boostrap.md file

* add clause on Pagination to bootstrap.md

* add SUMMARY documentation

* add lookback_window_days connector parameter

* Add support for start_date parameter

* Add ability to transform record in order to un-nest IDs

* Add support for extracting event properties based on connector configuration
2022-02-21 14:14:10 -03:00
augan-rymkhan
2282a4ae02 🎉 Source Hubspot: Migrate to CDK (#10177)
* migrate SourceHubspot to cdk

* refactor discover method

* change method name

* deleted Client class

* remove comment

* added get_updated_state

* fix setting initial state

* fix stream_state dict key

* fix cursor_field

* change check test case status

* refactor streams method

* remove comment

* remove TODOs

* remove comments

* fix get_updated_state

* refactor chunk_read

* override _read_incremental

* fix unit tests

* remove comments

* fix test_check_connection_backoff_on_server_error

* fix test_check_connection_backoff_on_server_error 2

* fix test_check_connection_backoff_on_limit_reached

* fix unit tests

* clear comments

* override read method on Source

* added comments to overriding methods

* some improvements

* reafactor overridden _read_incremental

* format code

* refactor discovery

* remove discover

* format code 2

* added return types

* refactor template stream classes

* remove comments

* remove _name field

* rename api.py to streams.py

* move to HttpStream

* refactor FormSubmissions

* refactor Campaings

* refactor ContactsListMemberships

* CRMSearchStream refactor

* CRMSearchStream refactor 2

* CRMObjectStream refactor

* DealStageHistoryStream refactor

* Deals refactor

* Engagements refactor

* path method refactor

* refactor authentication

* fix check_connection

* fix call parse_response

* fix Engagements stream

* fix CRMSearchStream

* fix CRMObjectIncremental stream

* override _read_incremental

* remove commented codes

* format code

* update cdk version

* fix cursor field

* fix unit tests

* removed client

* clear comments

* clear comments 2

* clear comments 3

* clear comments 4

* override backoff_time

* remove comment

* format code

* backoff_time modified

* refactor backoff_time

* format code

* added return typing

* format code

* removed cursor_paths

* bump version

* updated spec and def yaml

Co-authored-by: auganbay <auganenu@gmail.com>
2022-02-21 22:03:05 +06:00
Anna Lvova
ce5c007540 🎉 Salesforce Source: Support "Replicate Incremental Deletes" (#10454) 2022-02-21 15:03:50 +02:00
Maksym Pavlenok
61b0e9e196 🎉 Single py checker (#10246) 2022-02-21 14:01:42 +02:00
Octavia Squidington III
2a7a045e94 Bump Airbyte version from 0.35.31-alpha to 0.35.32-alpha (#10461)
Co-authored-by: davinchia <davinchia@users.noreply.github.com>
2022-02-19 02:34:16 +08:00
andriikorotkov
9d401fc9f7 🎉 Updated timestamp transformation with microseconds (#10242)
* updated timestamp transformation with microseconds

* updated timestamp transformation with microseconds

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql tests

* removed extra logs

* fixed cursor for timestamp

* updated test

* updated type transformation

* updated type transformation

* updated type transformation

* updated tests

* updated oracle tests

* updated documentations and connectors versions

* updated documentations and connectors versions

* fix code style
2022-02-18 14:51:58 +02:00
Octavia Squidington III
6bc9240c9c Bump Airbyte version from 0.35.30-alpha to 0.35.31-alpha (#10446)
Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>
2022-02-17 16:11:39 -08:00
LiRen Tu
049a11b2bc 🎉 Snowflake destination: reduce memory footprint (#10394)
* Add detailed logging for flushing

* Log sentry transaction event id

* Adjust logging

* Log memory usage

* Add jvm monitoring

* Remove log

* Remove port 9010

* Remove host network mode

* Sample record size

* Remove profiling code

* Add unit tests

* Use average estimation

* Rename variable

* Format code

* Bump version

* Revert unnecessary change

* Update doc

* Fix format

* Bump version in seed
2022-02-17 12:55:35 -08:00
VitaliiMaltsev
c9bd5e9bbd Destination Snowflake Execute COPY in parallel (#10212)
* fix for jdk 17

* add parallel chunk copy S3

* add parallel chunk copy GCS

* fixed checkstyle

* refactoring

* add unit tests

* updated CHANGELOG

* fixed S3 bucket path generation

* refactoring

* refactoring

* fixed compilation error after merge

* add multitheading into S3 and GCS stream copiers

* fixed checkstyle

* fixed checkstyle

* update parallel copy with CompletableFuture

* refactoring

* add javadoc

* bump version

* update destination_specs.yaml

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
2022-02-17 21:16:09 +02:00
girarda
3db3e88187 🎉 Destination MySQL: Add jdbc_url_params support for optional JDBC parameters (#10362)
* pass through jdbc params

* fail if contains verifyServerCertificate

* do the same for all ssl params

* delete dead file

* slight refactor

* new method

* remove default value

* error message

* rename

* update as per comments

* Update exception message

* Bump version

* extract to method

* Update doc

* Revert "Update doc"

This reverts commit 097906fe64.

* Update doc

* delete dead code

* update doc

* Throw exception with better error message

* Add missing test

* Use MoreMaps::merge

* Add missing tests

* camel case

* Allow colliding parameters if values are equal

* Remove trailing &

* Throw IllegalArgumentException

* extract to constants

* Bump version in seed

* Update destination specs
2022-02-17 10:58:50 -08:00
Serhii Chvaliuk
3eb3367cc3 🐛 Source Quickbooks: update label Quickbooks -> QuickBooks (#10346)
* Quickbooks -> QuickBooks

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-02-17 16:52:20 +02:00
Eugene Kulak
a3aae8017a 🎉 🎉 Source FB Marketing: performance and reliability fixes (#9805)
* Facebook Marketing performance improvement

* add comments and little refactoring

* fix integration tests with the new config

* improve job status handling, limit concurrency to 10

* fix campaign jobs, refactor manager

* big refactoring of async jobs, support random order of slices

* update source _read_incremental to hook new state logic

* fix issues with timeout

* remove debugging and clean up, improve retry logic

* merge changes from #8234

* fix call super _read_increment

* generalize batch execution, add use_batch flag

* improve coverage, do some refactoring of spec

* update test, remove overrides of source

* add split by AdSet

* add smaller insights

* fix end_date < start_date case

* add account_id to PK

* add notes

* fix new streams

* fix reversed incremental stream

* update spec.json for SAT

* upgrade CDK and bump version

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2022-02-17 08:35:05 +04:00
Eugene Kulak
d173ce741f CDK: add support for streams with state attribute (#9746)
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2022-02-17 00:20:33 +04:00
Hitesh Khandelwal
fd6aa69084 Source/Destination Kafka: added scram-sha-512 authentication (#10186)
* kafka, added scram-sha-512 authentication

* bump connectors version

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2022-02-16 13:08:57 -03:00
Nikolai Korolev
22810c3937 Source Clickhouse: Fix exception in case password field is not provided in configuration (#10214)
* Fix exception in case password is not provided for Clickhouse source

Exception while checking connection:
java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.asText()" because the return value of "com.fasterxml.jackson.databind.JsonNode.get(String)" is null

* Update changelog

* Bump version for Source Clickhouse

* Bump version of source-clickhouse-strict-encrypt

* Update version in docs

* correct conflict in docs

* bump connector version

* correct doc

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2022-02-16 13:06:20 -03:00
augan-rymkhan
741e672abf Source Google Ads: make streams incremental (#10315)
* make streams as incremental

* refactor Source

* fix typing

* fix primary_key

* fix primary key in configured_catalog

* bump the version

* fix primary key in catalog

* updated spec and def yaml

Co-authored-by: auganbay <auganenu@gmail.com>
2022-02-16 21:27:42 +06:00
Subodh Kant Chaturvedi
531ed1af28 update Snowflake destination docs with more info (#10213)
* update Snowflake destination docs with more info

* more edits

* address review comments
2022-02-16 16:30:21 +05:30
LiRen Tu
c27e2a0766 🐞 Snowflake destination: use pooled connections (#10342)
* Use data source conn supplier for snowflake database

* Format code

* Reuse the same database in integration tests

* Close query stream

* Refactor snowflake staging sql operations

* Close result set

* Add annotations

* Bump version

* Bump version in seed
2022-02-15 20:29:52 -08:00
Marcos Marx
7c93ef1a57 Source Chargebee: new stream coupon correct formating (#10329)
* added coupon stream

* correct small errors

* bump connector version

* format json file

* run format

Co-authored-by: aadityasinha-dotcom <aadityasinha009@gmail.com>
2022-02-15 20:02:57 -03:00
Aaditya Sinha
1c89320cc3 Source Chargebee: added coupon (#10269)
* added coupon stream

* changes

* Update airbyte-integrations/connectors/source-chargebee/source_chargebee/streams.py

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2022-02-15 19:54:15 -03:00
Octavia Squidington III
5054af0d96 Bump Airbyte version from 0.35.29-alpha to 0.35.30-alpha (#10358)
Co-authored-by: pmossman <pmossman@users.noreply.github.com>
2022-02-15 13:10:55 -08:00
andriikorotkov
b3916c987a 🐛 Snowflake Destination: use better file size with S3 staging files (#9920)
* split s3 staging files to files by 100 Mb and removed legacyS3StreamCopier

* split s3 staging files to files by 100 Mb and removed legacyS3StreamCopier

* updated code style

* fix remarks

* fix remarks

* fix code style

* fix remarks

* fix remarks

* fix remarks

* updated documentations and images versions

* updated documentation
2022-02-15 22:20:22 +02:00
Vladimir remar
fb647ad1e1 :tada:Source hubspot: add associations to tickets stream. (#10142)
* update: client add associations to tickets stream

* update: tickets json schema add deals and contacts

* bump connector version

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2022-02-15 16:08:24 -03:00
andriikorotkov
838ce1418f 🐛 Prevented double slashes in S3 destination path (#10318)
* Prevented double slashes in S3 destination path

* added test and update documentation

* updated documentation
2022-02-15 11:38:59 +02:00
LiRen Tu
6301cfa91f 🎉 Destination snowflake: reduce memory consumption (#10297)
* Avoid redundant adapter construction

* Remove unused logger

* Avoid redundant creation of buffer map

* Decrease max batch byte size to 128 mb

* Format code

* Move data adapter to an instance variable

* Bump version

* Bump version in seed
2022-02-14 23:37:54 -08:00
Octavia Squidington III
113dc993e9 Bump Airbyte version from 0.35.28-alpha to 0.35.29-alpha (#10324)
Co-authored-by: cgardens <cgardens@users.noreply.github.com>
2022-02-14 21:02:37 -08:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00
midavadim
ef4221784e 🎉 Source GitHub - added more meaningful error description for connector check (#10211)
* added human-readable errors in case of incorrect organization or repo names

* added change history

* added change history

* bumped github image version

* bumped github image version in source_specs.yaml file
2022-02-14 15:02:40 +02:00
LiRen Tu
28916dcda0 🎉 Destination GCS: fix connection check (#10299)
* Use stream transfer manager in check command

* Refactor code

* Move the check method to s3 destination

* Bump versions

* Revert s3 version bump

* Bump gcs version in seed
2022-02-12 16:30:02 -08:00
LiRen Tu
39049cbf24 🎉 End-to-end test source: support stream duplication (#10298)
* Add support to duplicate one stream multiple times

* Update cloud version

* Bump version

* Remove unused logger

* Add trace for each stream in mock source

* Fix cloud version

* Bump version in seed
2022-02-12 00:50:48 -08:00
Octavia Squidington III
3caeee7892 Bump Airbyte version from 0.35.27-alpha to 0.35.28-alpha (#10292)
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2022-02-11 16:57:29 -08:00
ron-damon
df5170ad20 🐛 Source BigCommerce: Fix date-time columns for orders (V2 stream) (#9935)
* fix datetime columns for orders

* bump connector's version

* add changelog entry

* set PR

* remove unused import

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2022-02-11 15:23:08 -03:00
Thanh Le
833690f757 Change streamr docs guide (#10267)
* Add streamr document

* add missing end line

* fix name: streamr > Streamr

* Update document

Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
2022-02-11 14:22:48 -03:00
Andrii Leonets
5795d1386d 🐛 Source MySQL CDC: sync failed when has Zero-date value in mandatory column (#10251)
* replace null to epoch if column is mandatory

* format

* add tests

* incr mysql encrypt version

* bump version

* bump version

* fix changelog
2022-02-11 17:53:41 +02:00
Yevhenii
7e4ec325f2 🎉 Source Google Ads: Implement multiple customer ids for google ads (#10150)
* Source Google Ads: Implement multiple Customer ID(s)

* Source Google Ads: update states
2022-02-11 17:42:38 +02:00
Joël Luijmes
25760e48e7 🎉 Source Chargebee: add event model stream(#10143)
* feat(chargebee): add event model

* fix(airbyte): update version Dockerfile

* docs(chargebee): update changelog

* bump connector version

* correct source seed

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2022-02-10 23:37:38 -03:00
Maksym Pavlenok
5bc8ec205e 🐛 Source salesforce: processing of failed jobs (#10141) 2022-02-10 19:49:08 +02:00
mkhokh-33
b230543ba2 MySQL CDC sync fails because starting binlog position not found in DB #6425 (#9514)
* Check binlog position on mysql server before run sync job, add error description into log

* fix MySqlStrictEncryptSourceAcceptanceTest

* fix formatting

* fix review comments

* added java docs and fixed few minor comments

* fix formatting

* update versions

* update source_specs.yaml
2022-02-10 16:41:52 +02:00
augan-rymkhan
b22efc03a1 Source Hubspot: fix "quotes" key error exception (#10055)
* check if stream exists in source

* check if stream exists in source, added comment

* test skipping reading quotes stream

* format code

* airbyte-cdk version

* added __init__.py to unit_tests

* fix importing airbyte models

* bump the version

* update spec and def yamls

Co-authored-by: auganbay <auganenu@gmail.com>
2022-02-10 10:39:50 +06:00
Octavia Squidington III
1ed0cc0243 Bump Airbyte version from 0.35.26-alpha to 0.35.27-alpha (#10234)
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2022-02-09 15:39:11 -08:00