1
0
mirror of synced 2026-01-10 09:04:48 -05:00
Commit Graph

1144 Commits

Author SHA1 Message Date
Vladimir remar
ccb3fe2a3d 🎉 Source TikTok Marketing: Add Audience reports (#10398)
* Update: add new streams(campaings by country, ads, ad_group, advertiser) for audience reports

* update: bump connector version
2022-02-21 18:56:47 -03:00
Baz
67b7fc96ad 🎉 Source Shopify: add tender_transactions stream (#10449) 2022-02-21 20:22:02 +02:00
Kshitij Grover
1e0ac30ebd 🎉 New Source: Orb (#9985)
* V1 of source_orb connector

* add boostrap.md file

* add clause on Pagination to bootstrap.md

* add SUMMARY documentation

* add lookback_window_days connector parameter

* Add support for start_date parameter

* Add ability to transform record in order to un-nest IDs

* Add support for extracting event properties based on connector configuration
2022-02-21 14:14:10 -03:00
augan-rymkhan
2282a4ae02 🎉 Source Hubspot: Migrate to CDK (#10177)
* migrate SourceHubspot to cdk

* refactor discover method

* change method name

* deleted Client class

* remove comment

* added get_updated_state

* fix setting initial state

* fix stream_state dict key

* fix cursor_field

* change check test case status

* refactor streams method

* remove comment

* remove TODOs

* remove comments

* fix get_updated_state

* refactor chunk_read

* override _read_incremental

* fix unit tests

* remove comments

* fix test_check_connection_backoff_on_server_error

* fix test_check_connection_backoff_on_server_error 2

* fix test_check_connection_backoff_on_limit_reached

* fix unit tests

* clear comments

* override read method on Source

* added comments to overriding methods

* some improvements

* reafactor overridden _read_incremental

* format code

* refactor discovery

* remove discover

* format code 2

* added return types

* refactor template stream classes

* remove comments

* remove _name field

* rename api.py to streams.py

* move to HttpStream

* refactor FormSubmissions

* refactor Campaings

* refactor ContactsListMemberships

* CRMSearchStream refactor

* CRMSearchStream refactor 2

* CRMObjectStream refactor

* DealStageHistoryStream refactor

* Deals refactor

* Engagements refactor

* path method refactor

* refactor authentication

* fix check_connection

* fix call parse_response

* fix Engagements stream

* fix CRMSearchStream

* fix CRMObjectIncremental stream

* override _read_incremental

* remove commented codes

* format code

* update cdk version

* fix cursor field

* fix unit tests

* removed client

* clear comments

* clear comments 2

* clear comments 3

* clear comments 4

* override backoff_time

* remove comment

* format code

* backoff_time modified

* refactor backoff_time

* format code

* added return typing

* format code

* removed cursor_paths

* bump version

* updated spec and def yaml

Co-authored-by: auganbay <auganenu@gmail.com>
2022-02-21 22:03:05 +06:00
Anna Lvova
ce5c007540 🎉 Salesforce Source: Support "Replicate Incremental Deletes" (#10454) 2022-02-21 15:03:50 +02:00
andriikorotkov
9d401fc9f7 🎉 Updated timestamp transformation with microseconds (#10242)
* updated timestamp transformation with microseconds

* updated timestamp transformation with microseconds

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql tests

* removed extra logs

* fixed cursor for timestamp

* updated test

* updated type transformation

* updated type transformation

* updated type transformation

* updated tests

* updated oracle tests

* updated documentations and connectors versions

* updated documentations and connectors versions

* fix code style
2022-02-18 14:51:58 +02:00
LiRen Tu
049a11b2bc 🎉 Snowflake destination: reduce memory footprint (#10394)
* Add detailed logging for flushing

* Log sentry transaction event id

* Adjust logging

* Log memory usage

* Add jvm monitoring

* Remove log

* Remove port 9010

* Remove host network mode

* Sample record size

* Remove profiling code

* Add unit tests

* Use average estimation

* Rename variable

* Format code

* Bump version

* Revert unnecessary change

* Update doc

* Fix format

* Bump version in seed
2022-02-17 12:55:35 -08:00
VitaliiMaltsev
c9bd5e9bbd Destination Snowflake Execute COPY in parallel (#10212)
* fix for jdk 17

* add parallel chunk copy S3

* add parallel chunk copy GCS

* fixed checkstyle

* refactoring

* add unit tests

* updated CHANGELOG

* fixed S3 bucket path generation

* refactoring

* refactoring

* fixed compilation error after merge

* add multitheading into S3 and GCS stream copiers

* fixed checkstyle

* fixed checkstyle

* update parallel copy with CompletableFuture

* refactoring

* add javadoc

* bump version

* update destination_specs.yaml

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
2022-02-17 21:16:09 +02:00
girarda
3db3e88187 🎉 Destination MySQL: Add jdbc_url_params support for optional JDBC parameters (#10362)
* pass through jdbc params

* fail if contains verifyServerCertificate

* do the same for all ssl params

* delete dead file

* slight refactor

* new method

* remove default value

* error message

* rename

* update as per comments

* Update exception message

* Bump version

* extract to method

* Update doc

* Revert "Update doc"

This reverts commit 097906fe64.

* Update doc

* delete dead code

* update doc

* Throw exception with better error message

* Add missing test

* Use MoreMaps::merge

* Add missing tests

* camel case

* Allow colliding parameters if values are equal

* Remove trailing &

* Throw IllegalArgumentException

* extract to constants

* Bump version in seed

* Update destination specs
2022-02-17 10:58:50 -08:00
Serhii Chvaliuk
3eb3367cc3 🐛 Source Quickbooks: update label Quickbooks -> QuickBooks (#10346)
* Quickbooks -> QuickBooks

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-02-17 16:52:20 +02:00
Eugene Kulak
a3aae8017a 🎉 🎉 Source FB Marketing: performance and reliability fixes (#9805)
* Facebook Marketing performance improvement

* add comments and little refactoring

* fix integration tests with the new config

* improve job status handling, limit concurrency to 10

* fix campaign jobs, refactor manager

* big refactoring of async jobs, support random order of slices

* update source _read_incremental to hook new state logic

* fix issues with timeout

* remove debugging and clean up, improve retry logic

* merge changes from #8234

* fix call super _read_increment

* generalize batch execution, add use_batch flag

* improve coverage, do some refactoring of spec

* update test, remove overrides of source

* add split by AdSet

* add smaller insights

* fix end_date < start_date case

* add account_id to PK

* add notes

* fix new streams

* fix reversed incremental stream

* update spec.json for SAT

* upgrade CDK and bump version

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2022-02-17 08:35:05 +04:00
Hitesh Khandelwal
fd6aa69084 Source/Destination Kafka: added scram-sha-512 authentication (#10186)
* kafka, added scram-sha-512 authentication

* bump connectors version

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2022-02-16 13:08:57 -03:00
Nikolai Korolev
22810c3937 Source Clickhouse: Fix exception in case password field is not provided in configuration (#10214)
* Fix exception in case password is not provided for Clickhouse source

Exception while checking connection:
java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.asText()" because the return value of "com.fasterxml.jackson.databind.JsonNode.get(String)" is null

* Update changelog

* Bump version for Source Clickhouse

* Bump version of source-clickhouse-strict-encrypt

* Update version in docs

* correct conflict in docs

* bump connector version

* correct doc

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2022-02-16 13:06:20 -03:00
augan-rymkhan
741e672abf Source Google Ads: make streams incremental (#10315)
* make streams as incremental

* refactor Source

* fix typing

* fix primary_key

* fix primary key in configured_catalog

* bump the version

* fix primary key in catalog

* updated spec and def yaml

Co-authored-by: auganbay <auganenu@gmail.com>
2022-02-16 21:27:42 +06:00
Subodh Kant Chaturvedi
531ed1af28 update Snowflake destination docs with more info (#10213)
* update Snowflake destination docs with more info

* more edits

* address review comments
2022-02-16 16:30:21 +05:30
LiRen Tu
c27e2a0766 🐞 Snowflake destination: use pooled connections (#10342)
* Use data source conn supplier for snowflake database

* Format code

* Reuse the same database in integration tests

* Close query stream

* Refactor snowflake staging sql operations

* Close result set

* Add annotations

* Bump version

* Bump version in seed
2022-02-15 20:29:52 -08:00
Marcos Marx
7c93ef1a57 Source Chargebee: new stream coupon correct formating (#10329)
* added coupon stream

* correct small errors

* bump connector version

* format json file

* run format

Co-authored-by: aadityasinha-dotcom <aadityasinha009@gmail.com>
2022-02-15 20:02:57 -03:00
Aaditya Sinha
1c89320cc3 Source Chargebee: added coupon (#10269)
* added coupon stream

* changes

* Update airbyte-integrations/connectors/source-chargebee/source_chargebee/streams.py

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2022-02-15 19:54:15 -03:00
andriikorotkov
b3916c987a 🐛 Snowflake Destination: use better file size with S3 staging files (#9920)
* split s3 staging files to files by 100 Mb and removed legacyS3StreamCopier

* split s3 staging files to files by 100 Mb and removed legacyS3StreamCopier

* updated code style

* fix remarks

* fix remarks

* fix code style

* fix remarks

* fix remarks

* fix remarks

* updated documentations and images versions

* updated documentation
2022-02-15 22:20:22 +02:00
Vladimir remar
fb647ad1e1 :tada:Source hubspot: add associations to tickets stream. (#10142)
* update: client add associations to tickets stream

* update: tickets json schema add deals and contacts

* bump connector version

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2022-02-15 16:08:24 -03:00
andriikorotkov
838ce1418f 🐛 Prevented double slashes in S3 destination path (#10318)
* Prevented double slashes in S3 destination path

* added test and update documentation

* updated documentation
2022-02-15 11:38:59 +02:00
LiRen Tu
6301cfa91f 🎉 Destination snowflake: reduce memory consumption (#10297)
* Avoid redundant adapter construction

* Remove unused logger

* Avoid redundant creation of buffer map

* Decrease max batch byte size to 128 mb

* Format code

* Move data adapter to an instance variable

* Bump version

* Bump version in seed
2022-02-14 23:37:54 -08:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00
midavadim
ef4221784e 🎉 Source GitHub - added more meaningful error description for connector check (#10211)
* added human-readable errors in case of incorrect organization or repo names

* added change history

* added change history

* bumped github image version

* bumped github image version in source_specs.yaml file
2022-02-14 15:02:40 +02:00
LiRen Tu
28916dcda0 🎉 Destination GCS: fix connection check (#10299)
* Use stream transfer manager in check command

* Refactor code

* Move the check method to s3 destination

* Bump versions

* Revert s3 version bump

* Bump gcs version in seed
2022-02-12 16:30:02 -08:00
LiRen Tu
39049cbf24 🎉 End-to-end test source: support stream duplication (#10298)
* Add support to duplicate one stream multiple times

* Update cloud version

* Bump version

* Remove unused logger

* Add trace for each stream in mock source

* Fix cloud version

* Bump version in seed
2022-02-12 00:50:48 -08:00
ron-damon
df5170ad20 🐛 Source BigCommerce: Fix date-time columns for orders (V2 stream) (#9935)
* fix datetime columns for orders

* bump connector's version

* add changelog entry

* set PR

* remove unused import

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2022-02-11 15:23:08 -03:00
Thanh Le
833690f757 Change streamr docs guide (#10267)
* Add streamr document

* add missing end line

* fix name: streamr > Streamr

* Update document

Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
2022-02-11 14:22:48 -03:00
Andrii Leonets
5795d1386d 🐛 Source MySQL CDC: sync failed when has Zero-date value in mandatory column (#10251)
* replace null to epoch if column is mandatory

* format

* add tests

* incr mysql encrypt version

* bump version

* bump version

* fix changelog
2022-02-11 17:53:41 +02:00
Yevhenii
7e4ec325f2 🎉 Source Google Ads: Implement multiple customer ids for google ads (#10150)
* Source Google Ads: Implement multiple Customer ID(s)

* Source Google Ads: update states
2022-02-11 17:42:38 +02:00
Joël Luijmes
25760e48e7 🎉 Source Chargebee: add event model stream(#10143)
* feat(chargebee): add event model

* fix(airbyte): update version Dockerfile

* docs(chargebee): update changelog

* bump connector version

* correct source seed

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2022-02-10 23:37:38 -03:00
Maksym Pavlenok
5bc8ec205e 🐛 Source salesforce: processing of failed jobs (#10141) 2022-02-10 19:49:08 +02:00
mkhokh-33
b230543ba2 MySQL CDC sync fails because starting binlog position not found in DB #6425 (#9514)
* Check binlog position on mysql server before run sync job, add error description into log

* fix MySqlStrictEncryptSourceAcceptanceTest

* fix formatting

* fix review comments

* added java docs and fixed few minor comments

* fix formatting

* update versions

* update source_specs.yaml
2022-02-10 16:41:52 +02:00
augan-rymkhan
b22efc03a1 Source Hubspot: fix "quotes" key error exception (#10055)
* check if stream exists in source

* check if stream exists in source, added comment

* test skipping reading quotes stream

* format code

* airbyte-cdk version

* added __init__.py to unit_tests

* fix importing airbyte models

* bump the version

* update spec and def yamls

Co-authored-by: auganbay <auganenu@gmail.com>
2022-02-10 10:39:50 +06:00
Serhii Chvaliuk
4aecd3fe6d 🎉 Source Stripe:: improve 404 handling for CheckoutSessionsLineItems stream (#10165)
* Improve 404 handling for CheckoutSessionsLineItems stream

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-02-09 21:53:18 +02:00
oneshcheret
84d1456fcf JDBC sources: update for sources that not have permissions issues (#10173)
* JDBC sources: update for sources that not have permissions issues

* Postgres, CockroachDB and DB2 sources: updated case when user has none permitted table

* Postgres, CockroachDB and DB2 sources: updated case when user has none permitted table

* Postgres, CockroachDB and DB2 sources: bump versions

* Postgres, CockroachDB and DB2 sources: bump source specs

* Postgres, CockroachDB and DB2 sources: revert changes for postgres-strict-encrypt
2022-02-09 19:23:23 +02:00
Yevhenii
f01b0c769f 🎉 Source Github: add handling the secondary rate limits (#10193)
* Source Github: add handling the secondary rate limits
2022-02-09 15:26:53 +02:00
Serhii Lazebnyi
2573fa145a 🎉 Source Facebook Marketing: Add AdAccount and Images stream implementation (#10180)
* Add AdAccount and Images stream implementation

* Update PR number

* Updated docker version

* Updated to linter

* Update to review

* Add comment to AdAccount read_records method

* Bumped version in seed, definitions and specs files
2022-02-09 15:26:19 +02:00
Anna Lvova
8749c7ab2c 🐛Source Github: remove BAD_GATEWAY code from backoff_time (#9999)
* remove BAD_GATEWAY code from backoff_time

* bump version

* bump version

* add unit test

* bump version
2022-02-09 12:10:16 +02:00
Baz
26934802a0 🐛 Source Shopify: fixed compatibility issues for legacy user config (#10175) 2022-02-08 16:37:28 +02:00
oleh.zorenko
66137f6454 🎉 Source Facebook Marketing: add primary_key for all insights streams (#10138)
* 6557 Add `primary_key` for all insights streams

* 6557 Update docs

* 6557 Fix integration_tests

* 6557 Bump connector's version
2022-02-07 10:59:04 +02:00
Vladimir remar
983d6ae2ba :tada:Source hubspot add new streams for engagements(calls, emails, meetings, notes and tasks) (#9786) 2022-02-04 20:09:29 +01:00
Serhii Chvaliuk
84d7323d43 🐛 Source Salesforce: increase CSV field_size_limit (#10012)
* Increase CSV field_size_limit

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-02-04 21:06:33 +02:00
augan-rymkhan
359fcd8011 Source Google Ads: handle page token expired exception (#9812)
* dynamic date range

* raise exception if exites the cycle without error

* if range days is 1 already do not retry

* added unit tests

* added comments

* added comments

* common mock classes are moved to common module

* change read_records

* refactored get_date_params

* handle corner case

* added parse_dates function

* added test_streams

* check mock calls

* fix unit tests for chunk date range refactoring

* removed commented codes

* remove commented line

* refactor test_streams

* refactor CustomQuery.get_query

* remove TODO

* deleted unused json

* format

* fix chunk_date_range

* added docstring

* set range_days to 15 for ShoppingPerformanceReport

* refactor chunk_date_range

* format code 2

* call parent read_records method

* add return type in get_date_params

* change e to exception

* set start_date as end_date

* log page token has expired

* bump version

* updated spec and def yaml

Co-authored-by: auganbay <auganenu@gmail.com>
2022-02-04 22:48:58 +06:00
midavadim
9e6da4692f 🎉 Source smartsheets: oauth support (#9792)
* Added SAT, fixed tests, fixed schema validation error

* Added Oauth support for smartsheets source

* minor changes after review

* added advanced_auth property instead of authSpecification

* Updated connector version

* removed empty unittests

* updated version in source_specs.yaml
2022-02-04 17:57:45 +02:00
augan-rymkhan
d7b9ae2918 Source Google Ads: upgrade google-ads package version (#9996)
* upgrade google-ads package version

* add comment about use_proto_plus

* bump version

* updated spec and def yaml

Co-authored-by: auganbay <auganenu@gmail.com>
2022-02-04 11:28:13 +06:00
Serhii Chvaliuk
f0546ed84f Update fields in source-connectors specifications: recharge (#9808)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-02-03 15:07:38 +02:00
LiRen Tu
dc20baf9d7 Remove redundant github issue links in postgres doc (#10027)
* Remove redundant github issue links

* Remove redundant word

* Update wording

* Update wording
2022-02-02 23:46:42 -08:00
LiRen Tu
fa6ebdf3e3 📝 Source postgres: add workaround to sync a subset of columns (#10024)
* Add workaround for partial table permission issue in doc

* Update doc

* Update doc

* Update doc

* Remove dedundant sentence

* Update doc
2022-02-02 21:36:02 -08:00
Harshith Mullapudi
7428f7c89b chore: add confluence and firestore to definition yamls (#9831)
* chore: add confluence and firestore to definition yamls

* fix: push spec to cache

* chore: added specs for confluence and firestore
2022-02-02 23:57:15 +05:30