1
0
mirror of synced 2025-12-30 12:04:43 -05:00
Commit Graph

4969 Commits

Author SHA1 Message Date
Edward Gao
9f6963ccfc Destination Bigquery 1s1t: handle cursor change (#28721)
* handle new cursor column

* sync2 is actualy weird, apparently

* logistics

* Automated Commit - Format and Process Resources Changes

---------

Co-authored-by: edgao <edgao@users.noreply.github.com>
2023-07-26 18:36:15 -06:00
Edward Gao
d9f2444c9e 1s1t: fix timestamps in T+D tests (#28749)
* fix timestamps in T+D tests

* lmao these tests have been wrong the entire time
2023-07-26 18:09:16 -05:00
Edward Gao
1aea368ce2 1s1t: move expected records into destination-bigquery (#28752) 2023-07-26 16:16:01 -06:00
Catherine Noll
dbb08ccfa5 Revert "pin stripe version in cloud to 3.13.0 (#28694)" (#28722)
This reverts commit 860e6af933.
2023-07-26 15:06:32 -04:00
Joe Reuter
94e8ef8b1e Langchain destination: Add Chroma support (#28605)
* add chroma support

* prepare release

* prepare release

* adjust docs

* adjust docs

* normalize metadata for Chroma
2023-07-26 19:29:15 +02:00
Jonathan Pearlin
3eb222fc89 Source MongoDB V2: Clean fieldPath before use in projection (#28733)
* Clean fieldPath before use in projection

* Update release notes

* Automated Commit - Format and Process Resources Changes

* Remove prefix $ from field name

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
2023-07-26 12:11:46 -05:00
Ben Church
fb7258e2bd Move tools/ci_* projects to airbyte-ci, update to use Poetry, bump to python 3.10 (#27957)
* Move ci_connector_ops

* Move ci_credentials

* Move tools/ci_common_utils

* Rename tools to airbyte-ci

* Move to ci

* Convert ci_credentials

* Convert ci_common_utls

* Convert ci_connector_ops

* Get pipelines running

* Move pipelines to own poetry project

* Update readme

* Delete

* Add ci_code_validator

* Use pipx to install gha deps

* Fix'

* Ensure every thing is running

* Automated Commit - Formatting Changes

* Gitignore miss

* Add pipx installer

* Get local pipx dependencies

* Fix paths

* Install pipx

* ceremonial source-faker change

* Add installation step for ci_code_validator

* Add comment

* remove ci_code_validator

* Address code review comments

* add pipx install to acceptance-test-docker.sh

* Run formater

* Revert "ceremonial source-faker change"

This reverts commit 26884cd0db.

* gitignore lecacy pipeline report path

* update poetry.lock

* skip upload if logs do not exist

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
2023-07-26 15:49:59 +00:00
Baz
343a45c3b7 🐛 Source Shopify: improve error messages, refactored code (#28700) 2023-07-26 15:44:52 +03:00
Serhii Lazebnyi
77251e445a 🐛 Source Hubspot: Add backoff for handler request (#28705)
* Add backoff for handler request

* Fixed code formating and expected records
2023-07-26 14:06:58 +02:00
Subodh Kant Chaturvedi
bcb5a4df8d mysql-cdc: implementation to construct initial debezium state (#28561)
* wip

* wip 2

* add tests + finalise implementation

* undo unwanted changes

* Automated Commit - Format and Process Resources Changes

* minor refactor

* add comment

* make method public

---------

Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
2023-07-26 06:18:23 -05:00
Artem Inzhyyants
efd8bc2745 Source Zendesk Support: add to cloud (#28711) 2023-07-26 01:35:30 +02:00
Catherine Noll
b95def8889 Source Stripe: remove duplicate streams (#28709) 2023-07-25 19:11:47 -04:00
Edward Gao
a8b099c49e cython >.> (#28675) 2023-07-25 17:31:42 -05:00
Rodi Reich Zilberman
fb7cacb321 Pre release ctid (#28339)
* enable ctid feature flag for testing.
bump versions

* Removed PostgresSourceTest pre-ctid and turn CtidEnabledPostgresSourceTest back into PostgresJdbcSourceAcceptanceTest (#28341)

Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>

* Update PostgresSourceTest to no longer use Legacy AirbyteStateType

* Fix test.
Uncovered a bug in handling of money[] and float[]

* revert version change

* Orderly removal of ctid flag for cursor

* Handle correctly full refresh streams. (#28369)

* Handle correctly full refresh streams.
Handle xmin full refresh (currently blocked in ui).
remove unnecessary debug logs

* Test catalog with a mix of incremental and full refresh streams

* Filter out full refresh streams since they are not expected to be retuned by getIncrementalIterators()

* Filter out full refresh existing streams as well

* revert name to reflect incremental only streams

* 28505 xmin + ctid reclassify connection when ctid is not possible (#28548)

* reclassify to xmin stream we cannot read with ctid

* fix reclassify

* improve test

* Add initial load using xmin flow

* typo

* Added test for xmin initial load flow

* Improve testing for initial load with xmin flow

* sanity

* sanity

* bump versions and release notes

---------

Co-authored-by: Duy Nguyen <duy@airbyte.io>
2023-07-25 16:56:29 -05:00
Rodi Reich Zilberman
f4bf4a8292 Revert "source-postgres: enable ctid+cdc implementation (#28044)" (#28707)
This reverts commit 886ab6b09b.
2023-07-25 16:51:04 -05:00
Subodh Kant Chaturvedi
886ab6b09b source-postgres: enable ctid+cdc implementation (#28044)
* source-postgres: enable ctid+cdc implementation

* 100% ctid rollout for cdc
2023-07-25 16:31:27 -05:00
Jonathan Pearlin
28bedf9675 Source MongoDB V2: Handle statistics collection failure gracefully (#28692)
* Fix bug introduced by statistics collection

* Fix syntax

* Formatting

* Fix failing test

* Fix failing test

* Update release notes
2023-07-25 15:22:02 -05:00
Ella Rohm-Ensing
860e6af933 pin stripe version in cloud to 3.13.0 (#28694)
* pin stripe version in cloud to 3.13.0

* add ref
2023-07-25 15:18:41 -05:00
Serhii Lazebnyi
8f92798092 🐛 Source Zendesk Support: Add handle 404 error (#28397)
* Add handle 404 error

* Updated PR number

* Updated version
2023-07-25 19:20:56 +02:00
btkcodedev
66938e1e69 Source Stripe: New Stream - ShippingRates (#27217)
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* Add stream ShippingRates

* Resolve conflicts, Bump version

* Fix unit tests

* Resolve conflict

* Fix unit_tests

* Remove unwanted files

* Run code formattingh

* Fix unit tests

* Fix unit_tests

* Fix unit_tests

* update expected records

* update expected records

* update expected records

* one more update for expected records

---------

Co-authored-by: Denys Davydov <davydov.den18@gmail.com>
2023-07-25 18:07:31 +03:00
Artem Inzhyyants
66b428bbda Source Zendesk Support: fix pagination for stream TicketMetricEvents (#28612)
* Source Zendesk Support: fix pagination for stream TicketMetricEvents

* Source Zendesk Support: update docs

* Source Zendesk Support: fix formatting

* Source Zendesk Support: add test for ticket_metric_events
2023-07-25 16:06:05 +02:00
Elliot T
9e1e198e18 🎉 New Source: Captain Data [Configuration based CDK] (#25230)
* initiate connector

* add doc

* fix schema

* add docs

* fix comments

* add logo

* add finished job filter

* suggested changes

* add metadata.yml

* add definition id

* update schemas

---------

Co-authored-by: Sajarin <sajarindider@gmail.com>
2023-07-25 08:03:09 -05:00
Efim Matytsin
d314dfd0ad 🐛 Source TikTok: fix default filters params - include deleted ad_groups, ad, campaigns to reports (#27910)
* SCNTALS-1949 fix filters for tiktok

* review fixes

* increment tiktok version in metadata file

* tiktok linter fix

* add changelog record

* small fix for changelog

* Added include_deleted param to TikTok source

* Update airbyte-integrations/connectors/source-tiktok-marketing/source_tiktok_marketing/streams.py

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* tiktok source: review fixes

* tiktok source, code format

---------

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2023-07-25 15:47:35 +03:00
Marcos Marx
53aade6333 🎉 Source Amazon Seller Partner: Add GET_FLAT_FILE_ACTIONABLE_ORDER_DATA_SHIPPING and GET_ORDER_REPORT_DATA_SHIPPING streams (#28578)
* add new streams

* update acceptancte test config

* format files

* fix format and update docs

* remove for

* update expected records

* update expected records
2023-07-24 17:01:20 -03:00
Edward Gao
df274b7f40 Destination snowflake: destinations v2 scaffolding (#28584)
* deps

* scaffolding

* logistics

* base-jdbc depends on base-td
2023-07-24 14:58:50 -05:00
Joe Bell
6159452f8f 🐛 Destination BigQuery - Limit Clustering Column Amount (#28625)
* Add limit to the number of clustering columns

* update bq version
2023-07-24 13:48:56 -06:00
Marcos Marx
6b588d04e2 Source Tiktok Marketing: add new fields to ad reports streams (#25423)
* add new fields to ad reports streams

* fix unit test

* fix test and check connection

* fix unit test

* add conversions too

* add additional fields

* fix: format python files with flake

* fix: add cta_conversion, vta_conversion to streams.py

* fix: python format

* fix: updated expected_records for ad_reports stream

---------

Co-authored-by: sajarin <sajarindider@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
2023-07-24 15:44:00 -04:00
Jonathan Pearlin
496678df5f Source MongoDB V2: Record statistics about MongoDB source connection (#28527)
* Record statistics about source database/collection

* Formatting

* Remove unnecessary log message

* Only record statistics during sync

* Emit structured Google log data

* Revert use of System.out.println

* Try using Log4j ObjectMessage

* Revert "Try using Log4j ObjectMessage"

This reverts commit d518c87aa8.

* Prepare release

* Update changelog

* Correct release version
2023-07-24 12:41:40 -05:00
Octavia Squidington III
b252a995d1 🤖 Add Gainsight Px connector to cloud (#28592)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-07-24 09:11:00 -07:00
Catherine Noll
7a18cc1535 Revert "Source Zendesk Support: remove version pin (#28587)" (#28611)
This reverts commit f402044dfb.
2023-07-24 08:37:33 -05:00
Denys Davydov
ccac29af1b 🚨 🚨 Source Amazon Ads: fix schemas (#27868)
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* #1955 source amazon ads: fix stream schemas

* source amazon ads: upd changelog

* source amazon ads: upd CAT config

* upd backwards incompatible version

* update metadata with breakingChanges entry

* fix CATs

* fix md
2023-07-24 16:29:42 +03:00
Roman Yermilov [GL]
7496163998 Source Google Ads: fix timezone (#28510) 2023-07-24 17:21:02 +04:00
Daryna Ishchenko
624bb815d0 increased timeout for incremental tests (#28559) 2023-07-24 14:05:03 +03:00
Daryna Ishchenko
a6cedb80cb increased timeout for discover test (#28560) 2023-07-24 14:03:44 +03:00
Daryna Ishchenko
d757c2ecf5 updated expected records (#28565) 2023-07-24 14:03:08 +03:00
Daryna Ishchenko
3f6b6eee8e updated expected records (#28595) 2023-07-24 14:01:57 +03:00
Daryna Ishchenko
36bb7aa0ee updated expected records (#28596) 2023-07-24 13:59:58 +03:00
Daryna Ishchenko
00ac8f44de fixed records comparison (#28597) 2023-07-24 13:58:59 +03:00
Artem Inzhyyants
f402044dfb Source Zendesk Support: remove version pin (#28587) 2023-07-24 12:13:50 +02:00
Edward Gao
adf8870de0 🐛 Destination bigquery 1s1t: respect dataset location (#28580)
* set dataset location during creation

* logistics
2023-07-21 16:54:45 -06:00
Davin Chia
b9a3c0817e 🐛 Release Snowflake Async State Bug. (#28581)
Release #28342 for Snowflake.
2023-07-21 15:38:54 -07:00
Davin Chia
496854caf4 Fix Async Framework Race Condition. (#28342)
While running the Snowflake certification test, we noticed NPE on this line, indicating the state id queue was empty.

This should never happen as we always keep an 'open' state id with a running counter to associate the next state the source emits to.

I eventually realised this happens because of a race condition in the FlushWorkers, where multiple threads flush the same queue not realising the state id the flushing logic was conducted on, was already flushed by another thread.
2023-07-21 14:41:38 -07:00
Artem Inzhyyants
a7b5a9c6b7 Source Zendesk Support: Fix cursor pagination (#28487)
* Source Zendesk Support: Remove extra page from params

* Source Zendesk Support: bump version

* Source Zendesk Support: Refactor + Fix pagination issues

* Source Zendesk Support: fix object count

* Source Zendesk Support: update timeout for tests

* Source Zendesk Support: update expected_records

* Source Zendesk Support: update expected_records

* Source Zendesk Support: update ignore pagination
2023-07-21 22:06:51 +02:00
Edward Gao
53da5baa7d Destination bigquery 1s1t: fix 1s1t schema change logic; extract TyperDeduper (#28490)
* rename for clarity

* fix cleanup method

* giant commit because I'm irresponsible

* rename constant

* better raw table creation

* fix build?

* move code around

* tweaks

* more code shuffling

* Automated Commit - Format and Process Resources Changes

* add tests

* minor tweak

* remove unimportant methods

* cleanup

* Automated Commit - Format and Process Resources Changes

* derp

* clean up tests

* some more fixes post-merge

* botched merge

* create NoopTyperDeduper

* try and update everything to work?

* tweak comment

* move suffix args to end of list

* fix exception message

* Automated Commit - Format and Process Resources Changes

* add sqlgenerator test for softReset

* only prepare once

* update log message

* do what intellij says

* implement one more test

* less indirection

* Automated Commit - Format and Process Resources Changes

* rename test

* use noop in test

* version bump + changelog

* use stringutils

* fix typo

* flip if-statement

* typo

* simplify logic

* fix schema change logic

* typo

* use spy for clarity

* Automated Commit - Format and Process Resources Changes

* better test teardown

* slightly better logs

* fix exception message

* softReset returns single string

* Automated Commit - Format and Process Resources Changes

* simplify if chain

---------

Co-authored-by: edgao <edgao@users.noreply.github.com>
2023-07-21 13:45:19 -06:00
Maxime Carbonneau-Leclerc
2464106459 Version bump to release CATs (#28577) 2023-07-21 15:12:25 -04:00
Ben Church
da8596f907 Remove invalid public field from metadata.yaml (#28572) 2023-07-21 11:31:19 -06:00
Joe Reuter
6540fa7c91 Langchain destination: Support composite and nested primary keys for deduping (#28556)
* support composite primary keys for deduping

* prepare release

* format

* fix

---------

Co-authored-by: Augustin <augustin@airbyte.io>
2023-07-21 17:17:29 +02:00
Maxime Carbonneau-Leclerc
675175a50e Allow for sources without spec.json or spec.yaml (#28519)
* Allow for sources without spec.json or spec.yaml

* Automated Commit - Format and Process Resources Changes
2023-07-21 08:32:32 -04:00
Daryna Ishchenko
f6a294712d :bug:Source PostHog: fixed EventsSimpleRetriever declaration (#28461)
* fixed EventsSimpleRetriever declaration

* updated changelog
2023-07-21 14:32:57 +03:00
Daryna Ishchenko
b3d223e24e updated expected records (#28525) 2023-07-21 14:30:18 +03:00