jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-30 03:02:21 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,968 Commits 5,123 Branches 518 Tags
9f6963ccfcddd7a573c4ca0ff94de41ce6175cc0
Commit Graph

11968 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Edward Gao
9f6963ccfc ✨Destination Bigquery 1s1t: handle cursor change (#28721) 
* handle new cursor column

* sync2 is actualy weird, apparently

* logistics

* Automated Commit - Format and Process Resources Changes

---------

Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-07-26 18:36:15 -06:00
Edward Gao
d9f2444c9e 1s1t: fix timestamps in T+D tests (#28749) 
* fix timestamps in T+D tests

* lmao these tests have been wrong the entire time
 2023-07-26 18:09:16 -05:00
Ben Church
5c9f0fa34e Dagster: Add a schedule and auto materialization to handle resource limits (#28190) 
* Initial schedule

* Add dynamic partition

* Add auto materialization policy

* Add partition add job

* Add logging and format

* Add comment

* Add MAX_METADATA_PARTITION_RUN_REQUEST to config
 2023-07-26 15:36:53 -07:00
Edward Gao
1aea368ce2 1s1t: move expected records into destination-bigquery (#28752) 2023-07-26 16:16:01 -06:00
Catherine Noll
dbb08ccfa5 Revert "pin stripe version in cloud to 3.13.0 (#28694)" (#28722) 
This reverts commit 860e6af933.
 2023-07-26 15:06:32 -04:00
Pedro S. Lopez
f1ffa2e148 rename method (#28741) 2023-07-26 14:31:08 -04:00
Joe Reuter
94e8ef8b1e Langchain destination: Add Chroma support (#28605) 
* add chroma support

* prepare release

* prepare release

* adjust docs

* adjust docs

* normalize metadata for Chroma
 2023-07-26 19:29:15 +02:00
Jonathan Pearlin
3eb222fc89 ✨ Source MongoDB V2: Clean fieldPath before use in projection (#28733) 
* Clean fieldPath before use in projection

* Update release notes

* Automated Commit - Format and Process Resources Changes

* Remove prefix $ from field name

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
 2023-07-26 12:11:46 -05:00
Pedro S. Lopez
f753dd04b6 fix: publish strict-encrypt connectors before regular variant (#28737) 
* sort strict encrypt specially

* add secure variant handling
 2023-07-26 13:10:43 -04:00
Ben Church
fb7258e2bd Move tools/ci_* projects to airbyte-ci, update to use Poetry, bump to python 3.10 (#27957) 
* Move ci_connector_ops

* Move ci_credentials

* Move tools/ci_common_utils

* Rename tools to airbyte-ci

* Move to ci

* Convert ci_credentials

* Convert ci_common_utls

* Convert ci_connector_ops

* Get pipelines running

* Move pipelines to own poetry project

* Update readme

* Delete

* Add ci_code_validator

* Use pipx to install gha deps

* Fix'

* Ensure every thing is running

* Automated Commit - Formatting Changes

* Gitignore miss

* Add pipx installer

* Get local pipx dependencies

* Fix paths

* Install pipx

* ceremonial source-faker change

* Add installation step for ci_code_validator

* Add comment

* remove ci_code_validator

* Address code review comments

* add pipx install to acceptance-test-docker.sh

* Run formater

* Revert "ceremonial source-faker change"

This reverts commit 26884cd0db.

* gitignore lecacy pipeline report path

* update poetry.lock

* skip upload if logs do not exist

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
 2023-07-26 15:49:59 +00:00
Marcos Marx
bdaddd4c20 Docs: update PR template and remove python destination as accepted contributions (#28665) 
* update PR template and remove python destination as accepted contributions

* fix warning callout

* improve speeling
 2023-07-26 10:09:53 -03:00
Baz
343a45c3b7 🐛 Source Shopify: improve error messages, refactored code (#28700) 2023-07-26 15:44:52 +03:00
Serhii Lazebnyi
77251e445a 🐛 Source Hubspot: Add backoff for handler request (#28705) 
* Add backoff for handler request

* Fixed code formating and expected records
 2023-07-26 14:06:58 +02:00
Subodh Kant Chaturvedi
bcb5a4df8d mysql-cdc: implementation to construct initial debezium state (#28561) 
* wip

* wip 2

* add tests + finalise implementation

* undo unwanted changes

* Automated Commit - Format and Process Resources Changes

* minor refactor

* add comment

* make method public

---------

Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
 2023-07-26 06:18:23 -05:00
Sherif A. Nada
7f4a90a5a1 specify extensibility as Python CDK CODEOWNER not all CDK 2023-07-25 19:06:39 -07:00
Artem Inzhyyants
efd8bc2745 Source Zendesk Support: add to cloud (#28711) 2023-07-26 01:35:30 +02:00
Lake Mossman
f66079cdb1 Capitalize title of request_authentication field (#28585) 
* capitalize title of request_authentication field

* Automated Commit - Formatting Changes

---------

Co-authored-by: lmossman <lmossman@users.noreply.github.com>
 2023-07-25 17:17:27 -06:00
Catherine Noll
b95def8889 Source Stripe: remove duplicate streams (#28709) 2023-07-25 19:11:47 -04:00
Ben Church
3404878a41 Update all metadata sensor from 10 hours to 10 min (#28717) 2023-07-25 16:02:08 -07:00
Edward Gao
a8b099c49e cython >.> (#28675) 2023-07-25 17:31:42 -05:00
Alexandre Girard
221cbdc093 Update the cloud dockerfile too (#28640) 2023-07-25 15:25:09 -07:00
Conor
71211e8591 add a stub to test a java cdk workflow (#28703) 2023-07-25 17:18:27 -05:00
Rodi Reich Zilberman
fb7cacb321 Pre release ctid (#28339) 
* enable ctid feature flag for testing.
bump versions

* Removed PostgresSourceTest pre-ctid and turn CtidEnabledPostgresSourceTest back into PostgresJdbcSourceAcceptanceTest (#28341)

Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>

* Update PostgresSourceTest to no longer use Legacy AirbyteStateType

* Fix test.
Uncovered a bug in handling of money[] and float[]

* revert version change

* Orderly removal of ctid flag for cursor

* Handle correctly full refresh streams. (#28369)

* Handle correctly full refresh streams.
Handle xmin full refresh (currently blocked in ui).
remove unnecessary debug logs

* Test catalog with a mix of incremental and full refresh streams

* Filter out full refresh streams since they are not expected to be retuned by getIncrementalIterators()

* Filter out full refresh existing streams as well

* revert name to reflect incremental only streams

* 28505 xmin + ctid reclassify connection when ctid is not possible (#28548)

* reclassify to xmin stream we cannot read with ctid

* fix reclassify

* improve test

* Add initial load using xmin flow

* typo

* Added test for xmin initial load flow

* Improve testing for initial load with xmin flow

* sanity

* sanity

* bump versions and release notes

---------

Co-authored-by: Duy Nguyen <duy@airbyte.io>
 2023-07-25 16:56:29 -05:00
Rodi Reich Zilberman
f4bf4a8292 Revert "source-postgres: enable ctid+cdc implementation (#28044)" (#28707) 
This reverts commit 886ab6b09b.
 2023-07-25 16:51:04 -05:00
girarda
6e75a7812e 🤖 Bump patch version of Airbyte CDK 2023-07-25 21:48:36 +00:00
Subodh Kant Chaturvedi
886ab6b09b source-postgres: enable ctid+cdc implementation (#28044) 
* source-postgres: enable ctid+cdc implementation

* 100% ctid rollout for cdc
 2023-07-25 16:31:27 -05:00
Alexandre Girard
df01616951 [Issue #23497] Deduplicate query parameters for declarative connectors (#28550) 
* remove duplicate param

* remove duplicate params

* fix some of the typing issues

* fix typing issues

* newline

* format

* Enable by default

* Add missing file

* refactor and remove flag

* none check

* move line of code

* fix typing in rate_limiting

* comment

* use typedef

* else branch

* format

* gate the feature

* rename test

* fix the test

* only dedupe if the values are the same

* Add some tests

* convert values to strings

* Document the change

* implement in requester too
 2023-07-25 14:22:25 -07:00
Jonathan Pearlin
28bedf9675 ✨ Source MongoDB V2: Handle statistics collection failure gracefully (#28692) 
* Fix bug introduced by statistics collection

* Fix syntax

* Formatting

* Fix failing test

* Fix failing test

* Update release notes
 2023-07-25 15:22:02 -05:00
Ella Rohm-Ensing
860e6af933 pin stripe version in cloud to 3.13.0 (#28694) 
* pin stripe version in cloud to 3.13.0

* add ref
 2023-07-25 15:18:41 -05:00
Serhii Lazebnyi
8f92798092 🐛 Source Zendesk Support: Add handle 404 error (#28397) 
* Add handle 404 error

* Updated PR number

* Updated version
 2023-07-25 19:20:56 +02:00
Brian Lai
59300093b1 [file-based cdk] Add avro parser for inferring schema and reading records (#28500) 
* add avro parser for inferring schema and reading records

* fix mypy check not caught locally

* pr feedback and some additional types

* add decimal_as_float for avro

* formatting + mypy
 2023-07-25 12:54:16 -04:00
btkcodedev
66938e1e69 ✨Source Stripe: New Stream - ShippingRates (#27217) 
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* Add stream ShippingRates

* Resolve conflicts, Bump version

* Fix unit tests

* Resolve conflict

* Fix unit_tests

* Remove unwanted files

* Run code formattingh

* Fix unit tests

* Fix unit_tests

* Fix unit_tests

* update expected records

* update expected records

* update expected records

* one more update for expected records

---------

Co-authored-by: Denys Davydov <davydov.den18@gmail.com>
 2023-07-25 18:07:31 +03:00
Augustin
9915bcd17a connectors-ci: extend nightly build timeout (#28669) 2023-07-25 09:35:25 -05:00
Augustin
232b6272ff connectors-ci: unique secret name at session level (#28656) 2023-07-25 16:31:16 +02:00
Artem Inzhyyants
66b428bbda Source Zendesk Support: fix pagination for stream TicketMetricEvents (#28612) 
* Source Zendesk Support: fix pagination for stream TicketMetricEvents

* Source Zendesk Support: update docs

* Source Zendesk Support: fix formatting

* Source Zendesk Support: add test for ticket_metric_events
 2023-07-25 16:06:05 +02:00
Elliot T
9e1e198e18 🎉 New Source: Captain Data [Configuration based CDK] (#25230) 
* initiate connector

* add doc

* fix schema

* add docs

* fix comments

* add logo

* add finished job filter

* suggested changes

* add metadata.yml

* add definition id

* update schemas

---------

Co-authored-by: Sajarin <sajarindider@gmail.com>
 2023-07-25 08:03:09 -05:00
Efim Matytsin
d314dfd0ad 🐛 Source TikTok: fix default filters params - include deleted ad_groups, ad, campaigns to reports (#27910) 
* SCNTALS-1949 fix filters for tiktok

* review fixes

* increment tiktok version in metadata file

* tiktok linter fix

* add changelog record

* small fix for changelog

* Added include_deleted param to TikTok source

* Update airbyte-integrations/connectors/source-tiktok-marketing/source_tiktok_marketing/streams.py

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* tiktok source: review fixes

* tiktok source, code format

---------

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2023-07-25 15:47:35 +03:00
Baz
335f99fd8d 🐛 📄 Source Shopify-OAuth: remove public docs (#28660) 2023-07-25 14:26:06 +03:00
Rodi Reich Zilberman
ff9fd07675 fix changelog table (#28649) 2023-07-24 19:53:17 -05:00
Davin Chia
1cb8cec907 Run AB Platform script fix. (#28647) 
Curl doesn't like it if the directory does not exists. Make sure to create it first.
 2023-07-24 16:43:44 -07:00
Davin Chia
8a62added0 Run Platform Script should also copy this file over. (#28644) 
The run platform script was not updated to account for new config files we need after updating Temporal. This PR adjusts the script so we do so.
 2023-07-24 16:01:11 -07:00
Ben Church
e53034921e Metadata: Ensure we run upload validate on the overrode metadata file (#28583) 
* Ensure we do an upload validate on the overrode metadata file

* Change return value of upload

* Format
 2023-07-24 15:35:46 -06:00
Marcos Marx
53aade6333 🎉 Source Amazon Seller Partner: Add GET_FLAT_FILE_ACTIONABLE_ORDER_DATA_SHIPPING and GET_ORDER_REPORT_DATA_SHIPPING streams (#28578) 
* add new streams

* update acceptancte test config

* format files

* fix format and update docs

* remove for

* update expected records

* update expected records
 2023-07-24 17:01:20 -03:00
Edward Gao
df274b7f40 Destination snowflake: destinations v2 scaffolding (#28584) 
* deps

* scaffolding

* logistics

* base-jdbc depends on base-td
 2023-07-24 14:58:50 -05:00
Joe Bell
6159452f8f 🐛 Destination BigQuery - Limit Clustering Column Amount (#28625) 
* Add limit to the number of clustering columns

* update bq version
 2023-07-24 13:48:56 -06:00
Marcos Marx
6b588d04e2 Source Tiktok Marketing: add new fields to ad reports streams (#25423) 
* add new fields to ad reports streams

* fix unit test

* fix test and check connection

* fix unit test

* add conversions too

* add additional fields

* fix: format python files with flake

* fix: add cta_conversion, vta_conversion to streams.py

* fix: python format

* fix: updated expected_records for ad_reports stream

---------

Co-authored-by: sajarin <sajarindider@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
 2023-07-24 15:44:00 -04:00
Jonathan Pearlin
496678df5f ✨ Source MongoDB V2: Record statistics about MongoDB source connection (#28527) 
* Record statistics about source database/collection

* Formatting

* Remove unnecessary log message

* Only record statistics during sync

* Emit structured Google log data

* Revert use of System.out.println

* Try using Log4j ObjectMessage

* Revert "Try using Log4j ObjectMessage"

This reverts commit d518c87aa8.

* Prepare release

* Update changelog

* Correct release version
 2023-07-24 12:41:40 -05:00
Parker Mossman
9c01e86f64 add Airbyte Pro helm installation instructions (#28582) 
* add Airbyte Pro helm installation instructions

* specify RELEASE_NAME behavior
 2023-07-24 10:33:21 -07:00
Octavia Squidington III
b252a995d1 🤖 Add Gainsight Px connector to cloud (#28592) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-07-24 09:11:00 -07:00
Catherine Noll
7a18cc1535 Revert "Source Zendesk Support: remove version pin (#28587)" (#28611) 
This reverts commit f402044dfb.
 2023-07-24 08:37:33 -05:00