jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-07 00:05:48 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,566 Commits 5,180 Branches 518 Tags
02e4bd07f7fd80ce07bdf522adabcea1e7687f67
Commit Graph

344 Commits

Author SHA1 Message Date
Brian Lai
02e4bd07f7 [26989] Add request filter for cloud and integration test fixtures for e2e sync testing (#27534) 
* add the request filters and integration test fixtures

* pr feedback and some tweaks to the testing framework

* optimize the cache for more hits

* formatting

* remove cache
 2023-06-22 12:14:07 -04:00
Catherine Noll
a8e99a46e6 File CDK: define streams via glob list (#27476) 2023-06-22 11:50:35 -04:00
Maxime Carbonneau-Leclerc
d9a5e2d873 🐛 Source zenloop: update to state per partition (#27556) 
* [ISSUE #26581] per partition cursor

* [ISSUE #26581] format

* [ISSUE #26581] clean up state management

* [ISSUE #26581] improving Hashabledict

* [ISSUE #26581] format cdk

* [ISSUE #26581] fix tests

* [ISSUE #26581] code review from girarda

* Retrigger pipeline

* [ISSUE #26581] code review

* Automated Commit - Formatting Changes

* [ISSUE #26581] validation overlapping keys

* [ISSUE #26581] add typing

* [ISSUE #26581] code review

* [ISSUE #26607] zenloop migration (#27243)

* [ISSUE #26607] zenloop migration implementation without tests

* [ISSUE #26607] zenloop migration adding edge cases

* [ISSUE #26607] add cursor field for state

* [ISSUE #26607] update abnormal state

* [ISSUE #26607] ensure default state

* [ISSUE #26607] updating CATs state

* [ISSUE #26607] revert migrating cursor

* [ISSUE #26607] remove default cursor value

* [ISSUE #26607] improve error message

* [ISSUE #26607] changelog

---------

Co-authored-by: Augustin <augustin@airbyte.io>

* 🤖 Auto format source-zenloop code [skip ci]

* Automated Commit - Formatting Changes

* [ISSUE #26581] move partition serialization to JSON

* Revert "[ISSUE #26607] zenloop migration (#27243)"

This reverts commit 5c6f19b775.

* Revert "Revert "[ISSUE #26607] zenloop migration (#27243)""

This reverts commit e363fd6cb8.

* [ISSUE #26607] update zenloop version

* TMP specify cdk version

* [ISSUE #26607] do not lock zenloop airbyte_cdk version

* trigger pipeline

* Automated Commit - Formatting Changes

* trigger pipeline

---------

Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
 2023-06-22 08:42:20 -05:00
Maxime Carbonneau-Leclerc
8926970c86 Issue 26581/per partition cursor (#27223) 
* [ISSUE #26581] per partition cursor

* [ISSUE #26581] format

* [ISSUE #26581] clean up state management

* [ISSUE #26581] improving Hashabledict

* [ISSUE #26581] format cdk

* [ISSUE #26581] fix tests

* [ISSUE #26581] code review from girarda

* Retrigger pipeline

* [ISSUE #26581] code review

* Automated Commit - Formatting Changes

* [ISSUE #26581] validation overlapping keys

* [ISSUE #26581] add typing

* [ISSUE #26581] code review

* [ISSUE #26607] zenloop migration (#27243)

* [ISSUE #26607] zenloop migration implementation without tests

* [ISSUE #26607] zenloop migration adding edge cases

* [ISSUE #26607] add cursor field for state

* [ISSUE #26607] update abnormal state

* [ISSUE #26607] ensure default state

* [ISSUE #26607] updating CATs state

* [ISSUE #26607] revert migrating cursor

* [ISSUE #26607] remove default cursor value

* [ISSUE #26607] improve error message

* [ISSUE #26607] changelog

---------

Co-authored-by: Augustin <augustin@airbyte.io>

* 🤖 Auto format source-zenloop code [skip ci]

* Automated Commit - Formatting Changes

* [ISSUE #26581] move partition serialization to JSON

* Revert "[ISSUE #26607] zenloop migration (#27243)"

This reverts commit 5c6f19b775.

* [ISSUE #26607] revert zenloop

---------

Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
 2023-06-21 12:59:11 -04:00
Alexandre Girard
dee2e9a905 🐛 Use url encoding in oauth refresh request (#27523) 
* Revert "🐛 CDK: replace `data` with `json` when making OAuth calls (#27350)"

This reverts commit 780f4415d9.

* Revert "Set content-type header on oauth request (#27225)"

This reverts commit 2864f72ff4.
 2023-06-20 14:41:06 -07:00
Catherine Noll
f464a330f8 File-based CDK module scaffolding (#27122) 
Includes CSV schema inference & record parser (#27176)

---------

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2023-06-19 11:01:11 -04:00
Lake Mossman
8be753df02 add link to record selector docs (#27291) 
* add link to record selector docs

* Automated Commit - Formatting Changes

---------

Co-authored-by: lmossman <lmossman@users.noreply.github.com>
 2023-06-16 12:09:59 -07:00
Denys Davydov
780f4415d9 🐛 CDK: replace data with json when making OAuth calls (#27350) 
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* Airbyte CDK: replace data with json when making OAuth calls
 2023-06-14 22:51:37 +03:00
Maxime Carbonneau-Leclerc
f48849fdb4 [ISSUE #26909] adding message repository (#27158) 
* [ISSUE #26909] adding message repository

* Automated Commit - Formatting Changes

* [ISSUE #26909] improve entrypoint error handling

* format CDK

* [ISSUE #26909] adding an integration test
 2023-06-13 08:40:55 -04:00
Joe Reuter
27635ba26a Low code CDK: Datetime format documentation (#27149) 
* add format documentation

* fix

* improve
 2023-06-12 17:39:12 -07:00
Alexandre Girard
2864f72ff4 Set content-type header on oauth request (#27225) 
* Set content-type header on oauth authenticator

* Revert "Set content-type header on oauth authenticator"

This reverts commit 1e6815e9bb.

* Set header on oauth request

* Fix test

* Verify header is set

* Automated Commit - Formatting Changes

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-06-12 13:29:53 -04:00
Joe Reuter
d6512dea2c CDK: Datetime format inferrer (#27071) 
* datetime inferrer class

* format

* pass inferred date formats along

* review comments
 2023-06-09 10:33:54 +02:00
Lake Mossman
a2c57c4fd4 Add metadata field to low-code schema (#27094) 
* add metadata field to low-code schema

* Automated Commit - Formatting Changes

* update wording of description

* fix formatting

---------

Co-authored-by: lmossman <lmossman@users.noreply.github.com>
 2023-06-08 08:14:42 -07:00
Maxime Carbonneau-Leclerc
13834395bb [ISSUE #26568] make DatetimeBasedCursor.end_datetime optional (#27031) 
* [ISSUE #26568] make DatetimeBasedCursor.end_datetime optional

* [ISSUE #26568] ensure model_to_component_factory create end_datetime properly

* [ISSUE #26568] fix typing

* [ISSUE #26568] improve end_datetime documentation
 2023-06-08 09:51:40 -04:00
Joe Reuter
c4981a72db Low code CDK: Allow query param / body injection for api key authenticator (#26953) 
* make authenticator more flexible

* format

* format

* format

* format

* format

* fix problem

* code review
 2023-06-07 15:08:13 +02:00
Joe Reuter
e19c498b00 Remove lowcode SingleUseRefreshTokenOauthAuthenticator (#27053) 
* wip

* Automated Commit - Formatting Changes

* add documentation

* tests and fixes

* fix tests

* more documentation

* revert

* changes as discussed

* fix case

* add docstring

* add details to schema

* format

* remove old singleuserefreshtokenauthenticator from lowcode

* prepare quickbooks connector

* format

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-06-07 15:07:50 +02:00
Maxime Carbonneau-Leclerc
4625cef571 [ISSUE #26909] add latest connector config control message to connect… (#26922) 
* [ISSUE #26909] add latest connector config control message to connector builder API

* [ISSUE #26909] flake

* Automated Commit - Formatting Changes

* [ISSUE #26909] fallback on in-memory dict if no config control message

* [ISSUE #26909] update and add tests
 2023-06-07 08:31:45 -04:00
Joe Reuter
b34fb00660 Extend low code OAuthAuthenticator with token refresh capabilities (#26966) 
* wip

* Automated Commit - Formatting Changes

* add documentation

* tests and fixes

* fix tests

* more documentation

* revert

* changes as discussed

* fix case

* add docstring

* add details to schema

* format

* fix bug

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-06-07 10:51:59 +02:00
Maxime Carbonneau-Leclerc
b5c0ac15ec [ISSUE #26570] make step and cursor_granularity optional (#26952) 
* [ISSUE #26570] make step and cursor_granularity optional

* [ISSUE #26570] fix typos
 2023-06-05 09:10:48 -04:00
Maxime Carbonneau-Leclerc
d54a68640f Improving error messages to have better messaging in datadog and the … (#26860) 
* Improving error messages to have better messaging in datadog and the frontend

* fixing tests
 2023-05-31 15:36:27 -04:00
Sherif A. Nada
8c4132eb7b lowcode: Better describe what the cursor field is for (#26815) 2023-05-31 09:24:40 -07:00
Joe Reuter
ec5aa7bab6 CDK: Improve schema detection (#26741) 
* improve schema detection

* improve schema detection

* review comment

* Automated Commit - Formatting Changes

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-05-31 09:57:08 -04:00
Joe Reuter
4a041bf77d Low code CDK: Allow nested objects for request_body_json (#26474) 
* allow nested JSON

* add test for boolean

* review comment

* change for testing

* try fix

* try another fix

* Revert "change for testing"

This reverts commit 931b935778.

* Revert "try fix"

This reverts commit 6f1c6c0e4b.
 2023-05-26 10:52:24 +02:00
Brian Lai
5707e477ad low-code cdk: make page_size optional for offset and page increment strategies (#26056) 
* make page_size optional

* Automated Commit - Formatting Changes

---------

Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
 2023-05-24 17:21:41 -04:00
Alexandre Girard
fa626c6f18 Update low-code reference docs to mention that %s should be used for unix timestamp (#26096) 
* Mention that %s should be used for unix timestamp

* update docs and add unit test

* Update test

* Update
 2023-05-22 17:05:56 -07:00
Joe Reuter
13752e29af Low-code CDK: Improve grant type documentation (#26335) 
* improve grant type documentation

* for the other

* Automated Commit - Formatting Changes

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-05-22 12:06:19 -04:00
Joe Reuter
dbb766c255 Low code CDK: Make refresh token in oauth authenticator optional (#26031) 
* make refresh_token optional

* format

* add back type annotation
 2023-05-15 14:47:48 +02:00
Catherine Noll
051c656aba [lowcode] OAuth Authenticator: cast token expiry time to number (#26020) 2023-05-12 14:31:20 -04:00
Alexandre Girard
7443970de3 low-code: Use Jinja sandbox environment and prevent use of range method (#25589) 
* secure the jinja environment

* format

* Update comment

* remove extra test

* remove lambda

* Update

* Raise an error on undefined variables

* remove unused import

* add unit tests to missing context vars and adjust error message

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
 2023-05-08 19:34:53 -04:00
Maxime Carbonneau-Leclerc
0efc18a114 [ISSUE #24720] connector builder set slice descriptor (#25677) 2023-05-01 12:18:22 -04:00
Maxime Carbonneau-Leclerc
b26c897a8b [ISSUE #25646] support parsing of non utc dates (#25665) 
* [ISSUE #25646] support parsing of non utc dates

* [ISSUE #25646] improve parsing

* [ISSUE #25646] removing timezone for DatetimeParser interface

* [ISSUE #25646] fix tests
 2023-05-01 12:16:44 -04:00
Alexandre Girard
a4b518fb30 low-code: Update docs on incremental syncs (#25696) 
* Update docs

* Automated Commit - Formatting Changes

* update

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-28 18:55:07 -07:00
Joe Reuter
5ec81cc8d1 Connector builder: Restructure documentation (#25353) 
* authentication documentation

* some fixes

* fix broken link

* remove session token auth

* review comments

* first version of tutorial

* Revert "first version of tutorial"

This reverts commit bbdeb997f2.

* first version of tutorial

* review comments

* improvements

* partitioning documentation

* incremental sync

* review comments

* improvements

* record processing doc

* improvements

* error handler connector builder ui documentation

* add missing link

* overview page

* mention current limitation

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
 2023-04-27 12:32:26 +02:00
Alexandre Girard
e41060c02c low-code: Fix type check in DeclarativeStream (#25533) 
* Set right type

* Update the comment

* Update

* format

* Update comment
 2023-04-26 15:49:10 -07:00
Jonathan Pearlin
2ebfa459cf Publish stream status messages in CDK (#24994) 
* Publish stream status messages in CDK

* Automated Commit - Formatting Changes

* Convert to StreamDescriptor

* Automated Commit - Formatting Changes

* Bump to latest protocol model

* Automated Commit - Formatting Changes

* Bump protocol version

* Add tests for stream status message creation

* Formatting

* Formatting

* Fix failing test

* Actually emit state message

* Automated Commit - Formatting Changes

* Bump airbyte-protocol

* PR feedback

* Fix parameter input

* Correctly yield status message

* PR feedback

* Formatting

* Fix failing tests

* Automated Commit - Formatting Changes

* Revert accidental change

* Automated Change

* Replace STOPPED with COMPLETE/INCOMPLETE

* Update source-facebook-marketing changelog

* Revert "Update source-facebook-marketing changelog"

This reverts commit 709edb800c.

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
 2023-04-26 10:30:36 -04:00
Maxime Carbonneau-Leclerc
3cc67a6d9e [ISSUE #23382] ignore backoff configuration on test reads (#25429) 2023-04-26 08:36:59 -04:00
Alexandre Girard
1c13a0af6a Update jinja variable description (#25440) 
* Update description

* Add examples

* format

* update

* update

* Update declarative_component_schema.yaml

* remove superfluous -

---------

Co-authored-by: Joe Reuter <joe@airbyte.io>
 2023-04-25 20:44:05 -07:00
Alexandre Girard
250c3b1c87 low-code: Delete now_local macro (#25404) 
* Delete now_local macro

* Remove from reference docs

* remove example
 2023-04-25 20:40:46 -07:00
Alexandre Girard
645763588c low-code: Alias stream_interval and stream_partition to stream_slice in interpolation context (#25373) 
* add aliases

* Raise error if the alias is found in the context

* format

* Comment

* Automated Commit - Formatting Changes

* rename to stream partition in greenhouse manifest

* Revert "rename to stream partition in greenhouse manifest"

This reverts commit d513ef418f.

* Clean up test

* Other test

* last test

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-24 18:25:54 -07:00
Maxime Carbonneau-Leclerc
4d65fa1b98 [ISSUE #23994] make MessageGrouper use AirbyteEntrypoint (#25402) 
* [ISSUE #23994] make MessageGrouper use AirbyteEntrypoint

* [ISSUE #23994] code review
 2023-04-24 11:24:15 -04:00
Alexandre Girard
15f90e3a2f Fix and document macros and interpolation variables (#25305) 
* Fix and document macros

* cleanup

* dots

* Add tests and refactor

* Update

* Add an example

* Document variables

* Mention now_local is not recommended
 2023-04-21 10:58:53 -07:00
Alexandre Girard
0a91a98370 Add descriptions and examples to component schema (#25117) 
* Reference docs for backoff strategies

* Docs for most authentication mechansims

* Docs for CheckStream

* MinMaxDatetime

* DeclarativeStream

* DefaultErrorHandler

* CompositeErrorHandler

* update

* Update token expiry date description

* DPath extractor

* Add interpolation_context

* HttpResponseFilter

* RecordFilter

* RecordSelector

* DefaultPaginator

* CursorPagination

* OffsetIncrement

* Page Increment

* PrimaryKey

* HttpRequester

* request option and request path

* Schemas

* Spec

* Add field

* remove fields

* no auth and no pagination

* Delete deprecated comment

* Missing description

* Json Decoder

* OAuthConfigSpecification

* reorder

* add titles, examples, and descriptions for partition routers and datetime based cursor

* updates

* Update

* fix indentation

* Automated Commit - Formatting Changes

* Update as per feedback

* html tag

* generate models

* Update name

* do not use title as class name

* Update

* Add stream_interval and stream_partition to interpolation_context

* Automated Commit - Formatting Changes

* fix path

* format

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-21 10:58:23 -07:00
Alexandre Girard
1e8cf8f5d5 low-code: Do not apply transforms on AirbyteLogMessages and AirbyteTraceMessages (#25290) 
* Check the input type before applying transformations

* format

* remove debug prints
 2023-04-20 14:12:22 -07:00
Alexandre Girard
fc3655c12a low-code: Clean up SessionTokenAuthenticator interface (#25086) 
* Username and session token are optional fields

* update

* Add titles, descriptions, and examples

* Automated Commit - Formatting Changes

* fix a small typo

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-17 14:42:49 -07:00
Alexandre Girard
3841141913 Fix manifest_declarative_source + add unit tests (#25217) 
* Fix + unit test

* Add a test with pagination

* Add a test with partition router

* Make sure _fetch_next_page is called with the right arguments

* Automated Commit - Formatting Changes

* pagination with partitions

* refactor

* clean up

* format

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-14 14:05:22 -07:00
Alexandre Girard
f3799280f2 connector builder: Emit message at start of slice (#25180) 
* Move condition for yielding the slice message to an overwritable method

* Automated Commit - Formatting Changes

* yield the slice log messages

* same for incremental

* refactor

* Revert "refactor"

This reverts commit c594365bd8.

* move flag from factory to source

* set the flag

* remove debug print

* halfmock

* clean up

* Add a test for a single page

* Add another test

* Pass the flag

* rename

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-14 10:23:59 -07:00
Alexandre Girard
71fc3dd517 Connector builder: set pages and slices limits (#25121) 
* Set limits

* refactor and add unit tests

* Update as per comments
 2023-04-12 14:46:43 -07:00
Brian Lai
3ba15b5fcb Decouple flags for debug messages from connector builder log messages (#24881) 
* decouple debug message flags from connector builder grouping messages

* Automated Commit - Formatting Changes

* pr feedback simplifying configs a bit

---------

Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
 2023-04-06 12:16:05 -04:00
Denys Davydov
30e2447ceb Lowcode CDK: implement single use refresh token oauth authenticator (#24891) 
* #24658 Lowcode CDK: implement single use refresh token oauth authenticator

* #24658 lowcode cdk: review fixes
 2023-04-06 12:15:36 +03:00
Alexandre Girard
4b324c3084 Low-code: fix duplicate stream_slicer update (#24827) 
* first tentative fix

* cleaner fix

* refactor test

* format

* format

* move to utils file

* use simpler implementation
 2023-04-04 15:40:33 -07:00