jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-02-01 22:01:57 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,521 Commits 5,453 Branches 518 Tags
ebd738e4ac2cb98e7d45d42e87a0e12c82ea7595
Commit Graph

338 Commits

Author SHA1 Message Date
Lake Mossman
8be753df02 add link to record selector docs (#27291) 
* add link to record selector docs

* Automated Commit - Formatting Changes

---------

Co-authored-by: lmossman <lmossman@users.noreply.github.com>
 2023-06-16 12:09:59 -07:00
Denys Davydov
780f4415d9 🐛 CDK: replace data with json when making OAuth calls (#27350) 
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* Airbyte CDK: replace data with json when making OAuth calls
 2023-06-14 22:51:37 +03:00
Maxime Carbonneau-Leclerc
f48849fdb4 [ISSUE #26909] adding message repository (#27158) 
* [ISSUE #26909] adding message repository

* Automated Commit - Formatting Changes

* [ISSUE #26909] improve entrypoint error handling

* format CDK

* [ISSUE #26909] adding an integration test
 2023-06-13 08:40:55 -04:00
Joe Reuter
27635ba26a Low code CDK: Datetime format documentation (#27149) 
* add format documentation

* fix

* improve
 2023-06-12 17:39:12 -07:00
Alexandre Girard
2864f72ff4 Set content-type header on oauth request (#27225) 
* Set content-type header on oauth authenticator

* Revert "Set content-type header on oauth authenticator"

This reverts commit 1e6815e9bb.

* Set header on oauth request

* Fix test

* Verify header is set

* Automated Commit - Formatting Changes

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-06-12 13:29:53 -04:00
Joe Reuter
d6512dea2c CDK: Datetime format inferrer (#27071) 
* datetime inferrer class

* format

* pass inferred date formats along

* review comments
 2023-06-09 10:33:54 +02:00
Lake Mossman
a2c57c4fd4 Add metadata field to low-code schema (#27094) 
* add metadata field to low-code schema

* Automated Commit - Formatting Changes

* update wording of description

* fix formatting

---------

Co-authored-by: lmossman <lmossman@users.noreply.github.com>
 2023-06-08 08:14:42 -07:00
Maxime Carbonneau-Leclerc
13834395bb [ISSUE #26568] make DatetimeBasedCursor.end_datetime optional (#27031) 
* [ISSUE #26568] make DatetimeBasedCursor.end_datetime optional

* [ISSUE #26568] ensure model_to_component_factory create end_datetime properly

* [ISSUE #26568] fix typing

* [ISSUE #26568] improve end_datetime documentation
 2023-06-08 09:51:40 -04:00
Joe Reuter
c4981a72db Low code CDK: Allow query param / body injection for api key authenticator (#26953) 
* make authenticator more flexible

* format

* format

* format

* format

* format

* fix problem

* code review
 2023-06-07 15:08:13 +02:00
Joe Reuter
e19c498b00 Remove lowcode SingleUseRefreshTokenOauthAuthenticator (#27053) 
* wip

* Automated Commit - Formatting Changes

* add documentation

* tests and fixes

* fix tests

* more documentation

* revert

* changes as discussed

* fix case

* add docstring

* add details to schema

* format

* remove old singleuserefreshtokenauthenticator from lowcode

* prepare quickbooks connector

* format

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-06-07 15:07:50 +02:00
Maxime Carbonneau-Leclerc
4625cef571 [ISSUE #26909] add latest connector config control message to connect… (#26922) 
* [ISSUE #26909] add latest connector config control message to connector builder API

* [ISSUE #26909] flake

* Automated Commit - Formatting Changes

* [ISSUE #26909] fallback on in-memory dict if no config control message

* [ISSUE #26909] update and add tests
 2023-06-07 08:31:45 -04:00
Joe Reuter
b34fb00660 Extend low code OAuthAuthenticator with token refresh capabilities (#26966) 
* wip

* Automated Commit - Formatting Changes

* add documentation

* tests and fixes

* fix tests

* more documentation

* revert

* changes as discussed

* fix case

* add docstring

* add details to schema

* format

* fix bug

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-06-07 10:51:59 +02:00
Maxime Carbonneau-Leclerc
b5c0ac15ec [ISSUE #26570] make step and cursor_granularity optional (#26952) 
* [ISSUE #26570] make step and cursor_granularity optional

* [ISSUE #26570] fix typos
 2023-06-05 09:10:48 -04:00
Maxime Carbonneau-Leclerc
d54a68640f Improving error messages to have better messaging in datadog and the … (#26860) 
* Improving error messages to have better messaging in datadog and the frontend

* fixing tests
 2023-05-31 15:36:27 -04:00
Sherif A. Nada
8c4132eb7b lowcode: Better describe what the cursor field is for (#26815) 2023-05-31 09:24:40 -07:00
Joe Reuter
ec5aa7bab6 CDK: Improve schema detection (#26741) 
* improve schema detection

* improve schema detection

* review comment

* Automated Commit - Formatting Changes

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-05-31 09:57:08 -04:00
Joe Reuter
4a041bf77d Low code CDK: Allow nested objects for request_body_json (#26474) 
* allow nested JSON

* add test for boolean

* review comment

* change for testing

* try fix

* try another fix

* Revert "change for testing"

This reverts commit 931b935778.

* Revert "try fix"

This reverts commit 6f1c6c0e4b.
 2023-05-26 10:52:24 +02:00
Brian Lai
5707e477ad low-code cdk: make page_size optional for offset and page increment strategies (#26056) 
* make page_size optional

* Automated Commit - Formatting Changes

---------

Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
 2023-05-24 17:21:41 -04:00
Alexandre Girard
fa626c6f18 Update low-code reference docs to mention that %s should be used for unix timestamp (#26096) 
* Mention that %s should be used for unix timestamp

* update docs and add unit test

* Update test

* Update
 2023-05-22 17:05:56 -07:00
Joe Reuter
13752e29af Low-code CDK: Improve grant type documentation (#26335) 
* improve grant type documentation

* for the other

* Automated Commit - Formatting Changes

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-05-22 12:06:19 -04:00
Joe Reuter
dbb766c255 Low code CDK: Make refresh token in oauth authenticator optional (#26031) 
* make refresh_token optional

* format

* add back type annotation
 2023-05-15 14:47:48 +02:00
Catherine Noll
051c656aba [lowcode] OAuth Authenticator: cast token expiry time to number (#26020) 2023-05-12 14:31:20 -04:00
Alexandre Girard
7443970de3 low-code: Use Jinja sandbox environment and prevent use of range method (#25589) 
* secure the jinja environment

* format

* Update comment

* remove extra test

* remove lambda

* Update

* Raise an error on undefined variables

* remove unused import

* add unit tests to missing context vars and adjust error message

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
 2023-05-08 19:34:53 -04:00
Maxime Carbonneau-Leclerc
0efc18a114 [ISSUE #24720] connector builder set slice descriptor (#25677) 2023-05-01 12:18:22 -04:00
Maxime Carbonneau-Leclerc
b26c897a8b [ISSUE #25646] support parsing of non utc dates (#25665) 
* [ISSUE #25646] support parsing of non utc dates

* [ISSUE #25646] improve parsing

* [ISSUE #25646] removing timezone for DatetimeParser interface

* [ISSUE #25646] fix tests
 2023-05-01 12:16:44 -04:00
Alexandre Girard
a4b518fb30 low-code: Update docs on incremental syncs (#25696) 
* Update docs

* Automated Commit - Formatting Changes

* update

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-28 18:55:07 -07:00
Joe Reuter
5ec81cc8d1 Connector builder: Restructure documentation (#25353) 
* authentication documentation

* some fixes

* fix broken link

* remove session token auth

* review comments

* first version of tutorial

* Revert "first version of tutorial"

This reverts commit bbdeb997f2.

* first version of tutorial

* review comments

* improvements

* partitioning documentation

* incremental sync

* review comments

* improvements

* record processing doc

* improvements

* error handler connector builder ui documentation

* add missing link

* overview page

* mention current limitation

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
 2023-04-27 12:32:26 +02:00
Alexandre Girard
e41060c02c low-code: Fix type check in DeclarativeStream (#25533) 
* Set right type

* Update the comment

* Update

* format

* Update comment
 2023-04-26 15:49:10 -07:00
Jonathan Pearlin
2ebfa459cf Publish stream status messages in CDK (#24994) 
* Publish stream status messages in CDK

* Automated Commit - Formatting Changes

* Convert to StreamDescriptor

* Automated Commit - Formatting Changes

* Bump to latest protocol model

* Automated Commit - Formatting Changes

* Bump protocol version

* Add tests for stream status message creation

* Formatting

* Formatting

* Fix failing test

* Actually emit state message

* Automated Commit - Formatting Changes

* Bump airbyte-protocol

* PR feedback

* Fix parameter input

* Correctly yield status message

* PR feedback

* Formatting

* Fix failing tests

* Automated Commit - Formatting Changes

* Revert accidental change

* Automated Change

* Replace STOPPED with COMPLETE/INCOMPLETE

* Update source-facebook-marketing changelog

* Revert "Update source-facebook-marketing changelog"

This reverts commit 709edb800c.

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
 2023-04-26 10:30:36 -04:00
Maxime Carbonneau-Leclerc
3cc67a6d9e [ISSUE #23382] ignore backoff configuration on test reads (#25429) 2023-04-26 08:36:59 -04:00
Alexandre Girard
1c13a0af6a Update jinja variable description (#25440) 
* Update description

* Add examples

* format

* update

* update

* Update declarative_component_schema.yaml

* remove superfluous -

---------

Co-authored-by: Joe Reuter <joe@airbyte.io>
 2023-04-25 20:44:05 -07:00
Alexandre Girard
250c3b1c87 low-code: Delete now_local macro (#25404) 
* Delete now_local macro

* Remove from reference docs

* remove example
 2023-04-25 20:40:46 -07:00
Alexandre Girard
645763588c low-code: Alias stream_interval and stream_partition to stream_slice in interpolation context (#25373) 
* add aliases

* Raise error if the alias is found in the context

* format

* Comment

* Automated Commit - Formatting Changes

* rename to stream partition in greenhouse manifest

* Revert "rename to stream partition in greenhouse manifest"

This reverts commit d513ef418f.

* Clean up test

* Other test

* last test

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-24 18:25:54 -07:00
Maxime Carbonneau-Leclerc
4d65fa1b98 [ISSUE #23994] make MessageGrouper use AirbyteEntrypoint (#25402) 
* [ISSUE #23994] make MessageGrouper use AirbyteEntrypoint

* [ISSUE #23994] code review
 2023-04-24 11:24:15 -04:00
Alexandre Girard
15f90e3a2f Fix and document macros and interpolation variables (#25305) 
* Fix and document macros

* cleanup

* dots

* Add tests and refactor

* Update

* Add an example

* Document variables

* Mention now_local is not recommended
 2023-04-21 10:58:53 -07:00
Alexandre Girard
0a91a98370 Add descriptions and examples to component schema (#25117) 
* Reference docs for backoff strategies

* Docs for most authentication mechansims

* Docs for CheckStream

* MinMaxDatetime

* DeclarativeStream

* DefaultErrorHandler

* CompositeErrorHandler

* update

* Update token expiry date description

* DPath extractor

* Add interpolation_context

* HttpResponseFilter

* RecordFilter

* RecordSelector

* DefaultPaginator

* CursorPagination

* OffsetIncrement

* Page Increment

* PrimaryKey

* HttpRequester

* request option and request path

* Schemas

* Spec

* Add field

* remove fields

* no auth and no pagination

* Delete deprecated comment

* Missing description

* Json Decoder

* OAuthConfigSpecification

* reorder

* add titles, examples, and descriptions for partition routers and datetime based cursor

* updates

* Update

* fix indentation

* Automated Commit - Formatting Changes

* Update as per feedback

* html tag

* generate models

* Update name

* do not use title as class name

* Update

* Add stream_interval and stream_partition to interpolation_context

* Automated Commit - Formatting Changes

* fix path

* format

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-21 10:58:23 -07:00
Alexandre Girard
1e8cf8f5d5 low-code: Do not apply transforms on AirbyteLogMessages and AirbyteTraceMessages (#25290) 
* Check the input type before applying transformations

* format

* remove debug prints
 2023-04-20 14:12:22 -07:00
Alexandre Girard
fc3655c12a low-code: Clean up SessionTokenAuthenticator interface (#25086) 
* Username and session token are optional fields

* update

* Add titles, descriptions, and examples

* Automated Commit - Formatting Changes

* fix a small typo

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-17 14:42:49 -07:00
Alexandre Girard
3841141913 Fix manifest_declarative_source + add unit tests (#25217) 
* Fix + unit test

* Add a test with pagination

* Add a test with partition router

* Make sure _fetch_next_page is called with the right arguments

* Automated Commit - Formatting Changes

* pagination with partitions

* refactor

* clean up

* format

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-14 14:05:22 -07:00
Alexandre Girard
f3799280f2 connector builder: Emit message at start of slice (#25180) 
* Move condition for yielding the slice message to an overwritable method

* Automated Commit - Formatting Changes

* yield the slice log messages

* same for incremental

* refactor

* Revert "refactor"

This reverts commit c594365bd8.

* move flag from factory to source

* set the flag

* remove debug print

* halfmock

* clean up

* Add a test for a single page

* Add another test

* Pass the flag

* rename

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-14 10:23:59 -07:00
Alexandre Girard
71fc3dd517 Connector builder: set pages and slices limits (#25121) 
* Set limits

* refactor and add unit tests

* Update as per comments
 2023-04-12 14:46:43 -07:00
Brian Lai
3ba15b5fcb Decouple flags for debug messages from connector builder log messages (#24881) 
* decouple debug message flags from connector builder grouping messages

* Automated Commit - Formatting Changes

* pr feedback simplifying configs a bit

---------

Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
 2023-04-06 12:16:05 -04:00
Denys Davydov
30e2447ceb Lowcode CDK: implement single use refresh token oauth authenticator (#24891) 
* #24658 Lowcode CDK: implement single use refresh token oauth authenticator

* #24658 lowcode cdk: review fixes
 2023-04-06 12:15:36 +03:00
Alexandre Girard
4b324c3084 Low-code: fix duplicate stream_slicer update (#24827) 
* first tentative fix

* cleaner fix

* refactor test

* format

* format

* move to utils file

* use simpler implementation
 2023-04-04 15:40:33 -07:00
Serhii Chvaliuk
032f9b8045 Low-Code CDK: make RecordFilter.filter_records as generator (#24772) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2023-04-03 13:29:24 +03:00
Alexandre Girard
c3b017c7b5 Add auth flow to declarative manifest schema (#24441) 
* Add auth flow to declarative manifest schema

* Rename

* fix rename

* set advanced_auth

* Automated Commit - Formatting Changes

* update unit test

* format

* Add examples

* example -> examples

* add missing examples

* Automated Commit - Formatting Changes

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-03-29 16:06:56 -05:00
Augustin
bad5bce8ce CDK: remove unexpected error swallowing on abstract source's check method (#24240) 2023-03-23 13:04:51 +00:00
Alexandre Girard
aaf00deb38 Add missing type hints (#24345) 2023-03-22 07:15:54 -07:00
Alexandre Girard
edfc59533d Connector builder: Port "send stacktrace when error on read" to CDK connector builder module (#24173) 
* wip

* fix unit test

* fix other unit test

* format

* reset

* format

* missing unit test

* yield a LogMessage on error

* format

* format

* fix unit tests

* yield a trace message instead of a log message

* format

* fix bad merge
 2023-03-21 17:22:08 -07:00
Catherine Noll
f4fd4d98a2 Connector Builder: Make connector_builder part of the CDK package (#24280) 2023-03-21 13:31:16 -04:00