jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-03 15:04:01 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,968 Commits 5,149 Branches 518 Tags
9f6963ccfcddd7a573c4ca0ff94de41ce6175cc0
Commit Graph

377 Commits

Author SHA1 Message Date
Joe Reuter
b34fb00660 Extend low code OAuthAuthenticator with token refresh capabilities (#26966) 
* wip

* Automated Commit - Formatting Changes

* add documentation

* tests and fixes

* fix tests

* more documentation

* revert

* changes as discussed

* fix case

* add docstring

* add details to schema

* format

* fix bug

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-06-07 10:51:59 +02:00
Maxime Carbonneau-Leclerc
b5c0ac15ec [ISSUE #26570] make step and cursor_granularity optional (#26952) 
* [ISSUE #26570] make step and cursor_granularity optional

* [ISSUE #26570] fix typos
 2023-06-05 09:10:48 -04:00
Maxime Carbonneau-Leclerc
d54a68640f Improving error messages to have better messaging in datadog and the … (#26860) 
* Improving error messages to have better messaging in datadog and the frontend

* fixing tests
 2023-05-31 15:36:27 -04:00
Sherif A. Nada
8c4132eb7b lowcode: Better describe what the cursor field is for (#26815) 2023-05-31 09:24:40 -07:00
Joe Reuter
ec5aa7bab6 CDK: Improve schema detection (#26741) 
* improve schema detection

* improve schema detection

* review comment

* Automated Commit - Formatting Changes

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-05-31 09:57:08 -04:00
Joe Reuter
4a041bf77d Low code CDK: Allow nested objects for request_body_json (#26474) 
* allow nested JSON

* add test for boolean

* review comment

* change for testing

* try fix

* try another fix

* Revert "change for testing"

This reverts commit 931b935778.

* Revert "try fix"

This reverts commit 6f1c6c0e4b.
 2023-05-26 10:52:24 +02:00
Brian Lai
5707e477ad low-code cdk: make page_size optional for offset and page increment strategies (#26056) 
* make page_size optional

* Automated Commit - Formatting Changes

---------

Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
 2023-05-24 17:21:41 -04:00
Alexandre Girard
fa626c6f18 Update low-code reference docs to mention that %s should be used for unix timestamp (#26096) 
* Mention that %s should be used for unix timestamp

* update docs and add unit test

* Update test

* Update
 2023-05-22 17:05:56 -07:00
Joe Reuter
13752e29af Low-code CDK: Improve grant type documentation (#26335) 
* improve grant type documentation

* for the other

* Automated Commit - Formatting Changes

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-05-22 12:06:19 -04:00
Joe Reuter
dbb766c255 Low code CDK: Make refresh token in oauth authenticator optional (#26031) 
* make refresh_token optional

* format

* add back type annotation
 2023-05-15 14:47:48 +02:00
Catherine Noll
051c656aba [lowcode] OAuth Authenticator: cast token expiry time to number (#26020) 2023-05-12 14:31:20 -04:00
Alexandre Girard
7443970de3 low-code: Use Jinja sandbox environment and prevent use of range method (#25589) 
* secure the jinja environment

* format

* Update comment

* remove extra test

* remove lambda

* Update

* Raise an error on undefined variables

* remove unused import

* add unit tests to missing context vars and adjust error message

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
 2023-05-08 19:34:53 -04:00
Maxime Carbonneau-Leclerc
0efc18a114 [ISSUE #24720] connector builder set slice descriptor (#25677) 2023-05-01 12:18:22 -04:00
Maxime Carbonneau-Leclerc
b26c897a8b [ISSUE #25646] support parsing of non utc dates (#25665) 
* [ISSUE #25646] support parsing of non utc dates

* [ISSUE #25646] improve parsing

* [ISSUE #25646] removing timezone for DatetimeParser interface

* [ISSUE #25646] fix tests
 2023-05-01 12:16:44 -04:00
Alexandre Girard
a4b518fb30 low-code: Update docs on incremental syncs (#25696) 
* Update docs

* Automated Commit - Formatting Changes

* update

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-28 18:55:07 -07:00
Joe Reuter
5ec81cc8d1 Connector builder: Restructure documentation (#25353) 
* authentication documentation

* some fixes

* fix broken link

* remove session token auth

* review comments

* first version of tutorial

* Revert "first version of tutorial"

This reverts commit bbdeb997f2.

* first version of tutorial

* review comments

* improvements

* partitioning documentation

* incremental sync

* review comments

* improvements

* record processing doc

* improvements

* error handler connector builder ui documentation

* add missing link

* overview page

* mention current limitation

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
 2023-04-27 12:32:26 +02:00
Alexandre Girard
e41060c02c low-code: Fix type check in DeclarativeStream (#25533) 
* Set right type

* Update the comment

* Update

* format

* Update comment
 2023-04-26 15:49:10 -07:00
Jonathan Pearlin
2ebfa459cf Publish stream status messages in CDK (#24994) 
* Publish stream status messages in CDK

* Automated Commit - Formatting Changes

* Convert to StreamDescriptor

* Automated Commit - Formatting Changes

* Bump to latest protocol model

* Automated Commit - Formatting Changes

* Bump protocol version

* Add tests for stream status message creation

* Formatting

* Formatting

* Fix failing test

* Actually emit state message

* Automated Commit - Formatting Changes

* Bump airbyte-protocol

* PR feedback

* Fix parameter input

* Correctly yield status message

* PR feedback

* Formatting

* Fix failing tests

* Automated Commit - Formatting Changes

* Revert accidental change

* Automated Change

* Replace STOPPED with COMPLETE/INCOMPLETE

* Update source-facebook-marketing changelog

* Revert "Update source-facebook-marketing changelog"

This reverts commit 709edb800c.

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
 2023-04-26 10:30:36 -04:00
Maxime Carbonneau-Leclerc
3cc67a6d9e [ISSUE #23382] ignore backoff configuration on test reads (#25429) 2023-04-26 08:36:59 -04:00
Alexandre Girard
1c13a0af6a Update jinja variable description (#25440) 
* Update description

* Add examples

* format

* update

* update

* Update declarative_component_schema.yaml

* remove superfluous -

---------

Co-authored-by: Joe Reuter <joe@airbyte.io>
 2023-04-25 20:44:05 -07:00
Alexandre Girard
250c3b1c87 low-code: Delete now_local macro (#25404) 
* Delete now_local macro

* Remove from reference docs

* remove example
 2023-04-25 20:40:46 -07:00
Alexandre Girard
645763588c low-code: Alias stream_interval and stream_partition to stream_slice in interpolation context (#25373) 
* add aliases

* Raise error if the alias is found in the context

* format

* Comment

* Automated Commit - Formatting Changes

* rename to stream partition in greenhouse manifest

* Revert "rename to stream partition in greenhouse manifest"

This reverts commit d513ef418f.

* Clean up test

* Other test

* last test

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-24 18:25:54 -07:00
Maxime Carbonneau-Leclerc
4d65fa1b98 [ISSUE #23994] make MessageGrouper use AirbyteEntrypoint (#25402) 
* [ISSUE #23994] make MessageGrouper use AirbyteEntrypoint

* [ISSUE #23994] code review
 2023-04-24 11:24:15 -04:00
Alexandre Girard
15f90e3a2f Fix and document macros and interpolation variables (#25305) 
* Fix and document macros

* cleanup

* dots

* Add tests and refactor

* Update

* Add an example

* Document variables

* Mention now_local is not recommended
 2023-04-21 10:58:53 -07:00
Alexandre Girard
0a91a98370 Add descriptions and examples to component schema (#25117) 
* Reference docs for backoff strategies

* Docs for most authentication mechansims

* Docs for CheckStream

* MinMaxDatetime

* DeclarativeStream

* DefaultErrorHandler

* CompositeErrorHandler

* update

* Update token expiry date description

* DPath extractor

* Add interpolation_context

* HttpResponseFilter

* RecordFilter

* RecordSelector

* DefaultPaginator

* CursorPagination

* OffsetIncrement

* Page Increment

* PrimaryKey

* HttpRequester

* request option and request path

* Schemas

* Spec

* Add field

* remove fields

* no auth and no pagination

* Delete deprecated comment

* Missing description

* Json Decoder

* OAuthConfigSpecification

* reorder

* add titles, examples, and descriptions for partition routers and datetime based cursor

* updates

* Update

* fix indentation

* Automated Commit - Formatting Changes

* Update as per feedback

* html tag

* generate models

* Update name

* do not use title as class name

* Update

* Add stream_interval and stream_partition to interpolation_context

* Automated Commit - Formatting Changes

* fix path

* format

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-21 10:58:23 -07:00
Alexandre Girard
1e8cf8f5d5 low-code: Do not apply transforms on AirbyteLogMessages and AirbyteTraceMessages (#25290) 
* Check the input type before applying transformations

* format

* remove debug prints
 2023-04-20 14:12:22 -07:00
Alexandre Girard
fc3655c12a low-code: Clean up SessionTokenAuthenticator interface (#25086) 
* Username and session token are optional fields

* update

* Add titles, descriptions, and examples

* Automated Commit - Formatting Changes

* fix a small typo

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-17 14:42:49 -07:00
Alexandre Girard
3841141913 Fix manifest_declarative_source + add unit tests (#25217) 
* Fix + unit test

* Add a test with pagination

* Add a test with partition router

* Make sure _fetch_next_page is called with the right arguments

* Automated Commit - Formatting Changes

* pagination with partitions

* refactor

* clean up

* format

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-14 14:05:22 -07:00
Alexandre Girard
f3799280f2 connector builder: Emit message at start of slice (#25180) 
* Move condition for yielding the slice message to an overwritable method

* Automated Commit - Formatting Changes

* yield the slice log messages

* same for incremental

* refactor

* Revert "refactor"

This reverts commit c594365bd8.

* move flag from factory to source

* set the flag

* remove debug print

* halfmock

* clean up

* Add a test for a single page

* Add another test

* Pass the flag

* rename

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-04-14 10:23:59 -07:00
Alexandre Girard
71fc3dd517 Connector builder: set pages and slices limits (#25121) 
* Set limits

* refactor and add unit tests

* Update as per comments
 2023-04-12 14:46:43 -07:00
Brian Lai
3ba15b5fcb Decouple flags for debug messages from connector builder log messages (#24881) 
* decouple debug message flags from connector builder grouping messages

* Automated Commit - Formatting Changes

* pr feedback simplifying configs a bit

---------

Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
 2023-04-06 12:16:05 -04:00
Denys Davydov
30e2447ceb Lowcode CDK: implement single use refresh token oauth authenticator (#24891) 
* #24658 Lowcode CDK: implement single use refresh token oauth authenticator

* #24658 lowcode cdk: review fixes
 2023-04-06 12:15:36 +03:00
Alexandre Girard
4b324c3084 Low-code: fix duplicate stream_slicer update (#24827) 
* first tentative fix

* cleaner fix

* refactor test

* format

* format

* move to utils file

* use simpler implementation
 2023-04-04 15:40:33 -07:00
Serhii Chvaliuk
032f9b8045 Low-Code CDK: make RecordFilter.filter_records as generator (#24772) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2023-04-03 13:29:24 +03:00
Alexandre Girard
c3b017c7b5 Add auth flow to declarative manifest schema (#24441) 
* Add auth flow to declarative manifest schema

* Rename

* fix rename

* set advanced_auth

* Automated Commit - Formatting Changes

* update unit test

* format

* Add examples

* example -> examples

* add missing examples

* Automated Commit - Formatting Changes

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-03-29 16:06:56 -05:00
Augustin
bad5bce8ce CDK: remove unexpected error swallowing on abstract source's check method (#24240) 2023-03-23 13:04:51 +00:00
Alexandre Girard
aaf00deb38 Add missing type hints (#24345) 2023-03-22 07:15:54 -07:00
Alexandre Girard
edfc59533d Connector builder: Port "send stacktrace when error on read" to CDK connector builder module (#24173) 
* wip

* fix unit test

* fix other unit test

* format

* reset

* format

* missing unit test

* yield a LogMessage on error

* format

* format

* fix unit tests

* yield a trace message instead of a log message

* format

* fix bad merge
 2023-03-21 17:22:08 -07:00
Catherine Noll
f4fd4d98a2 Connector Builder: Make connector_builder part of the CDK package (#24280) 2023-03-21 13:31:16 -04:00
Brian Lai
903d34e5f1 [Low-Code CDK] Enforce manifest against the airbyte-cdk version and the Beta version 0.29.0 (#23796) 
* enforce manifest version correctness against the CDK package being used

* parse versions into parts for better comparisons and error checking

* fix pr feedback and derp forgot to actually add the commit with the low-code manifests updated to the beta version

* pr feedback and fix new tests since last rebase
 2023-03-16 00:50:30 -04:00
Alexandre Girard
bb5741a0c0 Connector builder: support for test read with message grouping per slices (#23925) 
* New connector_builder module for handling requests from the Connector Builder.

Also implements `resolve_manifest` handler

* Automated Commit - Formatting Changes

* Rename ConnectorBuilderSource to ConnectorBuilderHandler

* Update source_declarative_manifest README

* Reorganize

* read records

* paste unit tests from connector builder server

* compiles but tests fail

* first test passes

* Second test passes

* 3rd test passes

* one more test

* another test

* one more test

* test

* return StreamRead

* test

* test

* rename

* test

* test

* test

* main seems to work

* Update

* Update

* Update

* Update

* update

* error message

* rename

* update

* Update

* CR improvements

* fix test_source_declarative_manifest

* fix tests

* Update

* Update

* Update

* Update

* rename

* rename

* rename

* format

* Give connector_builder its own main.py

* Update

* reset

* delete dead code

* remove debug print

* update test

* Update

* set right stream

* Add --catalog argument

* Remove unneeded preparse

* Update README

* handle error

* tests pass

* more explicit test

* reset

* format

* fix merge

* raise exception

* fix

* black format

* raise with config

* update

* fix flake

* __test_read_config is optional

* fix

* Automated Commit - Formatting Changes

* fix

* exclude_unset

---------

Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: clnoll <clnoll@users.noreply.github.com>
Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-03-15 17:12:37 -07:00
Maxime Carbonneau-Leclerc
2e3412d9dd [ISSUE #24059] fix reset pagination issue (#24069) 
* [ISSUE #24059] fix reset pagination issue

* Automated Commit - Formatting Changes
 2023-03-14 18:51:12 +00:00
Alexandre Girard
02f771b422 Do not remove trailing slash from path (#24003) 
* Do not remove trailing slash from path

* Add a breaking test

* Add some tests on HttpStream
 2023-03-13 23:59:24 +00:00
Sherif A. Nada
8d6f130666 remove outdated code comment 2023-03-13 09:54:59 -07:00
Leo Schick
09955423db support reading "expires_in" when the API passes the value as string (#23921) 
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2023-03-10 09:02:14 -08:00
Sherif A. Nada
ec9707c71f Low-code CDK: Override refresh_access_token logic DeclarativeOAuthAuthenticator (#23927) 
Co-authored-by: sherifnada <sherifnada@users.noreply.github.com>
 2023-03-09 17:28:33 -08:00
Sherif A. Nada
b2f748b9a3 Revert "fix(python-cdk): convert expires_in to int when refreshing (#20301)" (#23924) 
This reverts commit d67afbbd17.
 2023-03-09 15:54:07 -08:00
Joël Luijmes
d67afbbd17 fix(python-cdk): convert expires_in to int when refreshing (#20301) 2023-03-09 13:46:10 -08:00
Denys Davydov
f9f140239a CDK: retry token refresh requests (#23815) 
* #23767 CDK: retry token refresh requests

* Automated Commit - Formatting Changes

---------

Co-authored-by: davydov-d <davydov-d@users.noreply.github.com>
 2023-03-07 19:41:12 +02:00
Serhii Chvaliuk
bfe8ccbaac duration macro added (#23690) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2023-03-02 23:54:02 +02:00