1
0
mirror of synced 2025-12-21 02:51:29 -05:00
Commit Graph

1266 Commits

Author SHA1 Message Date
Alexandre Girard
15f90e3a2f Fix and document macros and interpolation variables (#25305)
* Fix and document macros

* cleanup

* dots

* Add tests and refactor

* Update

* Add an example

* Document variables

* Mention now_local is not recommended
2023-04-21 10:58:53 -07:00
Alexandre Girard
0a91a98370 Add descriptions and examples to component schema (#25117)
* Reference docs for backoff strategies

* Docs for most authentication mechansims

* Docs for CheckStream

* MinMaxDatetime

* DeclarativeStream

* DefaultErrorHandler

* CompositeErrorHandler

* update

* Update token expiry date description

* DPath extractor

* Add interpolation_context

* HttpResponseFilter

* RecordFilter

* RecordSelector

* DefaultPaginator

* CursorPagination

* OffsetIncrement

* Page Increment

* PrimaryKey

* HttpRequester

* request option and request path

* Schemas

* Spec

* Add field

* remove fields

* no auth and no pagination

* Delete deprecated comment

* Missing description

* Json Decoder

* OAuthConfigSpecification

* reorder

* add titles, examples, and descriptions for partition routers and datetime based cursor

* updates

* Update

* fix indentation

* Automated Commit - Formatting Changes

* Update as per feedback

* html tag

* generate models

* Update name

* do not use title as class name

* Update

* Add stream_interval and stream_partition to interpolation_context

* Automated Commit - Formatting Changes

* fix path

* format

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-04-21 10:58:23 -07:00
Alexandre Girard
1e8cf8f5d5 low-code: Do not apply transforms on AirbyteLogMessages and AirbyteTraceMessages (#25290)
* Check the input type before applying transformations

* format

* remove debug prints
2023-04-20 14:12:22 -07:00
girarda
dd68bb17b0 🤖 Bump minor version of Airbyte CDK 2023-04-17 21:49:21 +00:00
Alexandre Girard
fc3655c12a low-code: Clean up SessionTokenAuthenticator interface (#25086)
* Username and session token are optional fields

* update

* Add titles, descriptions, and examples

* Automated Commit - Formatting Changes

* fix a small typo

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-04-17 14:42:49 -07:00
girarda
6fa2f21b3d 🤖 Bump patch version of Airbyte CDK 2023-04-14 21:11:36 +00:00
Alexandre Girard
3841141913 Fix manifest_declarative_source + add unit tests (#25217)
* Fix + unit test

* Add a test with pagination

* Add a test with partition router

* Make sure _fetch_next_page is called with the right arguments

* Automated Commit - Formatting Changes

* pagination with partitions

* refactor

* clean up

* format

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-04-14 14:05:22 -07:00
girarda
0d9392e4ae 🤖 Bump patch version of Airbyte CDK 2023-04-14 17:30:23 +00:00
Alexandre Girard
f3799280f2 connector builder: Emit message at start of slice (#25180)
* Move condition for yielding the slice message to an overwritable method

* Automated Commit - Formatting Changes

* yield the slice log messages

* same for incremental

* refactor

* Revert "refactor"

This reverts commit c594365bd8.

* move flag from factory to source

* set the flag

* remove debug print

* halfmock

* clean up

* Add a test for a single page

* Add another test

* Pass the flag

* rename

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-04-14 10:23:59 -07:00
girarda
08067008ca 🤖 Bump patch version of Airbyte CDK 2023-04-12 21:54:31 +00:00
Alexandre Girard
71fc3dd517 Connector builder: set pages and slices limits (#25121)
* Set limits

* refactor and add unit tests

* Update as per comments
2023-04-12 14:46:43 -07:00
Alexandre Girard
f969ebd63b low-code: Add some unit tests for CompositeErrorHandler (#24930)
* Add some unit tests

* update

* fix indent

* fix warnings
2023-04-10 12:12:36 -07:00
Brian Lai
3ba15b5fcb Decouple flags for debug messages from connector builder log messages (#24881)
* decouple debug message flags from connector builder grouping messages

* Automated Commit - Formatting Changes

* pr feedback simplifying configs a bit

---------

Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
2023-04-06 12:16:05 -04:00
davydov-d
2b09f4faa0 🤖 Bump minor version of Airbyte CDK 2023-04-06 09:21:37 +00:00
Denys Davydov
30e2447ceb Lowcode CDK: implement single use refresh token oauth authenticator (#24891)
* #24658 Lowcode CDK: implement single use refresh token oauth authenticator

* #24658 lowcode cdk: review fixes
2023-04-06 12:15:36 +03:00
girarda
e1fab6c98c 🤖 Bump patch version of Airbyte CDK 2023-04-05 00:06:47 +00:00
Alexandre Girard
4b324c3084 Low-code: fix duplicate stream_slicer update (#24827)
* first tentative fix

* cleaner fix

* refactor test

* format

* format

* move to utils file

* use simpler implementation
2023-04-04 15:40:33 -07:00
grubberr
28c49d4048 🤖 Bump patch version of Airbyte CDK 2023-04-03 10:36:32 +00:00
Serhii Chvaliuk
032f9b8045 Low-Code CDK: make RecordFilter.filter_records as generator (#24772)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2023-04-03 13:29:24 +03:00
girarda
cf94c14ae5 🤖 Bump minor version of Airbyte CDK 2023-03-29 21:24:30 +00:00
Alexandre Girard
c3b017c7b5 Add auth flow to declarative manifest schema (#24441)
* Add auth flow to declarative manifest schema

* Rename

* fix rename

* set advanced_auth

* Automated Commit - Formatting Changes

* update unit test

* format

* Add examples

* example -> examples

* add missing examples

* Automated Commit - Formatting Changes

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-03-29 16:06:56 -05:00
alafanechere
69736d9f40 🤖 Bump minor version of Airbyte CDK 2023-03-23 14:09:17 +00:00
Augustin
bad5bce8ce CDK: remove unexpected error swallowing on abstract source's check method (#24240) 2023-03-23 13:04:51 +00:00
Alexandre Girard
aaf00deb38 Add missing type hints (#24345) 2023-03-22 07:15:54 -07:00
girarda
62caa92f22 🤖 Bump patch version of Airbyte CDK 2023-03-22 00:44:23 +00:00
Alexandre Girard
edfc59533d Connector builder: Port "send stacktrace when error on read" to CDK connector builder module (#24173)
* wip

* fix unit test

* fix other unit test

* format

* reset

* format

* missing unit test

* yield a LogMessage on error

* format

* format

* fix unit tests

* yield a trace message instead of a log message

* format

* fix bad merge
2023-03-21 17:22:08 -07:00
clnoll
99873c3a6c 🤖 Bump minor version of Airbyte CDK 2023-03-21 17:39:39 +00:00
Catherine Noll
f4fd4d98a2 Connector Builder: Make connector_builder part of the CDK package (#24280) 2023-03-21 13:31:16 -04:00
maxi297
d47ccd168e 🤖 Bump patch version of Airbyte CDK 2023-03-21 13:06:52 +00:00
Maxime Carbonneau-Leclerc
98719cf3f3 [ISSUE #23794] CDK's read command handler supports Connector Builder … (#24204)
* [ISSUE #23794] CDK's read command handler supports Connector Builder list_streams requests

* [ISSUE #23794] code review
2023-03-21 09:01:33 -04:00
Catherine Noll
deadeca0cf Connector Builder: assert catalog exists on test_read requests (#24199) 2023-03-20 19:35:11 -04:00
Catherine Noll
e890d01d55 Connector builder: handle empty catalog (#24184) 2023-03-17 12:51:10 -04:00
Brian Lai
903d34e5f1 [Low-Code CDK] Enforce manifest against the airbyte-cdk version and the Beta version 0.29.0 (#23796)
* enforce manifest version correctness against the CDK package being used

* parse versions into parts for better comparisons and error checking

* fix pr feedback and derp forgot to actually add the commit with the low-code manifests updated to the beta version

* pr feedback and fix new tests since last rebase
2023-03-16 00:50:30 -04:00
Alexandre Girard
bb5741a0c0 Connector builder: support for test read with message grouping per slices (#23925)
* New connector_builder module for handling requests from the Connector Builder.

Also implements `resolve_manifest` handler

* Automated Commit - Formatting Changes

* Rename ConnectorBuilderSource to ConnectorBuilderHandler

* Update source_declarative_manifest README

* Reorganize

* read records

* paste unit tests from connector builder server

* compiles but tests fail

* first test passes

* Second test passes

* 3rd test passes

* one more test

* another test

* one more test

* test

* return StreamRead

* test

* test

* rename

* test

* test

* test

* main seems to work

* Update

* Update

* Update

* Update

* update

* error message

* rename

* update

* Update

* CR improvements

* fix test_source_declarative_manifest

* fix tests

* Update

* Update

* Update

* Update

* rename

* rename

* rename

* format

* Give connector_builder its own main.py

* Update

* reset

* delete dead code

* remove debug print

* update test

* Update

* set right stream

* Add --catalog argument

* Remove unneeded preparse

* Update README

* handle error

* tests pass

* more explicit test

* reset

* format

* fix merge

* raise exception

* fix

* black format

* raise with config

* update

* fix flake

* __test_read_config is optional

* fix

* Automated Commit - Formatting Changes

* fix

* exclude_unset

---------

Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: clnoll <clnoll@users.noreply.github.com>
Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-03-15 17:12:37 -07:00
Catherine Noll
50849fc49a Connector Builder: print jsonified AirbyteMessage (#24111) 2023-03-15 18:59:23 +00:00
maxi297
dd7da58759 🤖 Bump patch version of Airbyte CDK 2023-03-14 18:58:59 +00:00
Maxime Carbonneau-Leclerc
2e3412d9dd [ISSUE #24059] fix reset pagination issue (#24069)
* [ISSUE #24059] fix reset pagination issue

* Automated Commit - Formatting Changes
2023-03-14 18:51:12 +00:00
Catherine Noll
8ee32b1132 New connector_builder module for handling requests from the Connector Builder (#23888)
Also implements `resolve_manifest` handler
2023-03-14 13:51:27 -04:00
Alexandre Girard
5fecc7e626 add missing newline (#24035)
* add missing newline

* Automated Commit - Formatting Changes

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-03-14 00:37:34 +00:00
Alexandre Girard
02f771b422 Do not remove trailing slash from path (#24003)
* Do not remove trailing slash from path

* Add a breaking test

* Add some tests on HttpStream
2023-03-13 23:59:24 +00:00
Sherif A. Nada
8d6f130666 remove outdated code comment 2023-03-13 09:54:59 -07:00
Leo Schick
09955423db support reading "expires_in" when the API passes the value as string (#23921)
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2023-03-10 09:02:14 -08:00
sherifnada
df31b814e8 🤖 Bump patch version of Airbyte CDK 2023-03-10 01:35:25 +00:00
Sherif A. Nada
ec9707c71f Low-code CDK: Override refresh_access_token logic DeclarativeOAuthAuthenticator (#23927)
Co-authored-by: sherifnada <sherifnada@users.noreply.github.com>
2023-03-09 17:28:33 -08:00
Sherif A. Nada
b2f748b9a3 Revert "fix(python-cdk): convert expires_in to int when refreshing (#20301)" (#23924)
This reverts commit d67afbbd17.
2023-03-09 15:54:07 -08:00
Joël Luijmes
d67afbbd17 fix(python-cdk): convert expires_in to int when refreshing (#20301) 2023-03-09 13:46:10 -08:00
Alexandre Girard
f47219037e dont publish to test pypi (#23709) 2023-03-09 13:40:22 -08:00
Brian Lai
52f83d4cda [source-monday] [source-square] [source-greenhouse] [source-tempo] [source-zenloop] fix a few connectors with failing custom component unit tests + release some connectors w/ beta (#23231)
* fix a few connectors with failing custom component unit tests

* fix outdated expected records

* make square orders stream incremental

* Updated square SAT and delete custom extractor for monday.

* Added allowedHosts

* Delete DpathStringExtractor from init

* Updated abnormal records

* bump versions of greenhouse, tempo, woocommerce, and zenloop to use beta low code version

* remove woocommerce because it's already published with beta

* add allowedHosts for tempo and zenloop

* fix abnormal state file to use created_at for some streams

* auto-bump connector version

* bump versions for square and monday

* auto-bump connector version

* auto-bump connector version

* manually update square definition generation due to failure

* auto-bump connector version

---------

Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-08 14:44:27 -05:00
maxi297
454906e7fa 🤖 Bump patch version of Airbyte CDK 2023-03-07 19:53:52 +00:00
davydov-d
f25f9eccc5 🤖 Bump minor version of Airbyte CDK 2023-03-07 17:47:45 +00:00