1
0
mirror of synced 2026-02-02 16:02:07 -05:00
Commit Graph

199 Commits

Author SHA1 Message Date
Marius Posta
3e680675a4 github workflows: repo-wide auto-format (#29798)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-08-25 10:20:41 -07:00
Alexandre Girard
b4ce532762 low-code: Allow formatting datetimes as milliseconds since unix epoch (#29504)
Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-08-17 18:49:28 -07:00
Lake Mossman
4bf6b8e15a Fix title & description of datetime_format field (#29025)
* fix description of datetime_format field

* Automated Commit - Formatting Changes

* improve description of cursor datetime formats field

* Automated Commit - Formatting Changes

---------

Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2023-08-04 14:04:24 -07:00
Joe Reuter
df3b1d9c8d 🚨🚨 Low code CDK: Decouple SimpleRetriever and HttpStream (#28657)
* fix tests

* format

* review comments

* Automated Commit - Formatting Changes

* review comments

* review comments

* review comments

* log all messages

* log all message

* review comments

* review comments

* Automated Commit - Formatting Changes

* add comment

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
2023-08-03 12:30:59 +02:00
Catherine Noll
09ebb47b24 File cdk parser and cursor updates (#28900)
* File-based CDK: update parquet parser to handle partitions

* File-based CDK: make the record output & cursor date time format consistent
2023-08-01 21:47:58 -04:00
Maxime Carbonneau-Leclerc
e158bec4b2 [ISSUE #28782] support multiple cursor field datetime formats (#28936)
* [ISSUE #28782] support multiple cursor field datetime formats

* Making sure we use the proper format for creating slices

* Code review
2023-08-01 17:59:17 -04:00
Lake Mossman
f66079cdb1 Capitalize title of request_authentication field (#28585)
* capitalize title of request_authentication field

* Automated Commit - Formatting Changes

---------

Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2023-07-25 17:17:27 -06:00
Alexandre Girard
df01616951 [Issue #23497] Deduplicate query parameters for declarative connectors (#28550)
* remove duplicate param

* remove duplicate params

* fix some of the typing issues

* fix typing issues

* newline

* format

* Enable by default

* Add missing file

* refactor and remove flag

* none check

* move line of code

* fix typing in rate_limiting

* comment

* use typedef

* else branch

* format

* gate the feature

* rename test

* fix the test

* only dedupe if the values are the same

* Add some tests

* convert values to strings

* Document the change

* implement in requester too
2023-07-25 14:22:25 -07:00
Maxime Carbonneau-Leclerc
b1a5f270ae Fix remove field transform (#28518)
* Fix remove field transform

* mypy
2023-07-20 10:15:42 -04:00
Joe Reuter
58cc540c6b 🚨 Low code CDK: Add session token authenticator (#28050)
This PR adds a new authenticator: The SessionTokenAuthenticator. The existing authenticator under the same name is renamed to LegacySessionTokenAuthenticator.
2023-07-19 17:10:24 +02:00
Joe Reuter
78728410f4 Low code CDK: Fix mypy errors (#28386)
* ingore unit tests in mypy check

* Update airbyte-cdk/python/bin/run-mypy-on-modified-files.sh

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>

* work through mypy errors

* fix a bunch of stuff

* fix more type hints

* fix model_to_component_factory types

* format

* ignore list instead of allow list

---------

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
2023-07-19 15:08:35 +02:00
Maxime Carbonneau-Leclerc
df2a6e50bb Issue 21014/oauth requests (#27973)
* [ISSUE #27494] fix type issue caused by connector builder logging

* [ISSUE #21014] log request/response for oauth as 'global_requests'

* formatcdk

* [ISSUE #21014] support DeclarativeOauth2Authenticator as well

* [ISSUE #21014] improving message grouper tests

* formatcdk

* Test solution with logic in MessageRepository (#27990)

* Test solution with logic in MessageRepository

* Solution without creating a new ModelToComponentFactory

* [ISSUE #21014] adding tests

* [ISSUE #21014] add title and description to global requests

* Revert "Solution without creating a new ModelToComponentFactory"

This reverts commit f17799ecff.

* Automated Commit - Formatting Changes

* [ISSUE #21014] code review

* [ISSUE #21014] do not break on log appender conflict

* Automated Commit - Formatting Changes

* [ISSUE #21014] code review

* formatcdk

* [ISSUE #21014] moving is_global to is_auxiliary
2023-07-11 13:37:38 -04:00
Maxime Carbonneau-Leclerc
c609897848 Stream state is not recorded if cursor field is result of transformation (#27915)
* [ISSUE #27494] move transformation for record selection

* formatcdk

* [ISSUE #27494] fix type issue caused by connector builder logging

* formatcdk

* [ISSUE #27494] code review
2023-07-06 08:02:16 -04:00
Lake Mossman
b78762f641 Update the request_body_data description to remove typos and be more readable (#27783)
* fix typo in schema

* remove hyphen
2023-07-03 15:45:59 -07:00
Maxime Carbonneau-Leclerc
4376527266 Fixing an issue with square as CATs only compare string and not datetime (#27840)
* Fixing an issue with square as CATs only compare string and not datetime

* formatcdk
2023-06-29 13:23:06 -04:00
Maxime Carbonneau-Leclerc
91a56171a0 [ISSUE #26343] update close_slice to use the greater record (#27818)
* [ISSUE #26343] update close_slice to use the greater record

* Renaming parameter for close_slice

* code review
2023-06-29 10:09:12 -04:00
Maxime Carbonneau-Leclerc
a013fad5a9 [ISSUE-26343] data feed (#27475)
* [ISSUE #26581] per partition cursor

* [ISSUE #26581] format

* [ISSUE #26581] clean up state management

* [ISSUE #26581] improving Hashabledict

* [ISSUE #26581] format cdk

* [ISSUE #26581] fix tests

* [ISSUE #26581] code review from girarda

* Retrigger pipeline

* Decouple cursor and stream slicer and pushing state management as far up cursor as possible

* Format cdk

* Small fixes/comments

* DatetimeBasedCursor should not update state based on slice (for now at least since it wasn't doing this before)

* [ISSUE #26581] code review

* Automated Commit - Formatting Changes

* [ISSUE #26581] validation overlapping keys

* [ISSUE #26581] add typing

* [ISSUE #26581] code review

* Remove SyncMode from stream_slices

* Removing SyncMode from stream_slices up until SimpleRetriever and fixing typing

* [ISSUE-26434] replacing Record primitive by class

* [ISSUE-26434] update Cursor.update_state to use new record object

* Issue 26343/data feed incremental sync solution 2 (#27481)

* TMP [ISSUE-26434] first solution to enable stop condition on pagination

* TMP [ISSUE-26434] second solution to enable stop condition on pagination

* TMP [ISSUE-26434] second solution fix

* [ISSUE #26343] fixing behavior and adding tests

* [ISSUE #26343] only updating state once a slice to allow for data feed

* [ISSUE #26343] removing freezing of cursor

* format cdk

* [ISSUE #26343] ensure data_feed doesn't have end_datetime

* [ISSUE #26343] self review

* [ISSUE #26343] code review

* [ISSUE #26343] code review clean up

* [ISSUE #26343] code review clean up

* Code review

* [ISSUE #26343] add warn log message in DatetimeBasedCursor

* format

* Format
2023-06-28 16:53:00 -04:00
Alexandre Girard
4d08781d04 Revert "Low-Code CDK: make RecordFilter.filter_records as generator (#24772)" (#27789)
This reverts commit 032f9b8045.
2023-06-28 09:23:05 -07:00
Joe Reuter
8aba48810c Low-code CDK: Serialize request body as string for connector builder module (#27657)
* serialize request body as string

* fix some bugs
2023-06-27 08:27:16 +02:00
midavadim
c44c3eae48 CDK: availability check - handle HttpErrors which happen during slice extraction (#26630)
* for availability check - handle  HttError happens during slice extraction (reading of parent stream),
updated reason messages,
moved check availability call under common try/except which handles errors during usual stream read,
moved log messages which indicate start of the stream sync before availability check in to make to understand which stream is the source of errors

* why do we return here and not try next stream?

* fixed bug in CheckStream, now we try to check availability for all streams
2023-06-23 13:15:25 -04:00
Joe Reuter
c53d1fa29d Datetime inferrer: Improve detected formats (#27546)
* consolidate formats

* Automated Commit - Formatting Changes

* consolidate formats

* consolidate formats

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
2023-06-23 05:23:33 -04:00
Maxime Carbonneau-Leclerc
a45a1e3341 Maxi297/refactoring declarative state management (#27445)
* [ISSUE #26581] per partition cursor

* [ISSUE #26581] format

* [ISSUE #26581] clean up state management

* [ISSUE #26581] improving Hashabledict

* [ISSUE #26581] format cdk

* [ISSUE #26581] fix tests

* [ISSUE #26581] code review from girarda

* Retrigger pipeline

* Decouple cursor and stream slicer and pushing state management as far up cursor as possible

* Format cdk

* Small fixes/comments

* DatetimeBasedCursor should not update state based on slice (for now at least since it wasn't doing this before)

* [ISSUE #26581] code review

* Automated Commit - Formatting Changes

* [ISSUE #26581] validation overlapping keys

* [ISSUE #26581] add typing

* [ISSUE #26581] code review

* Remove SyncMode from stream_slices

* Removing SyncMode from stream_slices up until SimpleRetriever and fixing typing

* format cdk
2023-06-22 12:54:36 -04:00
Alexandre Girard
d548587161 [ISSUE #27289] Document macros output in the manifest schema (#27600)
* Add example for macros

* Update changelog

* Revert "Update changelog"

This reverts commit 2993e5820e.
2023-06-22 09:14:17 -07:00
Maxime Carbonneau-Leclerc
d9a5e2d873 🐛 Source zenloop: update to state per partition (#27556)
* [ISSUE #26581] per partition cursor

* [ISSUE #26581] format

* [ISSUE #26581] clean up state management

* [ISSUE #26581] improving Hashabledict

* [ISSUE #26581] format cdk

* [ISSUE #26581] fix tests

* [ISSUE #26581] code review from girarda

* Retrigger pipeline

* [ISSUE #26581] code review

* Automated Commit - Formatting Changes

* [ISSUE #26581] validation overlapping keys

* [ISSUE #26581] add typing

* [ISSUE #26581] code review

* [ISSUE #26607] zenloop migration (#27243)

* [ISSUE #26607] zenloop migration implementation without tests

* [ISSUE #26607] zenloop migration adding edge cases

* [ISSUE #26607] add cursor field for state

* [ISSUE #26607] update abnormal state

* [ISSUE #26607] ensure default state

* [ISSUE #26607] updating CATs state

* [ISSUE #26607] revert migrating cursor

* [ISSUE #26607] remove default cursor value

* [ISSUE #26607] improve error message

* [ISSUE #26607] changelog

---------

Co-authored-by: Augustin <augustin@airbyte.io>

* 🤖 Auto format source-zenloop code [skip ci]

* Automated Commit - Formatting Changes

* [ISSUE #26581] move partition serialization to JSON

* Revert "[ISSUE #26607] zenloop migration (#27243)"

This reverts commit 5c6f19b775.

* Revert "Revert "[ISSUE #26607] zenloop migration (#27243)""

This reverts commit e363fd6cb8.

* [ISSUE #26607] update zenloop version

* TMP specify cdk version

* [ISSUE #26607] do not lock zenloop airbyte_cdk version

* trigger pipeline

* Automated Commit - Formatting Changes

* trigger pipeline

---------

Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
2023-06-22 08:42:20 -05:00
Maxime Carbonneau-Leclerc
8926970c86 Issue 26581/per partition cursor (#27223)
* [ISSUE #26581] per partition cursor

* [ISSUE #26581] format

* [ISSUE #26581] clean up state management

* [ISSUE #26581] improving Hashabledict

* [ISSUE #26581] format cdk

* [ISSUE #26581] fix tests

* [ISSUE #26581] code review from girarda

* Retrigger pipeline

* [ISSUE #26581] code review

* Automated Commit - Formatting Changes

* [ISSUE #26581] validation overlapping keys

* [ISSUE #26581] add typing

* [ISSUE #26581] code review

* [ISSUE #26607] zenloop migration (#27243)

* [ISSUE #26607] zenloop migration implementation without tests

* [ISSUE #26607] zenloop migration adding edge cases

* [ISSUE #26607] add cursor field for state

* [ISSUE #26607] update abnormal state

* [ISSUE #26607] ensure default state

* [ISSUE #26607] updating CATs state

* [ISSUE #26607] revert migrating cursor

* [ISSUE #26607] remove default cursor value

* [ISSUE #26607] improve error message

* [ISSUE #26607] changelog

---------

Co-authored-by: Augustin <augustin@airbyte.io>

* 🤖 Auto format source-zenloop code [skip ci]

* Automated Commit - Formatting Changes

* [ISSUE #26581] move partition serialization to JSON

* Revert "[ISSUE #26607] zenloop migration (#27243)"

This reverts commit 5c6f19b775.

* [ISSUE #26607] revert zenloop

---------

Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
2023-06-21 12:59:11 -04:00
Lake Mossman
8be753df02 add link to record selector docs (#27291)
* add link to record selector docs

* Automated Commit - Formatting Changes

---------

Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2023-06-16 12:09:59 -07:00
Maxime Carbonneau-Leclerc
f48849fdb4 [ISSUE #26909] adding message repository (#27158)
* [ISSUE #26909] adding message repository

* Automated Commit - Formatting Changes

* [ISSUE #26909] improve entrypoint error handling

* format CDK

* [ISSUE #26909] adding an integration test
2023-06-13 08:40:55 -04:00
Joe Reuter
27635ba26a Low code CDK: Datetime format documentation (#27149)
* add format documentation

* fix

* improve
2023-06-12 17:39:12 -07:00
Lake Mossman
a2c57c4fd4 Add metadata field to low-code schema (#27094)
* add metadata field to low-code schema

* Automated Commit - Formatting Changes

* update wording of description

* fix formatting

---------

Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2023-06-08 08:14:42 -07:00
Maxime Carbonneau-Leclerc
13834395bb [ISSUE #26568] make DatetimeBasedCursor.end_datetime optional (#27031)
* [ISSUE #26568] make DatetimeBasedCursor.end_datetime optional

* [ISSUE #26568] ensure model_to_component_factory create end_datetime properly

* [ISSUE #26568] fix typing

* [ISSUE #26568] improve end_datetime documentation
2023-06-08 09:51:40 -04:00
Joe Reuter
c4981a72db Low code CDK: Allow query param / body injection for api key authenticator (#26953)
* make authenticator more flexible

* format

* format

* format

* format

* format

* fix problem

* code review
2023-06-07 15:08:13 +02:00
Joe Reuter
e19c498b00 Remove lowcode SingleUseRefreshTokenOauthAuthenticator (#27053)
* wip

* Automated Commit - Formatting Changes

* add documentation

* tests and fixes

* fix tests

* more documentation

* revert

* changes as discussed

* fix case

* add docstring

* add details to schema

* format

* remove old singleuserefreshtokenauthenticator from lowcode

* prepare quickbooks connector

* format

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
2023-06-07 15:07:50 +02:00
Joe Reuter
b34fb00660 Extend low code OAuthAuthenticator with token refresh capabilities (#26966)
* wip

* Automated Commit - Formatting Changes

* add documentation

* tests and fixes

* fix tests

* more documentation

* revert

* changes as discussed

* fix case

* add docstring

* add details to schema

* format

* fix bug

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
2023-06-07 10:51:59 +02:00
Maxime Carbonneau-Leclerc
b5c0ac15ec [ISSUE #26570] make step and cursor_granularity optional (#26952)
* [ISSUE #26570] make step and cursor_granularity optional

* [ISSUE #26570] fix typos
2023-06-05 09:10:48 -04:00
Sherif A. Nada
8c4132eb7b lowcode: Better describe what the cursor field is for (#26815) 2023-05-31 09:24:40 -07:00
Joe Reuter
4a041bf77d Low code CDK: Allow nested objects for request_body_json (#26474)
* allow nested JSON

* add test for boolean

* review comment

* change for testing

* try fix

* try another fix

* Revert "change for testing"

This reverts commit 931b935778.

* Revert "try fix"

This reverts commit 6f1c6c0e4b.
2023-05-26 10:52:24 +02:00
Brian Lai
5707e477ad low-code cdk: make page_size optional for offset and page increment strategies (#26056)
* make page_size optional

* Automated Commit - Formatting Changes

---------

Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
2023-05-24 17:21:41 -04:00
Alexandre Girard
fa626c6f18 Update low-code reference docs to mention that %s should be used for unix timestamp (#26096)
* Mention that %s should be used for unix timestamp

* update docs and add unit test

* Update test

* Update
2023-05-22 17:05:56 -07:00
Joe Reuter
13752e29af Low-code CDK: Improve grant type documentation (#26335)
* improve grant type documentation

* for the other

* Automated Commit - Formatting Changes

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
2023-05-22 12:06:19 -04:00
Joe Reuter
dbb766c255 Low code CDK: Make refresh token in oauth authenticator optional (#26031)
* make refresh_token optional

* format

* add back type annotation
2023-05-15 14:47:48 +02:00
Catherine Noll
051c656aba [lowcode] OAuth Authenticator: cast token expiry time to number (#26020) 2023-05-12 14:31:20 -04:00
Alexandre Girard
7443970de3 low-code: Use Jinja sandbox environment and prevent use of range method (#25589)
* secure the jinja environment

* format

* Update comment

* remove extra test

* remove lambda

* Update

* Raise an error on undefined variables

* remove unused import

* add unit tests to missing context vars and adjust error message

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
2023-05-08 19:34:53 -04:00
Maxime Carbonneau-Leclerc
b26c897a8b [ISSUE #25646] support parsing of non utc dates (#25665)
* [ISSUE #25646] support parsing of non utc dates

* [ISSUE #25646] improve parsing

* [ISSUE #25646] removing timezone for DatetimeParser interface

* [ISSUE #25646] fix tests
2023-05-01 12:16:44 -04:00
Alexandre Girard
a4b518fb30 low-code: Update docs on incremental syncs (#25696)
* Update docs

* Automated Commit - Formatting Changes

* update

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-04-28 18:55:07 -07:00
Alexandre Girard
e41060c02c low-code: Fix type check in DeclarativeStream (#25533)
* Set right type

* Update the comment

* Update

* format

* Update comment
2023-04-26 15:49:10 -07:00
Maxime Carbonneau-Leclerc
3cc67a6d9e [ISSUE #23382] ignore backoff configuration on test reads (#25429) 2023-04-26 08:36:59 -04:00
Alexandre Girard
1c13a0af6a Update jinja variable description (#25440)
* Update description

* Add examples

* format

* update

* update

* Update declarative_component_schema.yaml

* remove superfluous -

---------

Co-authored-by: Joe Reuter <joe@airbyte.io>
2023-04-25 20:44:05 -07:00
Alexandre Girard
250c3b1c87 low-code: Delete now_local macro (#25404)
* Delete now_local macro

* Remove from reference docs

* remove example
2023-04-25 20:40:46 -07:00
Alexandre Girard
645763588c low-code: Alias stream_interval and stream_partition to stream_slice in interpolation context (#25373)
* add aliases

* Raise error if the alias is found in the context

* format

* Comment

* Automated Commit - Formatting Changes

* rename to stream partition in greenhouse manifest

* Revert "rename to stream partition in greenhouse manifest"

This reverts commit d513ef418f.

* Clean up test

* Other test

* last test

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-04-24 18:25:54 -07:00
Alexandre Girard
15f90e3a2f Fix and document macros and interpolation variables (#25305)
* Fix and document macros

* cleanup

* dots

* Add tests and refactor

* Update

* Add an example

* Document variables

* Mention now_local is not recommended
2023-04-21 10:58:53 -07:00