1
0
mirror of synced 2026-01-26 22:02:03 -05:00
Commit Graph

31 Commits

Author SHA1 Message Date
Alexandre Girard
86ee91ed5d Connector builder: read input state if it exists (#37495) 2024-04-24 15:53:09 -07:00
Alexandre Girard
a2e908dc17 connector builder: Set state on stream slices (#37109) 2024-04-18 16:16:02 -07:00
Ella Rohm-Ensing
b7819d9f6c python: assert actual == expected ordering (#36980) 2024-04-11 15:16:33 +00:00
Alexandre Girard
b27ddfe19e connector-builder: return full url-encoded URL instead of separating parameters (#36680) 2024-03-28 18:49:35 -07:00
Maxime Carbonneau-Leclerc
2f34f084e4 [ISSUE #6548] make all fields nullable except from pk and cursor field (#36201) 2024-03-20 09:48:38 -04:00
Alex Birdsall
385a70d89d Support user-specified test read limits in connector_builder code (#35312) 2024-02-16 15:53:26 -08:00
Artem Inzhyyants
9c6aea19cd Airbyte CDK: handle private network exception as config error (#33751) 2024-01-10 15:20:40 +01:00
Marius Posta
7ae97175a6 gradle: fix repo wide behaviour (#30607) 2023-09-28 05:01:13 -07:00
Maxime Carbonneau-Leclerc
b335880fda jira invalid user-provided urls generating sentry issues (#30672) 2023-09-21 15:01:17 -04:00
Joe Reuter
f8de9d12df CDK: Remove list endpoint (#29581) 2023-08-21 12:43:44 +02:00
Joe Reuter
df3b1d9c8d 🚨🚨 Low code CDK: Decouple SimpleRetriever and HttpStream (#28657)
* fix tests

* format

* review comments

* Automated Commit - Formatting Changes

* review comments

* review comments

* review comments

* log all messages

* log all message

* review comments

* review comments

* Automated Commit - Formatting Changes

* add comment

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
2023-08-03 12:30:59 +02:00
Maxime Carbonneau-Leclerc
48bf520d87 Fix stream read given stream doesn't have any slice (#28746)
* Fix stream read given stream doesn't have any slice

* Not return slices if there are none

* Fix test
2023-07-27 10:05:35 -04:00
Alexandre Girard
3ae73fb0ff connector builder: Set test_read_limit_reached to true if we hit the max records limit (#28293)
* set test_read_limit_reached to true if we hit the max records limit

* rename slice to _slice to avoid shadowing a builtin keyword

* newline

* fix some of the typing issues

* fix some more typing issues

* another fix

* fix last typing issue

* format

* Automated Commit - Formatting Changes

* reset type

* fix the type

* Update for clarity

* Update types

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-07-18 15:53:53 -07:00
Maxime Carbonneau-Leclerc
df2a6e50bb Issue 21014/oauth requests (#27973)
* [ISSUE #27494] fix type issue caused by connector builder logging

* [ISSUE #21014] log request/response for oauth as 'global_requests'

* formatcdk

* [ISSUE #21014] support DeclarativeOauth2Authenticator as well

* [ISSUE #21014] improving message grouper tests

* formatcdk

* Test solution with logic in MessageRepository (#27990)

* Test solution with logic in MessageRepository

* Solution without creating a new ModelToComponentFactory

* [ISSUE #21014] adding tests

* [ISSUE #21014] add title and description to global requests

* Revert "Solution without creating a new ModelToComponentFactory"

This reverts commit f17799ecff.

* Automated Commit - Formatting Changes

* [ISSUE #21014] code review

* [ISSUE #21014] do not break on log appender conflict

* Automated Commit - Formatting Changes

* [ISSUE #21014] code review

* formatcdk

* [ISSUE #21014] moving is_global to is_auxiliary
2023-07-11 13:37:38 -04:00
Brian Lai
02e4bd07f7 [26989] Add request filter for cloud and integration test fixtures for e2e sync testing (#27534)
* add the request filters and integration test fixtures

* pr feedback and some tweaks to the testing framework

* optimize the cache for more hits

* formatting

* remove cache
2023-06-22 12:14:07 -04:00
Maxime Carbonneau-Leclerc
f48849fdb4 [ISSUE #26909] adding message repository (#27158)
* [ISSUE #26909] adding message repository

* Automated Commit - Formatting Changes

* [ISSUE #26909] improve entrypoint error handling

* format CDK

* [ISSUE #26909] adding an integration test
2023-06-13 08:40:55 -04:00
Joe Reuter
d6512dea2c CDK: Datetime format inferrer (#27071)
* datetime inferrer class

* format

* pass inferred date formats along

* review comments
2023-06-09 10:33:54 +02:00
Maxime Carbonneau-Leclerc
4625cef571 [ISSUE #26909] add latest connector config control message to connect… (#26922)
* [ISSUE #26909] add latest connector config control message to connector builder API

* [ISSUE #26909] flake

* Automated Commit - Formatting Changes

* [ISSUE #26909] fallback on in-memory dict if no config control message

* [ISSUE #26909] update and add tests
2023-06-07 08:31:45 -04:00
Maxime Carbonneau-Leclerc
d54a68640f Improving error messages to have better messaging in datadog and the … (#26860)
* Improving error messages to have better messaging in datadog and the frontend

* fixing tests
2023-05-31 15:36:27 -04:00
Maxime Carbonneau-Leclerc
0efc18a114 [ISSUE #24720] connector builder set slice descriptor (#25677) 2023-05-01 12:18:22 -04:00
Maxime Carbonneau-Leclerc
3cc67a6d9e [ISSUE #23382] ignore backoff configuration on test reads (#25429) 2023-04-26 08:36:59 -04:00
Maxime Carbonneau-Leclerc
4d65fa1b98 [ISSUE #23994] make MessageGrouper use AirbyteEntrypoint (#25402)
* [ISSUE #23994] make MessageGrouper use AirbyteEntrypoint

* [ISSUE #23994] code review
2023-04-24 11:24:15 -04:00
Alexandre Girard
f3799280f2 connector builder: Emit message at start of slice (#25180)
* Move condition for yielding the slice message to an overwritable method

* Automated Commit - Formatting Changes

* yield the slice log messages

* same for incremental

* refactor

* Revert "refactor"

This reverts commit c594365bd8.

* move flag from factory to source

* set the flag

* remove debug print

* halfmock

* clean up

* Add a test for a single page

* Add another test

* Pass the flag

* rename

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-04-14 10:23:59 -07:00
Alexandre Girard
71fc3dd517 Connector builder: set pages and slices limits (#25121)
* Set limits

* refactor and add unit tests

* Update as per comments
2023-04-12 14:46:43 -07:00
Alexandre Girard
edfc59533d Connector builder: Port "send stacktrace when error on read" to CDK connector builder module (#24173)
* wip

* fix unit test

* fix other unit test

* format

* reset

* format

* missing unit test

* yield a LogMessage on error

* format

* format

* fix unit tests

* yield a trace message instead of a log message

* format

* fix bad merge
2023-03-21 17:22:08 -07:00
Catherine Noll
f4fd4d98a2 Connector Builder: Make connector_builder part of the CDK package (#24280) 2023-03-21 13:31:16 -04:00
Maxime Carbonneau-Leclerc
98719cf3f3 [ISSUE #23794] CDK's read command handler supports Connector Builder … (#24204)
* [ISSUE #23794] CDK's read command handler supports Connector Builder list_streams requests

* [ISSUE #23794] code review
2023-03-21 09:01:33 -04:00
Catherine Noll
e890d01d55 Connector builder: handle empty catalog (#24184) 2023-03-17 12:51:10 -04:00
Brian Lai
903d34e5f1 [Low-Code CDK] Enforce manifest against the airbyte-cdk version and the Beta version 0.29.0 (#23796)
* enforce manifest version correctness against the CDK package being used

* parse versions into parts for better comparisons and error checking

* fix pr feedback and derp forgot to actually add the commit with the low-code manifests updated to the beta version

* pr feedback and fix new tests since last rebase
2023-03-16 00:50:30 -04:00
Alexandre Girard
bb5741a0c0 Connector builder: support for test read with message grouping per slices (#23925)
* New connector_builder module for handling requests from the Connector Builder.

Also implements `resolve_manifest` handler

* Automated Commit - Formatting Changes

* Rename ConnectorBuilderSource to ConnectorBuilderHandler

* Update source_declarative_manifest README

* Reorganize

* read records

* paste unit tests from connector builder server

* compiles but tests fail

* first test passes

* Second test passes

* 3rd test passes

* one more test

* another test

* one more test

* test

* return StreamRead

* test

* test

* rename

* test

* test

* test

* main seems to work

* Update

* Update

* Update

* Update

* update

* error message

* rename

* update

* Update

* CR improvements

* fix test_source_declarative_manifest

* fix tests

* Update

* Update

* Update

* Update

* rename

* rename

* rename

* format

* Give connector_builder its own main.py

* Update

* reset

* delete dead code

* remove debug print

* update test

* Update

* set right stream

* Add --catalog argument

* Remove unneeded preparse

* Update README

* handle error

* tests pass

* more explicit test

* reset

* format

* fix merge

* raise exception

* fix

* black format

* raise with config

* update

* fix flake

* __test_read_config is optional

* fix

* Automated Commit - Formatting Changes

* fix

* exclude_unset

---------

Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: clnoll <clnoll@users.noreply.github.com>
Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-03-15 17:12:37 -07:00
Catherine Noll
8ee32b1132 New connector_builder module for handling requests from the Connector Builder (#23888)
Also implements `resolve_manifest` handler
2023-03-14 13:51:27 -04:00