1
0
mirror of synced 2025-12-31 06:05:12 -05:00
Commit Graph

19 Commits

Author SHA1 Message Date
Alexandre Girard
3ae73fb0ff connector builder: Set test_read_limit_reached to true if we hit the max records limit (#28293)
* set test_read_limit_reached to true if we hit the max records limit

* rename slice to _slice to avoid shadowing a builtin keyword

* newline

* fix some of the typing issues

* fix some more typing issues

* another fix

* fix last typing issue

* format

* Automated Commit - Formatting Changes

* reset type

* fix the type

* Update for clarity

* Update types

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-07-18 15:53:53 -07:00
Maxime Carbonneau-Leclerc
df2a6e50bb Issue 21014/oauth requests (#27973)
* [ISSUE #27494] fix type issue caused by connector builder logging

* [ISSUE #21014] log request/response for oauth as 'global_requests'

* formatcdk

* [ISSUE #21014] support DeclarativeOauth2Authenticator as well

* [ISSUE #21014] improving message grouper tests

* formatcdk

* Test solution with logic in MessageRepository (#27990)

* Test solution with logic in MessageRepository

* Solution without creating a new ModelToComponentFactory

* [ISSUE #21014] adding tests

* [ISSUE #21014] add title and description to global requests

* Revert "Solution without creating a new ModelToComponentFactory"

This reverts commit f17799ecff.

* Automated Commit - Formatting Changes

* [ISSUE #21014] code review

* [ISSUE #21014] do not break on log appender conflict

* Automated Commit - Formatting Changes

* [ISSUE #21014] code review

* formatcdk

* [ISSUE #21014] moving is_global to is_auxiliary
2023-07-11 13:37:38 -04:00
Brian Lai
02e4bd07f7 [26989] Add request filter for cloud and integration test fixtures for e2e sync testing (#27534)
* add the request filters and integration test fixtures

* pr feedback and some tweaks to the testing framework

* optimize the cache for more hits

* formatting

* remove cache
2023-06-22 12:14:07 -04:00
Maxime Carbonneau-Leclerc
f48849fdb4 [ISSUE #26909] adding message repository (#27158)
* [ISSUE #26909] adding message repository

* Automated Commit - Formatting Changes

* [ISSUE #26909] improve entrypoint error handling

* format CDK

* [ISSUE #26909] adding an integration test
2023-06-13 08:40:55 -04:00
Joe Reuter
d6512dea2c CDK: Datetime format inferrer (#27071)
* datetime inferrer class

* format

* pass inferred date formats along

* review comments
2023-06-09 10:33:54 +02:00
Maxime Carbonneau-Leclerc
4625cef571 [ISSUE #26909] add latest connector config control message to connect… (#26922)
* [ISSUE #26909] add latest connector config control message to connector builder API

* [ISSUE #26909] flake

* Automated Commit - Formatting Changes

* [ISSUE #26909] fallback on in-memory dict if no config control message

* [ISSUE #26909] update and add tests
2023-06-07 08:31:45 -04:00
Maxime Carbonneau-Leclerc
d54a68640f Improving error messages to have better messaging in datadog and the … (#26860)
* Improving error messages to have better messaging in datadog and the frontend

* fixing tests
2023-05-31 15:36:27 -04:00
Maxime Carbonneau-Leclerc
0efc18a114 [ISSUE #24720] connector builder set slice descriptor (#25677) 2023-05-01 12:18:22 -04:00
Maxime Carbonneau-Leclerc
3cc67a6d9e [ISSUE #23382] ignore backoff configuration on test reads (#25429) 2023-04-26 08:36:59 -04:00
Maxime Carbonneau-Leclerc
4d65fa1b98 [ISSUE #23994] make MessageGrouper use AirbyteEntrypoint (#25402)
* [ISSUE #23994] make MessageGrouper use AirbyteEntrypoint

* [ISSUE #23994] code review
2023-04-24 11:24:15 -04:00
Alexandre Girard
f3799280f2 connector builder: Emit message at start of slice (#25180)
* Move condition for yielding the slice message to an overwritable method

* Automated Commit - Formatting Changes

* yield the slice log messages

* same for incremental

* refactor

* Revert "refactor"

This reverts commit c594365bd8.

* move flag from factory to source

* set the flag

* remove debug print

* halfmock

* clean up

* Add a test for a single page

* Add another test

* Pass the flag

* rename

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-04-14 10:23:59 -07:00
Alexandre Girard
71fc3dd517 Connector builder: set pages and slices limits (#25121)
* Set limits

* refactor and add unit tests

* Update as per comments
2023-04-12 14:46:43 -07:00
Alexandre Girard
edfc59533d Connector builder: Port "send stacktrace when error on read" to CDK connector builder module (#24173)
* wip

* fix unit test

* fix other unit test

* format

* reset

* format

* missing unit test

* yield a LogMessage on error

* format

* format

* fix unit tests

* yield a trace message instead of a log message

* format

* fix bad merge
2023-03-21 17:22:08 -07:00
Catherine Noll
f4fd4d98a2 Connector Builder: Make connector_builder part of the CDK package (#24280) 2023-03-21 13:31:16 -04:00
Maxime Carbonneau-Leclerc
98719cf3f3 [ISSUE #23794] CDK's read command handler supports Connector Builder … (#24204)
* [ISSUE #23794] CDK's read command handler supports Connector Builder list_streams requests

* [ISSUE #23794] code review
2023-03-21 09:01:33 -04:00
Catherine Noll
e890d01d55 Connector builder: handle empty catalog (#24184) 2023-03-17 12:51:10 -04:00
Brian Lai
903d34e5f1 [Low-Code CDK] Enforce manifest against the airbyte-cdk version and the Beta version 0.29.0 (#23796)
* enforce manifest version correctness against the CDK package being used

* parse versions into parts for better comparisons and error checking

* fix pr feedback and derp forgot to actually add the commit with the low-code manifests updated to the beta version

* pr feedback and fix new tests since last rebase
2023-03-16 00:50:30 -04:00
Alexandre Girard
bb5741a0c0 Connector builder: support for test read with message grouping per slices (#23925)
* New connector_builder module for handling requests from the Connector Builder.

Also implements `resolve_manifest` handler

* Automated Commit - Formatting Changes

* Rename ConnectorBuilderSource to ConnectorBuilderHandler

* Update source_declarative_manifest README

* Reorganize

* read records

* paste unit tests from connector builder server

* compiles but tests fail

* first test passes

* Second test passes

* 3rd test passes

* one more test

* another test

* one more test

* test

* return StreamRead

* test

* test

* rename

* test

* test

* test

* main seems to work

* Update

* Update

* Update

* Update

* update

* error message

* rename

* update

* Update

* CR improvements

* fix test_source_declarative_manifest

* fix tests

* Update

* Update

* Update

* Update

* rename

* rename

* rename

* format

* Give connector_builder its own main.py

* Update

* reset

* delete dead code

* remove debug print

* update test

* Update

* set right stream

* Add --catalog argument

* Remove unneeded preparse

* Update README

* handle error

* tests pass

* more explicit test

* reset

* format

* fix merge

* raise exception

* fix

* black format

* raise with config

* update

* fix flake

* __test_read_config is optional

* fix

* Automated Commit - Formatting Changes

* fix

* exclude_unset

---------

Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: clnoll <clnoll@users.noreply.github.com>
Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-03-15 17:12:37 -07:00
Catherine Noll
8ee32b1132 New connector_builder module for handling requests from the Connector Builder (#23888)
Also implements `resolve_manifest` handler
2023-03-14 13:51:27 -04:00