1
0
mirror of synced 2025-12-31 15:03:11 -05:00
Commit Graph

260 Commits

Author SHA1 Message Date
Alexandre Girard
3bdd7f4f43 Standardize HttpRequester's url_base and path format (#17524)
* add trailing slash to base url if missing

* remove leading / from path

* Bump
2022-10-05 15:30:37 -07:00
Alexandre Girard
61a5a1e967 [#17107] Fix propagation of $options field in arrays (#17196)
* input can only be a string

* remove debug print

* cleanup tests

* fix in the factory

* Add a unit test

* fix for lists

* Update changelog

* Update changelog
2022-10-03 17:02:41 -07:00
Alexandre Girard
d6b79b8e52 Alex/limit paginator (#17197)
* remove page size from limit paginator

* Rename LimitPaginator to DefaultPaginator

* rename limit_options

* rename method

* cleanup

* comment

* update schema file

* Update changelog

* pin greenhouse connector

* fix doc in comment

* fix changelog

* missing -

* bump

* bump
2022-09-29 19:24:53 -07:00
Pedro S. Lopez
58f5128adb 🐞 CDK: fix error when logging warning for types that could not be transformed (#17367)
* fix error when logging warning for types that could not be transformed

* changelog and bump
2022-09-29 18:48:53 -04:00
Pedro S. Lopez
fe2238f3f7 🐞 CDK: properly emit state on empty slices when using iterators (#17296)
* fix: emit state on empty slices iterator

* bump version / update changelog

* format
2022-09-28 02:12:04 -04:00
Serhii Chvaliuk
a82f59ec1e CDK: Evaluate response.text only in debug mode (#16809)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-09-17 12:12:03 +03:00
Brian Lai
f283652313 [per-stream cdk] Emit per-stream state messages (#16628)
* update abstract source and connector state manager to update shared and stream state

* clean up some extra comments and extra lines

* a few changes based on PR feedback

* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob

* rename a few paramets and add testing for state update via stream.state override

* replace shared_state processing with an explicit error and fix a few comments and pr feedback

* add some polish and additional test cases

* Add create_state_message to manager and emit per-stream state for connectors

* pr feedback and restructuring parts of the connector state manager initialization

* fix unfinished comment

* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* use pytest params to annotate tests better

* change to fix changed class name

* format test params a little better

* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* a few bits of pr feedback

* do not emit namespace in the outbound message if it is None

* fix some issues w/ prior merge from master and pr feedback

* add new tests to ensure we never send None namespace to the platform

* bump CDK version number

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-09-15 23:42:49 -04:00
Serhii Chvaliuk
0d55a895cb CDK: TypeTransformer - warning message more informative (#16695)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-09-15 10:24:37 +03:00
Serhii Chvaliuk
5a3b6d82b6 CDK: make TypeTransformer more robust to incorrect incoming records (#16544)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-09-13 10:30:16 +03:00
Brian Lai
6253406de9 forgot to also emit dictionary instead of empty lists when state not specified (#16569) 2022-09-12 17:45:48 -04:00
Brian Lai
037e8ed1a9 fix cdk bug to send legacy format if connector overrides read() (#16566)
* fix cdk bug to send legacy format if connector overrides read()

* fix comment

* update changelog and setup.py
2022-09-09 21:09:50 -04:00
Alexandre Girard
0cb44ca071 release cdk with frozen dataclasses-jsonschema lib (#16525) 2022-09-09 07:03:18 -07:00
Brian Lai
dceeef4683 [cdk] pin dataclasses-jsonschema to 2.15.1 (#16253) 2022-09-01 21:08:32 -04:00
Serhii Chvaliuk
3cfa489234 CDK: Fix regression in _checkpoint_state arg (#16141)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-08-31 22:39:30 +03:00
Brian Lai
10a2bd1d3c [low code connectors] add NoAuth to class types registry (#15939)
* add no auth to class type registry

* NoAuth should receive options and fail normally

* forgot to pass options in

* update changelog
2022-08-25 18:06:02 -04:00
Brian Lai
09cddffd36 [low code connectors] replace file retrieval with pkgutil to fix getting schema files (#15814)
* replace file retrieval with pkgutil to fix getting schema files

* slightly better error handling on missing files

* filter our schema gen warnings for some classes that cannot generate schemas

* add comment for todo

* add changelog and setup before publish
2022-08-23 03:06:20 -04:00
Brian Lai
ca6513625d [low code connectors] read configs from package_data (#15810)
* read configs from package_data

* update changelog and setup

* commenting out failing tests in the short term
2022-08-19 21:16:20 -04:00
Brian Lai
ca80d3782a [low code connectors] perform schema validation of the input config against the declarative language schema (#15543)
* draft: first pass at complete schema language generation and factory validator

* actually a working validator and fixes to the schema that went uncaught

* remove extra spike file

* fix formatting file

* pr feedback and a little bit of refactoring

* fix some types that were erroneously marked as invalid schema

* some comments

* add jsonschemamixin to interfaces

* update changelog

* bump version
2022-08-18 15:29:26 -04:00
Alexandre Girard
313ac11e6d [low-code connectors] Get parent stream's full slice (#15631)
* always access parent stream using full_refresh mode

* Update test

* fix substream slicer

* bump
2022-08-18 10:24:10 -07:00
Serhii Chvaliuk
4e6cb05759 CDK: Improve filter_secrets skip empty string (#15684)
* Improve `filter_secrets` skip empty string

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-08-16 19:28:57 +03:00
Alexandre Girard
6332fd6527 [low-code-connectors] Replace JelloExtractor with DpathExtractor (#15514)
* Handle extracting no records from root

* handle missing keys

* record extractor interface

* dpath extractor

* docstring

* handle extract root array

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/extractors/jello.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/extractors/record_selector.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* update docstring

* respect extractor interface

* edge case handling

* document

* use dpath by default

* delete jello extractor

* bump cdk version

* delete jello dependency

* Update reference docs templates

* update template

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-08-11 12:34:54 -07:00
Alexandre Girard
29fafe26eb [low-code connectors] Extract datetime parser and handle %s format directive (#15429)
* fix parse

* Revert "fix parse"

This reverts commit 3c76c5a782.

* fix parse timestamp

* extract datetime parser

* remove print

* use parser

* top level docstring

* rename variable

* do not use timestamp()

* Revert "do not use timestamp()"

This reverts commit 016cb69193.

* update comment

* bump cdk version

* Update template
2022-08-10 16:35:29 -07:00
Alexandre Girard
9507d56be9 low-code connectors: fix parse and format methods (#15326)
* fix parse and format methods

* define constant

* remove timestamp magic keyword

* comment

* test for ci

* uncomment test

* use timestamp()

* Bump cdk version

* bump to 0.1.72
2022-08-08 19:02:02 -07:00
Alexandre Girard
6f1715eabe low-code connectors: convert request headers to string before submitting them (#15336)
* convert values to strings

* introduce variable

* kwargs

* bump version
2022-08-06 10:10:23 -07:00
Brian Lai
bd31100774 initial first pass converting every component to dataclasses (#15189)
* initial first pass converting every component to dataclasses

* replace the hackier options pass through

* get rid of the hackier way

* fix issues w/ type hints by making options required and lots of fixes to the language to fix compatability for dataclasses

* add dataclasses-jsonschema to setup

* fix oauth authenticator to avoid dataclass name collisions

* fix spacing for CI tests

* remove property from oauth and fix a interpolation bug

* pr feedback and cleaning up the code a bit, attempt at avoiding renaming

* fix templates and bugs surfaced during greenhouse spec testing

* fix tests

* fix missing options in some declarative components

* fix tests related to pulling latest master

* fix issue w/ passing state, slice, and token to subcomponents

* switch name back to get_access_token() since no name collision anymore
2022-08-05 17:39:27 -04:00
Alexandre Girard
6e59cfd7be low-code connectors: Set slicer's request options (#15283)
* requester is a request options provider

* get request options from slicer

* remove prints

* share interface

* actual fix with test

* small fix

* missing tests

* missing *

* simplify intersection logic

* bump cdk version
2022-08-04 16:18:28 -07:00
Alexandre Girard
79a54a81fd Emit a state message even if no records were read (#15067)
* Emit a state message even if no records were read

* newline

* merge

* comment

* implement logic in the abstract source

* remove logic from declarative source

* comment

* bump cdk version
2022-08-04 12:48:21 -07:00
Alexandre Girard
a3ff80c179 [low-code-connectors] Disable parse-time interpolation in favor of runtime-only (#14923)
* abstract auth token

* basichttp

* remove prints

* docstrings

* get rid of parse-time interpolation

* always pass options through

* delete print

* delete misleading comment

* delete note

* reset

* pass down options

* delete duplicate file

* missing test

* refactor test

* rename to '$options'

* rename to ''

* interpolatedauth

* fix tests

* fix

* docstrings

* update docstring

* docstring

* update docstring

* remove extra field

* undo

* rename to runtime_parameters

* docstring

* update

* / -> *

* update template

* rename to options

* Add examples

* update docstring

* Update test

* newlines

* rename kwargs to options

* options init param

* delete duplicate line

* type hints

* update docstring

* Revert "delete duplicate line"

This reverts commit 4255d5b346.

* delete duplicate code from bad merge

* rename file

* bump cdk version
2022-07-28 08:57:17 -07:00
Alexandre Girard
783923db76 [low-code CDK] Enable runtime string interpolation in authenticators (#14914)
* interpolatedauth

* fix tests

* fix import

* no need for default

* Bump version

* Missing docstrings

* example

* missing example

* more docstrings

* interpolated types
2022-07-25 19:04:05 -07:00
Augustin
b76b73bbfb cdk: do not call init_uncaught_exception_handler from modules' root (#14892) 2022-07-21 16:34:20 +02:00
Brian Lai
ff74998057 forgot to publish a new version of airbyte-cdk to PyPi (#14696) 2022-07-14 13:22:12 -04:00
Brian Lai
374e265fcb [Low Code CDK] configurable oauth request payload (#13993)
* configurable oauth request payload

* support interpolation for dictionaries that are not new subcomponents

* rewrite a declarative oauth authenticator that performs interpolation at runtime

* formatting

* whatever i don't know why factory gets flagged w/ the newline change

* we java now

* remove duplicate oauth

* add some comments

* parse time properly from string interpolation

* move declarative oauth to its own package in declarative module

* add changelog info
2022-07-08 16:49:16 -04:00
jordan-glitch
4395fbb4d2 cdk: Attach namespace to stream in catalog (#13923) 2022-07-04 18:04:31 +02:00
Sherif A. Nada
7acbcfbfb7 Fix CDK obfuscation of nested secrets (#14035) 2022-06-23 09:23:27 -07:00
George Claireaux
4d279f8238 Remove legacy sentry code from cdk (#14016)
* rip sentry out of cdk

* remove sentry dsn from gsc
2022-06-23 12:14:09 +01:00
Alexandre Girard
57dd7a9d4b [low-code connectors] use jello instead of jq (#13570)
* use jello instead of jq

* reset

* rename

* remove print
2022-06-07 14:10:26 -07:00
Alexandre Girard
d9fa24ffff Low code connectors: implement components for sendgrid (#12853)
* checkout from alex/cac

* checkout from alex/cac

* checkout from alex/cac

* Add missing tests

* Add missing files

* Add missing tests

* add missing file

* missing file

* missing file

* rename

* doc

* doc

* remove broken test

* rename

* jinja dependency

* Add comment

* comment

* comment

* pyjq dependency

* rename file

* delete unused file

* Revert "delete unused file"

This reverts commit 758e939367.

* fix

* rename

* abstract property

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* isort

* update state

* fix imports

* update dependency

* format

* rename file

* decoder

* Use decoder

* Update comment

* dict_state is actually backed by a dict

* Add a comment

* update state takes kwargs

* move state out of offset paginator

* update jq parameter order

* update

* remove incremental mixin

* delete comment

* update comments

* update comments

* remove no_state

* rename package

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* remove print

* move test

* delete duplicates

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* pass parameters through kwargs

* update interface to pass source in interface

* update interface to pass source in interface

* rename to stream_slicer

* Allow passing a string or an enum

* Define StateType enum

* convert state_type if not of type type

* convert state_type if not of type type

* Low code connectors: string interpolation with jinja (#12852)

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* move test to right module

* Add missing test

* Use authbase instead of deprecated class

* leverage generator

* Delete dead code

* rename methods

* rename to declarative

* rename the classes too

* Try to install packages to build jq

* isort

* only automake

* Revert "only automake"

This reverts commit c8fe154ffc.

* remove git

* format

* Add jq dependency

* Use request header provider

* rename

* rename field

* remove get_context method

* rename
2022-06-01 07:02:35 -07:00
Denis Davydov
acd1533cec 🎉 Python CDK: handle requests.exceptions.ChunkedEncodingError for broken connections (#13260) 2022-05-27 22:05:39 +03:00
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Alexandre Girard
150ab593f8 Low code connectors: core structure (#12850)
* checkout from alex/cac

* doc

* doc

* remove broken test

* rename

* rename file

* delete unused file

* rename

* abstract property

* isort

* update state

* Update comment

* remove incremental mixin

* delete comment

* update comments

* update comments

* remove no_state

* rename package

* pass parameters through kwargs

* update interface to pass source in interface

* update interface to pass source in interface

* rename to stream_slicer

* Low code connectors: string interpolation with jinja (#12852)

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* move test to right module

* Add missing test

* Use authbase instead of deprecated class

* leverage generator

* rename to declarative

* rename the classes too
2022-05-25 17:27:54 -07:00
Pedro S. Lopez
41dc82f056 CDK: provide better user-friendly error messages (#12944)
* initial implementation

* add parse_response_error_message tests

* move error handler to existing try/catch in AbstractSource

* formatting

* var rename

* use isinstance for type checking

* add docstrings

* add more abstract_source and httpstream tests

* fix wrong httperror usage

* more test cases

* bump version, update changelog
2022-05-19 17:28:24 -04:00
Serhii Chvaliuk
9a960c4ca6 CDK: TypeTransformer.default_convert catch TypeError (#12948)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-05-19 07:32:48 +03:00
Pedro S. Lopez
7ea27d0ecd CDK: publish version 0.1.57 (#12938) 2022-05-17 16:33:06 -04:00
Pedro S. Lopez
73c7fad7fc CDK: emit AirbyteTraceMessage with exception trace information (#12593) 2022-05-06 17:16:57 -04:00
Sherif A. Nada
cc46255190 cull base-python and base-python-test (#12596) 2022-05-05 13:48:15 -07:00
Pedro S. Lopez
53799cb9a2 🎉 CDK: support loading spec from yaml file (#12104)
* support loading spec from yaml file

* formatting

* remove commented code

* update comment

* remove unused file

* raise correct exception types

* bump version, update changelog
2022-04-20 13:18:46 -07:00
Pedro S. Lopez
a6a6be2588 allow importing IncrementalMixin from airbyte_cdk.sources.streams (#11858)
* allow importing IncrementalMixin from airbyte_cdk.sources.streams

* format file

* update docs with hint on where to import IncrementalMixin from

* bump cdk version to 0.1.54

* update changelog

* only build python 3.9

* update changelog with python dep bump
2022-04-14 11:42:57 -07:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Augustin
2d277747be cdk: remove false error logging in _send (#11650) 2022-04-04 10:36:05 -07:00
Serhii Chvaliuk
df12c9e4d4 Python CDK - Fix BaseBackoffException constructor (#11653)
* - Fix BaseBackoffException constructor

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-04-01 19:05:11 +03:00