1
0
mirror of synced 2026-01-22 10:01:28 -05:00
Commit Graph

6 Commits

Author SHA1 Message Date
Brian Lai
b7113a2e5e [Low-Code CDK] Write the component schema and use it during manifest validation (#20422)
* handwritten low code manifest example components

* add MinMaxDatetime to jsonschema

* add a basic gradle command to generate manifest components

* Add auth components to handwritten component schema

- ApiKeyAuthenticator
- BasicHttpAuthenticator
- BearerAuthenticator
- DeclarativeOauth2Authenticator
- NoAuth

* Respect optional properties in DeclarativeOauth2Authenticator

* Fix `Dict[str, Any]` mapping in auth components

* add default error handler composite error handler and http response filter components

* [low code component schema] adding backoff strategies to schema

* [low code component schema] fix float types

* [low code component schema] add RecordFilter

* Remove `config` from auth components

* [low code component schema] add Interpolation (with pending question on 'type' not being defined)

* Add CartesianProductStreamSlicer & DatetimeStreamSlicer

* Add ListStreamSlicer, and fix nesting of DatetimeStreamSlicer

* [low code component schema] add InterpolatedRequestOptionsProvider

* Add slicer components, and fix a couple of components after reviewing output

* [low code component schema] adding transformations and adding type to interpolators

* adding spec and a few small tweaks

* Add DefaultSchemaLoader

* [low code component schema] attempt on custom class

* Add descriptions for auth components

* add RequestOption

* remove interpolated objects from the schema in favor of strings only

* a few schema fixes and adding some custom pagination and stream slicer

* [low code component schema] fix CustomBackoffStrategy

* Add CustomRecordExtractor

* add some description and add additional properties

* insert a transformer to hydrate default manifest components and perform validation against the handwritten schema

* [low code component schema] validating existing schemas

* [low code component schema] clean validation script

* add manifest transformer tests and a few tweaks to the schema

* Revert "[low code component schema] clean validation script"

This reverts commit 2408f41cf6.

* Revert "[low code component schema] validating existing schemas"

This reverts commit 9d39977815.

* [low code component schema] integrate validation script to gradle

* [low code component schema] updating validation script permissions

* remove a few model gen spike files and clean up comments

* default types should take parent type into account and a few schema changes

* add inline schema components and fix validation bugs

* add types to every component's literal enum and more little schema fixes

* missing component prefix

* pr feedback, renaming the schema and updating various docs

* fixing schema bugs from testing and new components

* last few small tweaks and fixes and fix the script

* bump the version before publsih

* include the declarative_component_schema.yaml as package data so its included when package is published

* roll the version to publish

Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
2022-12-19 14:42:34 -06:00
Brian Lai
bd31100774 initial first pass converting every component to dataclasses (#15189)
* initial first pass converting every component to dataclasses

* replace the hackier options pass through

* get rid of the hackier way

* fix issues w/ type hints by making options required and lots of fixes to the language to fix compatability for dataclasses

* add dataclasses-jsonschema to setup

* fix oauth authenticator to avoid dataclass name collisions

* fix spacing for CI tests

* remove property from oauth and fix a interpolation bug

* pr feedback and cleaning up the code a bit, attempt at avoiding renaming

* fix templates and bugs surfaced during greenhouse spec testing

* fix tests

* fix missing options in some declarative components

* fix tests related to pulling latest master

* fix issue w/ passing state, slice, and token to subcomponents

* switch name back to get_access_token() since no name collision anymore
2022-08-05 17:39:27 -04:00
Alexandre Girard
a3ff80c179 [low-code-connectors] Disable parse-time interpolation in favor of runtime-only (#14923)
* abstract auth token

* basichttp

* remove prints

* docstrings

* get rid of parse-time interpolation

* always pass options through

* delete print

* delete misleading comment

* delete note

* reset

* pass down options

* delete duplicate file

* missing test

* refactor test

* rename to '$options'

* rename to ''

* interpolatedauth

* fix tests

* fix

* docstrings

* update docstring

* docstring

* update docstring

* remove extra field

* undo

* rename to runtime_parameters

* docstring

* update

* / -> *

* update template

* rename to options

* Add examples

* update docstring

* Update test

* newlines

* rename kwargs to options

* options init param

* delete duplicate line

* type hints

* update docstring

* Revert "delete duplicate line"

This reverts commit 4255d5b346.

* delete duplicate code from bad merge

* rename file

* bump cdk version
2022-07-28 08:57:17 -07:00
Alexandre Girard
08239abafd Alex/lowcode referencedocs (#14973)
* Add docstrings for auth package

* docstrings for the check package

* docstrings for the datetime package

* docstrings for the decoder package

* docstrings for extractors package and fix tests

* interpolation docstrings

* ref ->  and parser docstrings

* docstrings for parsers package

* error handler docstrings

* requester docstrings

* more docstrings

* docstrings

* docstrings

* docstrings

* Use defined type annotations

* update

* update docstrings

* Update docstrings

* update docstrings

* update docstrings

* update template

* Revert "update template"

This reverts commit eb4a11858b.

* update template

* update

* move to interpolated_string

* update docstring

* update

* fix tests

* format

* return type can also be an array

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/interpolated_boolean.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/interpolation.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/jinja.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/interpolated_boolean.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/backoff_strategy.py

* Update as per comments

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-07-25 18:10:32 -07:00
Alexandre Girard
e23789bbd6 Alex/declarative stream incremental fix (#14268)
* checkout files from test branch

* read_incremental works

* reset to master

* remove dead code

* comment

* fix

* Add test

* comments

* utc

* format

* small fix

* Add test with rfc3339

* remove unused param

* fix test
2022-07-01 08:39:19 -07:00
Brian Lai
ccbd7c6508 conditional paginator + interpolated boolean components (#13389)
* conditional paginator + interpolated boolean components

* fix bug where empty arrays and dicts evaluated to true

* add more falsey values

* adjust falsey values and tweak based on review comments
2022-06-01 20:24:32 -04:00