1
0
mirror of synced 2026-01-17 12:07:50 -05:00
Commit Graph

7 Commits

Author SHA1 Message Date
Brian Lai
cbf9ea76c1 [Low-Code CDK] Construct declarative components from Pydantic models (#21050)
* initial work to parse manifest objects into pydantic models

* pr feedback and some other validations

* rerun manifest schema generation

* remove field constraint due to bug

* initial work to construct most components from greenhouse

* custom components parse subcomponent fields correctly and adding a few more component constructors

* construct components from gnews

* first pass at posthog.yaml

* Handle nested custom components with list values.
Also includes updates to posthog.yaml, including autoformatting changes.

* adding constructors for slicers, filters, and transformations and a few bug fixes

* make sed work across multiple OS

* add NoAuth component

* fix handling of custom components with nested list

* Autogenerate `TYPE_NAME_TO_MODEL` mapping

* Handle default kwargs not defined on model for custom components

* Re-add `options` for CartesianProductStreamSlicer for backwards compat
with custom stream slicers

* add basic unit tests for the model component factory

* add back defaults and extra parameters like options to retain compatibility with legacy flow and backwards compatibility

* Remove `_get_defaults`; using actual default values on classes instead

* Add backoff strategy component creation functions

* add back defaults and extra parameters like options to retain compatibility with legacy flow and backwards compatibility

* add lots of tests to construct components from the pydantic models and a few bug fixes

* add a few tests for the model to component factory

* add catch

* fix a bug where propagated schema doesn't work with old factory

* clean up a few files

* add type inference for custom components, more tests and some refactoring of the model factory

* template, docs, manifest updates, pr feedback and some cleanup

* pr feedback and polish schema a bit

* fix tests from the latest rebase of master

* fix the last few bugs I found and adjust a few sources that weren't perfectly compatible with the new component flow

* fix CheckStream bug cleanup and a few small tweaks and polish

* add additional test to cover bug case

* fix formatting

* 🤖 Bump minor version of Airbyte CDK

Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
2023-01-12 21:02:08 -05:00
Lake Mossman
63ff3d75b4 Make documentation_url optional in a declarative connector spec (#21347)
* Make documentation_url optional in a declarative connector spec

* simplify if statement

* bump patch version of cdk

* Revert "bump patch version of cdk"

This reverts commit 1854bf3be1.

* 🤖 Bump patch version of Airbyte CDK

* Revert "🤖 Bump patch version of Airbyte CDK"

This reverts commit 85d5a989e2.

* 🤖 Bump patch version of Airbyte CDK

Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2023-01-12 14:23:04 -08:00
Brian Lai
3d98c41f0f [Low-Code CDK] Parse incoming manifest objects into Pydantic models (#20747)
* initial work to parse manifest objects into pydantic models

* pr feedback and some other validations

* rerun manifest schema generation

* remove field constraint due to bug

* add a few tests for the model to component factory

* add catch

* fix a bug where propagated schema doesn't work with old factory

* add additionaProperties: false to the handwritten schema

* regenerate pydantic models

* fix test
2023-01-11 17:46:14 -05:00
Brian Lai
8f21d0de1b Allow for custom requesters to be defined in low-code manifests (#21001)
* Allow for custom requesters to be defined in low-code manifests

* add test for custom requester component

* bump versions and changelog
2023-01-06 20:17:51 -05:00
Maxime Carbonneau-Leclerc
4f0aca536c [ISSUE #20322] add datetime_granularity logic to DatetimeStreamSlicer… (#20717)
* [ISSUE #20322] add datetime_granularity logic to DatetimeStreamSlicer and migrate duration to ISO8601

* [ISSUE #20322] fix tests

* [ISSUE #20322] code review based on clnoll's comments and fixed tests

* [ISSUE #20322] fix flake8 error

* [ISSUE #20322] fix source tests

* [ISSUE #20322] fixing yet another error in source

* [ISSUE #20322] code review

* [ISSUE #20322] adding new sources using datetime slicer

* [ISSUE #20322] fixing source-datascope and increasing version

* [ISSUE #20322] regenerate component schema

* [ISSUE #20322] fixing source-datascope

* [ISSUE #20322] extra field error
2023-01-06 12:29:37 -05:00
Brian Lai
b17df69dc5 fix a bug where the OAuthAuthenticator was misnamed (#20700)
* fix a bug where the OAuthAuthenticator was misnamed

* roll airbyte-cdk version

* add some missing component types to the existing factory type registry

* fix test
2022-12-20 15:16:14 -06:00
Brian Lai
b7113a2e5e [Low-Code CDK] Write the component schema and use it during manifest validation (#20422)
* handwritten low code manifest example components

* add MinMaxDatetime to jsonschema

* add a basic gradle command to generate manifest components

* Add auth components to handwritten component schema

- ApiKeyAuthenticator
- BasicHttpAuthenticator
- BearerAuthenticator
- DeclarativeOauth2Authenticator
- NoAuth

* Respect optional properties in DeclarativeOauth2Authenticator

* Fix `Dict[str, Any]` mapping in auth components

* add default error handler composite error handler and http response filter components

* [low code component schema] adding backoff strategies to schema

* [low code component schema] fix float types

* [low code component schema] add RecordFilter

* Remove `config` from auth components

* [low code component schema] add Interpolation (with pending question on 'type' not being defined)

* Add CartesianProductStreamSlicer & DatetimeStreamSlicer

* Add ListStreamSlicer, and fix nesting of DatetimeStreamSlicer

* [low code component schema] add InterpolatedRequestOptionsProvider

* Add slicer components, and fix a couple of components after reviewing output

* [low code component schema] adding transformations and adding type to interpolators

* adding spec and a few small tweaks

* Add DefaultSchemaLoader

* [low code component schema] attempt on custom class

* Add descriptions for auth components

* add RequestOption

* remove interpolated objects from the schema in favor of strings only

* a few schema fixes and adding some custom pagination and stream slicer

* [low code component schema] fix CustomBackoffStrategy

* Add CustomRecordExtractor

* add some description and add additional properties

* insert a transformer to hydrate default manifest components and perform validation against the handwritten schema

* [low code component schema] validating existing schemas

* [low code component schema] clean validation script

* add manifest transformer tests and a few tweaks to the schema

* Revert "[low code component schema] clean validation script"

This reverts commit 2408f41cf6.

* Revert "[low code component schema] validating existing schemas"

This reverts commit 9d39977815.

* [low code component schema] integrate validation script to gradle

* [low code component schema] updating validation script permissions

* remove a few model gen spike files and clean up comments

* default types should take parent type into account and a few schema changes

* add inline schema components and fix validation bugs

* add types to every component's literal enum and more little schema fixes

* missing component prefix

* pr feedback, renaming the schema and updating various docs

* fixing schema bugs from testing and new components

* last few small tweaks and fixes and fix the script

* bump the version before publsih

* include the declarative_component_schema.yaml as package data so its included when package is published

* roll the version to publish

Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
2022-12-19 14:42:34 -06:00