1
0
mirror of synced 2025-12-31 15:03:11 -05:00
Commit Graph

16 Commits

Author SHA1 Message Date
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594)
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
2023-02-08 13:01:16 -08:00
Brian Lai
e97ece5532 [low-code cdk] break resolving reference preprocessing into its own class so it can be reused (#19517)
* break resolving reference preprocessing into its own class so it can be reused

* move reference resolution into the ManifestDeclarativeSource and deprecate the parser

* formatting

* last formatting i promise

* rename

* bump version

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
2022-11-17 13:15:28 -08:00
Alexandre Girard
e029353b9f SimpleRetriever yield request and response as log messages (#18644)
* method yielding airbytemessage

* move to Stream

* update abstract source

* reset

* missing file

* Yield request and response as log messages

* only emit request and responses if the debug flag is on

* add test docker image

* script to run acceptance tests with local cdk

* Update conftest to use a different image

* extract to method

* dont use a different image tag

* Always install local cdk

* break the cdk

* get path from current working directory

* or

* ignore unit test

* debug log

* Revert "AMI change: ami-0f23be2f917510c26 -> ami-005924fb76f7477ce (#18689)"

This reverts commit bf06decf73.

* build from the top

* Update source-acceptance-test

* fix

* copy setup

* some work on the gradle plugin

* reset to master

* delete unused file

* delete unused file

* reset to master

* optional argument

* delete dead code

* use latest cdk with sendgrid

* fix sendgrid dockerfile

* break the cdk

* use local file

* Revert "break the cdk"

This reverts commit 600c195541.

* dont raise an exception

* reset to master

* unit tests

* missing test

* more unit tests

* remove deprecated comment

* newline

* reset to master

* remove files

* reset

* Update abstract source

* remove method from stream

* convert to airbytemessage

* unittests

* Update

* unit test

* remove debug logs

* Revert "remove debug logs"

This reverts commit a1a139ef37.

* Revert "Revert "remove debug logs""

This reverts commit b1d62cdb60.

* Revert "reset to master"

This reverts commit 3fa6a004c1.

* fix

* slightly better test

* typing

* extract method

* Revert "Revert "reset to master""

This reverts commit 5dac7c2804.

* reset to master

* reset to master

* Revert "reset to master"

This reverts commit 3fa6a004c1.

* Comment

* operate on the message

* Revert "Revert "reset to master""

This reverts commit 5833c84d0a.

* comment

* test

* Revert "test"

This reverts commit 2f91b803b0.

* test

* Revert "test"

This reverts commit 62d95ebbb5.

* test

* Revert "test"

This reverts commit 27150ba341.

* format

* format

* symlink

* Update setup

* update path

* reset to master

* update

* Add local files

* greenhouse

* format

* symlink

* try reordering

* better error message

* better log message

* reset to master

* Revert "merge for qa"

This reverts commit ad7128f2c5, reversing
changes made to 7196c22a73.

* reset to master

* reset to master

* reset to master

* format

* gradlew format

* right type hints

* reset to master

* reset to master

* gradlew format

* a bunch of small fixes

* Update output format

* fixes from feedback

* fixme comment

* streams cannot return AirbyteRecordMessage

* fix

* format

* only return logs when running on debug mode

* move branching

* update typing

* remove dead code

* fix simpleretriever name

* i think this is better

* log response.text

* debug flag

* comment

* pass config

* comments

* run SATs

* fix most of the unit tests

* fix unit test

* reset to master

* runFromPath

* Revert "runFromPath"

This reverts commit 85979a801a.

* Revert "run SATs"

This reverts commit a8a8a2da95.

* no need to convert to dict

* fix test
2022-11-09 13:26:50 -08:00
Brian Lai
5c9e5d9934 [low-code cdk] decouple parsing the yaml manifest from the declarative source implementation (#19095)
* decouple parsing the yaml manifest from the declarative source implementation

* bump version and changelog
2022-11-08 15:45:01 -05:00
Brian Lai
f9863d6374 [low-code cdk] Allow for spec file to be defined in the yaml manifest instead of an external file (#18411)
* allow for spec to be defined in the source.yaml manifest instead of an external file

* make spec a component within the language to get schema validation and rework the code for better testing

* fix formatting and extra method

* pr feedback and add some more test

* pr feedback

* bump airbyte-cdk version

* bump version

* gradle format

* remove  from manifest spec
2022-11-07 14:44:45 -05:00
Alexandre Girard
4b0fb6787e [low-code] Apply log level to stream loggers (#17284)
* Apply log level to stream loggers

* remove debug print
2022-09-30 12:49:08 -07:00
Brian Lai
ca6513625d [low code connectors] read configs from package_data (#15810)
* read configs from package_data

* update changelog and setup

* commenting out failing tests in the short term
2022-08-19 21:16:20 -04:00
Brian Lai
7e158ef9af [low code connectors] generate complete json schema from classes (#15647)
* draft: first pass at complete schema language generation and factory validator

* actually a working validator and fixes to the schema that went uncaught

* remove extra spike file

* fix formatting file

* Add method to generate the complete JSON schema of the low code declarative language

* add testing of a few components during schema gen

* pr feedback and a little bit of refactoring

* test for schema version

* fix some types that were erroneously marked as invalid schema

* some comments

* add jsonschemamixin to interfaces

* update tests now that interfaces are jsonschemamixin

* accidentally removed a mixin

* remove unneeded test

* make comment a little more clear

* update changelog

* bump version

* generic enum not enum class

* Add method to generate the complete JSON schema of the low code declarative language

* add testing of a few components during schema gen

* test for schema version

* update tests now that interfaces are jsonschemamixin

* accidentally removed a mixin

* remove unneeded test

* make comment a little more clear

* generic enum not enum class

* add generated json file and update docs to reference it

* verbage
2022-08-18 18:53:42 -04:00
Brian Lai
ca80d3782a [low code connectors] perform schema validation of the input config against the declarative language schema (#15543)
* draft: first pass at complete schema language generation and factory validator

* actually a working validator and fixes to the schema that went uncaught

* remove extra spike file

* fix formatting file

* pr feedback and a little bit of refactoring

* fix some types that were erroneously marked as invalid schema

* some comments

* add jsonschemamixin to interfaces

* update changelog

* bump version
2022-08-18 15:29:26 -04:00
Alexandre Girard
f540499f43 [low-code connectors]: Assert there are no custom top-level fields (#15489)
* move components to definitions field

* Also update the references

* validate the top level fields and add version

* raise exception on unknown fields

* newline

* unit tests

* set version to 0.1.0

* newline
2022-08-10 11:37:07 -07:00
Brian Lai
ef712f18aa [low-code connectors] fix so we don't display yaml when debug flag is turned off (#15383)
* fix so we don't display yaml when debug is turned off

* forgot to remove old debug level
2022-08-08 01:11:31 -04:00
Alexandre Girard
08239abafd Alex/lowcode referencedocs (#14973)
* Add docstrings for auth package

* docstrings for the check package

* docstrings for the datetime package

* docstrings for the decoder package

* docstrings for extractors package and fix tests

* interpolation docstrings

* ref ->  and parser docstrings

* docstrings for parsers package

* error handler docstrings

* requester docstrings

* more docstrings

* docstrings

* docstrings

* docstrings

* Use defined type annotations

* update

* update docstrings

* Update docstrings

* update docstrings

* update docstrings

* update template

* Revert "update template"

This reverts commit eb4a11858b.

* update template

* update

* move to interpolated_string

* update docstring

* update

* fix tests

* format

* return type can also be an array

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/interpolated_boolean.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/interpolation.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/jinja.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/interpolated_boolean.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/backoff_strategy.py

* Update as per comments

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-07-25 18:10:32 -07:00
Brian Lai
7bff12aea5 [#3078] [CDK] Add support for enabling debug from command line and some basic general debug logs (#14521)
* allow for command line debug option and basic debug statements + declarative

* feedback from pr comments

* fix some tests w/ req/res mixed up and fixing logging tests

* formatting

* pr feedback: cleaning up traces in logger.py and update docs with debug configuration

* remove unneeded trace logger test

* remove extra print statement
2022-07-13 18:01:07 -04:00
Sherif A. Nada
743e6c2d6c [Low-code CDK] Add ability to remove fields (#14402) 2022-07-11 19:04:57 -07:00
Alexandre Girard
85449c975c [low-code connectors] default types and default values (#14004)
* default types and default values

* cleanup

* fixes so read works

* remove prints and trycatch

* comment

* remove unused param

* split file

* extract method

* extract methods

* comment

* optional

* fix test

* cleanup

* delete interpolated request header provider

* simplify next page url paginator interface

* comment

* format
2022-06-27 20:43:03 -07:00
Alexandre Girard
aa92518721 Low-code connectors: configurable source from yaml (#13038)
* checkout from alex/cac

* checkout from alex/cac

* checkout from alex/cac

* checkout from alex/cac

* Add missing tests

* Add missing files

* Add missing tests

* add missing file

* missing file

* missing file

* sengrid low code connector

* rename

* doc

* doc

* remove broken test

* rename

* jinja dependency

* Add comment

* comment

* comment

* pyjq dependency

* update import

* rename file

* delete unused file

* Revert "delete unused file"

This reverts commit 758e939367.

* fix

* rename

* abstract property

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* isort

* update state

* fix imports

* update

* update dependency

* remove dead code

* remove dead code

* format

* rename file

* decoder

* Use decoder

* Update comment

* dict_state is actually backed by a dict

* Add a comment

* update state takes kwargs

* move state out of offset paginator

* fix

* update jq parameter order

* fix

* pass config

* update

* update

* remove incremental mixin

* delete comment

* start workin on yaml parser

* fix test

* progress

* refer and overwrite partials

* factory tests pass

* fix

* reset

* Assert http_method is an enum value

* fix auth

* read lists works

* fix test

* comment

* implement all streams

* build connection checker

* update comments

* update comments

* remove no_state

* rename package

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* remove print

* move test

* delete duplicates

* update

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* pass parameters through kwargs

* fix test

* update interface

* update interface to pass source in interface

* update interface to pass source in interface

* rename to stream_slicer

* Allow passing a string or an enum

* Define StateType enum

* unit tests pass

* update dict state

* update

* can read

* fix test

* fix from yaml update

* elif

* convert state_type if not of type type

* convert state_type if not of type type

* Add a test

* Low code connectors: string interpolation with jinja (#12852)

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* move test to right module

* Add missing test

* Use authbase instead of deprecated class

* leverage generator

* remove sendgrid specific code

* update

* update

* delete comment

* remove sendgrid specific file

* remove unused file

* Delete dead code

* rename methods

* rename to declarative

* rename the classes too

* select streams to check

* nit

* rename method

* rename class

* {} is faster than dict()

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/yaml_parser.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* more precise exception

* rename class

* add comment

* Try to install packages to build jq

* isort

* only automake

* Revert "only automake"

This reverts commit c8fe154ffc.

* remove git

* rename file

* create components in kwargs

* Use tuple of strings

* parser doesn't need to be stored

* move file and delete duplicates

* Revert "Use tuple of strings"

This reverts commit ab5a7afd08.

* raise error if streams to check are not in the catalog

* Revert "Revert "Use tuple of strings""

This reverts commit 7c9fb8eb33.

* traverse tree

* rename to options

* move docstring

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/checks/check_stream.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* fix tests and format

* format

* update

* better error message

* Add jq dependency

* Use request header provider

* rename

* rename field

* remove get_context method

* rename

* add a comment

* format

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-06-01 07:31:52 -07:00