1
0
mirror of synced 2026-01-01 18:02:53 -05:00
Commit Graph

30 Commits

Author SHA1 Message Date
Alexandre Girard
e029353b9f SimpleRetriever yield request and response as log messages (#18644)
* method yielding airbytemessage

* move to Stream

* update abstract source

* reset

* missing file

* Yield request and response as log messages

* only emit request and responses if the debug flag is on

* add test docker image

* script to run acceptance tests with local cdk

* Update conftest to use a different image

* extract to method

* dont use a different image tag

* Always install local cdk

* break the cdk

* get path from current working directory

* or

* ignore unit test

* debug log

* Revert "AMI change: ami-0f23be2f917510c26 -> ami-005924fb76f7477ce (#18689)"

This reverts commit bf06decf73.

* build from the top

* Update source-acceptance-test

* fix

* copy setup

* some work on the gradle plugin

* reset to master

* delete unused file

* delete unused file

* reset to master

* optional argument

* delete dead code

* use latest cdk with sendgrid

* fix sendgrid dockerfile

* break the cdk

* use local file

* Revert "break the cdk"

This reverts commit 600c195541.

* dont raise an exception

* reset to master

* unit tests

* missing test

* more unit tests

* remove deprecated comment

* newline

* reset to master

* remove files

* reset

* Update abstract source

* remove method from stream

* convert to airbytemessage

* unittests

* Update

* unit test

* remove debug logs

* Revert "remove debug logs"

This reverts commit a1a139ef37.

* Revert "Revert "remove debug logs""

This reverts commit b1d62cdb60.

* Revert "reset to master"

This reverts commit 3fa6a004c1.

* fix

* slightly better test

* typing

* extract method

* Revert "Revert "reset to master""

This reverts commit 5dac7c2804.

* reset to master

* reset to master

* Revert "reset to master"

This reverts commit 3fa6a004c1.

* Comment

* operate on the message

* Revert "Revert "reset to master""

This reverts commit 5833c84d0a.

* comment

* test

* Revert "test"

This reverts commit 2f91b803b0.

* test

* Revert "test"

This reverts commit 62d95ebbb5.

* test

* Revert "test"

This reverts commit 27150ba341.

* format

* format

* symlink

* Update setup

* update path

* reset to master

* update

* Add local files

* greenhouse

* format

* symlink

* try reordering

* better error message

* better log message

* reset to master

* Revert "merge for qa"

This reverts commit ad7128f2c5, reversing
changes made to 7196c22a73.

* reset to master

* reset to master

* reset to master

* format

* gradlew format

* right type hints

* reset to master

* reset to master

* gradlew format

* a bunch of small fixes

* Update output format

* fixes from feedback

* fixme comment

* streams cannot return AirbyteRecordMessage

* fix

* format

* only return logs when running on debug mode

* move branching

* update typing

* remove dead code

* fix simpleretriever name

* i think this is better

* log response.text

* debug flag

* comment

* pass config

* comments

* run SATs

* fix most of the unit tests

* fix unit test

* reset to master

* runFromPath

* Revert "runFromPath"

This reverts commit 85979a801a.

* Revert "run SATs"

This reverts commit a8a8a2da95.

* no need to convert to dict

* fix test
2022-11-09 13:26:50 -08:00
Alexandre Girard
1a608f846a Stream returns AirbyteMessages (#18572)
* method yielding airbytemessage

* move to Stream

* update abstract source

* reset

* missing file

* add test docker image

* script to run acceptance tests with local cdk

* Update conftest to use a different image

* extract to method

* dont use a different image tag

* Always install local cdk

* break the cdk

* get path from current working directory

* or

* ignore unit test

* debug log

* Revert "AMI change: ami-0f23be2f917510c26 -> ami-005924fb76f7477ce (#18689)"

This reverts commit bf06decf73.

* build from the top

* Update source-acceptance-test

* fix

* copy setup

* some work on the gradle plugin

* reset to master

* delete unused file

* delete unused file

* reset to master

* optional argument

* delete dead code

* use latest cdk with sendgrid

* fix sendgrid dockerfile

* break the cdk

* use local file

* Revert "break the cdk"

This reverts commit 600c195541.

* dont raise an exception

* reset to master

* unit tests

* missing test

* more unit tests

* newline

* reset to master

* remove files

* reset

* Update abstract source

* remove method from stream

* convert to airbytemessage

* unittests

* Update

* unit test

* remove debug logs

* Revert "remove debug logs"

This reverts commit a1a139ef37.

* Revert "Revert "remove debug logs""

This reverts commit b1d62cdb60.

* Revert "reset to master"

This reverts commit 3fa6a004c1.

* Revert "Revert "reset to master""

This reverts commit 5dac7c2804.

* reset to master

* reset to master

* test

* Revert "test"

This reverts commit 2f91b803b0.

* test

* Revert "test"

This reverts commit 62d95ebbb5.

* test

* Revert "test"

This reverts commit 27150ba341.

* format
2022-11-07 21:23:11 -08:00
Pedro S. Lopez
ad29513a43 CDK: Handle config validation errors as config_error and failed check status (#18214)
* schema helpers raises a config_error

* traced exceptions can be turned into connection status messages

* add tests for schema helpers

* return a failed status message rather than throwing in check command

* remove unused imports

* add comment

* bump version / update changelog
2022-10-20 13:23:06 -04:00
Pedro S. Lopez
58f5128adb 🐞 CDK: fix error when logging warning for types that could not be transformed (#17367)
* fix error when logging warning for types that could not be transformed

* changelog and bump
2022-09-29 18:48:53 -04:00
Serhii Chvaliuk
74e936bf7d CDK: improve TypeTransformer to convert simple types to array of simple types (#16636)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-09-15 11:34:17 +03:00
Serhii Chvaliuk
0d55a895cb CDK: TypeTransformer - warning message more informative (#16695)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-09-15 10:24:37 +03:00
Serhii Chvaliuk
5a3b6d82b6 CDK: make TypeTransformer more robust to incorrect incoming records (#16544)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-09-13 10:30:16 +03:00
Sherif A. Nada
7acbcfbfb7 Fix CDK obfuscation of nested secrets (#14035) 2022-06-23 09:23:27 -07:00
George Claireaux
4d279f8238 Remove legacy sentry code from cdk (#14016)
* rip sentry out of cdk

* remove sentry dsn from gsc
2022-06-23 12:14:09 +01:00
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Serhii Chvaliuk
9a960c4ca6 CDK: TypeTransformer.default_convert catch TypeError (#12948)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-05-19 07:32:48 +03:00
Pedro S. Lopez
73c7fad7fc CDK: emit AirbyteTraceMessage with exception trace information (#12593) 2022-05-06 17:16:57 -04:00
vitaliizazmic
936050847f 🐛 CDK: fixing type errors (#9926)
* CDK #9500 - fixing type errors

* CDK #9500 - exclude singer from check

* CDK #9500 - type ignore in utils.schema_helpers.py

* CDK #9500 - type ignore in utils.sentry.py

* CDK #9500 - type ignore in singer.singer_helpers.py

* CDK #9500 - type ignore in singer.source

* CDK #9500 - fix according to PR review

* CDK #9500 - fix according to PR review

* CDK #9500 - bump version
2022-02-14 09:24:04 +02:00
Eugene Kulak
f83eca58ea CDK: Fix typing errors (#9037)
* fix typing, drop AirbyteLogger

* format

* bump the version

* use logger instead of fixture logger

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: auganbay <auganenu@gmail.com>
2022-01-14 10:29:34 +06:00
Dmytro
fe954c1a16 Integrate Sentry for performance and errors tracking. (#8248)
* Integrate Sentry for performance and errors tracking.

* Add sentry sensitive data scrubbing.

* updated cdk version and changelog

* Integrate Sentry for performance and errors tracking

Add `SENTRY_DSN` environment variable

* Integrate Sentry for performance and errors tracking

Add `sentry_sdk` to install requirements

* format cdk

* enable Sentry for google-search-console

* updated connector version

* update spec and source yamls

Co-authored-by: auganbay <auganenu@gmail.com>
Co-authored-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com>
2021-12-21 20:02:02 +06:00
Eugene Kulak
d50ae47b49 🐛 CDK: Fix the logging of unhandled exceptions to show stacktrace (#8704)
* print stacktrace for unhandled exceptions

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-12-14 00:04:29 +02:00
Eugene Kulak
aa67604f09 CDK: Add base pydantic model for connector config and schemas (#8485)
* add base spec model

* fix usage of state_checkpoint_interval in case it is dynamic

* add schema base models, fix spelling, signatures and polishing

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-12-08 01:14:59 +02:00
Dmytro
e56e86dd28 Fix schema transformation false warnings. (#7863) 2021-11-12 18:20:53 +02:00
Dmytro
16b626cc91 Resolve discover catalog ref fields (#7734) 2021-11-09 12:42:15 +02:00
Yevhenii
5eb2af6c08 CDK: Autogenerate reference docs (#4759)
* CDK Autogenerated reference docs: base version

* Update docs config.
Add .readthedocs.yaml file.
Update build html files.

* Update .gitignore.
Remove sphinx build files.

* Add newline at the end of .gitignore

* Update setup.py requirements.
Update .readthedocs.yaml config.

* Update rst files.
Add Makefile rst config.

* Update CDK docstring format.
Change rst layout.
Update sphinx config.

* Add Sphinx docs.
Update index.rst.
Update abstract_source.py docstrings.

* Override master_doc and package templates.
Add docs schema enerator script.
Update sphinx docs.

* Add `Publishing to Read the Docs` section to sphinx-docs.md".
Replace sphinx-docs.md to `airbyte-cdk` module.

* Update sphinx-docs.md section name

* Bump airbyte-cdk version.
Update CHANGELOG.md.

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Vadym Hevlich <vege1wgw@gmail.com>
2021-10-22 20:47:48 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Dmytro
1c5ac5b5ea 🏗️ Python CDK: add schema transformer class (#6139)
* Python CDK: add schema transformer class
2021-09-27 13:07:17 +03:00
Dmytro
474b8f1065 CDK: support nested refs resolving (#6044)
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-09-16 18:38:08 +03:00
Marcos Eliziario Santos
1314a6a04c #5796 silence printing full config when config validation fails (#5879)
* - #5796 silence printing full config when config validation fails

* fix unit tests after config validation check changes

Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com>
2021-09-14 11:00:47 -03:00
Dmytro
7584440515 CDK: private configuration option _limit and _page_size (#5617)
* CDK: private configuration option _limit and _page_size
2021-08-31 12:16:48 +03:00
Dmytro
3aabd92137 Fix facebook marketing source SAT fail on invalid config file (#5621)
* Fix facebook marketing source SAT fail on invalid config file
2021-08-31 11:48:03 +03:00
Marcos Marx
7225187fa1 run gradlew format (#5552) 2021-08-20 15:38:28 -03:00
Dmytro
b1f2bf5665 4776: Python CDK: Validate input config.py against spec (#5457)
Python CDK: Validate input config.py against spec

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-08-19 13:14:37 +03:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Sherif A. Nada
184dab77eb CDK: overhaul directory structure (#3295) 2021-05-09 15:27:38 -07:00