Brian Lai
be01b476ce
Add new InterpolatedRequestOptionsProvider that encapsulates all variations of request arguments ( #13472 )
...
* write out new request options provider and refactor components and parts of the YAML config
* fix formatting
* pr feedback to consolidate body_data_provider to simplify the code
* pr feedback get rid of extraneous optional
2022-06-21 16:01:05 -04:00
Alexandre Girard
34e2dc36c6
[low-code connectors] Cartesian product stream slicer ( #13740 )
...
* list slicer
* Add comment
* product stream slicer
* comment
* rename
* format
* Update comment
* split on 2 lines for readability
* Revert "rename"
This reverts commit e801f2d1f8
2022-06-16 08:48:32 -07:00
Alexandre Girard
2c8fef8b7d
[low-code connectors] Support for list-based stream slicers ( #13668 )
...
* list slicer
* Add comment
* test
* ast eval
* refactor
* fix
* remove print
* rename
* return a generator
* smaller fix and reset to master
* union type hint
* assert
* comment
* format
2022-06-14 17:12:53 -07:00
Charles
22b727c0ea
Update Airbyte Protocol Docs ( #13709 )
2022-06-14 15:04:38 -07:00
Alexandre Girard
57dd7a9d4b
[low-code connectors] use jello instead of jq ( #13570 )
...
* use jello instead of jq
* reset
* rename
* remove print
2022-06-07 14:10:26 -07:00
Alexandre Girard
1225f80bd8
Source hubspot: filter out streams we don't have access to ( #13455 )
...
* filter out streams we don't have access to
* cleanup
* fix associations
* fix for api key auth
* this doesn't need to be a property method
* Bump
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com >
2022-06-03 12:06:51 -07:00
Alexandre Girard
7fbe6dab5c
[Low code CDK] add support for iterating over parent stream slices ( #13287 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944c8fe154ffc
2022-06-02 08:53:04 -07:00
Brian Lai
ccbd7c6508
conditional paginator + interpolated boolean components ( #13389 )
...
* conditional paginator + interpolated boolean components
* fix bug where empty arrays and dicts evaluated to true
* add more falsey values
* adjust falsey values and tweak based on review comments
2022-06-01 20:24:32 -04:00
Alexandre Girard
aa92518721
Low-code connectors: configurable source from yaml ( #13038 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* sengrid low code connector
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* update import
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944snadalive@gmail.com >
* more precise exception
* rename class
* add comment
* Try to install packages to build jq
* isort
* only automake
* Revert "only automake"
This reverts commit c8fe154ffcab5a7afd087c9fb8eb33snadalive@gmail.com >
* fix tests and format
* format
* update
* better error message
* Add jq dependency
* Use request header provider
* rename
* rename field
* remove get_context method
* rename
* add a comment
* format
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
2022-06-01 07:31:52 -07:00
Alexandre Girard
d9fa24ffff
Low code connectors: implement components for sendgrid ( #12853 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944c8fe154ffc
2022-06-01 07:02:35 -07:00
Denis Davydov
acd1533cec
🎉 Python CDK: handle requests.exceptions.ChunkedEncodingError for broken connections ( #13260 )
2022-05-27 22:05:39 +03:00
Marcos Marx
dca2256a7c
Bump 2022 license version ( #13233 )
...
* Bump year in license short to 2022
* remove protocol from cdk
2022-05-26 15:00:42 -03:00
Brian Lai
c4ef7db5eb
[Low Code CDK] add support for string interpolation for request headers ( #13215 )
...
* add support for string interpolation for request headers + small refactor
* fix formatting
* move parameter provider to folder for consistency
2022-05-25 22:15:19 -04:00
Alexandre Girard
3894134d11
Bump year in license short to 2022 ( #13191 )
...
* Bump to 2022
* format
2022-05-25 17:56:49 -07:00
Alexandre Girard
150ab593f8
Low code connectors: core structure ( #12850 )
...
* checkout from alex/cac
* doc
* doc
* remove broken test
* rename
* rename file
* delete unused file
* rename
* abstract property
* isort
* update state
* Update comment
* remove incremental mixin
* delete comment
* update comments
* update comments
* remove no_state
* rename package
* pass parameters through kwargs
* update interface to pass source in interface
* update interface to pass source in interface
* rename to stream_slicer
* Low code connectors: string interpolation with jinja (#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944
2022-05-25 17:27:54 -07:00
Pedro S. Lopez
41dc82f056
CDK: provide better user-friendly error messages ( #12944 )
...
* initial implementation
* add parse_response_error_message tests
* move error handler to existing try/catch in AbstractSource
* formatting
* var rename
* use isinstance for type checking
* add docstrings
* add more abstract_source and httpstream tests
* fix wrong httperror usage
* more test cases
* bump version, update changelog
2022-05-19 17:28:24 -04:00
Serhii Chvaliuk
9a960c4ca6
CDK: TypeTransformer.default_convert catch TypeError ( #12948 )
...
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com >
2022-05-19 07:32:48 +03:00
Pedro S. Lopez
73c7fad7fc
CDK: emit AirbyteTraceMessage with exception trace information ( #12593 )
2022-05-06 17:16:57 -04:00
Pedro S. Lopez
a6a6be2588
allow importing IncrementalMixin from airbyte_cdk.sources.streams ( #11858 )
...
* allow importing IncrementalMixin from airbyte_cdk.sources.streams
* format file
* update docs with hint on where to import IncrementalMixin from
* bump cdk version to 0.1.54
* update changelog
* only build python 3.9
* update changelog with python dep bump
2022-04-14 11:42:57 -07:00
Augustin
2d277747be
cdk: remove false error logging in _send ( #11650 )
2022-04-04 10:36:05 -07:00
Serhii Chvaliuk
df12c9e4d4
Python CDK - Fix BaseBackoffException constructor ( #11653 )
...
* - Fix BaseBackoffException constructor
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com >
2022-04-01 19:05:11 +03:00
Marcos Marx
78cafc58ab
CDK improve error handling responses ( #11629 )
...
* add response text and message to backoff error handling
* add response text and message to backoff error handling
* add response text case http error
* change response text to before raise error
* apply suggestions
* bump cdk version
2022-03-31 16:41:10 -03:00
LiRen Tu
049a11b2bc
🎉 Snowflake destination: reduce memory footprint ( #10394 )
...
* Add detailed logging for flushing
* Log sentry transaction event id
* Adjust logging
* Log memory usage
* Add jvm monitoring
* Remove log
* Remove port 9010
* Remove host network mode
* Sample record size
* Remove profiling code
* Add unit tests
* Use average estimation
* Rename variable
* Format code
* Bump version
* Revert unnecessary change
* Update doc
* Fix format
* Bump version in seed
2022-02-17 12:55:35 -08:00
Eugene Kulak
d173ce741f
CDK: add support for streams with state attribute ( #9746 )
...
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2022-02-17 00:20:33 +04:00
vitaliizazmic
936050847f
🐛 CDK: fixing type errors ( #9926 )
...
* CDK #9500 - fixing type errors
* CDK #9500 - exclude singer from check
* CDK #9500 - type ignore in utils.schema_helpers.py
* CDK #9500 - type ignore in utils.sentry.py
* CDK #9500 - type ignore in singer.singer_helpers.py
* CDK #9500 - type ignore in singer.source
* CDK #9500 - fix according to PR review
* CDK #9500 - fix according to PR review
* CDK #9500 - bump version
2022-02-14 09:24:04 +02:00
Eugene Kulak
f83eca58ea
CDK: Fix typing errors ( #9037 )
...
* fix typing, drop AirbyteLogger
* format
* bump the version
* use logger instead of fixture logger
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
Co-authored-by: auganbay <auganenu@gmail.com >
2022-01-14 10:29:34 +06:00
Dmytro
fe954c1a16
Integrate Sentry for performance and errors tracking. ( #8248 )
...
* Integrate Sentry for performance and errors tracking.
* Add sentry sensitive data scrubbing.
* updated cdk version and changelog
* Integrate Sentry for performance and errors tracking
Add `SENTRY_DSN` environment variable
* Integrate Sentry for performance and errors tracking
Add `sentry_sdk` to install requirements
* format cdk
* enable Sentry for google-search-console
* updated connector version
* update spec and source yamls
Co-authored-by: auganbay <auganenu@gmail.com >
Co-authored-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com >
2021-12-21 20:02:02 +06:00
augan-rymkhan
7fbdfc61d0
Airbyte CDK: Log http status code and content in backoff handlers ( #8829 )
...
* Log http status code and content in default backoff handler
* Log http status code and content in usef defined backoff handler
* updated cdk version and changelog
* make it clear: exc.response
Co-authored-by: auganbay <auganenu@gmail.com >
2021-12-16 20:20:12 +06:00
Eugene Kulak
d50ae47b49
🐛 CDK: Fix the logging of unhandled exceptions to show stacktrace ( #8704 )
...
* print stacktrace for unhandled exceptions
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2021-12-14 00:04:29 +02:00
Eugene Kulak
aa67604f09
CDK: Add base pydantic model for connector config and schemas ( #8485 )
...
* add base spec model
* fix usage of state_checkpoint_interval in case it is dynamic
* add schema base models, fix spelling, signatures and polishing
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2021-12-08 01:14:59 +02:00
Sergei Solonitcyn
6ee922cef2
Improve URL creation in the CDK ( #8513 )
...
Signed-off-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com >
2021-12-05 22:30:16 +02:00
Marcos Eliziario Santos
65d471abf4
Output run time per stream ( #8173 )
...
* Output run time per stream
* Code review changes: Remove multithreaded support.
Port code to cdk
* Formatting
* remove extraneous try block
2021-11-30 10:20:51 -03:00
Yevhenii
afaa963a7f
CDK: Fix 'emitted_at' from 'seconds * 1000' to correct milliseconds. ( #8295 )
2021-11-29 14:32:20 +02:00
Yevhenii
d64b1134d5
🐛 CDK: Add logger inheritance for streams from airbyte ( #8040 )
...
* Fix broken logger in streams: add logger inheritance for streams from airbyte
2021-11-17 11:37:03 +02:00
Dmytro
e56e86dd28
Fix schema transformation false warnings. ( #7863 )
2021-11-12 18:20:53 +02:00
LiRen Tu
c511abd20c
Fix master build ( #7873 )
...
* Check in format diff
* Remove cache http yaml files
* Ignore tmp files generated by unit tests
2021-11-11 09:47:51 -08:00
Dmytro
16b626cc91
Resolve discover catalog ref fields ( #7734 )
2021-11-09 12:42:15 +02:00
Yevhenii
5eb2af6c08
CDK: Autogenerate reference docs ( #4759 )
...
* CDK Autogenerated reference docs: base version
* Update docs config.
Add .readthedocs.yaml file.
Update build html files.
* Update .gitignore.
Remove sphinx build files.
* Add newline at the end of .gitignore
* Update setup.py requirements.
Update .readthedocs.yaml config.
* Update rst files.
Add Makefile rst config.
* Update CDK docstring format.
Change rst layout.
Update sphinx config.
* Add Sphinx docs.
Update index.rst.
Update abstract_source.py docstrings.
* Override master_doc and package templates.
Add docs schema enerator script.
Update sphinx docs.
* Add `Publishing to Read the Docs` section to sphinx-docs.md".
Replace sphinx-docs.md to `airbyte-cdk` module.
* Update sphinx-docs.md section name
* Bump airbyte-cdk version.
Update CHANGELOG.md.
Co-authored-by: ykurochkin <y.kurochkin@zazmic.com >
Co-authored-by: Vadym Hevlich <vege1wgw@gmail.com >
2021-10-22 20:47:48 +03:00
vitaliizazmic
d5c04991bc
🎉 CDK: use standard logger with custom handler
...
* CDK - using native logger with custom formatter
* CDK - using native logger(fix exception and add trace)
* CDK - using native logger in AirbyteEntrypoint
* CDK - CHANGELOG.md
* CDK - remove unnecessary imports
* CDK - fixing according to PR review
* CDK native logger #1279 - annotations
* CDK native logger #1279 - fixing according to PR review
* CDK standard logger #1279 - tests
* CDK standard logger #1279 - reformat
* Airbyte CDK #1279 - improve docstrings
* Airbyte CDK #1279 - improve log levels
* Airbyte CDK #1279 - fix init get source name
* Airbyte CDK #1279 - update test licence
* Airbyte CDK #1279 - bump version
2021-10-12 20:56:56 +03:00
Michel Tricot
1773e41e47
Shorten our headers + adds contributors file ( #6478 )
2021-09-27 10:45:50 -07:00
Dmytro
1c5ac5b5ea
🏗️ Python CDK: add schema transformer class ( #6139 )
...
* Python CDK: add schema transformer class
2021-09-27 13:07:17 +03:00
Marcos Marx
5861b6faf7
correct CI applying gradlew format ( #6390 )
...
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local >
2021-09-23 01:55:06 -03:00
Arthur Galuza
9aa5a5a52d
🎉 CDK: Added support for efficient parent/child streams using cache ( #6057 )
...
* Add caching
* Upd cache file handling
* Upd slices, sync mode, docs
* Bump version
* Use SyncMode.full_refresh for parent stream_slices
* Refactor
2021-09-22 20:23:27 +03:00
oleh.zorenko
4dca32713b
🎉 CDK: Add support for custom headers passing to the request in OAuth2Authenticator. refresh_access_token ( #6219 )
...
* Add support for headers to OAuth2Authenticator
Send custom headers in `refresh_access_token()`.
* Bump version + update CHANGELOG.md
* Add tests
* Update tests for refresh_access_token()
* Assert that mock_refresh_token_call was called
* Remove init file
2021-09-22 07:45:05 +03:00
Dmytro
474b8f1065
CDK: support nested refs resolving ( #6044 )
...
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com >
2021-09-16 18:38:08 +03:00
Vadym
4a0d364ea1
🎉 CDK: Add requests native authenticator support ( #5731 )
...
* Add requests native auth class
* Update init file.
Update type annotations.
Bump version.
* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.
* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md
* Add requests native authenticator tests
* Add CDK requests native __call__ method tests.
Update CHANGELOG.md
* Add outdated auth deprication messages
* Update requests native auth __call__ method tests
* Bump CDK version to 0.1.20
2021-09-15 19:23:31 +03:00
Marcos Eliziario Santos
1314a6a04c
#5796 silence printing full config when config validation fails ( #5879 )
...
* - #5796 silence printing full config when config validation fails
* fix unit tests after config validation check changes
Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com >
2021-09-14 11:00:47 -03:00
Dmytro
6041f3df37
🐛 CDK: fix bug with limit parameter for incremental stream ( #5833 )
...
* CDK: fix bug with limit parameter for incremental stream
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com >
2021-09-09 19:55:29 +03:00
oleh.zorenko
58c9b84c3d
🐛 CDK: fix number of records mismatch ( #5767 )
...
* CDK: fix number of records mismatch
Increment `record_counter` after we check whether we reached the limit or not.
* Update docs. Bump version to 0.1.17
2021-09-02 09:16:43 +03:00
Dmytro
7584440515
CDK: private configuration option _limit and _page_size ( #5617 )
...
* CDK: private configuration option _limit and _page_size
2021-08-31 12:16:48 +03:00
