Brian Lai
374e265fcb
[Low Code CDK] configurable oauth request payload ( #13993 )
...
* configurable oauth request payload
* support interpolation for dictionaries that are not new subcomponents
* rewrite a declarative oauth authenticator that performs interpolation at runtime
* formatting
* whatever i don't know why factory gets flagged w/ the newline change
* we java now
* remove duplicate oauth
* add some comments
* parse time properly from string interpolation
* move declarative oauth to its own package in declarative module
* add changelog info
2022-07-08 16:49:16 -04:00
Alexandre Girard
01fc7a48c7
move jinja macros to their own file ( #14515 )
...
* move jinja macros to their own file
* format
* export as dict
* format
2022-07-08 07:56:06 -07:00
Brian Lai
35ca7c8d10
[low code cdk] Add MinMaxDate component and allow for it to be specified for DatetimeStreamSlicer ( #14288 )
...
* implement MinMaxDate and incorporate it into the DatetimeStreamSlicer
* rename to datetime and restrict start/end to minmaxdatetime object
* get rid of eval_date which isn't needed anymore and allow for datetimestreamslicer to set datetime format
* add MinMaxDatetime to class types registry
2022-07-07 01:05:13 -04:00
Alexandre Girard
37acb9289f
[low-code cdk] Enable configurable state checkpointing ( #14317 )
...
* checkout files from test branch
* read_incremental works
* reset to master
* remove dead code
* comment
* fix
* Add test
* comments
* utc
* format
* small fix
* Add test with rfc3339
* remove unused param
* fix test
* configurable state checkpointing
* update test
2022-07-05 19:20:23 -07:00
jordan-glitch
4395fbb4d2
cdk: Attach namespace to stream in catalog ( #13923 )
2022-07-04 18:04:31 +02:00
Alexandre Girard
e23789bbd6
Alex/declarative stream incremental fix ( #14268 )
...
* checkout files from test branch
* read_incremental works
* reset to master
* remove dead code
* comment
* fix
* Add test
* comments
* utc
* format
* small fix
* Add test with rfc3339
* remove unused param
* fix test
2022-07-01 08:39:19 -07:00
Alexandre Girard
85449c975c
[low-code connectors] default types and default values ( #14004 )
...
* default types and default values
* cleanup
* fixes so read works
* remove prints and trycatch
* comment
* remove unused param
* split file
* extract method
* extract methods
* comment
* optional
* fix test
* cleanup
* delete interpolated request header provider
* simplify next page url paginator interface
* comment
* format
2022-06-27 20:43:03 -07:00
Sherif A. Nada
7acbcfbfb7
Fix CDK obfuscation of nested secrets ( #14035 )
2022-06-23 09:23:27 -07:00
George Claireaux
4d279f8238
Remove legacy sentry code from cdk ( #14016 )
...
* rip sentry out of cdk
* remove sentry dsn from gsc
2022-06-23 12:14:09 +01:00
Brian Lai
a61224887e
support semi incremental by adding extractor record filter ( #13520 )
...
* support semi incremental by adding extractor record filter
* refactor extractor into a record_selector that supports extraction and filtering of response records
2022-06-23 00:09:44 -04:00
Alexandre Girard
94abef3acc
[low-code connectors] BasicHttpAuthenticator ( #13733 )
...
* implement basichttpauthenticator
* add optional refresh access token authenticator
* remove prints
* type hints
* Fix and unit test
* missing test
* Add class to __init__ file
* Add comment
2022-06-22 07:35:32 -07:00
Brian Lai
be01b476ce
Add new InterpolatedRequestOptionsProvider that encapsulates all variations of request arguments ( #13472 )
...
* write out new request options provider and refactor components and parts of the YAML config
* fix formatting
* pr feedback to consolidate body_data_provider to simplify the code
* pr feedback get rid of extraneous optional
2022-06-21 16:01:05 -04:00
Alexandre Girard
34e2dc36c6
[low-code connectors] Cartesian product stream slicer ( #13740 )
...
* list slicer
* Add comment
* product stream slicer
* comment
* rename
* format
* Update comment
* split on 2 lines for readability
* Revert "rename"
This reverts commit e801f2d1f8
2022-06-16 08:48:32 -07:00
Alexandre Girard
2c8fef8b7d
[low-code connectors] Support for list-based stream slicers ( #13668 )
...
* list slicer
* Add comment
* test
* ast eval
* refactor
* fix
* remove print
* rename
* return a generator
* smaller fix and reset to master
* union type hint
* assert
* comment
* format
2022-06-14 17:12:53 -07:00
Alexandre Girard
15fe51b788
[low-code connectors] add a few unit tests ( #13666 )
...
* add a couple of unit tests
* refactor tests
2022-06-13 16:51:36 -07:00
Pedro S. Lopez
4f8f979045
Fix Connectors Base Build: CDK tests fail after dependency update ( #13652 )
2022-06-09 16:05:36 -04:00
Alexandre Girard
57dd7a9d4b
[low-code connectors] use jello instead of jq ( #13570 )
...
* use jello instead of jq
* reset
* rename
* remove print
2022-06-07 14:10:26 -07:00
Alexandre Girard
7fbe6dab5c
[Low code CDK] add support for iterating over parent stream slices ( #13287 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944c8fe154ffc
2022-06-02 08:53:04 -07:00
Brian Lai
ccbd7c6508
conditional paginator + interpolated boolean components ( #13389 )
...
* conditional paginator + interpolated boolean components
* fix bug where empty arrays and dicts evaluated to true
* add more falsey values
* adjust falsey values and tweak based on review comments
2022-06-01 20:24:32 -04:00
Alexandre Girard
aa92518721
Low-code connectors: configurable source from yaml ( #13038 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* sengrid low code connector
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* update import
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944snadalive@gmail.com >
* more precise exception
* rename class
* add comment
* Try to install packages to build jq
* isort
* only automake
* Revert "only automake"
This reverts commit c8fe154ffcab5a7afd087c9fb8eb33snadalive@gmail.com >
* fix tests and format
* format
* update
* better error message
* Add jq dependency
* Use request header provider
* rename
* rename field
* remove get_context method
* rename
* add a comment
* format
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
2022-06-01 07:31:52 -07:00
Alexandre Girard
d9fa24ffff
Low code connectors: implement components for sendgrid ( #12853 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944c8fe154ffc
2022-06-01 07:02:35 -07:00
Denis Davydov
acd1533cec
🎉 Python CDK: handle requests.exceptions.ChunkedEncodingError for broken connections ( #13260 )
2022-05-27 22:05:39 +03:00
Marcos Marx
dca2256a7c
Bump 2022 license version ( #13233 )
...
* Bump year in license short to 2022
* remove protocol from cdk
2022-05-26 15:00:42 -03:00
Brian Lai
c4ef7db5eb
[Low Code CDK] add support for string interpolation for request headers ( #13215 )
...
* add support for string interpolation for request headers + small refactor
* fix formatting
* move parameter provider to folder for consistency
2022-05-25 22:15:19 -04:00
Alexandre Girard
3894134d11
Bump year in license short to 2022 ( #13191 )
...
* Bump to 2022
* format
2022-05-25 17:56:49 -07:00
Alexandre Girard
150ab593f8
Low code connectors: core structure ( #12850 )
...
* checkout from alex/cac
* doc
* doc
* remove broken test
* rename
* rename file
* delete unused file
* rename
* abstract property
* isort
* update state
* Update comment
* remove incremental mixin
* delete comment
* update comments
* update comments
* remove no_state
* rename package
* pass parameters through kwargs
* update interface to pass source in interface
* update interface to pass source in interface
* rename to stream_slicer
* Low code connectors: string interpolation with jinja (#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944
2022-05-25 17:27:54 -07:00
Pedro S. Lopez
41dc82f056
CDK: provide better user-friendly error messages ( #12944 )
...
* initial implementation
* add parse_response_error_message tests
* move error handler to existing try/catch in AbstractSource
* formatting
* var rename
* use isinstance for type checking
* add docstrings
* add more abstract_source and httpstream tests
* fix wrong httperror usage
* more test cases
* bump version, update changelog
2022-05-19 17:28:24 -04:00
Pedro S. Lopez
73c7fad7fc
CDK: emit AirbyteTraceMessage with exception trace information ( #12593 )
2022-05-06 17:16:57 -04:00
Pedro S. Lopez
53799cb9a2
🎉 CDK: support loading spec from yaml file ( #12104 )
...
* support loading spec from yaml file
* formatting
* remove commented code
* update comment
* remove unused file
* raise correct exception types
* bump version, update changelog
2022-04-20 13:18:46 -07:00
Augustin
2d277747be
cdk: remove false error logging in _send ( #11650 )
2022-04-04 10:36:05 -07:00
Serhii Chvaliuk
df12c9e4d4
Python CDK - Fix BaseBackoffException constructor ( #11653 )
...
* - Fix BaseBackoffException constructor
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com >
2022-04-01 19:05:11 +03:00
Marcos Marx
78cafc58ab
CDK improve error handling responses ( #11629 )
...
* add response text and message to backoff error handling
* add response text and message to backoff error handling
* add response text case http error
* change response text to before raise error
* apply suggestions
* bump cdk version
2022-03-31 16:41:10 -03:00
Eugene Kulak
d173ce741f
CDK: add support for streams with state attribute ( #9746 )
...
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2022-02-17 00:20:33 +04:00
Eugene Kulak
f83eca58ea
CDK: Fix typing errors ( #9037 )
...
* fix typing, drop AirbyteLogger
* format
* bump the version
* use logger instead of fixture logger
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
Co-authored-by: auganbay <auganenu@gmail.com >
2022-01-14 10:29:34 +06:00
Dmytro
fe954c1a16
Integrate Sentry for performance and errors tracking. ( #8248 )
...
* Integrate Sentry for performance and errors tracking.
* Add sentry sensitive data scrubbing.
* updated cdk version and changelog
* Integrate Sentry for performance and errors tracking
Add `SENTRY_DSN` environment variable
* Integrate Sentry for performance and errors tracking
Add `sentry_sdk` to install requirements
* format cdk
* enable Sentry for google-search-console
* updated connector version
* update spec and source yamls
Co-authored-by: auganbay <auganenu@gmail.com >
Co-authored-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com >
2021-12-21 20:02:02 +06:00
Eugene Kulak
d50ae47b49
🐛 CDK: Fix the logging of unhandled exceptions to show stacktrace ( #8704 )
...
* print stacktrace for unhandled exceptions
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2021-12-14 00:04:29 +02:00
Eugene Kulak
aa67604f09
CDK: Add base pydantic model for connector config and schemas ( #8485 )
...
* add base spec model
* fix usage of state_checkpoint_interval in case it is dynamic
* add schema base models, fix spelling, signatures and polishing
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2021-12-08 01:14:59 +02:00
Marcos Eliziario Santos
34ff075ba7
flake8 fix 3764 obfuscate secrets logging ( #8583 )
...
* Secure logger implementation minus still broken new tests
* Secure logger implementation and unit tests
* code review changes
* filter text on uncaught exceptions
* auto-formatting
* Mistaken change left in code
* filter text on uncaught exceptions
* Simplify code, remove LoggingFilter and move obfuscation functionality to Formatter
* Update airbyte-cdk/python/airbyte_cdk/entrypoint.py
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com >
* Obfuscate Secrets in Logging, code review changes
* Obfuscate Secrets in Logging, code review changes, unit test fixes
* CHANGELOG.md
* Format and flake8
* Fix build error/bump version/format
* Add airbyte prefix on test logger
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com >
2021-12-07 11:35:50 -03:00
Marcos Eliziario Santos
1d493ec620
CDK: #3764 obfuscate secrets in logging ( #8211 )
...
* Secure logger implementation minus still broken new tests
* Secure logger implementation and unit tests
* code review changes
* filter text on uncaught exceptions
* auto-formatting
* Mistaken change left in code
* filter text on uncaught exceptions
* Simplify code, remove LoggingFilter and move obfuscation functionality to Formatter
* Update airbyte-cdk/python/airbyte_cdk/entrypoint.py
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com >
* Obfuscate Secrets in Logging, code review changes
* Obfuscate Secrets in Logging, code review changes, unit test fixes
* CHANGELOG.md
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com >
2021-12-07 10:45:22 -03:00
Marcos Eliziario Santos
65d471abf4
Output run time per stream ( #8173 )
...
* Output run time per stream
* Code review changes: Remove multithreaded support.
Port code to cdk
* Formatting
* remove extraneous try block
2021-11-30 10:20:51 -03:00
Dmytro
e56e86dd28
Fix schema transformation false warnings. ( #7863 )
2021-11-12 18:20:53 +02:00
LiRen Tu
c511abd20c
Fix master build ( #7873 )
...
* Check in format diff
* Remove cache http yaml files
* Ignore tmp files generated by unit tests
2021-11-11 09:47:51 -08:00
Dmytro
16b626cc91
Resolve discover catalog ref fields ( #7734 )
2021-11-09 12:42:15 +02:00
vitaliizazmic
0c7f06018b
🐛 Airbyte CDK: transforming Python log levels to Airbyte protocol log levels
...
* Airbyte CDK native logger #1279 - transforming Python log levels to Airbyte protocol log levels
* Airbyte CDK native logger #1279 - add test for level types transforming
* CDK standard logger #1279 - test for transform critical level to fatal
2021-10-22 17:14:01 +03:00
vitaliizazmic
d5c04991bc
🎉 CDK: use standard logger with custom handler
...
* CDK - using native logger with custom formatter
* CDK - using native logger(fix exception and add trace)
* CDK - using native logger in AirbyteEntrypoint
* CDK - CHANGELOG.md
* CDK - remove unnecessary imports
* CDK - fixing according to PR review
* CDK native logger #1279 - annotations
* CDK native logger #1279 - fixing according to PR review
* CDK standard logger #1279 - tests
* CDK standard logger #1279 - reformat
* Airbyte CDK #1279 - improve docstrings
* Airbyte CDK #1279 - improve log levels
* Airbyte CDK #1279 - fix init get source name
* Airbyte CDK #1279 - update test licence
* Airbyte CDK #1279 - bump version
2021-10-12 20:56:56 +03:00
Michel Tricot
1773e41e47
Shorten our headers + adds contributors file ( #6478 )
2021-09-27 10:45:50 -07:00
Dmytro
1c5ac5b5ea
🏗️ Python CDK: add schema transformer class ( #6139 )
...
* Python CDK: add schema transformer class
2021-09-27 13:07:17 +03:00
Marcos Marx
5861b6faf7
correct CI applying gradlew format ( #6390 )
...
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local >
2021-09-23 01:55:06 -03:00
Arthur Galuza
9aa5a5a52d
🎉 CDK: Added support for efficient parent/child streams using cache ( #6057 )
...
* Add caching
* Upd cache file handling
* Upd slices, sync mode, docs
* Bump version
* Use SyncMode.full_refresh for parent stream_slices
* Refactor
2021-09-22 20:23:27 +03:00
LiRen Tu
548a3a3063
Format code ( #6381 )
2021-09-22 09:30:14 -07:00
