Brian Lai
be01b476ce
Add new InterpolatedRequestOptionsProvider that encapsulates all variations of request arguments ( #13472 )
...
* write out new request options provider and refactor components and parts of the YAML config
* fix formatting
* pr feedback to consolidate body_data_provider to simplify the code
* pr feedback get rid of extraneous optional
2022-06-21 16:01:05 -04:00
Alexandre Girard
34e2dc36c6
[low-code connectors] Cartesian product stream slicer ( #13740 )
...
* list slicer
* Add comment
* product stream slicer
* comment
* rename
* format
* Update comment
* split on 2 lines for readability
* Revert "rename"
This reverts commit e801f2d1f8
2022-06-16 08:48:32 -07:00
Alexandre Girard
2c8fef8b7d
[low-code connectors] Support for list-based stream slicers ( #13668 )
...
* list slicer
* Add comment
* test
* ast eval
* refactor
* fix
* remove print
* rename
* return a generator
* smaller fix and reset to master
* union type hint
* assert
* comment
* format
2022-06-14 17:12:53 -07:00
Charles
22b727c0ea
Update Airbyte Protocol Docs ( #13709 )
2022-06-14 15:04:38 -07:00
Alexandre Girard
15fe51b788
[low-code connectors] add a few unit tests ( #13666 )
...
* add a couple of unit tests
* refactor tests
2022-06-13 16:51:36 -07:00
Benoit Moriceau
704dd8b534
Update schema ( #13573 )
...
* Update schema
* generate python
* Stream as an object
* PR comments
* generate python
* rm unused required
* Description the state with no type
* Fix connector build
* Format
* format
Co-authored-by: cgardens <charles@airbyte.io >
2022-06-10 16:35:59 -07:00
Pedro S. Lopez
4f8f979045
Fix Connectors Base Build: CDK tests fail after dependency update ( #13652 )
2022-06-09 16:05:36 -04:00
Alexandre Girard
57dd7a9d4b
[low-code connectors] use jello instead of jq ( #13570 )
...
* use jello instead of jq
* reset
* rename
* remove print
2022-06-07 14:10:26 -07:00
Benoit Moriceau
5a7b1aaacd
Bmoric/add namespace to protocol ( #13356 )
...
* Add namespace
* Matching python update
* Pr comments
* regenerate and format python cdk
* format
Co-authored-by: Pedro Lopez <pedroslopez@me.com >
2022-06-06 15:41:56 -07:00
Davin Chia
dd6d263dd4
Update generate script to use new protocol models directory. ( #13479 )
...
Missed this change from https://github.com/airbytehq/airbyte/pull/13476 .
2022-06-06 01:31:38 +08:00
Alexandre Girard
1225f80bd8
Source hubspot: filter out streams we don't have access to ( #13455 )
...
* filter out streams we don't have access to
* cleanup
* fix associations
* fix for api key auth
* this doesn't need to be a property method
* Bump
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com >
2022-06-03 12:06:51 -07:00
Alexandre Girard
7fbe6dab5c
[Low code CDK] add support for iterating over parent stream slices ( #13287 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944c8fe154ffc
2022-06-02 08:53:04 -07:00
Brian Lai
ccbd7c6508
conditional paginator + interpolated boolean components ( #13389 )
...
* conditional paginator + interpolated boolean components
* fix bug where empty arrays and dicts evaluated to true
* add more falsey values
* adjust falsey values and tweak based on review comments
2022-06-01 20:24:32 -04:00
Alexandre Girard
aa92518721
Low-code connectors: configurable source from yaml ( #13038 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* sengrid low code connector
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* update import
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944snadalive@gmail.com >
* more precise exception
* rename class
* add comment
* Try to install packages to build jq
* isort
* only automake
* Revert "only automake"
This reverts commit c8fe154ffcab5a7afd087c9fb8eb33snadalive@gmail.com >
* fix tests and format
* format
* update
* better error message
* Add jq dependency
* Use request header provider
* rename
* rename field
* remove get_context method
* rename
* add a comment
* format
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
2022-06-01 07:31:52 -07:00
Alexandre Girard
d9fa24ffff
Low code connectors: implement components for sendgrid ( #12853 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944c8fe154ffc
2022-06-01 07:02:35 -07:00
Denis Davydov
acd1533cec
🎉 Python CDK: handle requests.exceptions.ChunkedEncodingError for broken connections ( #13260 )
2022-05-27 22:05:39 +03:00
Alexandre Girard
731c8bf178
Add airbyte protocol file ( #13234 )
2022-05-26 11:22:46 -07:00
Marcos Marx
dca2256a7c
Bump 2022 license version ( #13233 )
...
* Bump year in license short to 2022
* remove protocol from cdk
2022-05-26 15:00:42 -03:00
Brian Lai
c4ef7db5eb
[Low Code CDK] add support for string interpolation for request headers ( #13215 )
...
* add support for string interpolation for request headers + small refactor
* fix formatting
* move parameter provider to folder for consistency
2022-05-25 22:15:19 -04:00
Alexandre Girard
3894134d11
Bump year in license short to 2022 ( #13191 )
...
* Bump to 2022
* format
2022-05-25 17:56:49 -07:00
Alexandre Girard
150ab593f8
Low code connectors: core structure ( #12850 )
...
* checkout from alex/cac
* doc
* doc
* remove broken test
* rename
* rename file
* delete unused file
* rename
* abstract property
* isort
* update state
* Update comment
* remove incremental mixin
* delete comment
* update comments
* update comments
* remove no_state
* rename package
* pass parameters through kwargs
* update interface to pass source in interface
* update interface to pass source in interface
* rename to stream_slicer
* Low code connectors: string interpolation with jinja (#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944
2022-05-25 17:27:54 -07:00
Pedro S. Lopez
41dc82f056
CDK: provide better user-friendly error messages ( #12944 )
...
* initial implementation
* add parse_response_error_message tests
* move error handler to existing try/catch in AbstractSource
* formatting
* var rename
* use isinstance for type checking
* add docstrings
* add more abstract_source and httpstream tests
* fix wrong httperror usage
* more test cases
* bump version, update changelog
2022-05-19 17:28:24 -04:00
Serhii Chvaliuk
9a960c4ca6
CDK: TypeTransformer.default_convert catch TypeError ( #12948 )
...
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com >
2022-05-19 07:32:48 +03:00
Pedro S. Lopez
7ea27d0ecd
CDK: publish version 0.1.57 ( #12938 )
2022-05-17 16:33:06 -04:00
Pedro S. Lopez
b8703ea6a1
update CDK airbyte protocol models to fix master build ( #12829 )
2022-05-12 22:38:30 -04:00
Pedro S. Lopez
73c7fad7fc
CDK: emit AirbyteTraceMessage with exception trace information ( #12593 )
2022-05-06 17:16:57 -04:00
Sherif A. Nada
49810f8c13
sweep airbyte-integrations/bases/airbyte-protocol ( #12598 )
2022-05-05 13:51:16 -07:00
Sherif A. Nada
cc46255190
cull base-python and base-python-test ( #12596 )
2022-05-05 13:48:15 -07:00
Pedro S. Lopez
3c1eab3e27
generate AirbyeTraceMessage.type enum with descriptive class name ( #12581 )
...
* generate AirbyeTraceMessage `type` enum with descriptive class name
* add comment on `title` usage
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
* apply changes to bases/airbyte-protocol
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
2022-05-04 18:48:34 -04:00
LiRen Tu
a294f40383
Use jsongenerator 1.0.1 ( #12554 )
...
* Use jsongenerator 1.0.1
* Fix connectors base
2022-05-03 21:31:12 -07:00
Brian Lai
ceec454e30
update python cdk tutorial with updates to exchange rates api ( #12427 )
...
* update python cdk tutorial with updates to exchange rates api
* remove extra comment in source.py tutorial sample code
2022-04-29 13:44:04 -07:00
Pedro S. Lopez
53799cb9a2
🎉 CDK: support loading spec from yaml file ( #12104 )
...
* support loading spec from yaml file
* formatting
* remove commented code
* update comment
* remove unused file
* raise correct exception types
* bump version, update changelog
2022-04-20 13:18:46 -07:00
Pedro S. Lopez
a6a6be2588
allow importing IncrementalMixin from airbyte_cdk.sources.streams ( #11858 )
...
* allow importing IncrementalMixin from airbyte_cdk.sources.streams
* format file
* update docs with hint on where to import IncrementalMixin from
* bump cdk version to 0.1.54
* update changelog
* only build python 3.9
* update changelog with python dep bump
2022-04-14 11:42:57 -07:00
Brian Leonard
c302af45ff
Upgrade to Python 3.9 ( #11763 )
...
* Dockerfile to 3.9
* Python version
* More python updates
* 3.9 on GitHub actions and lint updates
* Test out 3.9.11 on GitHub actions
* install python with an action
* formatting: newline
* Also has python code
* only check first level for changed modules
Previous example (source-google-search-console/credentials)
* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Augustin
2d277747be
cdk: remove false error logging in _send ( #11650 )
2022-04-04 10:36:05 -07:00
Serhii Chvaliuk
df12c9e4d4
Python CDK - Fix BaseBackoffException constructor ( #11653 )
...
* - Fix BaseBackoffException constructor
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com >
2022-04-01 19:05:11 +03:00
Marcos Marx
78cafc58ab
CDK improve error handling responses ( #11629 )
...
* add response text and message to backoff error handling
* add response text and message to backoff error handling
* add response text case http error
* change response text to before raise error
* apply suggestions
* bump cdk version
2022-03-31 16:41:10 -03:00
LiRen Tu
81417e6728
Add connector metadata as sentry tags ( #10475 )
...
* Pass worker metadata to connector
* Fix compilation
* Pass in job id and image from worker
* Remove application version
* Add default job environment variables
* Add back removed comment
* Rename env map to job metadata
* Fix env configs
* Read connector from application
* Use empty string
* Remove println
* Fix unit test
* Fix compilation error
* Introduce constants for worker env
* Add worker env to ENV_VARS_TO_TRANSFER
* Pass into getWorkerMetadata map to all constructions
* Format code
* Format octavia cli
* Fix test compilation
* Fix typos
2022-03-09 07:36:03 -08:00
girarda
1edd6d7684
run gradle format ( #10746 )
2022-02-28 18:00:41 -08:00
jdclarke5
be8f124ea9
CDK: Ensure AirbyteLogger is thread-safe using Lock ( #9943 )
...
* Ensure AirbyteLogger is thread-safe
- Introduce a global lock to ensure `AirbyteLogger` is thread-safe.
- The `logging` module is thread-safe, however `print` is not, and is currently used. This means that messages sent to stdout can clash if connectors use threading. This is obviously a huge problem when the IPC between the source/destination is stdout!
- A `multiprocessing.Lock` could have been introduced however given that `logging` module is not multiprocess-safe I thought that thread-safety should be first goal.
- IMO the `AirbyteLogger` should be a subclass of the `logging.Logger` so you have thread-safety automatically, however I didn't want to make a huge wholesale change here.
* Revert lock and add deprecation warning instead
2022-02-28 19:55:30 -03:00
LiRen Tu
049a11b2bc
🎉 Snowflake destination: reduce memory footprint ( #10394 )
...
* Add detailed logging for flushing
* Log sentry transaction event id
* Adjust logging
* Log memory usage
* Add jvm monitoring
* Remove log
* Remove port 9010
* Remove host network mode
* Sample record size
* Remove profiling code
* Add unit tests
* Use average estimation
* Rename variable
* Format code
* Bump version
* Revert unnecessary change
* Update doc
* Fix format
* Bump version in seed
2022-02-17 12:55:35 -08:00
Eugene Kulak
d173ce741f
CDK: add support for streams with state attribute ( #9746 )
...
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2022-02-17 00:20:33 +04:00
vitaliizazmic
936050847f
🐛 CDK: fixing type errors ( #9926 )
...
* CDK #9500 - fixing type errors
* CDK #9500 - exclude singer from check
* CDK #9500 - type ignore in utils.schema_helpers.py
* CDK #9500 - type ignore in utils.sentry.py
* CDK #9500 - type ignore in singer.singer_helpers.py
* CDK #9500 - type ignore in singer.source
* CDK #9500 - fix according to PR review
* CDK #9500 - fix according to PR review
* CDK #9500 - bump version
2022-02-14 09:24:04 +02:00
Eugene Kulak
f83eca58ea
CDK: Fix typing errors ( #9037 )
...
* fix typing, drop AirbyteLogger
* format
* bump the version
* use logger instead of fixture logger
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
Co-authored-by: auganbay <auganenu@gmail.com >
2022-01-14 10:29:34 +06:00
Eugene
44cb30aca5
🐛 Destination-gcs\destination-bigquery(gcs) - updated check() method to handle that user has both storage.objects.create and storage.multipartUploads.create roles ( #9121 )
...
* [9044] Destination-gcs\destination-bigquery(gcs) - updated check() method to handle that user has both storage.objects.create and storage.multipartUploads.create roles
2022-01-10 22:00:16 +02:00
LiRen Tu
0ee4d09eed
Revert "Fix connector base test ( #9348 )" ( #9373 )
...
This commit reverts #9348 (9bb28939ee
2022-01-09 22:17:24 -08:00
LiRen Tu
9bb28939ee
Fix connector base test ( #9348 )
...
* Format code
* Add sleep back
* Revert wait time
* Decrease waiting time
* Adjust waiting time
* Delay container command
2022-01-09 20:06:13 -08:00
Marcos Eliziario Santos
fd716d7a10
Adding unit tests to SAT full refresh test ( #8589 )
...
* WIP - Adding unit tests to SAT full refresh test
* unit tests to SAT full refresh test
* Format
* Update airbyte-integrations/bases/source-acceptance-test/unit_tests/test_test_full_refresh.py
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com >
* use symmetric_difference to check for differences between expected/actual when testing stream read on SAT
* bump version
* bump version
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com >
2021-12-21 12:09:09 -03:00
Dmytro
fe954c1a16
Integrate Sentry for performance and errors tracking. ( #8248 )
...
* Integrate Sentry for performance and errors tracking.
* Add sentry sensitive data scrubbing.
* updated cdk version and changelog
* Integrate Sentry for performance and errors tracking
Add `SENTRY_DSN` environment variable
* Integrate Sentry for performance and errors tracking
Add `sentry_sdk` to install requirements
* format cdk
* enable Sentry for google-search-console
* updated connector version
* update spec and source yamls
Co-authored-by: auganbay <auganenu@gmail.com >
Co-authored-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com >
2021-12-21 20:02:02 +06:00
augan-rymkhan
7fbdfc61d0
Airbyte CDK: Log http status code and content in backoff handlers ( #8829 )
...
* Log http status code and content in default backoff handler
* Log http status code and content in usef defined backoff handler
* updated cdk version and changelog
* make it clear: exc.response
Co-authored-by: auganbay <auganenu@gmail.com >
2021-12-16 20:20:12 +06:00
