* write out new request options provider and refactor components and parts of the YAML config
* fix formatting
* pr feedback to consolidate body_data_provider to simplify the code
* pr feedback get rid of extraneous optional
* list slicer
* Add comment
* test
* ast eval
* refactor
* fix
* remove print
* rename
* return a generator
* smaller fix and reset to master
* union type hint
* assert
* comment
* format
* Update schema
* generate python
* Stream as an object
* PR comments
* generate python
* rm unused required
* Description the state with no type
* Fix connector build
* Format
* format
Co-authored-by: cgardens <charles@airbyte.io>
* filter out streams we don't have access to
* cleanup
* fix associations
* fix for api key auth
* this doesn't need to be a property method
* Bump
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* conditional paginator + interpolated boolean components
* fix bug where empty arrays and dicts evaluated to true
* add more falsey values
* adjust falsey values and tweak based on review comments
* generate AirbyeTraceMessage `type` enum with descriptive class name
* add comment on `title` usage
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* apply changes to bases/airbyte-protocol
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* allow importing IncrementalMixin from airbyte_cdk.sources.streams
* format file
* update docs with hint on where to import IncrementalMixin from
* bump cdk version to 0.1.54
* update changelog
* only build python 3.9
* update changelog with python dep bump
* add response text and message to backoff error handling
* add response text and message to backoff error handling
* add response text case http error
* change response text to before raise error
* apply suggestions
* bump cdk version
* Pass worker metadata to connector
* Fix compilation
* Pass in job id and image from worker
* Remove application version
* Add default job environment variables
* Add back removed comment
* Rename env map to job metadata
* Fix env configs
* Read connector from application
* Use empty string
* Remove println
* Fix unit test
* Fix compilation error
* Introduce constants for worker env
* Add worker env to ENV_VARS_TO_TRANSFER
* Pass into getWorkerMetadata map to all constructions
* Format code
* Format octavia cli
* Fix test compilation
* Fix typos
* Ensure AirbyteLogger is thread-safe
- Introduce a global lock to ensure `AirbyteLogger` is thread-safe.
- The `logging` module is thread-safe, however `print` is not, and is currently used. This means that messages sent to stdout can clash if connectors use threading. This is obviously a huge problem when the IPC between the source/destination is stdout!
- A `multiprocessing.Lock` could have been introduced however given that `logging` module is not multiprocess-safe I thought that thread-safety should be first goal.
- IMO the `AirbyteLogger` should be a subclass of the `logging.Logger` so you have thread-safety automatically, however I didn't want to make a huge wholesale change here.
* Revert lock and add deprecation warning instead
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
* CDK #9500 - fixing type errors
* CDK #9500 - exclude singer from check
* CDK #9500 - type ignore in utils.schema_helpers.py
* CDK #9500 - type ignore in utils.sentry.py
* CDK #9500 - type ignore in singer.singer_helpers.py
* CDK #9500 - type ignore in singer.source
* CDK #9500 - fix according to PR review
* CDK #9500 - fix according to PR review
* CDK #9500 - bump version
* fix typing, drop AirbyteLogger
* format
* bump the version
* use logger instead of fixture logger
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: auganbay <auganenu@gmail.com>
* [9044] Destination-gcs\destination-bigquery(gcs) - updated check() method to handle that user has both storage.objects.create and storage.multipartUploads.create roles
This commit reverts #9348 (9bb28939ee) because it does not work. The `test_docker_runner[standard]` and `test_docker_runner[waiting]` test cases still fail transiently.
* Integrate Sentry for performance and errors tracking.
* Add sentry sensitive data scrubbing.
* updated cdk version and changelog
* Integrate Sentry for performance and errors tracking
Add `SENTRY_DSN` environment variable
* Integrate Sentry for performance and errors tracking
Add `sentry_sdk` to install requirements
* format cdk
* enable Sentry for google-search-console
* updated connector version
* update spec and source yamls
Co-authored-by: auganbay <auganenu@gmail.com>
Co-authored-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com>
* Log http status code and content in default backoff handler
* Log http status code and content in usef defined backoff handler
* updated cdk version and changelog
* make it clear: exc.response
Co-authored-by: auganbay <auganenu@gmail.com>
* add base spec model
* fix usage of state_checkpoint_interval in case it is dynamic
* add schema base models, fix spelling, signatures and polishing
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
* Secure logger implementation minus still broken new tests
* Secure logger implementation and unit tests
* code review changes
* filter text on uncaught exceptions
* auto-formatting
* Mistaken change left in code
* filter text on uncaught exceptions
* Simplify code, remove LoggingFilter and move obfuscation functionality to Formatter
* Update airbyte-cdk/python/airbyte_cdk/entrypoint.py
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
* Obfuscate Secrets in Logging, code review changes
* Obfuscate Secrets in Logging, code review changes, unit test fixes
* CHANGELOG.md
* Format and flake8
* Fix build error/bump version/format
* Add airbyte prefix on test logger
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
* Secure logger implementation minus still broken new tests
* Secure logger implementation and unit tests
* code review changes
* filter text on uncaught exceptions
* auto-formatting
* Mistaken change left in code
* filter text on uncaught exceptions
* Simplify code, remove LoggingFilter and move obfuscation functionality to Formatter
* Update airbyte-cdk/python/airbyte_cdk/entrypoint.py
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
* Obfuscate Secrets in Logging, code review changes
* Obfuscate Secrets in Logging, code review changes, unit test fixes
* CHANGELOG.md
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
