* Restore AvailabilityStrategy
* Add test for http availability strategy on empty stream from original bug fix 7c17351631
* fix flake errors
* Change CheckStream to use availability strategy
* Refactor test from bug fix
* fix flake errors
* Remove extra helper file from cherry-pick
* Merge tests for default http availability strategy
* turn off HttpAvailabilityStrategy as default (for now) (#21488)
* turn off HttpAvailabilityStrategy as default (for now)
* Update imports accordingly
* Add test for failure case
* Except StopIteration - make test pass
* Don't attempt to connect to a stream if we get no stream slices
* Make helper method for getting first record for a slice
* Add comments and exit early if stream to check isn't in list of source streams
* move helpers to helper module
* Clarify what it means when StopIteration is returned by helper methods
* Rough first implememtation of AvailabilityStrategy s
* Basic unit tests for AvailabilityStrategy and ScopedAvailabilityStrategy
* Make availability_strategy a property, separate out tests
* Remove from DeclarativeSource, remove Source parameter from methods, make default no AvailabilityStrategy
* Add skip stream if not available to read()
* Changes to CDK to get source-github working using AvailabilityStrategy, flakecheck
* reorganize cdk class, add HTTPAvailabilityStrategy test
* cleanup, docstrings
* pull out error handling into separate method
* Pass source and logger to check_connection method
* Add documentation links, handle 403 specifically
* Fix circular import
* Add AvailabilityStrategy to Stream and HTTPStream classes
* Remove AS from abstract_source, add to Stream, HTTPStream, AvailabilityStrategy unit tests passing for per-stream strategies
* Modify MockHttpStream to set no AvailabilityStrategy since source test mocking doesn't support this
* Move AvailabilityStrategy class to sources.streams
* Move HTTPAvailabilityStrategy to http module
* Use pascal case for HttpAvailabilityStrategy
* Remove docs message method :( and default to True availability on unhandled HTTPErrors
* add check_availability method to stream class
* Add optional source parameter
* Add test for connector-specific documentation, small tests refactor
* Add test that performs the read() function for stream with default availability strategy
* Add test for read function behavior when stream is unavailable
* Add 403 info in logger message
* Don't return error for other HTTPErrors
* Split up error handling into methods 'unavailable_error_codes' and 'get_reason_for_error'
* rework overrideable list of status codes to be a dict with reasons, to enforce that users provide reasons for all listed errors
* Fix incorrect typing
* Move HttpAvailability to its own module, fix flake errors
* Fix ScopedAvailabilityStrategy, docstrings and types for streams/availability_strategy.py
* Docstrings and types for core.py and http/availability_strategy.py
* Move _get_stream_slices to a StreamHelper class
* Docstrings + types for stream_helpers.py, cleanup test_availability.py
* Clean up test_source.py
* Move logic of getting the initial record from a stream to StreamHelper class
* Add changelog and bump minor version
* change 'is True' and 'is False' behavior
* use mocker.MagicMock
* Remove ScopedAvailabilityStrategy
* Don't except non-403 errors, check_stream uses availability_strategy if possible
* CDK: pass error to reasons_for_error_codes
* make get_stream_slice public
* Add tests for raising unhandled errors and retries are handled
* Add tests for CheckStream via AvailabilityStrategy
* Add documentation for stream availability of http streams
* Move availability unit tests to correct modules, report error message if possible
* Add test for reporting specific error if available
* Raise exception with error message
* read from detail field if no other value was found
* format
* dont test the error message
* format
* error code
* bump
* method yielding airbytemessage
* move to Stream
* update abstract source
* reset
* missing file
* Yield request and response as log messages
* only emit request and responses if the debug flag is on
* add test docker image
* script to run acceptance tests with local cdk
* Update conftest to use a different image
* extract to method
* dont use a different image tag
* Always install local cdk
* break the cdk
* get path from current working directory
* or
* ignore unit test
* debug log
* Revert "AMI change: ami-0f23be2f917510c26 -> ami-005924fb76f7477ce (#18689)"
This reverts commit bf06decf73.
* build from the top
* Update source-acceptance-test
* fix
* copy setup
* some work on the gradle plugin
* reset to master
* delete unused file
* delete unused file
* reset to master
* optional argument
* delete dead code
* use latest cdk with sendgrid
* fix sendgrid dockerfile
* break the cdk
* use local file
* Revert "break the cdk"
This reverts commit 600c195541.
* dont raise an exception
* reset to master
* unit tests
* missing test
* more unit tests
* remove deprecated comment
* newline
* reset to master
* remove files
* reset
* Update abstract source
* remove method from stream
* convert to airbytemessage
* unittests
* Update
* unit test
* remove debug logs
* Revert "remove debug logs"
This reverts commit a1a139ef37.
* Revert "Revert "remove debug logs""
This reverts commit b1d62cdb60.
* Revert "reset to master"
This reverts commit 3fa6a004c1.
* fix
* slightly better test
* typing
* extract method
* Revert "Revert "reset to master""
This reverts commit 5dac7c2804.
* reset to master
* reset to master
* Revert "reset to master"
This reverts commit 3fa6a004c1.
* Comment
* operate on the message
* Revert "Revert "reset to master""
This reverts commit 5833c84d0a.
* comment
* test
* Revert "test"
This reverts commit 2f91b803b0.
* test
* Revert "test"
This reverts commit 62d95ebbb5.
* test
* Revert "test"
This reverts commit 27150ba341.
* format
* format
* symlink
* Update setup
* update path
* reset to master
* update
* Add local files
* greenhouse
* format
* symlink
* try reordering
* better error message
* better log message
* reset to master
* Revert "merge for qa"
This reverts commit ad7128f2c5, reversing
changes made to 7196c22a73.
* reset to master
* reset to master
* reset to master
* format
* gradlew format
* right type hints
* reset to master
* reset to master
* gradlew format
* a bunch of small fixes
* Update output format
* fixes from feedback
* fixme comment
* streams cannot return AirbyteRecordMessage
* fix
* format
* only return logs when running on debug mode
* move branching
* update typing
* remove dead code
* fix simpleretriever name
* i think this is better
* log response.text
* debug flag
* comment
* pass config
* comments
* run SATs
* fix most of the unit tests
* fix unit test
* reset to master
* runFromPath
* Revert "runFromPath"
This reverts commit 85979a801a.
* Revert "run SATs"
This reverts commit a8a8a2da95.
* no need to convert to dict
* fix test
* method yielding airbytemessage
* move to Stream
* update abstract source
* reset
* missing file
* add test docker image
* script to run acceptance tests with local cdk
* Update conftest to use a different image
* extract to method
* dont use a different image tag
* Always install local cdk
* break the cdk
* get path from current working directory
* or
* ignore unit test
* debug log
* Revert "AMI change: ami-0f23be2f917510c26 -> ami-005924fb76f7477ce (#18689)"
This reverts commit bf06decf73.
* build from the top
* Update source-acceptance-test
* fix
* copy setup
* some work on the gradle plugin
* reset to master
* delete unused file
* delete unused file
* reset to master
* optional argument
* delete dead code
* use latest cdk with sendgrid
* fix sendgrid dockerfile
* break the cdk
* use local file
* Revert "break the cdk"
This reverts commit 600c195541.
* dont raise an exception
* reset to master
* unit tests
* missing test
* more unit tests
* newline
* reset to master
* remove files
* reset
* Update abstract source
* remove method from stream
* convert to airbytemessage
* unittests
* Update
* unit test
* remove debug logs
* Revert "remove debug logs"
This reverts commit a1a139ef37.
* Revert "Revert "remove debug logs""
This reverts commit b1d62cdb60.
* Revert "reset to master"
This reverts commit 3fa6a004c1.
* Revert "Revert "reset to master""
This reverts commit 5dac7c2804.
* reset to master
* reset to master
* test
* Revert "test"
This reverts commit 2f91b803b0.
* test
* Revert "test"
This reverts commit 62d95ebbb5.
* test
* Revert "test"
This reverts commit 27150ba341.
* format
* support custom error messaging for error response + retryable errors
* remove changed backoff i was using for testing
* refactor filter to construct response status internally
* pr feedback
* bump version and update changelog
* initial first pass converting every component to dataclasses
* replace the hackier options pass through
* get rid of the hackier way
* fix issues w/ type hints by making options required and lots of fixes to the language to fix compatability for dataclasses
* add dataclasses-jsonschema to setup
* fix oauth authenticator to avoid dataclass name collisions
* fix spacing for CI tests
* remove property from oauth and fix a interpolation bug
* pr feedback and cleaning up the code a bit, attempt at avoiding renaming
* fix templates and bugs surfaced during greenhouse spec testing
* fix tests
* fix missing options in some declarative components
* fix tests related to pulling latest master
* fix issue w/ passing state, slice, and token to subcomponents
* switch name back to get_access_token() since no name collision anymore
* interpolatedauth
* fix tests
* fix import
* no need for default
* Bump version
* Missing docstrings
* example
* missing example
* more docstrings
* interpolated types
* [#14361] Adding new generator for configuration based source template
* remove unit tests and update a few doc files generated by the templates that aren't relevant to config based connectors
* use 0.1.65 as the latest available CDK version we have
* checkout files from test branch
* read_incremental works
* reset to master
* remove dead code
* comment
* fix
* Add test
* comments
* utc
* format
* small fix
* Add test with rfc3339
* remove unused param
* fix test
* configurable state checkpointing
* update test
* start working on retrier
* retry predicate
* return response status
* look in error message
* cleanup test
* constant backoff strategy
* chain backoff strategy
* chain retrier
* Add to class types registry
* extract backoff time from header
* wait until
* update
* split file
* parse_records
* classmethod
* delete dead code
* comment
* comment
* comments
* fix
* test for instantiating chain retrier
* fix parsing
* cleanup
* fix
* reset
* never raise on http error
* remove print
* comment
* comment
* comment
* comment
* remove prints
* add declarative stream to registry
* Delete dead code
* Add docstrings
* quick fix
* exponential backoff
* fix test
* fix
* delete unused properties
* fix
* missing unit tests
* uppercase
* docstrings
* rename to success
* compare full request instead of just url
* renmae module
* rename test file
* rename interface
* rename default retrier
* rename to compositeerrorhandler
* fix missing renames
* move action to filter
* str -> minmaxdatetime
* small fixes
* plural
* add example
* handle header variations
* also fix wait time from
* allow using a regex to extract the value
* group()
* docstring
* add docs
* update comment
* docstrings
* update comment
* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/http_requester.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* version: Update Parquet library to latest release (#14502)
The upstream Parquet library that is currently pinned for use in the S3 destination plugin is over a year old. The current version is generating invalid schemas for date-time with time-zone fields which appears to be addressed in the `1.12.3` release of the library in commit c72862b613
* merge
* 🎉 Source Github: improve schema for stream `pull_request_commits` added "null" (#14613)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
* Docs: Fixed broken links (#14622)
* fixing broken links
* more broken links
* source-hubspot: change mentioning of Mailchimp into HubSpot doc (#14620)
* Helm Chart: Add external temporal option (#14597)
* conflict env configmap and chart lock
* reverting lock
* add eof lines and documentation on values yaml
* conflict json file
* rollback json
* solve conflict
* correct minio with new version
Co-authored-by: Guy Feldman <gfeldman@86labs.com>
* 🎉 Add YAML format to source-file reader (#14588)
* Add yaml reader
* Update docs
* Bumpversion of connector
* bump docs
* Update pyarrow dependency
* Upgrade pandas dependency
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* 🎉 Source Okta: add GroupMembers stream (#14380)
* add Group_Members stream to okta source
- Group_Members return a list of users, the same schema of Users stream.
- Create a shared schema users, and both group_members and users sechema use it as a reference.
- Add Group_Members stream to source connector
* add tests and fix logs schema
- fix the test error: None is not one of enums though the enum type includes both string and null, it comes from json schema validator
ddb87afad8/jsonschema/_validators.py (L279-L285)
- change grouop_members to use id as the cursor field since `filter` is not supported in the query string
- fix the abnormal state test on logs stream, when since is abnormally large, until has to defined, an equal or a larger value
- remove logs stream from full sync test, because 2 full sync always has a gap -- at least a new log about users or groups api.
* last polish before submit the PR
- bump docker version
- update changelog
- add the right abnormal value for logs stream
- correct the sample catalog
* address comments::
- improve comments for until parameter under the logs stream
- add use_cache on groupMembers
* add use_cache to Group_Members
* change configured_catalog to test
* auto-bump connector version
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* split test files
* renames
* missing unit test
* add missing unit tests
* rename
* assert isinstance
* start extracting to their own files
* use final instead of classmethod
* assert we retry 429 errors
* Add log
* replace asserts with valueexceptions
* delete superfluous print statement
* fix factory so we don't need to union everything with strings
* get class_name from type
* remove from class types registry
* process error handlers one at a time
* sort
* delete print statement
* comment
* comment
* format
* delete unused file
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Tobias Macey <tmacey@boundlessnotions.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com>
Co-authored-by: Bas Beelen <bjgbeelen@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Guy Feldman <gfeldman@86labs.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Yiyang Li <yiyangli2010@gmail.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
* allow for command line debug option and basic debug statements + declarative
* feedback from pr comments
* fix some tests w/ req/res mixed up and fixing logging tests
* formatting
* pr feedback: cleaning up traces in logger.py and update docs with debug configuration
* remove unneeded trace logger test
* remove extra print statement
* configurable oauth request payload
* support interpolation for dictionaries that are not new subcomponents
* rewrite a declarative oauth authenticator that performs interpolation at runtime
* formatting
* whatever i don't know why factory gets flagged w/ the newline change
* we java now
* remove duplicate oauth
* add some comments
* parse time properly from string interpolation
* move declarative oauth to its own package in declarative module
* add changelog info
* implement basichttpauthenticator
* add optional refresh access token authenticator
* remove prints
* type hints
* Fix and unit test
* missing test
* Add class to __init__ file
* Add comment
* write out new request options provider and refactor components and parts of the YAML config
* fix formatting
* pr feedback to consolidate body_data_provider to simplify the code
* pr feedback get rid of extraneous optional
* filter out streams we don't have access to
* cleanup
* fix associations
* fix for api key auth
* this doesn't need to be a property method
* Bump
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* allow importing IncrementalMixin from airbyte_cdk.sources.streams
* format file
* update docs with hint on where to import IncrementalMixin from
* bump cdk version to 0.1.54
* update changelog
* only build python 3.9
* update changelog with python dep bump
* add response text and message to backoff error handling
* add response text and message to backoff error handling
* add response text case http error
* change response text to before raise error
* apply suggestions
* bump cdk version
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
* fix typing, drop AirbyteLogger
* format
* bump the version
* use logger instead of fixture logger
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: auganbay <auganenu@gmail.com>
* Integrate Sentry for performance and errors tracking.
* Add sentry sensitive data scrubbing.
* updated cdk version and changelog
* Integrate Sentry for performance and errors tracking
Add `SENTRY_DSN` environment variable
* Integrate Sentry for performance and errors tracking
Add `sentry_sdk` to install requirements
* format cdk
* enable Sentry for google-search-console
* updated connector version
* update spec and source yamls
Co-authored-by: auganbay <auganenu@gmail.com>
Co-authored-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com>
* Log http status code and content in default backoff handler
* Log http status code and content in usef defined backoff handler
* updated cdk version and changelog
* make it clear: exc.response
Co-authored-by: auganbay <auganenu@gmail.com>
