* [ISSUE #20771] limiting the number of requests performed to the backend without flag
* [ISSUE #20771] code reviewing my own code
* [ISSUE #20771] adding ABC to paginator
* [ISSUE #20771] format code
* [ISSUE #20771] adding slices to connector builder read request (#21605)
* [ISSUE #20771] adding slices to connector builder read request
* [ISSUE #20771] formatting
* [ISSUE #20771] set flag when limit requests reached (#21619)
* [ISSUE #20771] set flag when limit requests reached
* [ISSUE #20771] assert proper value on test read objects __init__
* [ISSUE #20771] code review and fix edge case
* [ISSUE #20771] fix flake8 error
* [ISSUE #20771] code review
* 🤖 Bump minor version of Airbyte CDK
* to run the CI
* Restore AvailabilityStrategy
* Add test for http availability strategy on empty stream from original bug fix 7c17351631
* fix flake errors
* Change CheckStream to use availability strategy
* Refactor test from bug fix
* fix flake errors
* Remove extra helper file from cherry-pick
* Merge tests for default http availability strategy
* turn off HttpAvailabilityStrategy as default (for now) (#21488)
* turn off HttpAvailabilityStrategy as default (for now)
* Update imports accordingly
* Add test for failure case
* Except StopIteration - make test pass
* Don't attempt to connect to a stream if we get no stream slices
* Make helper method for getting first record for a slice
* Add comments and exit early if stream to check isn't in list of source streams
* move helpers to helper module
* Clarify what it means when StopIteration is returned by helper methods
* initial work to parse manifest objects into pydantic models
* pr feedback and some other validations
* rerun manifest schema generation
* remove field constraint due to bug
* initial work to construct most components from greenhouse
* custom components parse subcomponent fields correctly and adding a few more component constructors
* construct components from gnews
* first pass at posthog.yaml
* Handle nested custom components with list values.
Also includes updates to posthog.yaml, including autoformatting changes.
* adding constructors for slicers, filters, and transformations and a few bug fixes
* make sed work across multiple OS
* add NoAuth component
* fix handling of custom components with nested list
* Autogenerate `TYPE_NAME_TO_MODEL` mapping
* Handle default kwargs not defined on model for custom components
* Re-add `options` for CartesianProductStreamSlicer for backwards compat
with custom stream slicers
* add basic unit tests for the model component factory
* add back defaults and extra parameters like options to retain compatibility with legacy flow and backwards compatibility
* Remove `_get_defaults`; using actual default values on classes instead
* Add backoff strategy component creation functions
* add back defaults and extra parameters like options to retain compatibility with legacy flow and backwards compatibility
* add lots of tests to construct components from the pydantic models and a few bug fixes
* add a few tests for the model to component factory
* add catch
* fix a bug where propagated schema doesn't work with old factory
* clean up a few files
* add type inference for custom components, more tests and some refactoring of the model factory
* template, docs, manifest updates, pr feedback and some cleanup
* pr feedback and polish schema a bit
* fix tests from the latest rebase of master
* fix the last few bugs I found and adjust a few sources that weren't perfectly compatible with the new component flow
* fix CheckStream bug cleanup and a few small tweaks and polish
* add additional test to cover bug case
* fix formatting
* 🤖 Bump minor version of Airbyte CDK
Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
* Make documentation_url optional in a declarative connector spec
* simplify if statement
* bump patch version of cdk
* Revert "bump patch version of cdk"
This reverts commit 1854bf3be1.
* 🤖 Bump patch version of Airbyte CDK
* Revert "🤖 Bump patch version of Airbyte CDK"
This reverts commit 85d5a989e2.
* 🤖 Bump patch version of Airbyte CDK
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
* initial work to parse manifest objects into pydantic models
* pr feedback and some other validations
* rerun manifest schema generation
* remove field constraint due to bug
* add a few tests for the model to component factory
* add catch
* fix a bug where propagated schema doesn't work with old factory
* add additionaProperties: false to the handwritten schema
* regenerate pydantic models
* fix test
* [ISSUE #19981] testing version bump
* [ISSUE #19981] debugging part-to-bump
* [ISSUE #19981] further debugging on part-to-bump
* [ISSUE #19981] further debugging on part-to-bump yet again
* [ISSUE #19981] further debugging on part-to-bump yet again and again
* [ISSUE #19981] improving on first successful attempt
* [ISSUE #19981] adding changelog and concurrency
* [ISSUE #19981] update version in .bumpversion.cfg
* [ISSUE #19981] testing Slack notification with working channel
* [ISSUE #19981] documentation and clean up before PR
* [ISSUE #19981] make changelog-message optional
* [ISSUE #19981] align version with new release
* [ISSUE #19981] code review
* convert request body
* fix tests
* test body data
* more tests
* more tests
* _
* return stacktrace
* pretty print
* Revert "pretty print"
This reverts commit 091253803c.
* Revert "Revert "pretty print""
This reverts commit b6f62d645b.
* replace \n
* missing type hint
* fix a bug where the OAuthAuthenticator was misnamed
* roll airbyte-cdk version
* add some missing component types to the existing factory type registry
* fix test
* handwritten low code manifest example components
* add MinMaxDatetime to jsonschema
* add a basic gradle command to generate manifest components
* Add auth components to handwritten component schema
- ApiKeyAuthenticator
- BasicHttpAuthenticator
- BearerAuthenticator
- DeclarativeOauth2Authenticator
- NoAuth
* Respect optional properties in DeclarativeOauth2Authenticator
* Fix `Dict[str, Any]` mapping in auth components
* add default error handler composite error handler and http response filter components
* [low code component schema] adding backoff strategies to schema
* [low code component schema] fix float types
* [low code component schema] add RecordFilter
* Remove `config` from auth components
* [low code component schema] add Interpolation (with pending question on 'type' not being defined)
* Add CartesianProductStreamSlicer & DatetimeStreamSlicer
* Add ListStreamSlicer, and fix nesting of DatetimeStreamSlicer
* [low code component schema] add InterpolatedRequestOptionsProvider
* Add slicer components, and fix a couple of components after reviewing output
* [low code component schema] adding transformations and adding type to interpolators
* adding spec and a few small tweaks
* Add DefaultSchemaLoader
* [low code component schema] attempt on custom class
* Add descriptions for auth components
* add RequestOption
* remove interpolated objects from the schema in favor of strings only
* a few schema fixes and adding some custom pagination and stream slicer
* [low code component schema] fix CustomBackoffStrategy
* Add CustomRecordExtractor
* add some description and add additional properties
* insert a transformer to hydrate default manifest components and perform validation against the handwritten schema
* [low code component schema] validating existing schemas
* [low code component schema] clean validation script
* add manifest transformer tests and a few tweaks to the schema
* Revert "[low code component schema] clean validation script"
This reverts commit 2408f41cf6.
* Revert "[low code component schema] validating existing schemas"
This reverts commit 9d39977815.
* [low code component schema] integrate validation script to gradle
* [low code component schema] updating validation script permissions
* remove a few model gen spike files and clean up comments
* default types should take parent type into account and a few schema changes
* add inline schema components and fix validation bugs
* add types to every component's literal enum and more little schema fixes
* missing component prefix
* pr feedback, renaming the schema and updating various docs
* fixing schema bugs from testing and new components
* last few small tweaks and fixes and fix the script
* bump the version before publsih
* include the declarative_component_schema.yaml as package data so its included when package is published
* roll the version to publish
Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Enable low-code CDK users to specify schema in the manifest
Also update documentation:
* Add inline schema loader info to yaml-overview.md
* Include inline schema info in tutorial
* Rough first implememtation of AvailabilityStrategy s
* Basic unit tests for AvailabilityStrategy and ScopedAvailabilityStrategy
* Make availability_strategy a property, separate out tests
* Remove from DeclarativeSource, remove Source parameter from methods, make default no AvailabilityStrategy
* Add skip stream if not available to read()
* Changes to CDK to get source-github working using AvailabilityStrategy, flakecheck
* reorganize cdk class, add HTTPAvailabilityStrategy test
* cleanup, docstrings
* pull out error handling into separate method
* Pass source and logger to check_connection method
* Add documentation links, handle 403 specifically
* Fix circular import
* Add AvailabilityStrategy to Stream and HTTPStream classes
* Remove AS from abstract_source, add to Stream, HTTPStream, AvailabilityStrategy unit tests passing for per-stream strategies
* Modify MockHttpStream to set no AvailabilityStrategy since source test mocking doesn't support this
* Move AvailabilityStrategy class to sources.streams
* Move HTTPAvailabilityStrategy to http module
* Use pascal case for HttpAvailabilityStrategy
* Remove docs message method :( and default to True availability on unhandled HTTPErrors
* add check_availability method to stream class
* Add optional source parameter
* Add test for connector-specific documentation, small tests refactor
* Add test that performs the read() function for stream with default availability strategy
* Add test for read function behavior when stream is unavailable
* Add 403 info in logger message
* Don't return error for other HTTPErrors
* Split up error handling into methods 'unavailable_error_codes' and 'get_reason_for_error'
* rework overrideable list of status codes to be a dict with reasons, to enforce that users provide reasons for all listed errors
* Fix incorrect typing
* Move HttpAvailability to its own module, fix flake errors
* Fix ScopedAvailabilityStrategy, docstrings and types for streams/availability_strategy.py
* Docstrings and types for core.py and http/availability_strategy.py
* Move _get_stream_slices to a StreamHelper class
* Docstrings + types for stream_helpers.py, cleanup test_availability.py
* Clean up test_source.py
* Move logic of getting the initial record from a stream to StreamHelper class
* Add changelog and bump minor version
* change 'is True' and 'is False' behavior
* use mocker.MagicMock
* Remove ScopedAvailabilityStrategy
* Don't except non-403 errors, check_stream uses availability_strategy if possible
* CDK: pass error to reasons_for_error_codes
* make get_stream_slice public
* Add tests for raising unhandled errors and retries are handled
* Add tests for CheckStream via AvailabilityStrategy
* Add documentation for stream availability of http streams
* Move availability unit tests to correct modules, report error message if possible
* Add test for reporting specific error if available
* [ISSUE #15628] apply lookback window on earliest datetime between start and cursor
* [ISSUE #15628] update release information and clean return statement
