* [ISSUE #26581] per partition cursor
* [ISSUE #26581] format
* [ISSUE #26581] clean up state management
* [ISSUE #26581] improving Hashabledict
* [ISSUE #26581] format cdk
* [ISSUE #26581] fix tests
* [ISSUE #26581] code review from girarda
* Retrigger pipeline
* Decouple cursor and stream slicer and pushing state management as far up cursor as possible
* Format cdk
* Small fixes/comments
* DatetimeBasedCursor should not update state based on slice (for now at least since it wasn't doing this before)
* [ISSUE #26581] code review
* Automated Commit - Formatting Changes
* [ISSUE #26581] validation overlapping keys
* [ISSUE #26581] add typing
* [ISSUE #26581] code review
* Remove SyncMode from stream_slices
* Removing SyncMode from stream_slices up until SimpleRetriever and fixing typing
* format cdk
* [ISSUE #19410] remove request_options_provider from the … (#21403)
* [ISSUE #19410] (incomplete) remove request_options_provider from the manifest
* [ISSUE #19410] (incomplete) incomplete cleanup config_component_schema.json as well
* [ISSUE #19410] update source-monday
* [ISSUE #19410] code review
* [ISSUE #19410] formatting files
* [Low-Code CDK] Replace the $options keyword with $parameters (#21632)
* refactor flows and tests to use parameters instead of options
* update documentation to reflect the change from options to parameters
* create migration script to replace options with parameters in existing manifests
* update template to use parameters instead of options
* fix tests after rebasing from the branch
* address pr feedback and extra uses of options that I missed
* additional changes needed after rebasing from master
* migrate low-code connectors to use parameters instead of options
* 🚨🚨 [Low Code CDK] Update `*ref` format to `#/` (#21434)
* [Low-Code CDK] Remove JsonSchema type in favor of JsonSchemaFileLoader (#21832)
* fully deprecate JsonSchema in favor of JsonFileSchemaLoader
* remove usage in the legacy registry
* Update migration scripts according to manifest file rename (#21920)
* Issue 21866 remove legacy factory and validation flow (#21878)
* [ISSUE #21866] clean ManifestDeclarativeSource validation
* [ISSUE #21866] remove dataclasses-jsonschema
* [ISSUE #21866] code review
* [ISSUE-21866] flake8
* [ISSUE #21559] remove DefaultPaginator.url_base (#21823)
* [ISSUE #21559] remove DefaultPaginator.url_base
* [ISSUE #21559] code review
* [ISSUE #21559] update migration script
* [ISSUE #21559] code review
* [ISSUE #21559] update documentation
* [ISSUE #21559] run migration (#21824)
* [ISSUE #21559] remove DefaultPaginator.url_base (#21823)
* [ISSUE #21559] remove DefaultPaginator.url_base
* [ISSUE #21559] code review
* [ISSUE #21559] update migration script
* [ISSUE #21559] code review
* [ISSUE #21559] update documentation
* [ISSUE #21559] run migration (#21824)
* [ISSUE #21559] fix manifests
* [ISSUE #21926] setup server to allow for local tests (#21974)
* [Low Code CDK] remove checkpoint_interval from DeclarativeStream component (#22120)
* Issue #21576 rename dpathextractor fieldpointer (#21990)
* [ISSUE #21926] setup server to allow for local tests
* [ISSUE #21576] Rename DpathExtractor.field_pointer to field_path
* [ISSUE #21576] migration script
* [ISSUE #21576] update source-monday and source-pocket as well
* [ISSUE #21576] migration (#21997)
* [ISSUE #21576] code review
* Remove checkpoint_interval from source-prestashop manifest (#22141)
* replacing options with parameters for a few connectors I missed or were newly added
* [Low-Code CDK] Rremove stream_cursor_field from stream and derive it from stream_slicer (#22294)
* update schema to derive cursor_field from a stream slicer if it exists
* remove usage of stream_cursor_field on simple connector use cases
* fixing some of the more complex usage of stream_cursor_field that rely on cartesian product stream slicers
* fix documentation to replace references to stream_cursor_field
* Low Code CDK: Remove `name` and `primary_key` from non-DeclarativeStream components (#21891)
* fix eslint issues for webapp (#22462)
* 🪟🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22375)
* bump connector builder server to latest CDK version
* fix breaking CDK changes in connector builder FE
* [Low-Code CDK] Separate request path from RequestOption component (#22398)
* split apart path from RequestOption and fix usages and cleanup the code
* replace usage of path with RequestPath and get rid of default to RequestOption
* fix bug where stream_slice_field was used in outbound request instead of request_option field_name
* organize yaml schema names and update documentation for RequestOption and RequestPath
* clean up tests
* regenerate models
* [ISSUE #19961] refactor stream slices (#22225)
* [ISSUE #19961] add 'incremental' and partially remove CartesianProductStreamSlicer - Google PageSpeed Insights not working yet
* [ISSUE #19961] fixing Google PageSpeed Insights
* move incremental_sync field to the stream level and perform merging into one stream slicer at that level
* add tests to merging incremental and iterable into cartesian
* rewrite documentation to separate incremental sync and iterator concepts
* update documentation to use partition router and revise the tutorial to reflect the new changes to the components
* [ISSUE #19961] update code to newest CDK version and clean autogenerated files (#22670)
* [ISSUE #19961] rename stream_slicer to partition_router and update ma… (#22590)
* [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well)
* [ISSUE 19961] rename CustomStreamSlicer (#22598)
* [ISSUE 19961] rename CustomStreamSlicer
* [ISSUE #19961] code review CustomStreamSlicer
* [ISSUE #19961] fix source_square incremental sync
* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591)
* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter
* remove SinglePartitionRouter from the schema
---------
Co-authored-by: brianjlai <brian.lai@airbyte.io>
* [ISSUE #19961] rename SubstreamSlicer to SubstreamPartitionRouter (#22596)
* [ISSUE #19961] TMP rename SubstreamSlicer to SubstreamPartitionRouter
* [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end
* [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593)
---------
Co-authored-by: brianjlai <brian.lai@airbyte.io>
* [ISSUE #19961] clean faulty merge
* [ISSUE #19961] rename DatetimeStreamSlicer (#22617)
* [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well)
* [ISSUE 19961] rename CustomStreamSlicer (#22598)
* [ISSUE 19961] rename CustomStreamSlicer
* [ISSUE #19961] code review CustomStreamSlicer
* [ISSUE #19961] fix source_square incremental sync
* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591)
* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter
* remove SinglePartitionRouter from the schema
---------
Co-authored-by: brianjlai <brian.lai@airbyte.io>
* [ISSUE #19961] rename DatetimeStreamSlicer
* [ISSUE #19961] rename SubstreamSlicer to SubstreamPartitionRouter (#22596)
* [ISSUE #19961] TMP rename SubstreamSlicer to SubstreamPartitionRouter
* [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end
* [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593)
---------
Co-authored-by: brianjlai <brian.lai@airbyte.io>
* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* Update docs/connector-development/config-based/understanding-the-yaml-file/yaml-overview.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* Update docs/connector-development/config-based/understanding-the-yaml-file/incremental-syncs.md
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
* update docs
* [ISSUE #19961] clean unit tests files
* [ISSUE #19961] code review
---------
Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
* [Low-Code CDK] Allow for children of custom components to specify parameters that are normally derived (#22379)
* Fix a bug where child components of a custom component cannot receive fields from other components
* add tests, documentation and commenting
* fix test from merge
* add better error message for nested initialization failures
* 🪟🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22880)
* restrict name to stream level
* remove checkpoint interval
* adjust logic for new request options
* refactor slicers
* wording
* review comments
* make oldest supported version explicit
* separate the frontend and connector builder changes from the low-code to beta release
* [Low-Code CDK] Add script to run low code unit tests and address issues with a few connectors (#23123)
* consolidate all the changes into a new PR after I messed up the merge on the side branch
* add set to allow this to be called externally if necessary later
* remove last few extra fields i found and fix docs links
* fix docs one more time
---------
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: Joe Reuter <joe@airbyte.io>
* initial work to parse manifest objects into pydantic models
* pr feedback and some other validations
* rerun manifest schema generation
* remove field constraint due to bug
* initial work to construct most components from greenhouse
* custom components parse subcomponent fields correctly and adding a few more component constructors
* construct components from gnews
* first pass at posthog.yaml
* Handle nested custom components with list values.
Also includes updates to posthog.yaml, including autoformatting changes.
* adding constructors for slicers, filters, and transformations and a few bug fixes
* make sed work across multiple OS
* add NoAuth component
* fix handling of custom components with nested list
* Autogenerate `TYPE_NAME_TO_MODEL` mapping
* Handle default kwargs not defined on model for custom components
* Re-add `options` for CartesianProductStreamSlicer for backwards compat
with custom stream slicers
* add basic unit tests for the model component factory
* add back defaults and extra parameters like options to retain compatibility with legacy flow and backwards compatibility
* Remove `_get_defaults`; using actual default values on classes instead
* Add backoff strategy component creation functions
* add back defaults and extra parameters like options to retain compatibility with legacy flow and backwards compatibility
* add lots of tests to construct components from the pydantic models and a few bug fixes
* add a few tests for the model to component factory
* add catch
* fix a bug where propagated schema doesn't work with old factory
* clean up a few files
* add type inference for custom components, more tests and some refactoring of the model factory
* template, docs, manifest updates, pr feedback and some cleanup
* pr feedback and polish schema a bit
* fix tests from the latest rebase of master
* fix the last few bugs I found and adjust a few sources that weren't perfectly compatible with the new component flow
* fix CheckStream bug cleanup and a few small tweaks and polish
* add additional test to cover bug case
* fix formatting
* 🤖 Bump minor version of Airbyte CDK
Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
* [ISSUE #15628] apply lookback window on earliest datetime between start and cursor
* [ISSUE #15628] update release information and clean return statement
* method yielding airbytemessage
* move to Stream
* update abstract source
* reset
* missing file
* Yield request and response as log messages
* only emit request and responses if the debug flag is on
* add test docker image
* script to run acceptance tests with local cdk
* Update conftest to use a different image
* extract to method
* dont use a different image tag
* Always install local cdk
* break the cdk
* get path from current working directory
* or
* ignore unit test
* debug log
* Revert "AMI change: ami-0f23be2f917510c26 -> ami-005924fb76f7477ce (#18689)"
This reverts commit bf06decf73.
* build from the top
* Update source-acceptance-test
* fix
* copy setup
* some work on the gradle plugin
* reset to master
* delete unused file
* delete unused file
* reset to master
* optional argument
* delete dead code
* use latest cdk with sendgrid
* fix sendgrid dockerfile
* break the cdk
* use local file
* Revert "break the cdk"
This reverts commit 600c195541.
* dont raise an exception
* reset to master
* unit tests
* missing test
* more unit tests
* remove deprecated comment
* newline
* reset to master
* remove files
* reset
* Update abstract source
* remove method from stream
* convert to airbytemessage
* unittests
* Update
* unit test
* remove debug logs
* Revert "remove debug logs"
This reverts commit a1a139ef37.
* Revert "Revert "remove debug logs""
This reverts commit b1d62cdb60.
* Revert "reset to master"
This reverts commit 3fa6a004c1.
* fix
* slightly better test
* typing
* extract method
* Revert "Revert "reset to master""
This reverts commit 5dac7c2804.
* reset to master
* reset to master
* Revert "reset to master"
This reverts commit 3fa6a004c1.
* Comment
* operate on the message
* Revert "Revert "reset to master""
This reverts commit 5833c84d0a.
* comment
* test
* Revert "test"
This reverts commit 2f91b803b0.
* test
* Revert "test"
This reverts commit 62d95ebbb5.
* test
* Revert "test"
This reverts commit 27150ba341.
* format
* format
* symlink
* Update setup
* update path
* reset to master
* update
* Add local files
* greenhouse
* format
* symlink
* try reordering
* better error message
* better log message
* reset to master
* Revert "merge for qa"
This reverts commit ad7128f2c5, reversing
changes made to 7196c22a73.
* reset to master
* reset to master
* reset to master
* format
* gradlew format
* right type hints
* reset to master
* reset to master
* gradlew format
* a bunch of small fixes
* Update output format
* fixes from feedback
* fixme comment
* streams cannot return AirbyteRecordMessage
* fix
* format
* only return logs when running on debug mode
* move branching
* update typing
* remove dead code
* fix simpleretriever name
* i think this is better
* log response.text
* debug flag
* comment
* pass config
* comments
* run SATs
* fix most of the unit tests
* fix unit test
* reset to master
* runFromPath
* Revert "runFromPath"
This reverts commit 85979a801a.
* Revert "run SATs"
This reverts commit a8a8a2da95.
* no need to convert to dict
* fix test
* allow for spec to be defined in the source.yaml manifest instead of an external file
* make spec a component within the language to get schema validation and rework the code for better testing
* fix formatting and extra method
* pr feedback and add some more test
* pr feedback
* bump airbyte-cdk version
* bump version
* gradle format
* remove from manifest spec
* draft: first pass at complete schema language generation and factory validator
* actually a working validator and fixes to the schema that went uncaught
* remove extra spike file
* fix formatting file
* pr feedback and a little bit of refactoring
* fix some types that were erroneously marked as invalid schema
* some comments
* add jsonschemamixin to interfaces
* update changelog
* bump version
* fix parse and format methods
* define constant
* remove timestamp magic keyword
* comment
* test for ci
* uncomment test
* use timestamp()
* Bump cdk version
* bump to 0.1.72
* initial first pass converting every component to dataclasses
* replace the hackier options pass through
* get rid of the hackier way
* fix issues w/ type hints by making options required and lots of fixes to the language to fix compatability for dataclasses
* add dataclasses-jsonschema to setup
* fix oauth authenticator to avoid dataclass name collisions
* fix spacing for CI tests
* remove property from oauth and fix a interpolation bug
* pr feedback and cleaning up the code a bit, attempt at avoiding renaming
* fix templates and bugs surfaced during greenhouse spec testing
* fix tests
* fix missing options in some declarative components
* fix tests related to pulling latest master
* fix issue w/ passing state, slice, and token to subcomponents
* switch name back to get_access_token() since no name collision anymore
* requester is a request options provider
* get request options from slicer
* remove prints
* share interface
* actual fix with test
* small fix
* missing tests
* missing *
* simplify intersection logic
* bump cdk version
* comment
* comment
* comments
* fix
* test for instantiating chain retrier
* fix parsing
* cleanup
* fix
* reset
* never raise on http error
* remove print
* comment
* comment
* comment
* comment
* remove prints
* add declarative stream to registry
* start working on limit paginator
* support for offset pagination
* tests
* move limit value
* extract request option
* boilerplate
* page increment
* delete offset paginator
* update conditional paginator
* refactor and fix test
* fix test
* small fix
* Delete dead code
* Add docstrings
* quick fix
* exponential backoff
* fix test
* fix
* delete unused properties
* fix
* missing unit tests
* uppercase
* docstrings
* rename to success
* compare full request instead of just url
* renmae module
* rename test file
* rename interface
* rename default retrier
* rename to compositeerrorhandler
* fix missing renames
* move action to filter
* str -> minmaxdatetime
* small fixes
* plural
* add example
* handle header variations
* also fix wait time from
* allow using a regex to extract the value
* group()
* docstring
* add docs
* update comment
* docstrings
* fix tests
* rename param
* cleanup stop_condition
* cleanup
* Add examples
* interpolated pagination strategy
* dont need duplicate class
* docstrings
* more docstrings
* docstrings
* fix tests
* first pass at substream
* seems to work for a single stream
* can also be defined in requester with stream_state
* tmp update
* update comment
* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/http_requester.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* version: Update Parquet library to latest release (#14502)
The upstream Parquet library that is currently pinned for use in the S3 destination plugin is over a year old. The current version is generating invalid schemas for date-time with time-zone fields which appears to be addressed in the `1.12.3` release of the library in commit c72862b613
* merge
* 🎉 Source Github: improve schema for stream `pull_request_commits` added "null" (#14613)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
* Docs: Fixed broken links (#14622)
* fixing broken links
* more broken links
* source-hubspot: change mentioning of Mailchimp into HubSpot doc (#14620)
* Helm Chart: Add external temporal option (#14597)
* conflict env configmap and chart lock
* reverting lock
* add eof lines and documentation on values yaml
* conflict json file
* rollback json
* solve conflict
* correct minio with new version
Co-authored-by: Guy Feldman <gfeldman@86labs.com>
* 🎉 Add YAML format to source-file reader (#14588)
* Add yaml reader
* Update docs
* Bumpversion of connector
* bump docs
* Update pyarrow dependency
* Upgrade pandas dependency
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* 🎉 Source Okta: add GroupMembers stream (#14380)
* add Group_Members stream to okta source
- Group_Members return a list of users, the same schema of Users stream.
- Create a shared schema users, and both group_members and users sechema use it as a reference.
- Add Group_Members stream to source connector
* add tests and fix logs schema
- fix the test error: None is not one of enums though the enum type includes both string and null, it comes from json schema validator
ddb87afad8/jsonschema/_validators.py (L279-L285)
- change grouop_members to use id as the cursor field since `filter` is not supported in the query string
- fix the abnormal state test on logs stream, when since is abnormally large, until has to defined, an equal or a larger value
- remove logs stream from full sync test, because 2 full sync always has a gap -- at least a new log about users or groups api.
* last polish before submit the PR
- bump docker version
- update changelog
- add the right abnormal value for logs stream
- correct the sample catalog
* address comments::
- improve comments for until parameter under the logs stream
- add use_cache on groupMembers
* add use_cache to Group_Members
* change configured_catalog to test
* auto-bump connector version
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* split test files
* renames
* missing unit test
* add missing unit tests
* rename
* assert isinstance
* start extracting to their own files
* use final instead of classmethod
* assert we retry 429 errors
* Add log
* replace asserts with valueexceptions
* delete superfluous print statement
* only accept minmaxdatetime
* fix factory so we don't need to union everything with strings
* get class_name from type
* remove from class types registry
* process error handlers one at a time
* sort
* delete print statement
* comment
* comment
* format
* delete unused file
* comment
* interpolatedboolean
* comment
* not optional
* not optional
* unit tests
* fix request body data
* add test
* move file to right module
* update
* reset to master
* format
* rename to pass_by
* rename to page size
* fix
* fix some tests
* reset
* fix
* fix some of the tests
* fix test
* fix more tests
* all tests pass
* path is not optional
* reset
* reset
* reset
* delete print
* remove prints
* delete duplicate method
* add test
* fix body data
* delete extra newlines
* move to subpackage
* fix imports
* handle str body data
* simplify
* Update tests
* filter dates before stream state
* Revert "Update tests"
This reverts commit c0808c8009.
* update
* fix test
* state management
* add test
* delete dead code
* update cursor
* update cursor cartesian
* delete unused state class
* fix
* missing test
* update cursor substreams
* missing test
* fix typing
* fix typing
* delete unused field
* delete unused method
* update datetime stream slice
* cleanup
* assert
* request options
* request option cartesian
* assert when passing by path
* request options for substreams
* always return a map
* pass stream_state
* refactor and almost done fixing tests
* fix tests
* rename to inject_into
* only accept enum
* delete conditional paginator
* only return body data
* missing test
* update docstrings
* update docstrings
* update comment
* rename
* tests
* class_name -> type
* improve interface
* fix some of the tests
* fix more of the tests
* fix tests
* reset
* reset
* Revert "reset"
This reverts commit eb9a918a09.
* remove extra argument
* docstring
* update
* delete unused file
* reset
* reset
* rename
* fix timewindow
* create InterpolatedString
* helper method
* assert on request option
* better asserts
* format
* docstrings
* docstrings
* remove optional from type hint
* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/stream_slicers/cartesian_product_stream_slicer.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* inherit from request options provider
* inherit from request options provider
* remove optional from type hint
* remove extra parameter
* none check
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Tobias Macey <tmacey@boundlessnotions.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com>
Co-authored-by: Bas Beelen <bjgbeelen@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Guy Feldman <gfeldman@86labs.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Yiyang Li <yiyangli2010@gmail.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
* checkout files from test branch
* read_incremental works
* reset to master
* remove dead code
* comment
* fix
* Add test
* comments
* utc
* format
* small fix
* Add test with rfc3339
* remove unused param
* fix test
* configurable state checkpointing
* update test
* start working on retrier
* retry predicate
* return response status
* look in error message
* cleanup test
* constant backoff strategy
* chain backoff strategy
* chain retrier
* Add to class types registry
* extract backoff time from header
* wait until
* update
* split file
* parse_records
* classmethod
* delete dead code
* comment
* comment
* comments
* fix
* test for instantiating chain retrier
* fix parsing
* cleanup
* fix
* reset
* never raise on http error
* remove print
* comment
* comment
* comment
* comment
* remove prints
* add declarative stream to registry
* Delete dead code
* Add docstrings
* quick fix
* exponential backoff
* fix test
* fix
* delete unused properties
* fix
* missing unit tests
* uppercase
* docstrings
* rename to success
* compare full request instead of just url
* renmae module
* rename test file
* rename interface
* rename default retrier
* rename to compositeerrorhandler
* fix missing renames
* move action to filter
* str -> minmaxdatetime
* small fixes
* plural
* add example
* handle header variations
* also fix wait time from
* allow using a regex to extract the value
* group()
* docstring
* add docs
* update comment
* docstrings
* update comment
* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/http_requester.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* version: Update Parquet library to latest release (#14502)
The upstream Parquet library that is currently pinned for use in the S3 destination plugin is over a year old. The current version is generating invalid schemas for date-time with time-zone fields which appears to be addressed in the `1.12.3` release of the library in commit c72862b613
* merge
* 🎉 Source Github: improve schema for stream `pull_request_commits` added "null" (#14613)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
* Docs: Fixed broken links (#14622)
* fixing broken links
* more broken links
* source-hubspot: change mentioning of Mailchimp into HubSpot doc (#14620)
* Helm Chart: Add external temporal option (#14597)
* conflict env configmap and chart lock
* reverting lock
* add eof lines and documentation on values yaml
* conflict json file
* rollback json
* solve conflict
* correct minio with new version
Co-authored-by: Guy Feldman <gfeldman@86labs.com>
* 🎉 Add YAML format to source-file reader (#14588)
* Add yaml reader
* Update docs
* Bumpversion of connector
* bump docs
* Update pyarrow dependency
* Upgrade pandas dependency
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* 🎉 Source Okta: add GroupMembers stream (#14380)
* add Group_Members stream to okta source
- Group_Members return a list of users, the same schema of Users stream.
- Create a shared schema users, and both group_members and users sechema use it as a reference.
- Add Group_Members stream to source connector
* add tests and fix logs schema
- fix the test error: None is not one of enums though the enum type includes both string and null, it comes from json schema validator
ddb87afad8/jsonschema/_validators.py (L279-L285)
- change grouop_members to use id as the cursor field since `filter` is not supported in the query string
- fix the abnormal state test on logs stream, when since is abnormally large, until has to defined, an equal or a larger value
- remove logs stream from full sync test, because 2 full sync always has a gap -- at least a new log about users or groups api.
* last polish before submit the PR
- bump docker version
- update changelog
- add the right abnormal value for logs stream
- correct the sample catalog
* address comments::
- improve comments for until parameter under the logs stream
- add use_cache on groupMembers
* add use_cache to Group_Members
* change configured_catalog to test
* auto-bump connector version
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* split test files
* renames
* missing unit test
* add missing unit tests
* rename
* assert isinstance
* start extracting to their own files
* use final instead of classmethod
* assert we retry 429 errors
* Add log
* replace asserts with valueexceptions
* delete superfluous print statement
* fix factory so we don't need to union everything with strings
* get class_name from type
* remove from class types registry
* process error handlers one at a time
* sort
* delete print statement
* comment
* comment
* format
* delete unused file
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Tobias Macey <tmacey@boundlessnotions.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com>
Co-authored-by: Bas Beelen <bjgbeelen@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Guy Feldman <gfeldman@86labs.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Yiyang Li <yiyangli2010@gmail.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
* generate pdoc
* set python version to 3.8
* mimic doc
* set the os
* Revert "generate pdoc"
This reverts commit 82bdeabe8f.
* add missing docs
* add declarative to index
* missing init files
* generate rst files
* implement MinMaxDate and incorporate it into the DatetimeStreamSlicer
* rename to datetime and restrict start/end to minmaxdatetime object
* get rid of eval_date which isn't needed anymore and allow for datetimestreamslicer to set datetime format
* add MinMaxDatetime to class types registry
* checkout files from test branch
* read_incremental works
* reset to master
* remove dead code
* comment
* fix
* Add test
* comments
* utc
* format
* small fix
* Add test with rfc3339
* remove unused param
* fix test
* list slicer
* Add comment
* test
* ast eval
* refactor
* fix
* remove print
* rename
* return a generator
* smaller fix and reset to master
* union type hint
* assert
* comment
* format
