9f6963ccfcddd7a573c4ca0ff94de41ce6175cc0
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Brian Lai
|
1c482ee339 |
Low code cdk to beta (#22853)
* [ISSUE #19410] remove request_options_provider from the … (#21403) * [ISSUE #19410] (incomplete) remove request_options_provider from the manifest * [ISSUE #19410] (incomplete) incomplete cleanup config_component_schema.json as well * [ISSUE #19410] update source-monday * [ISSUE #19410] code review * [ISSUE #19410] formatting files * [Low-Code CDK] Replace the $options keyword with $parameters (#21632) * refactor flows and tests to use parameters instead of options * update documentation to reflect the change from options to parameters * create migration script to replace options with parameters in existing manifests * update template to use parameters instead of options * fix tests after rebasing from the branch * address pr feedback and extra uses of options that I missed * additional changes needed after rebasing from master * migrate low-code connectors to use parameters instead of options * 🚨🚨 [Low Code CDK] Update `*ref` format to `#/` (#21434) * [Low-Code CDK] Remove JsonSchema type in favor of JsonSchemaFileLoader (#21832) * fully deprecate JsonSchema in favor of JsonFileSchemaLoader * remove usage in the legacy registry * Update migration scripts according to manifest file rename (#21920) * Issue 21866 remove legacy factory and validation flow (#21878) * [ISSUE #21866] clean ManifestDeclarativeSource validation * [ISSUE #21866] remove dataclasses-jsonschema * [ISSUE #21866] code review * [ISSUE-21866] flake8 * [ISSUE #21559] remove DefaultPaginator.url_base (#21823) * [ISSUE #21559] remove DefaultPaginator.url_base * [ISSUE #21559] code review * [ISSUE #21559] update migration script * [ISSUE #21559] code review * [ISSUE #21559] update documentation * [ISSUE #21559] run migration (#21824) * [ISSUE #21559] remove DefaultPaginator.url_base (#21823) * [ISSUE #21559] remove DefaultPaginator.url_base * [ISSUE #21559] code review * [ISSUE #21559] update migration script * [ISSUE #21559] code review * [ISSUE #21559] update documentation * [ISSUE #21559] run migration (#21824) * [ISSUE #21559] fix manifests * [ISSUE #21926] setup server to allow for local tests (#21974) * [Low Code CDK] remove checkpoint_interval from DeclarativeStream component (#22120) * Issue #21576 rename dpathextractor fieldpointer (#21990) * [ISSUE #21926] setup server to allow for local tests * [ISSUE #21576] Rename DpathExtractor.field_pointer to field_path * [ISSUE #21576] migration script * [ISSUE #21576] update source-monday and source-pocket as well * [ISSUE #21576] migration (#21997) * [ISSUE #21576] code review * Remove checkpoint_interval from source-prestashop manifest (#22141) * replacing options with parameters for a few connectors I missed or were newly added * [Low-Code CDK] Rremove stream_cursor_field from stream and derive it from stream_slicer (#22294) * update schema to derive cursor_field from a stream slicer if it exists * remove usage of stream_cursor_field on simple connector use cases * fixing some of the more complex usage of stream_cursor_field that rely on cartesian product stream slicers * fix documentation to replace references to stream_cursor_field * Low Code CDK: Remove `name` and `primary_key` from non-DeclarativeStream components (#21891) * fix eslint issues for webapp (#22462) * 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22375) * bump connector builder server to latest CDK version * fix breaking CDK changes in connector builder FE * [Low-Code CDK] Separate request path from RequestOption component (#22398) * split apart path from RequestOption and fix usages and cleanup the code * replace usage of path with RequestPath and get rid of default to RequestOption * fix bug where stream_slice_field was used in outbound request instead of request_option field_name * organize yaml schema names and update documentation for RequestOption and RequestPath * clean up tests * regenerate models * [ISSUE #19961] refactor stream slices (#22225) * [ISSUE #19961] add 'incremental' and partially remove CartesianProductStreamSlicer - Google PageSpeed Insights not working yet * [ISSUE #19961] fixing Google PageSpeed Insights * move incremental_sync field to the stream level and perform merging into one stream slicer at that level * add tests to merging incremental and iterable into cartesian * rewrite documentation to separate incremental sync and iterator concepts * update documentation to use partition router and revise the tutorial to reflect the new changes to the components * [ISSUE #19961] update code to newest CDK version and clean autogenerated files (#22670) * [ISSUE #19961] rename stream_slicer to partition_router and update ma… (#22590) * [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well) * [ISSUE 19961] rename CustomStreamSlicer (#22598) * [ISSUE 19961] rename CustomStreamSlicer * [ISSUE #19961] code review CustomStreamSlicer * [ISSUE #19961] fix source_square incremental sync * [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591) * [ISSUE #19961] rename SingleSlice to SinglePartitionRouter * remove SinglePartitionRouter from the schema --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> * [ISSUE #19961] rename SubstreamSlicer to SubstreamPartitionRouter (#22596) * [ISSUE #19961] TMP rename SubstreamSlicer to SubstreamPartitionRouter * [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end * [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593) --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> * [ISSUE #19961] clean faulty merge * [ISSUE #19961] rename DatetimeStreamSlicer (#22617) * [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well) * [ISSUE 19961] rename CustomStreamSlicer (#22598) * [ISSUE 19961] rename CustomStreamSlicer * [ISSUE #19961] code review CustomStreamSlicer * [ISSUE #19961] fix source_square incremental sync * [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591) * [ISSUE #19961] rename SingleSlice to SinglePartitionRouter * remove SinglePartitionRouter from the schema --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> * [ISSUE #19961] rename DatetimeStreamSlicer * [ISSUE #19961] rename SubstreamSlicer to SubstreamPartitionRouter (#22596) * [ISSUE #19961] TMP rename SubstreamSlicer to SubstreamPartitionRouter * [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end * [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593) --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/yaml-overview.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/incremental-syncs.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * update docs * [ISSUE #19961] clean unit tests files * [ISSUE #19961] code review --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * [Low-Code CDK] Allow for children of custom components to specify parameters that are normally derived (#22379) * Fix a bug where child components of a custom component cannot receive fields from other components * add tests, documentation and commenting * fix test from merge * add better error message for nested initialization failures * 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22880) * restrict name to stream level * remove checkpoint interval * adjust logic for new request options * refactor slicers * wording * review comments * make oldest supported version explicit * separate the frontend and connector builder changes from the low-code to beta release * [Low-Code CDK] Add script to run low code unit tests and address issues with a few connectors (#23123) * consolidate all the changes into a new PR after I messed up the merge on the side branch * add set to allow this to be called externally if necessary later * remove last few extra fields i found and fix docs links * fix docs one more time --------- Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com> Co-authored-by: maxi297 <maxime@airbyte.io> Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: Joe Reuter <joe@airbyte.io> |
||
|
Cole Snodgrass
|
2e099acc52 |
update headers from 2022 -> 2023 (#22594)
* It's 2023! * 2022 -> 2023 --------- Co-authored-by: evantahler <evan@airbyte.io> |
||
|
Alexandre Girard
|
0ffd503b21 |
🐛 [low-code] $options shouldn't overwrite values that are already defined (#18060)
* fix * Add missing test * remove prints * extract to method * rename * Add missing test * rename * bump |
||
|
Brian Lai
|
bd31100774 |
initial first pass converting every component to dataclasses (#15189)
* initial first pass converting every component to dataclasses * replace the hackier options pass through * get rid of the hackier way * fix issues w/ type hints by making options required and lots of fixes to the language to fix compatability for dataclasses * add dataclasses-jsonschema to setup * fix oauth authenticator to avoid dataclass name collisions * fix spacing for CI tests * remove property from oauth and fix a interpolation bug * pr feedback and cleaning up the code a bit, attempt at avoiding renaming * fix templates and bugs surfaced during greenhouse spec testing * fix tests * fix missing options in some declarative components * fix tests related to pulling latest master * fix issue w/ passing state, slice, and token to subcomponents * switch name back to get_access_token() since no name collision anymore |
||
|
Alexandre Girard
|
a3ff80c179 |
[low-code-connectors] Disable parse-time interpolation in favor of runtime-only (#14923)
* abstract auth token
* basichttp
* remove prints
* docstrings
* get rid of parse-time interpolation
* always pass options through
* delete print
* delete misleading comment
* delete note
* reset
* pass down options
* delete duplicate file
* missing test
* refactor test
* rename to '$options'
* rename to ''
* interpolatedauth
* fix tests
* fix
* docstrings
* update docstring
* docstring
* update docstring
* remove extra field
* undo
* rename to runtime_parameters
* docstring
* update
* / -> *
* update template
* rename to options
* Add examples
* update docstring
* Update test
* newlines
* rename kwargs to options
* options init param
* delete duplicate line
* type hints
* update docstring
* Revert "delete duplicate line"
This reverts commit
|
||
|
Alexandre Girard
|
08239abafd |
Alex/lowcode referencedocs (#14973)
* Add docstrings for auth package
* docstrings for the check package
* docstrings for the datetime package
* docstrings for the decoder package
* docstrings for extractors package and fix tests
* interpolation docstrings
* ref -> and parser docstrings
* docstrings for parsers package
* error handler docstrings
* requester docstrings
* more docstrings
* docstrings
* docstrings
* docstrings
* Use defined type annotations
* update
* update docstrings
* Update docstrings
* update docstrings
* update docstrings
* update template
* Revert "update template"
This reverts commit
|
||
|
Alexandre Girard
|
09aa685aad |
Alex/configurable retrier (#14330)
* checkout files from test branch * read_incremental works * reset to master * remove dead code * comment * fix * Add test * comments * utc * format * small fix * Add test with rfc3339 * remove unused param * fix test * configurable state checkpointing * update test * start working on retrier * retry predicate * return response status * look in error message * cleanup test * constant backoff strategy * chain backoff strategy * chain retrier * Add to class types registry * extract backoff time from header * wait until * update * split file * parse_records * classmethod * delete dead code * comment * comment * comments * fix * test for instantiating chain retrier * fix parsing * cleanup * fix * reset * never raise on http error * remove print * comment * comment * comment * comment * remove prints * add declarative stream to registry * Delete dead code * Add docstrings * quick fix * exponential backoff * fix test * fix * delete unused properties * fix * missing unit tests * uppercase * docstrings * rename to success * compare full request instead of just url * renmae module * rename test file * rename interface * rename default retrier * rename to compositeerrorhandler * fix missing renames * move action to filter * str -> minmaxdatetime * small fixes * plural * add example * handle header variations * also fix wait time from * allow using a regex to extract the value * group() * docstring * add docs * update comment * docstrings * update comment * Update airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/http_requester.py Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * version: Update Parquet library to latest release (#14502) The upstream Parquet library that is currently pinned for use in the S3 destination plugin is over a year old. The current version is generating invalid schemas for date-time with time-zone fields which appears to be addressed in the `1.12.3` release of the library in commit |
||
|
Alexandre Girard
|
aa92518721 |
Low-code connectors: configurable source from yaml (#13038)
* checkout from alex/cac * checkout from alex/cac * checkout from alex/cac * checkout from alex/cac * Add missing tests * Add missing files * Add missing tests * add missing file * missing file * missing file * sengrid low code connector * rename * doc * doc * remove broken test * rename * jinja dependency * Add comment * comment * comment * pyjq dependency * update import * rename file * delete unused file * Revert "delete unused file" This reverts commit |