1
0
mirror of synced 2025-12-21 19:11:14 -05:00
Commit Graph

24 Commits

Author SHA1 Message Date
Natik Gadzhi
63ab010eaa docs: Connector Builder contribution flow text-based docs (#44616) 2024-08-27 00:42:47 +00:00
Jeroen de Vries
92f299de55 Update low-code-cdk-overview.md (#41338)
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
2024-07-12 14:52:19 -03:00
Natik Gadzhi
cb80e6922a [tools] prettier rules for .md + formatting cleanup 2024-05-07 08:19:33 -07:00
Natik Gadzhi
0e9bdf314d connector-templates: move generated code from src so airbyte-ci builds it (#36428)
### What

For newly generated python and low-code connectors, move sources from `src` to `source_%CONN_NAME%` so `airbyte-ci build` can pick them up correctly. Closes #35893.

### What changed?

- Moved the sources from `src`
- Updated package paths in `pyproject.toml` templates

### How to test?

You can make a new connector with `./generate.sh` and verify that it works with Poetry and that it builds with `airbyte-ci build --name source-test`

---
2024-03-25 10:01:30 -07:00
Alexandre Girard
6fb2591643 low-code docs: Link to schema from the overview page (#32893) 2023-11-28 15:57:14 -08:00
Alexandre Girard
5aa90cc379 Revert "Remove reference page from low-code docs" (#32093) 2023-11-02 14:56:59 +00:00
Alexandre Girard
19b60addec Remove reference page from low-code docs (#31829) 2023-10-25 15:13:35 -07:00
Ben Church
26fe7886eb 📚 Docs: Add Support Level, remove release stage (#30501)
Co-authored-by: Lake Mossman <lake@airbyte.io>
2023-09-18 15:55:13 +00:00
Dzmitry Belash
2e014a1e95 Fix typo in low-code-cdk-overview.md (#30503) 2023-09-18 10:43:53 -05:00
Evan Tahler
090c1a9701 help-connector-development > using-the-cdk (#25902) 2023-05-09 08:45:06 -07:00
Sherif A. Nada
9959e8d437 docs: low-code is now in beta 2023-05-02 17:11:14 -07:00
Joe Reuter
5ec81cc8d1 Connector builder: Restructure documentation (#25353)
* authentication documentation

* some fixes

* fix broken link

* remove session token auth

* review comments

* first version of tutorial

* Revert "first version of tutorial"

This reverts commit bbdeb997f2.

* first version of tutorial

* review comments

* improvements

* partitioning documentation

* incremental sync

* review comments

* improvements

* record processing doc

* improvements

* error handler connector builder ui documentation

* add missing link

* overview page

* mention current limitation

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
2023-04-27 12:32:26 +02:00
Sophia Wiley
b19998eeb7 edited links in docs (#24108) 2023-03-15 19:57:54 +01:00
Sherif A. Nada
ccc1c6c890 Docs: Encourage CDK devs to join slack channel to get updates (#23682) 2023-03-02 11:11:46 -08:00
Brian Lai
1c482ee339 Low code cdk to beta (#22853)
* [ISSUE #19410] remove request_options_provider from the … (#21403)

* [ISSUE #19410] (incomplete) remove request_options_provider from the manifest

* [ISSUE #19410] (incomplete) incomplete cleanup config_component_schema.json as well

* [ISSUE #19410] update source-monday

* [ISSUE #19410] code review

* [ISSUE #19410] formatting files

* [Low-Code CDK] Replace the $options keyword with $parameters (#21632)

* refactor flows and tests to use parameters instead of options

* update documentation to reflect the change from options to parameters

* create migration script to replace options with parameters in existing manifests

* update template to use parameters instead of options

* fix tests after rebasing from the branch

* address pr feedback and extra uses of options that I missed

* additional changes needed after rebasing from master

* migrate low-code connectors to use parameters instead of options

* 🚨🚨 [Low Code CDK] Update `*ref` format to `#/` (#21434)

* [Low-Code CDK] Remove JsonSchema type in favor of JsonSchemaFileLoader (#21832)

* fully deprecate JsonSchema in favor of JsonFileSchemaLoader

* remove usage in the legacy registry

* Update migration scripts according to manifest file rename (#21920)

* Issue 21866 remove legacy factory and validation flow (#21878)

* [ISSUE #21866] clean ManifestDeclarativeSource validation

* [ISSUE #21866] remove dataclasses-jsonschema

* [ISSUE #21866] code review

* [ISSUE-21866] flake8

* [ISSUE #21559] remove DefaultPaginator.url_base (#21823)

* [ISSUE #21559] remove DefaultPaginator.url_base

* [ISSUE #21559] code review

* [ISSUE #21559] update migration script

* [ISSUE #21559] code review

* [ISSUE #21559] update documentation

* [ISSUE #21559] run migration (#21824)

* [ISSUE #21559] remove DefaultPaginator.url_base (#21823)

* [ISSUE #21559] remove DefaultPaginator.url_base

* [ISSUE #21559] code review

* [ISSUE #21559] update migration script

* [ISSUE #21559] code review

* [ISSUE #21559] update documentation

* [ISSUE #21559] run migration (#21824)

* [ISSUE #21559] fix manifests

* [ISSUE #21926] setup server to allow for local tests (#21974)

* [Low Code CDK] remove checkpoint_interval from DeclarativeStream component (#22120)

* Issue #21576 rename dpathextractor fieldpointer (#21990)

* [ISSUE #21926] setup server to allow for local tests

* [ISSUE #21576] Rename DpathExtractor.field_pointer to field_path

* [ISSUE #21576] migration script

* [ISSUE #21576] update source-monday and source-pocket as well

* [ISSUE #21576] migration (#21997)

* [ISSUE #21576] code review

* Remove checkpoint_interval from source-prestashop manifest (#22141)

* replacing options with parameters for a few connectors I missed or were newly added

* [Low-Code CDK] Rremove stream_cursor_field from stream and derive it from stream_slicer (#22294)

* update schema to derive cursor_field from a stream slicer if it exists

* remove usage of stream_cursor_field on simple connector use cases

* fixing some of the more complex usage of stream_cursor_field that rely on cartesian product stream slicers

* fix documentation to replace references to stream_cursor_field

* Low Code CDK: Remove `name` and `primary_key` from non-DeclarativeStream components (#21891)

* fix eslint issues for webapp (#22462)

* 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22375)

* bump connector builder server to latest CDK version

* fix breaking CDK changes in connector builder FE

* [Low-Code CDK] Separate request path from RequestOption component (#22398)

* split apart path from RequestOption and fix usages and cleanup the code

* replace usage of path with RequestPath and get rid of default to RequestOption

* fix bug where stream_slice_field was used in outbound request instead of request_option field_name

* organize yaml schema names and update documentation for RequestOption and RequestPath

* clean up tests

* regenerate models

* [ISSUE #19961] refactor stream slices (#22225)

* [ISSUE #19961] add 'incremental' and partially remove CartesianProductStreamSlicer - Google PageSpeed Insights not working yet

* [ISSUE #19961] fixing Google PageSpeed Insights

* move incremental_sync field to the stream level and perform merging into one stream slicer at that level

* add tests to merging incremental and iterable into cartesian

* rewrite documentation to separate incremental sync and iterator concepts

* update documentation to use partition router and revise the tutorial to reflect the new changes to the components

* [ISSUE #19961] update code to newest CDK version and clean autogenerated files (#22670)

* [ISSUE #19961] rename stream_slicer to partition_router and update ma… (#22590)

* [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well)

* [ISSUE 19961] rename CustomStreamSlicer (#22598)

* [ISSUE 19961] rename CustomStreamSlicer

* [ISSUE #19961] code review CustomStreamSlicer

* [ISSUE #19961] fix source_square incremental sync

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591)

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter

* remove SinglePartitionRouter from the schema

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* [ISSUE #19961] rename SubstreamSlicer  to SubstreamPartitionRouter (#22596)

* [ISSUE #19961] TMP rename SubstreamSlicer  to SubstreamPartitionRouter

* [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end

* [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593)

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* [ISSUE #19961] clean faulty merge

* [ISSUE #19961] rename DatetimeStreamSlicer (#22617)

* [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well)

* [ISSUE 19961] rename CustomStreamSlicer (#22598)

* [ISSUE 19961] rename CustomStreamSlicer

* [ISSUE #19961] code review CustomStreamSlicer

* [ISSUE #19961] fix source_square incremental sync

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591)

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter

* remove SinglePartitionRouter from the schema

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* [ISSUE #19961] rename DatetimeStreamSlicer

* [ISSUE #19961] rename SubstreamSlicer  to SubstreamPartitionRouter (#22596)

* [ISSUE #19961] TMP rename SubstreamSlicer  to SubstreamPartitionRouter

* [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end

* [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593)

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/yaml-overview.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/incremental-syncs.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* update docs

* [ISSUE #19961] clean unit tests files

* [ISSUE #19961] code review

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* [Low-Code CDK] Allow for children of custom components to specify parameters that are normally derived (#22379)

* Fix a bug where child components of a custom component cannot receive fields from other components

* add tests, documentation and commenting

* fix test from merge

* add better error message for nested initialization failures

* 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22880)

* restrict name to stream level

* remove checkpoint interval

* adjust logic for new request options

* refactor slicers

* wording

* review comments

* make oldest supported version explicit

* separate the frontend and connector builder changes from the low-code to beta release

* [Low-Code CDK] Add script to run low code unit tests and address issues with a few connectors (#23123)

* consolidate all the changes into a new PR after I messed up the merge on the side branch

* add set to allow this to be called externally if necessary later

* remove last few extra fields i found and fix docs links

* fix docs one more time

---------

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: Joe Reuter <joe@airbyte.io>
2023-02-15 21:46:49 -05:00
John Aitchison
a73f3de5e9 📚 Docs: fix low code example links (#22189)
* Docs, fix low code example links

* link to manifest

* forgot to push link fix...
2023-02-14 18:16:24 -06:00
Brian Lai
b7113a2e5e [Low-Code CDK] Write the component schema and use it during manifest validation (#20422)
* handwritten low code manifest example components

* add MinMaxDatetime to jsonschema

* add a basic gradle command to generate manifest components

* Add auth components to handwritten component schema

- ApiKeyAuthenticator
- BasicHttpAuthenticator
- BearerAuthenticator
- DeclarativeOauth2Authenticator
- NoAuth

* Respect optional properties in DeclarativeOauth2Authenticator

* Fix `Dict[str, Any]` mapping in auth components

* add default error handler composite error handler and http response filter components

* [low code component schema] adding backoff strategies to schema

* [low code component schema] fix float types

* [low code component schema] add RecordFilter

* Remove `config` from auth components

* [low code component schema] add Interpolation (with pending question on 'type' not being defined)

* Add CartesianProductStreamSlicer & DatetimeStreamSlicer

* Add ListStreamSlicer, and fix nesting of DatetimeStreamSlicer

* [low code component schema] add InterpolatedRequestOptionsProvider

* Add slicer components, and fix a couple of components after reviewing output

* [low code component schema] adding transformations and adding type to interpolators

* adding spec and a few small tweaks

* Add DefaultSchemaLoader

* [low code component schema] attempt on custom class

* Add descriptions for auth components

* add RequestOption

* remove interpolated objects from the schema in favor of strings only

* a few schema fixes and adding some custom pagination and stream slicer

* [low code component schema] fix CustomBackoffStrategy

* Add CustomRecordExtractor

* add some description and add additional properties

* insert a transformer to hydrate default manifest components and perform validation against the handwritten schema

* [low code component schema] validating existing schemas

* [low code component schema] clean validation script

* add manifest transformer tests and a few tweaks to the schema

* Revert "[low code component schema] clean validation script"

This reverts commit 2408f41cf6.

* Revert "[low code component schema] validating existing schemas"

This reverts commit 9d39977815.

* [low code component schema] integrate validation script to gradle

* [low code component schema] updating validation script permissions

* remove a few model gen spike files and clean up comments

* default types should take parent type into account and a few schema changes

* add inline schema components and fix validation bugs

* add types to every component's literal enum and more little schema fixes

* missing component prefix

* pr feedback, renaming the schema and updating various docs

* fixing schema bugs from testing and new components

* last few small tweaks and fixes and fix the script

* bump the version before publsih

* include the declarative_component_schema.yaml as package data so its included when package is published

* roll the version to publish

Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
2022-12-19 14:42:34 -06:00
Sherif A. Nada
f5d8e25519 Add connector builder UI docs (#19588) 2022-11-18 10:56:30 -08:00
Amruta Ranade
17932faebd Update low-code-cdk-overview.md 2022-11-16 14:02:12 -05:00
Amruta Ranade
57571ee94f Fixed broken links 2022-10-13 09:11:57 -05:00
Alexandre Girard
39220f4b1e Update lowcode docs (#17752)
* Update lowcode docs

* Update

* fix links

* more links

* Schemas

* indent

* Update docs/connector-development/config-based/advanced-topics.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/advanced-topics.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/low-code-cdk-overview.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/request-options.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/requester.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/stream-slicers.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/requester.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/error-handling.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* capitalization

* fix schema

* fix schema

* small changes so greenhouse validates

* fix brackets

* update code samples

* Add missing types

* fix

* reset

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
2022-10-07 19:15:27 -07:00
Amruta Ranade
66dafd85aa Updated Slack link 2022-10-03 19:39:40 -04:00
Amruta Ranade
984945d3aa Added Slack link 2022-10-03 19:22:03 -04:00
Amruta Ranade
1ac6df7537 low-code connector docs (#17534) 2022-10-03 18:22:29 -04:00