1
0
mirror of synced 2025-12-22 03:21:25 -05:00
Commit Graph

98 Commits

Author SHA1 Message Date
Anatolii Yatsuk
fb769bd0a0 feat(airbyte-cdk): Add Per Partition with Global fallback Cursor (#45125) 2024-10-28 14:03:41 +02:00
Anatolii Yatsuk
caa12ab0c7 docs(airbyte-cdk): Fix error in incremental sync docs (#45194) 2024-09-06 17:39:35 +03:00
Anatolii Yatsuk
2fa35ab30b feat(airbyte-cdk): Add Global Parent State Cursor (#39593) 2024-09-06 16:44:34 +03:00
Natik Gadzhi
63ab010eaa docs: Connector Builder contribution flow text-based docs (#44616) 2024-08-27 00:42:47 +00:00
Jeroen de Vries
bb4df149f6 feat(docs): oauth refresh token updater (#41555)
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2024-07-18 12:28:47 -03:00
Jeroen de Vries
92f299de55 Update low-code-cdk-overview.md (#41338)
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
2024-07-12 14:52:19 -03:00
Natik Gadzhi
4a62ae76a3 fix(docs): typo in low-code docs (#40694) 2024-07-02 18:08:48 -07:00
Jeroen de Vries
570f5f5954 Change docs based on CLI prompt (#40674) 2024-07-02 08:55:20 -07:00
Jeroen de Vries
74a052da53 Update 4-reading-data.md (#40677) 2024-07-02 07:53:09 -07:00
Natik Gadzhi
cb80e6922a [tools] prettier rules for .md + formatting cleanup 2024-05-07 08:19:33 -07:00
Patrick Nilan
b20cd1bd1d airbyte-cdk - Adds JwtAuthenticator to low-code (#37005) 2024-04-19 09:17:54 -07:00
Alexandre Girard
87f051c3a9 Add a walkthrough of building a custom Python connector (#36743)
Co-authored-by: Natik Gadzhi <natik@respawn.io>
2024-04-08 21:08:25 -07:00
graingert-coef
85ca7ec595 docs: fix pagination link header documentation (#36599) 2024-03-28 09:13:03 -07:00
Alexandre Girard
8a99c8e891 Add low-code filters to the reference docs (#36469)
Co-authored-by: Lake Mossman <lake@airbyte.io>
2024-03-25 18:07:28 -07:00
Natik Gadzhi
0e9bdf314d connector-templates: move generated code from src so airbyte-ci builds it (#36428)
### What

For newly generated python and low-code connectors, move sources from `src` to `source_%CONN_NAME%` so `airbyte-ci build` can pick them up correctly. Closes #35893.

### What changed?

- Moved the sources from `src`
- Updated package paths in `pyproject.toml` templates

### How to test?

You can make a new connector with `./generate.sh` and verify that it works with Poetry and that it builds with `airbyte-ci build --name source-test`

---
2024-03-25 10:01:30 -07:00
Augustin
84cf59216a doc: remove Node requirements on config based getting started tutorial (#35436) 2024-02-20 04:26:07 -06:00
Augustin
05a1512a7e update doc to reference poetry (#35414) 2024-02-20 07:57:53 +00:00
Joe Reuter
e2789b4f03 Update docusaurus to 3 (#33041) 2023-12-11 17:03:18 +01:00
Alexandre Girard
6fb2591643 low-code docs: Link to schema from the overview page (#32893) 2023-11-28 15:57:14 -08:00
Alexandre Girard
5aa90cc379 Revert "Remove reference page from low-code docs" (#32093) 2023-11-02 14:56:59 +00:00
Alexandre Girard
19b60addec Remove reference page from low-code docs (#31829) 2023-10-25 15:13:35 -07:00
Augustin
b35a2b2907 doc: update connector tutorials to reference airbyte-ci for connector build (#31650) 2023-10-23 22:42:53 +02:00
Ben Church
26fe7886eb 📚 Docs: Add Support Level, remove release stage (#30501)
Co-authored-by: Lake Mossman <lake@airbyte.io>
2023-09-18 15:55:13 +00:00
Dzmitry Belash
2e014a1e95 Fix typo in low-code-cdk-overview.md (#30503) 2023-09-18 10:43:53 -05:00
Saku3449
c1a7b961b9 change to use the Exchange Rate Data API access token instead of the … (#29322) 2023-09-04 07:33:38 -05:00
Evan Tahler
090c1a9701 help-connector-development > using-the-cdk (#25902) 2023-05-09 08:45:06 -07:00
Sherif A. Nada
9959e8d437 docs: low-code is now in beta 2023-05-02 17:11:14 -07:00
Deke Li
ad7c227c1b fix doc naming for DatetimeCursorBased to DatetimeBasedCursor (#25764) 2023-05-02 23:46:03 +00:00
Joe Reuter
5ec81cc8d1 Connector builder: Restructure documentation (#25353)
* authentication documentation

* some fixes

* fix broken link

* remove session token auth

* review comments

* first version of tutorial

* Revert "first version of tutorial"

This reverts commit bbdeb997f2.

* first version of tutorial

* review comments

* improvements

* partitioning documentation

* incremental sync

* review comments

* improvements

* record processing doc

* improvements

* error handler connector builder ui documentation

* add missing link

* overview page

* mention current limitation

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
2023-04-27 12:32:26 +02:00
Joe Reuter
e96fc22d6f Connector builder documentation: Auto-generate yaml reference and link up in fitting places (#25428)
* auto-generate yaml reference and link up

* Update description

* Add examples

* format

* update

* update

* add description for params

* Update declarative_component_schema.yaml

* render type and examples for interpolation variables

* fix schema bug

* Regenerate yarn lock file

---------

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
2023-04-26 16:08:35 +02:00
Joe Reuter
e19bc57352 remove session token auth (#25142) 2023-04-19 12:27:23 +02:00
Alexandre Girard
f7fa1de93b Add connector builder compatibility guide (#24916)
* Add connector builder compatibility guide

* Update

* Update

* Update

* Update

* Update docs/connector-development/config-based/connector-builder-compatibility.md

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update docs/connector-development/config-based/connector-builder-compatibility.md

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update docs/connector-development/config-based/connector-builder-compatibility.md

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update docs/connector-development/config-based/connector-builder-compatibility.md

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update docs/connector-development/config-based/connector-builder-compatibility.md

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update docs/connector-development/config-based/connector-builder-compatibility.md

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update

* Update

* Update

* Update whimsical link

---------

Co-authored-by: Lake Mossman <lake@airbyte.io>
2023-04-11 12:17:41 -07:00
Sophia Wiley
b19998eeb7 edited links in docs (#24108) 2023-03-15 19:57:54 +01:00
Alexandre Girard
9864e4bf19 Update links (#24100)
* Update links

* Add missing files
2023-03-15 10:20:44 -07:00
Sophia Wiley
5512befeb1 Docs: updated links from .io to .com (#23652)
* updated links

* edited contributors link

* deleted line about CDK in docs
2023-03-06 17:27:55 +01:00
Sherif A. Nada
ccc1c6c890 Docs: Encourage CDK devs to join slack channel to get updates (#23682) 2023-03-02 11:11:46 -08:00
Alexandre Girard
ef083500b2 Fix typo CursorPaginator -> CursorPagination (#23513) 2023-02-28 09:40:55 -05:00
Brian Lai
1c482ee339 Low code cdk to beta (#22853)
* [ISSUE #19410] remove request_options_provider from the … (#21403)

* [ISSUE #19410] (incomplete) remove request_options_provider from the manifest

* [ISSUE #19410] (incomplete) incomplete cleanup config_component_schema.json as well

* [ISSUE #19410] update source-monday

* [ISSUE #19410] code review

* [ISSUE #19410] formatting files

* [Low-Code CDK] Replace the $options keyword with $parameters (#21632)

* refactor flows and tests to use parameters instead of options

* update documentation to reflect the change from options to parameters

* create migration script to replace options with parameters in existing manifests

* update template to use parameters instead of options

* fix tests after rebasing from the branch

* address pr feedback and extra uses of options that I missed

* additional changes needed after rebasing from master

* migrate low-code connectors to use parameters instead of options

* 🚨🚨 [Low Code CDK] Update `*ref` format to `#/` (#21434)

* [Low-Code CDK] Remove JsonSchema type in favor of JsonSchemaFileLoader (#21832)

* fully deprecate JsonSchema in favor of JsonFileSchemaLoader

* remove usage in the legacy registry

* Update migration scripts according to manifest file rename (#21920)

* Issue 21866 remove legacy factory and validation flow (#21878)

* [ISSUE #21866] clean ManifestDeclarativeSource validation

* [ISSUE #21866] remove dataclasses-jsonschema

* [ISSUE #21866] code review

* [ISSUE-21866] flake8

* [ISSUE #21559] remove DefaultPaginator.url_base (#21823)

* [ISSUE #21559] remove DefaultPaginator.url_base

* [ISSUE #21559] code review

* [ISSUE #21559] update migration script

* [ISSUE #21559] code review

* [ISSUE #21559] update documentation

* [ISSUE #21559] run migration (#21824)

* [ISSUE #21559] remove DefaultPaginator.url_base (#21823)

* [ISSUE #21559] remove DefaultPaginator.url_base

* [ISSUE #21559] code review

* [ISSUE #21559] update migration script

* [ISSUE #21559] code review

* [ISSUE #21559] update documentation

* [ISSUE #21559] run migration (#21824)

* [ISSUE #21559] fix manifests

* [ISSUE #21926] setup server to allow for local tests (#21974)

* [Low Code CDK] remove checkpoint_interval from DeclarativeStream component (#22120)

* Issue #21576 rename dpathextractor fieldpointer (#21990)

* [ISSUE #21926] setup server to allow for local tests

* [ISSUE #21576] Rename DpathExtractor.field_pointer to field_path

* [ISSUE #21576] migration script

* [ISSUE #21576] update source-monday and source-pocket as well

* [ISSUE #21576] migration (#21997)

* [ISSUE #21576] code review

* Remove checkpoint_interval from source-prestashop manifest (#22141)

* replacing options with parameters for a few connectors I missed or were newly added

* [Low-Code CDK] Rremove stream_cursor_field from stream and derive it from stream_slicer (#22294)

* update schema to derive cursor_field from a stream slicer if it exists

* remove usage of stream_cursor_field on simple connector use cases

* fixing some of the more complex usage of stream_cursor_field that rely on cartesian product stream slicers

* fix documentation to replace references to stream_cursor_field

* Low Code CDK: Remove `name` and `primary_key` from non-DeclarativeStream components (#21891)

* fix eslint issues for webapp (#22462)

* 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22375)

* bump connector builder server to latest CDK version

* fix breaking CDK changes in connector builder FE

* [Low-Code CDK] Separate request path from RequestOption component (#22398)

* split apart path from RequestOption and fix usages and cleanup the code

* replace usage of path with RequestPath and get rid of default to RequestOption

* fix bug where stream_slice_field was used in outbound request instead of request_option field_name

* organize yaml schema names and update documentation for RequestOption and RequestPath

* clean up tests

* regenerate models

* [ISSUE #19961] refactor stream slices (#22225)

* [ISSUE #19961] add 'incremental' and partially remove CartesianProductStreamSlicer - Google PageSpeed Insights not working yet

* [ISSUE #19961] fixing Google PageSpeed Insights

* move incremental_sync field to the stream level and perform merging into one stream slicer at that level

* add tests to merging incremental and iterable into cartesian

* rewrite documentation to separate incremental sync and iterator concepts

* update documentation to use partition router and revise the tutorial to reflect the new changes to the components

* [ISSUE #19961] update code to newest CDK version and clean autogenerated files (#22670)

* [ISSUE #19961] rename stream_slicer to partition_router and update ma… (#22590)

* [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well)

* [ISSUE 19961] rename CustomStreamSlicer (#22598)

* [ISSUE 19961] rename CustomStreamSlicer

* [ISSUE #19961] code review CustomStreamSlicer

* [ISSUE #19961] fix source_square incremental sync

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591)

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter

* remove SinglePartitionRouter from the schema

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* [ISSUE #19961] rename SubstreamSlicer  to SubstreamPartitionRouter (#22596)

* [ISSUE #19961] TMP rename SubstreamSlicer  to SubstreamPartitionRouter

* [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end

* [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593)

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* [ISSUE #19961] clean faulty merge

* [ISSUE #19961] rename DatetimeStreamSlicer (#22617)

* [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well)

* [ISSUE 19961] rename CustomStreamSlicer (#22598)

* [ISSUE 19961] rename CustomStreamSlicer

* [ISSUE #19961] code review CustomStreamSlicer

* [ISSUE #19961] fix source_square incremental sync

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591)

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter

* remove SinglePartitionRouter from the schema

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* [ISSUE #19961] rename DatetimeStreamSlicer

* [ISSUE #19961] rename SubstreamSlicer  to SubstreamPartitionRouter (#22596)

* [ISSUE #19961] TMP rename SubstreamSlicer  to SubstreamPartitionRouter

* [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end

* [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593)

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/yaml-overview.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/incremental-syncs.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* update docs

* [ISSUE #19961] clean unit tests files

* [ISSUE #19961] code review

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* [Low-Code CDK] Allow for children of custom components to specify parameters that are normally derived (#22379)

* Fix a bug where child components of a custom component cannot receive fields from other components

* add tests, documentation and commenting

* fix test from merge

* add better error message for nested initialization failures

* 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22880)

* restrict name to stream level

* remove checkpoint interval

* adjust logic for new request options

* refactor slicers

* wording

* review comments

* make oldest supported version explicit

* separate the frontend and connector builder changes from the low-code to beta release

* [Low-Code CDK] Add script to run low code unit tests and address issues with a few connectors (#23123)

* consolidate all the changes into a new PR after I messed up the merge on the side branch

* add set to allow this to be called externally if necessary later

* remove last few extra fields i found and fix docs links

* fix docs one more time

---------

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: Joe Reuter <joe@airbyte.io>
2023-02-15 21:46:49 -05:00
Sherif A. Nada
26f5615924 lowcode CDK Docs: update references to manifest file names (#23041) 2023-02-14 21:17:17 -08:00
John Aitchison
a73f3de5e9 📚 Docs: fix low code example links (#22189)
* Docs, fix low code example links

* link to manifest

* forgot to push link fix...
2023-02-14 18:16:24 -06:00
Dominik Wagner
01ff0ab05f Fix typo in example in error handing doc (#22975) 2023-02-14 18:14:45 -06:00
Sherif A. Nada
140e0964c8 docs: comment out outdated lowcode builder UI video (#23022) 2023-02-14 10:30:46 -08:00
Lake Mossman
0cef7b0e55 🪟 🎉 Connector Builder Landing Page (#22122)
* save working version of minimal landing page

* move landing page to own component and show Toast on invalid yaml

* add default Untitled connector name

* save styling progress

* style the landing page

* use card component

* reword descriptions

* undo unnecessary syntax change

* use file name as connector name

* move showLandingPage to global state and set to true on reset

* fix e2e test

* move local storage hooks into a separate service/context

* move landing page to its own page

* update doc

* move full file input handling logic into callback

* use hidden

* clear out notification on unmount

* use url if filename is 'manifest'

* add comment

---------

Co-authored-by: Joe Reuter <joe@airbyte.io>
2023-02-06 17:19:18 +00:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846)
Rename source acceptance test to connector acceptance test
2023-02-02 11:38:19 +01:00
Artem Inzhyyants
e0e83e2215 Low Code CDK: DpathExtractor extend (#21690)
* Low Code CDK: Dpath Extractor extend

* Low Code CDK: Dpath Extractor update docs

* Low Code CDK: Update version

* Low Code CDK: fix formatting

* Low Code CDK: update docs
2023-01-25 15:49:50 +01:00
Lake Mossman
c6aea575ca Fix connector builder link once more (#21442)
* fix connector builder link once more

* use relative path that works in other places
2023-01-18 13:59:52 -08:00
Lake Mossman
08207e2181 Fix connector builder documentation link again (#21441)
* fix relative link in docs

* fix link again
2023-01-13 16:57:24 -08:00
Lake Mossman
d19b51c751 Fix links in connector builder UI docs (#21411) 2023-01-13 15:52:49 -08:00
Brian Lai
cbf9ea76c1 [Low-Code CDK] Construct declarative components from Pydantic models (#21050)
* initial work to parse manifest objects into pydantic models

* pr feedback and some other validations

* rerun manifest schema generation

* remove field constraint due to bug

* initial work to construct most components from greenhouse

* custom components parse subcomponent fields correctly and adding a few more component constructors

* construct components from gnews

* first pass at posthog.yaml

* Handle nested custom components with list values.
Also includes updates to posthog.yaml, including autoformatting changes.

* adding constructors for slicers, filters, and transformations and a few bug fixes

* make sed work across multiple OS

* add NoAuth component

* fix handling of custom components with nested list

* Autogenerate `TYPE_NAME_TO_MODEL` mapping

* Handle default kwargs not defined on model for custom components

* Re-add `options` for CartesianProductStreamSlicer for backwards compat
with custom stream slicers

* add basic unit tests for the model component factory

* add back defaults and extra parameters like options to retain compatibility with legacy flow and backwards compatibility

* Remove `_get_defaults`; using actual default values on classes instead

* Add backoff strategy component creation functions

* add back defaults and extra parameters like options to retain compatibility with legacy flow and backwards compatibility

* add lots of tests to construct components from the pydantic models and a few bug fixes

* add a few tests for the model to component factory

* add catch

* fix a bug where propagated schema doesn't work with old factory

* clean up a few files

* add type inference for custom components, more tests and some refactoring of the model factory

* template, docs, manifest updates, pr feedback and some cleanup

* pr feedback and polish schema a bit

* fix tests from the latest rebase of master

* fix the last few bugs I found and adjust a few sources that weren't perfectly compatible with the new component flow

* fix CheckStream bug cleanup and a few small tweaks and polish

* add additional test to cover bug case

* fix formatting

* 🤖 Bump minor version of Airbyte CDK

Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
2023-01-12 21:02:08 -05:00
Sherif A. Nada
09e4d99d79 lowcode docs: add link to YAML reference (#21231) 2023-01-10 14:15:32 -08:00