1
0
mirror of synced 2026-01-02 12:02:47 -05:00
Commit Graph

10 Commits

Author SHA1 Message Date
Alexandre Girard
288c3cabad Tutorial and documentation for config-based connectors (#15027)
* 5-step tutorial

* move

* tiny bit of editing

* Update tutorial

* update docs

* reset

* move files

* record selector, request options, and more links

* update

* update

* connector definition

* link

* links

* update example

* footnote

* typo

* document string interpolation

* note on string interpolation

* update

* fix code sample

* fix

* update sample

* fix

* use the actual config

* Update as per comments

* write as yaml

* typo

* Clarify options overloading

* clarify that docker must be running

* remove extra footnote

* use venv directly

* Apply suggestions from code review

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* signup instructions

* update

* clarify that both dot and bracket notations are interchangeable

* Clarify how check works

* create spec and config before updating connector definition

* clarify what now_local() is

* rename to yaml structure

* Go through tutorial and update end of section code samples

* fix link

* update

* update code samples

* Update code samples

* Update to bracket notation

* remove superfluous comments

* Update docs/connector-development/config-based/tutorial/2-install-dependencies.md

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* Update docs/connector-development/config-based/tutorial/4-reading-data.md

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* fix path

* update

* motivation blurp

* warning

* warning

* fix code block

* update code samples

* update code sample

* update code samples

* small updates

* update yaml structure

* custom class example

* language annotations

* update warning

* Update tutorial to use dpath extractor

* Update record selector docs

* unit test

* link to contributing

* tiny update

* $ in front of commands

* $ in front of commands

* More readings

* link to existing config-based connectors

* index

* update

* delete broken link

* supported features

* update

* Add some links

* Update docs/connector-development/config-based/overview.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/record-selector.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/overview.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/overview.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/overview.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* mention the unit

* headers

* remove mentions of interpolating on stream slice, etc.

* update

* exclude config-based docs

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
2022-08-12 15:50:54 -07:00
Alexandre Girard
6332fd6527 [low-code-connectors] Replace JelloExtractor with DpathExtractor (#15514)
* Handle extracting no records from root

* handle missing keys

* record extractor interface

* dpath extractor

* docstring

* handle extract root array

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/extractors/jello.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/extractors/record_selector.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* update docstring

* respect extractor interface

* edge case handling

* document

* use dpath by default

* delete jello extractor

* bump cdk version

* delete jello dependency

* Update reference docs templates

* update template

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-08-11 12:34:54 -07:00
Alexandre Girard
c5c13f05b4 low-code-connectors: handle single records (#15346)
* handle single records

* comment

* comment
2022-08-05 15:52:45 -07:00
Brian Lai
bd31100774 initial first pass converting every component to dataclasses (#15189)
* initial first pass converting every component to dataclasses

* replace the hackier options pass through

* get rid of the hackier way

* fix issues w/ type hints by making options required and lots of fixes to the language to fix compatability for dataclasses

* add dataclasses-jsonschema to setup

* fix oauth authenticator to avoid dataclass name collisions

* fix spacing for CI tests

* remove property from oauth and fix a interpolation bug

* pr feedback and cleaning up the code a bit, attempt at avoiding renaming

* fix templates and bugs surfaced during greenhouse spec testing

* fix tests

* fix missing options in some declarative components

* fix tests related to pulling latest master

* fix issue w/ passing state, slice, and token to subcomponents

* switch name back to get_access_token() since no name collision anymore
2022-08-05 17:39:27 -04:00
Alexandre Girard
a3ff80c179 [low-code-connectors] Disable parse-time interpolation in favor of runtime-only (#14923)
* abstract auth token

* basichttp

* remove prints

* docstrings

* get rid of parse-time interpolation

* always pass options through

* delete print

* delete misleading comment

* delete note

* reset

* pass down options

* delete duplicate file

* missing test

* refactor test

* rename to '$options'

* rename to ''

* interpolatedauth

* fix tests

* fix

* docstrings

* update docstring

* docstring

* update docstring

* remove extra field

* undo

* rename to runtime_parameters

* docstring

* update

* / -> *

* update template

* rename to options

* Add examples

* update docstring

* Update test

* newlines

* rename kwargs to options

* options init param

* delete duplicate line

* type hints

* update docstring

* Revert "delete duplicate line"

This reverts commit 4255d5b346.

* delete duplicate code from bad merge

* rename file

* bump cdk version
2022-07-28 08:57:17 -07:00
Alexandre Girard
08239abafd Alex/lowcode referencedocs (#14973)
* Add docstrings for auth package

* docstrings for the check package

* docstrings for the datetime package

* docstrings for the decoder package

* docstrings for extractors package and fix tests

* interpolation docstrings

* ref ->  and parser docstrings

* docstrings for parsers package

* error handler docstrings

* requester docstrings

* more docstrings

* docstrings

* docstrings

* docstrings

* Use defined type annotations

* update

* update docstrings

* Update docstrings

* update docstrings

* update docstrings

* update template

* Revert "update template"

This reverts commit eb4a11858b.

* update template

* update

* move to interpolated_string

* update docstring

* update

* fix tests

* format

* return type can also be an array

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/interpolated_boolean.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/interpolation.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/jinja.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/interpolated_boolean.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/backoff_strategy.py

* Update as per comments

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-07-25 18:10:32 -07:00
Brian Lai
a61224887e support semi incremental by adding extractor record filter (#13520)
* support semi incremental by adding extractor record filter

* refactor extractor into a record_selector that supports extraction and filtering of response records
2022-06-23 00:09:44 -04:00
Alexandre Girard
15fe51b788 [low-code connectors] add a few unit tests (#13666)
* add a couple of unit tests

* refactor tests
2022-06-13 16:51:36 -07:00
Alexandre Girard
57dd7a9d4b [low-code connectors] use jello instead of jq (#13570)
* use jello instead of jq

* reset

* rename

* remove print
2022-06-07 14:10:26 -07:00
Alexandre Girard
150ab593f8 Low code connectors: core structure (#12850)
* checkout from alex/cac

* doc

* doc

* remove broken test

* rename

* rename file

* delete unused file

* rename

* abstract property

* isort

* update state

* Update comment

* remove incremental mixin

* delete comment

* update comments

* update comments

* remove no_state

* rename package

* pass parameters through kwargs

* update interface to pass source in interface

* update interface to pass source in interface

* rename to stream_slicer

* Low code connectors: string interpolation with jinja (#12852)

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* move test to right module

* Add missing test

* Use authbase instead of deprecated class

* leverage generator

* rename to declarative

* rename the classes too
2022-05-25 17:27:54 -07:00