1
0
mirror of synced 2025-12-21 11:01:41 -05:00
Commit Graph

386 Commits

Author SHA1 Message Date
Maxime Carbonneau-Leclerc
f69a128a9c [ISSUE #19981] testing version bump (#21106)
* [ISSUE #19981] testing version bump

* [ISSUE #19981] debugging part-to-bump

* [ISSUE #19981] further debugging on part-to-bump

* [ISSUE #19981] further debugging on part-to-bump yet again

* [ISSUE #19981] further debugging on part-to-bump yet again and again

* [ISSUE #19981] improving on first successful attempt

* [ISSUE #19981] adding changelog and concurrency

* [ISSUE #19981] update version in .bumpversion.cfg

* [ISSUE #19981] testing Slack notification with working channel

* [ISSUE #19981] documentation and clean up before PR

* [ISSUE #19981] make changelog-message optional

* [ISSUE #19981] align version with new release

* [ISSUE #19981] code review
2023-01-11 08:28:07 -05:00
Sherif A. Nada
09e4d99d79 lowcode docs: add link to YAML reference (#21231) 2023-01-10 14:15:32 -08:00
Maxime Carbonneau-Leclerc
423cb8bdbc [ISSUE-20322] updating tutorial documentation (#21163)
* [ISSUE-20322] updating tutorial documentation

* [ISSUE-20322] updating test and improving docs
2023-01-10 10:03:35 -05:00
Lake Mossman
d414e3d4f7 Update Connector Builder docs to reflect configuration UI additions (#21129)
* update connector builder docs for UI MVP release

* remove note about separately declaring schemas
2023-01-06 13:01:34 -08:00
Maxime Carbonneau-Leclerc
4f0aca536c [ISSUE #20322] add datetime_granularity logic to DatetimeStreamSlicer… (#20717)
* [ISSUE #20322] add datetime_granularity logic to DatetimeStreamSlicer and migrate duration to ISO8601

* [ISSUE #20322] fix tests

* [ISSUE #20322] code review based on clnoll's comments and fixed tests

* [ISSUE #20322] fix flake8 error

* [ISSUE #20322] fix source tests

* [ISSUE #20322] fixing yet another error in source

* [ISSUE #20322] code review

* [ISSUE #20322] adding new sources using datetime slicer

* [ISSUE #20322] fixing source-datascope and increasing version

* [ISSUE #20322] regenerate component schema

* [ISSUE #20322] fixing source-datascope

* [ISSUE #20322] extra field error
2023-01-06 12:29:37 -05:00
Rodi Reich Zilberman
06ef1e90c6 source-postgres: enable SAT high test strictness level (#20549) 2023-01-04 09:08:43 +00:00
Evan Tahler
e39e1898c5 Expected Records to .jsonl format (#20850)
* Expected Records to `.jsonl` format

* fix formatting template

* remove endline

* update templates

* Update docs/connector-development/testing-connectors/source-acceptance-tests-reference.md

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
2023-01-03 15:55:36 -08:00
Brian Lai
b7113a2e5e [Low-Code CDK] Write the component schema and use it during manifest validation (#20422)
* handwritten low code manifest example components

* add MinMaxDatetime to jsonschema

* add a basic gradle command to generate manifest components

* Add auth components to handwritten component schema

- ApiKeyAuthenticator
- BasicHttpAuthenticator
- BearerAuthenticator
- DeclarativeOauth2Authenticator
- NoAuth

* Respect optional properties in DeclarativeOauth2Authenticator

* Fix `Dict[str, Any]` mapping in auth components

* add default error handler composite error handler and http response filter components

* [low code component schema] adding backoff strategies to schema

* [low code component schema] fix float types

* [low code component schema] add RecordFilter

* Remove `config` from auth components

* [low code component schema] add Interpolation (with pending question on 'type' not being defined)

* Add CartesianProductStreamSlicer & DatetimeStreamSlicer

* Add ListStreamSlicer, and fix nesting of DatetimeStreamSlicer

* [low code component schema] add InterpolatedRequestOptionsProvider

* Add slicer components, and fix a couple of components after reviewing output

* [low code component schema] adding transformations and adding type to interpolators

* adding spec and a few small tweaks

* Add DefaultSchemaLoader

* [low code component schema] attempt on custom class

* Add descriptions for auth components

* add RequestOption

* remove interpolated objects from the schema in favor of strings only

* a few schema fixes and adding some custom pagination and stream slicer

* [low code component schema] fix CustomBackoffStrategy

* Add CustomRecordExtractor

* add some description and add additional properties

* insert a transformer to hydrate default manifest components and perform validation against the handwritten schema

* [low code component schema] validating existing schemas

* [low code component schema] clean validation script

* add manifest transformer tests and a few tweaks to the schema

* Revert "[low code component schema] clean validation script"

This reverts commit 2408f41cf6.

* Revert "[low code component schema] validating existing schemas"

This reverts commit 9d39977815.

* [low code component schema] integrate validation script to gradle

* [low code component schema] updating validation script permissions

* remove a few model gen spike files and clean up comments

* default types should take parent type into account and a few schema changes

* add inline schema components and fix validation bugs

* add types to every component's literal enum and more little schema fixes

* missing component prefix

* pr feedback, renaming the schema and updating various docs

* fixing schema bugs from testing and new components

* last few small tweaks and fixes and fix the script

* bump the version before publsih

* include the declarative_component_schema.yaml as package data so its included when package is published

* roll the version to publish

Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
2022-12-19 14:42:34 -06:00
Nicolas MOREAU
c637aed1aa Typo fix in CDK docs (#20446) 2022-12-13 21:15:48 -03:00
Catherine Noll
9dae0986f9 Enable low-code CDK users to specify schema in the manifest (#20375)
Enable low-code CDK users to specify schema in the manifest

Also update documentation:
* Add inline schema loader info to yaml-overview.md
* Include inline schema info in tutorial
2022-12-12 23:44:11 -05:00
Philipp Wallhäuser
c4c26c0adc fix wrong indentation in error handling docs (#20288) 2022-12-09 15:16:42 -03:00
Julian
dc71194e39 Fix: Tutorial config-based invalid keywords #20071 (#20078)
* Update 5-incremental-reads.md in response of Issue #20047

Added the suggested solution from issuer.

* Updated 5-incremental-reads.md #2

Appended the suggested +00:00 into another occurence. Datetime format now corresponds to standards defined in ISO-8601.

* Fix: Low Code Connector tutorial yaml #20071
2022-12-08 14:33:18 -03:00
Alexandre Girard
61ddba697e Alex/lowcode custom components docs (#20176)
* Update label

* reset

* update

* update

* update

* Update

* update
2022-12-06 19:01:54 -08:00
Luke Hsiao
84ab12f93d Docs: Update building-a-python-destination.md (#19516)
If a user used `append_dedupe`, they get the error:

```
{
  "type": "LOG",
  "log": {
    "level": "FATAL",
    "message": "1 validation error for ConnectorSpecification\nsupported_destination_sync_modes -> 2\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])..."
  }
}
```
2022-11-29 17:42:36 -03:00
Augustin
b78fdadaf2 SAT: optionally disable discovered catalog caching (#19806) 2022-11-29 13:41:45 +01:00
Andy
0b6999993d docs: Include alpha call out in lowcode builder UI docs (#19841)
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-11-28 10:51:48 -08:00
Lake Mossman
f53bbb3c24 Docs: fix broken connector builder UI docs links (#19631) 2022-11-18 18:03:47 -08:00
Sherif Nada
7598ca0b34 fix docs build 2022-11-18 11:13:24 -08:00
Sherif A. Nada
f5d8e25519 Add connector builder UI docs (#19588) 2022-11-18 10:56:30 -08:00
Amruta Ranade
17932faebd Update low-code-cdk-overview.md 2022-11-16 14:02:12 -05:00
Sherif A. Nada
88da35c6f9 Fix callout formatting in CDK docs 2022-11-16 08:14:08 -08:00
Andy
56342d28cd Lowcode CDK docs: Update with CTA for creating feature requests (#19462) 2022-11-15 21:25:01 -08:00
JJ Nilbodee
33227f5812 Allows user to specify grant_type in OAuthAuthenticator (#19139)
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-11-10 22:22:17 -08:00
Denys Davydov
459856b73c #2576 oncall. SAT: test spec against exposed secrets (#19124)
* #2576 oncall. SAT: test spec against exposed secrets

* #2576 upd changelog

* #2576 review fixes

* #2576 more test fixes
2022-11-10 23:31:04 +02:00
Ella Rohm-Ensing
e73f535261 Add SAT test for no duplicates in enum properties (#19243)
* Make pytest aware of backward compatibility mark

* Unit tests for duplicate values in enum properties

* Add docs about airbyte enforcements to enum usage

* Add docs info to test
2022-11-09 15:47:05 -05:00
Sherif A. Nada
5bf211a796 docs: Add note about fields which don't have order field (#19189) 2022-11-08 15:23:19 -08:00
Ella Rohm-Ensing
f22485a908 Add improvements to "Building a connector the hard way" (#19093)
* Add improvements to "Building a connector the hard way"

* add log_error() formatting to pass SAT tests

* Update to new version of acceptance-test-config.yml

* Edit tutorial directory to match tutorial

* Change permissions on acceptance-test-docker.sh

This reverts commit 40b2d98053.
2022-11-08 17:13:21 -05:00
Augustin
a196cf712c SAT: make future_state configuration mandatory in high test strictness level (#19085) 2022-11-08 23:10:22 +01:00
Augustin
7d73b08606 SAT: basic read on full catalog when test_strictness_level == high (#18937) 2022-11-07 08:09:34 +01:00
Alexandre Girard
605fb921c4 [low-code]: Evaluate backoff strategies at runtime (#18053)
* pass options to wait time from header

* fix constant backoff

* parameterize test

* fix tests

* missing unit tests

* eval header at runtime

* eval regex at runtime

* evaluate min_wait at runtime

* eval factor at runtime

* missing unit tests

* remove debug print

* rename

* Add tests

* Add tests

* Update docs
2022-11-03 18:50:47 -07:00
Sherif A. Nada
f4e3b605c9 fix typo in lowcode tutorial (#18916) 2022-11-03 09:18:37 -07:00
Scott Lee Chua
926a26a02e Remove duplicate paragraph (#18662)
Removes a duplicate of the paragraph that begins "Note that we're also setting the `stream_cursor_field` in the stream's `$options`..." which followed an outdated schema.
2022-10-31 08:35:45 -07:00
Augustin
c25ac8a748 SAT: make expect_records mandatory in high test_strictness_level (#18497) 2022-10-28 11:45:29 +02:00
JC (Jonathan Chen)
042300751a docs: fix link reference (#18503) 2022-10-26 17:08:21 -03:00
Augustin
726d3bde4d SAT: enforce bypass_reason declaration on empty_streams when test_strictness_level == high (#18425) 2022-10-25 22:31:16 +02:00
Augustin
0865c62ee6 SAT: make all tests mandatory when test_strictness_level == high (#18414) 2022-10-25 21:45:12 +02:00
Pinhas Keizman
4a61e6aaa3 Update cdk-speedrun.md (#18416) 2022-10-25 08:31:56 -03:00
Augustin
e933de02e8 SAT: declare bypass_reason in acceptance-test-config.yml (#18364) 2022-10-25 08:52:35 +02:00
Tanvesh Takawale
da5aed8a31 CDK: fixed document links regarding airbyte specification (#18169) 2022-10-19 09:50:01 -07:00
Delena Malan
4b0d1d4ac6 Replace NoPaginator in the docs with NoPagination (#18142) 2022-10-19 09:20:40 -04:00
Delena Malan
3b12a585ec Docs: Fix backoff stategy docs (#18143)
* Fix backoff strategy docs

* Update config_component_schema.json
2022-10-18 16:28:38 -07:00
Amin
3f2af7d8e1 Update 4-reading-data.md (#18037) 2022-10-17 09:37:47 -03:00
Delena Malan
289a065401 Use page_token_option instead of page_token (#17892) 2022-10-13 19:00:20 -03:00
Delena Malan
6eb3893f14 Remove typo line from incremental reads docs (#17920) 2022-10-13 13:07:56 -03:00
Subham Sahu
2247038a52 DocS: Update authentication.md (#17931) 2022-10-13 13:07:35 -03:00
Amruta Ranade
57571ee94f Fixed broken links 2022-10-13 09:11:57 -05:00
Alexandre Girard
39220f4b1e Update lowcode docs (#17752)
* Update lowcode docs

* Update

* fix links

* more links

* Schemas

* indent

* Update docs/connector-development/config-based/advanced-topics.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/advanced-topics.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/low-code-cdk-overview.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/request-options.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/requester.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/stream-slicers.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/requester.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/error-handling.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* capitalization

* fix schema

* fix schema

* small changes so greenhouse validates

* fix brackets

* update code samples

* Add missing types

* fix

* reset

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
2022-10-07 19:15:27 -07:00
Alexandre Girard
141803a78a Fix link in docs (#17670) 2022-10-06 08:30:29 -07:00
Alexandre Girard
65be9b74f0 Simplify low-code connectors template (#17476)
* default schema loader

* Update generator

* update example

* remove more cruft

* Update code sampels in section 3 (but not the text)

* Update code sampels in 5

* Update instructions

* Update code samples

* Update samples

* Use now_utc instead of local

* isolate complexity to JsonSchema

* Bump cdk version

* Update changelog

* fix unit test

* slightly more helpful default

* Don't freeze cdk version in template

* remove newline

* newline

* bump

* Update changelog

* Bump
2022-10-06 08:12:21 -07:00
Amruta Ranade
66dafd85aa Updated Slack link 2022-10-03 19:39:40 -04:00