jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-01 09:02:59 -05:00
Code Issues Packages Projects Releases Wiki Activity
8,635 Commits 5,135 Branches 518 Tags
b4dfbdb67006d028c8fdbe0dfd2305273fa6df0d
Commit Graph

140 Commits

Author SHA1 Message Date
Serhii Chvaliuk
d5b66e9d7b CDK: make default_paginator.page_token_option optional (#19368) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-11-14 20:16:30 +02:00
Serhii Chvaliuk
5cd88e7c05 CDK: don't filter failed interpolated vars for request_options_provider.request_body_json (#19297) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-11-11 10:44:23 +02:00
Brian Lai
60c008f6cd don't update cursor for log messages and and default schema path coming from connector builder (#19271) 
* don't update cursor for log messages and and default schema path coming from connector builder

* replace check for connector_builder module with a basic default file path

* update changelog and patch version

* catch the correct exception when pkgutil can't load json file
 2022-11-10 19:23:05 -05:00
Brian Lai
f54f4af43f bump cdk version (#19255) 2022-11-09 17:53:29 -05:00
Brian Lai
5c9e5d9934 [low-code cdk] decouple parsing the yaml manifest from the declarative source implementation (#19095) 
* decouple parsing the yaml manifest from the declarative source implementation

* bump version and changelog
 2022-11-08 15:45:01 -05:00
Brian Lai
f9863d6374 [low-code cdk] Allow for spec file to be defined in the yaml manifest instead of an external file (#18411) 
* allow for spec to be defined in the source.yaml manifest instead of an external file

* make spec a component within the language to get schema validation and rework the code for better testing

* fix formatting and extra method

* pr feedback and add some more test

* pr feedback

* bump airbyte-cdk version

* bump version

* gradle format

* remove  from manifest spec
 2022-11-07 14:44:45 -05:00
Alexandre Girard
3ff3e9aeee Bump CDK version (#19033) 
* bump

* bump to 0.6.0
 2022-11-07 09:27:09 -08:00
Xabier Lahuerta Vazquez
aa5da75d81 Add support for monthly and yearly incremental updates (#18861) 
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2022-11-07 07:49:57 -08:00
Denys Davydov
8145be55a3 Low-code CDK: safe get response.json (#18931) 
* low-code cdk: safe get response.json

* flake fix
 2022-11-04 11:44:51 +02:00
Brian Lai
186580a6ee [low-code] replace emptySchemaLoader with DefaultSchemaLoader (#18947) 
* replace emptySchemaLoader with DefaultSchemaLoader

* fix test name

* fix test

* add logging for when we default to the empty schema

* increment patch version

* fix formatting

* update changelog
 2022-11-03 23:25:01 -04:00
Brian Lai
dd7184dbca [low-code cdk] Allow for read command to be run on low code connector streams w/o a schema file (#18532) 
* use mockschemaloader as the default and rename JsonSchema to JsonFileSchemaLoader

* rename mock to EmptySchemaLoader

* retain existing JsonSchema while hacktoberfest is still in progress

* bump version and changelog
 2022-11-03 12:05:04 -04:00
Alexandre Girard
cd1f492140 🐛 Lowcode: ListStreamSlicer and SubstreamSlicer should get the stream_slice from the arguments (#18574) 
* fix list stream slicer

* get the stream slice from the parameters

* delete unused code

* Add comments

* missing test

* newline

* with pytest.raise

* reset to master

* newline

* Update __init__.py

* Update __init__.py

* 🎉 New Source: ConvertKit (#18455)

* Init source omnisend

* Removed unnecessary files

* Init source convertkit

* Added forms, sequences streams

* Added tags stream

* Added subscribers, broadcasts streams

* Added documentation

* Removed unnecessary files

* Updated pull request information in documentation

* Added sample config

* Updated invalid config

* Formatting, removed abnormal state, uncommented full_refresh acceptance test

* Added pagination for subscribers stream

* fix: add source definition for convertkit

* auto-bump connector version

Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

* 🎉New Source: Google Webfonts [low-code cdk] (#18496)

* New source: Google webfonts

* chore: Add Docs

* chore: update changelog

* chore: resolved given comments for PR

* chore: unwanted files removed

* fix: generate and add source definitions

Co-authored-by: sajarin <sajarindider@gmail.com>

* Clean up build.gradle. (#18555)

Upstream has fixed this bug so we no longer need to host it internally. Remove this to clean up build.gradle.

* 🎉 New Destination: Typesense (#18349)

* Initial boilerplate

* 🎉 New Destination: Typesense

* remove .java-version

* fix doc

* add typesense to dest def

* add release stage

* add requirement to main

* auto-bump connector version

* add changelog

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

* 🎉 New Source: Waiteraid [low-code cdk] (#18165)

* added source waiteraid

* 🎉 New Source: Waiteraid

* add searchBookings stream

* add P/R number

* add SUMMARY entry

* add docs/integrations/README.md entry

* add builds.md entry

* add docs to each endpoint

* fix schema
~

* Update airbyte-integrations/builds.md

added web address

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* Delete catalog.json

* Delete TODO.md

* Update spec.yaml

* add waiteraid to source def seed

* auto-bump connector version

Co-authored-by: Sebastian Brickel <sebastianbrickel@Sebastians-MacBook-Air.local>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

* Replace `recipesLink` link with `tutorialsLink` (#18616)

* Protocol Change: `AirbyteControlMessage.ConnectorConfig` (#17907)

* Protocol Change: AirbyteConfigMessage

* update PR link in docs

* Lint

* Update python files

* Update docs/understanding-airbyte/airbyte-protocol.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/understanding-airbyte/airbyte-protocol.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* `AirbyteConfigMessage` -> `AirbyteConnectorConfigMessage`

* AirbyteOrchestratorMessage

* Update docs

* `AirbyteControlConnectorConfigMessage`

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Remove workflow version check (#18613)

* Improved the Oracle cloud deployment guide (#18615)

* Update OCI VM deployment guide

* Update OCI deployment guide

* Update on oci-vm.md

* bump

Co-authored-by: Dainius Salkauskas <dainiuxazz@gmail.com>
Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Balasubramanian T K <btk.codedev@gmail.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Cirdes <cirdes@linkana.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: sbrickel-nimble <113671803+sbrickel-nimble@users.noreply.github.com>
Co-authored-by: Sebastian Brickel <sebastianbrickel@Sebastians-MacBook-Air.local>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io>
Co-authored-by: Victor Ikechukwu <vickyikechukwu13@gmail.com>
 2022-10-28 13:09:19 -07:00
Alexandre Girard
1ddc8c6185 Lowcode connectors: Fix few issues with stream slicers (#18529) 
* Fix a few bugs

* bump
 2022-10-27 13:33:13 -07:00
Brian Lai
5222093b54 support custom error messaging for error response + retryable errors (#18204) 
* support custom error messaging for error response + retryable errors

* remove changed backoff i was using for testing

* refactor filter to construct response status internally

* pr feedback

* bump version and update changelog
 2022-10-26 15:39:36 -04:00
Sherif A. Nada
25e7a37de6 Publish new CDK version with typehints exported (#18398) 2022-10-25 09:02:06 -07:00
Nipunn Koorapati
d77f913373 Export python type information for airbyte-cdk (#18387) 
Fixes #18384
 2022-10-24 12:20:43 -07:00
Alexandre Girard
76acfb8192 [low-code] Propagate options to InterpolatedRequestInputProvider (#18050) 
* properly propagate options

* cleanup

* turn into dataclass

* rename

* no need for deepcopy

* fix test

* bump

* cleaner
 2022-10-21 10:16:25 -07:00
Pedro S. Lopez
ad29513a43 CDK: Handle config validation errors as config_error and failed check status (#18214) 
* schema helpers raises a config_error

* traced exceptions can be turned into connection status messages

* add tests for schema helpers

* return a failed status message rather than throwing in check command

* remove unused imports

* add comment

* bump version / update changelog
 2022-10-20 13:23:06 -04:00
Brian Lai
de24ddced5 make sure stream_slices is an iterator (#18092) 
* ensure output from stream_slices() is always an iterator

* bump cdk version

* bump version after merging latest cdk

* bump cdk version post rebase
 2022-10-19 15:37:00 -04:00
Serhii Chvaliuk
258b23cf79 CDK: VCR -> requests_cache + SQLite (#17990) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-10-19 19:31:28 +03:00
Augustin
b564f3eb78 Protocol: make supported_sync_modes a required not empty list on AirbyteStream (#15591) 2022-10-19 15:22:25 +02:00
Alexander Marquardt
26b6317198 Added new "filters" python file, along with a "hash" filter. This can… (#18000) 
* Added new "filters" python file, along with a "hash" filter. This can be extended to include other custom filters in the future.

* Added additional comments

* Moved usage of the hash_obj inside the conditional that confirms it exists

* Moved the hash function call inside a condition to ensure that it exists

* Fixed the application of the salt , so that it does not modify the hash unless it is actually passed in.

* Added unit tests to validate new jinja hash functionality

* Updated unit test to pass numeric value as a float instead of string

* Removed unreferenced import to pytest

* Updated version

* format

* format

* format

* format

* format

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2022-10-17 23:19:55 -07:00
Brian Lai
4714a1c004 fix check for streams that do not use a stream slicer (#18080) 
* fix check for streams that do not use a stream slicer

* increment version and changelog before publish
 2022-10-17 17:11:11 -04:00
Alexandre Girard
0ffd503b21 🐛 [low-code] $options shouldn't overwrite values that are already defined (#18060) 
* fix

* Add missing test

* remove prints

* extract to method

* rename

* Add missing test

* rename

* bump
 2022-10-17 11:07:04 -07:00
Alexandre Girard
df72bbdf2c Low-code: Pass stream_slice to read_records when reading from CheckStream (#17804) 
* Implement a test

* Implement fix

* rename

* extract method

* bump
 2022-10-17 05:47:56 -07:00
Alexandre Girard
62500af93b get module name from sys.modules (#17779) 
* get module name from sys.modules

* bump

* fix comment

* throw exception

* fix unittests

* Add missing files

* remove debug prints

* indent
 2022-10-10 13:54:09 -07:00
Alexandre Girard
89a9d6497a Add missing types to the registry (#17763) 
* Add missing types to the registry

* bump
 2022-10-09 01:25:38 -07:00
Alexandre Girard
18da341b91 Revert: #17047 Airbyte CDK: Improve error for returning non-iterable from connectors parse_response (#17707) 
* Bump cdk version

* Revert "#17047 Airbyte CDK: Improve error for returning non-iterable from connectors parse_response (#17626)"

This reverts commit d9ad2725b1.

* Bump
 2022-10-06 18:43:01 -07:00
Alexandre Girard
20ae75e4ee 🐛 : Add missing component types to the types registry (#17700) 
* Add page increment to type registry

* Add page increment to type registry
 2022-10-06 15:01:58 -07:00
Alexandre Girard
65be9b74f0 Simplify low-code connectors template (#17476) 
* default schema loader

* Update generator

* update example

* remove more cruft

* Update code sampels in section 3 (but not the text)

* Update code sampels in 5

* Update instructions

* Update code samples

* Update samples

* Use now_utc instead of local

* isolate complexity to JsonSchema

* Bump cdk version

* Update changelog

* fix unit test

* slightly more helpful default

* Don't freeze cdk version in template

* remove newline

* newline

* bump

* Update changelog

* Bump
 2022-10-06 08:12:21 -07:00
Alexandre Girard
3bdd7f4f43 Standardize HttpRequester's url_base and path format (#17524) 
* add trailing slash to base url if missing

* remove leading / from path

* Bump
 2022-10-05 15:30:37 -07:00
Alexandre Girard
61a5a1e967 [#17107] Fix propagation of $options field in arrays (#17196) 
* input can only be a string

* remove debug print

* cleanup tests

* fix in the factory

* Add a unit test

* fix for lists

* Update changelog

* Update changelog
 2022-10-03 17:02:41 -07:00
Alexandre Girard
d6b79b8e52 Alex/limit paginator (#17197) 
* remove page size from limit paginator

* Rename LimitPaginator to DefaultPaginator

* rename limit_options

* rename method

* cleanup

* comment

* update schema file

* Update changelog

* pin greenhouse connector

* fix doc in comment

* fix changelog

* missing -

* bump

* bump
 2022-09-29 19:24:53 -07:00
Pedro S. Lopez
58f5128adb 🐞 CDK: fix error when logging warning for types that could not be transformed (#17367) 
* fix error when logging warning for types that could not be transformed

* changelog and bump
 2022-09-29 18:48:53 -04:00
Pedro S. Lopez
fe2238f3f7 🐞 CDK: properly emit state on empty slices when using iterators (#17296) 
* fix: emit state on empty slices iterator

* bump version / update changelog

* format
 2022-09-28 02:12:04 -04:00
Serhii Chvaliuk
a82f59ec1e CDK: Evaluate response.text only in debug mode (#16809) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-09-17 12:12:03 +03:00
Brian Lai
f283652313 [per-stream cdk] Emit per-stream state messages (#16628) 
* update abstract source and connector state manager to update shared and stream state

* clean up some extra comments and extra lines

* a few changes based on PR feedback

* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob

* rename a few paramets and add testing for state update via stream.state override

* replace shared_state processing with an explicit error and fix a few comments and pr feedback

* add some polish and additional test cases

* Add create_state_message to manager and emit per-stream state for connectors

* pr feedback and restructuring parts of the connector state manager initialization

* fix unfinished comment

* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* use pytest params to annotate tests better

* change to fix changed class name

* format test params a little better

* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* a few bits of pr feedback

* do not emit namespace in the outbound message if it is None

* fix some issues w/ prior merge from master and pr feedback

* add new tests to ensure we never send None namespace to the platform

* bump CDK version number

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2022-09-15 23:42:49 -04:00
Serhii Chvaliuk
0d55a895cb CDK: TypeTransformer - warning message more informative (#16695) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-09-15 10:24:37 +03:00
Serhii Chvaliuk
5a3b6d82b6 CDK: make TypeTransformer more robust to incorrect incoming records (#16544) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-09-13 10:30:16 +03:00
Brian Lai
6253406de9 forgot to also emit dictionary instead of empty lists when state not specified (#16569) 2022-09-12 17:45:48 -04:00
Brian Lai
037e8ed1a9 fix cdk bug to send legacy format if connector overrides read() (#16566) 
* fix cdk bug to send legacy format if connector overrides read()

* fix comment

* update changelog and setup.py
 2022-09-09 21:09:50 -04:00
Alexandre Girard
0cb44ca071 release cdk with frozen dataclasses-jsonschema lib (#16525) 2022-09-09 07:03:18 -07:00
Brian Lai
dceeef4683 [cdk] pin dataclasses-jsonschema to 2.15.1 (#16253) 2022-09-01 21:08:32 -04:00
Serhii Chvaliuk
3cfa489234 CDK: Fix regression in _checkpoint_state arg (#16141) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-08-31 22:39:30 +03:00
Brian Lai
10a2bd1d3c [low code connectors] add NoAuth to class types registry (#15939) 
* add no auth to class type registry

* NoAuth should receive options and fail normally

* forgot to pass options in

* update changelog
 2022-08-25 18:06:02 -04:00
Brian Lai
09cddffd36 [low code connectors] replace file retrieval with pkgutil to fix getting schema files (#15814) 
* replace file retrieval with pkgutil to fix getting schema files

* slightly better error handling on missing files

* filter our schema gen warnings for some classes that cannot generate schemas

* add comment for todo

* add changelog and setup before publish
 2022-08-23 03:06:20 -04:00
Brian Lai
ca6513625d [low code connectors] read configs from package_data (#15810) 
* read configs from package_data

* update changelog and setup

* commenting out failing tests in the short term
 2022-08-19 21:16:20 -04:00
Brian Lai
ca80d3782a [low code connectors] perform schema validation of the input config against the declarative language schema (#15543) 
* draft: first pass at complete schema language generation and factory validator

* actually a working validator and fixes to the schema that went uncaught

* remove extra spike file

* fix formatting file

* pr feedback and a little bit of refactoring

* fix some types that were erroneously marked as invalid schema

* some comments

* add jsonschemamixin to interfaces

* update changelog

* bump version
 2022-08-18 15:29:26 -04:00
Alexandre Girard
313ac11e6d [low-code connectors] Get parent stream's full slice (#15631) 
* always access parent stream using full_refresh mode

* Update test

* fix substream slicer

* bump
 2022-08-18 10:24:10 -07:00
Serhii Chvaliuk
4e6cb05759 CDK: Improve filter_secrets skip empty string (#15684) 
* Improve `filter_secrets` skip empty string

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-08-16 19:28:57 +03:00