1
0
mirror of synced 2026-01-15 15:06:14 -05:00
Commit Graph

15 Commits

Author SHA1 Message Date
Alexandre Girard
18da341b91 Revert: #17047 Airbyte CDK: Improve error for returning non-iterable from connectors parse_response (#17707)
* Bump cdk version

* Revert "#17047 Airbyte CDK: Improve error for returning non-iterable from connectors parse_response (#17626)"

This reverts commit d9ad2725b1.

* Bump
2022-10-06 18:43:01 -07:00
Robert
d9ad2725b1 #17047 Airbyte CDK: Improve error for returning non-iterable from connectors parse_response (#17626)
* Improve airbyte cdk invalid message data type error message

* Test cdk invalid message data type custom error is raised

* Fix test to pass stream as a string

* Add valid record message data input type test

* Add object type and value to AirbyteRecordMessage validator message

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
2022-10-06 16:46:33 -07:00
Pedro S. Lopez
fe2238f3f7 🐞 CDK: properly emit state on empty slices when using iterators (#17296)
* fix: emit state on empty slices iterator

* bump version / update changelog

* format
2022-09-28 02:12:04 -04:00
Brian Lai
f283652313 [per-stream cdk] Emit per-stream state messages (#16628)
* update abstract source and connector state manager to update shared and stream state

* clean up some extra comments and extra lines

* a few changes based on PR feedback

* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob

* rename a few paramets and add testing for state update via stream.state override

* replace shared_state processing with an explicit error and fix a few comments and pr feedback

* add some polish and additional test cases

* Add create_state_message to manager and emit per-stream state for connectors

* pr feedback and restructuring parts of the connector state manager initialization

* fix unfinished comment

* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* use pytest params to annotate tests better

* change to fix changed class name

* format test params a little better

* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* a few bits of pr feedback

* do not emit namespace in the outbound message if it is None

* fix some issues w/ prior merge from master and pr feedback

* add new tests to ensure we never send None namespace to the platform

* bump CDK version number

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-09-15 23:42:49 -04:00
Brian Lai
db56c75f03 [per-stream cdk] Allow for reading in per-stream state and passing it to connectors (#16505)
* update abstract source and connector state manager to update shared and stream state

* clean up some extra comments and extra lines

* a few changes based on PR feedback

* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob

* rename a few paramets and add testing for state update via stream.state override

* replace shared_state processing with an explicit error and fix a few comments and pr feedback

* add some polish and additional test cases

* pr feedback and restructuring parts of the connector state manager initialization

* fix unfinished comment

* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* use pytest params to annotate tests better

* change to fix changed class name

* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* a few bits of pr feedback

* pr feedback and cleaning up some comments and variable renames

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-09-15 15:55:46 -04:00
Alexandre Girard
79a54a81fd Emit a state message even if no records were read (#15067)
* Emit a state message even if no records were read

* newline

* merge

* comment

* implement logic in the abstract source

* remove logic from declarative source

* comment

* bump cdk version
2022-08-04 12:48:21 -07:00
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Pedro S. Lopez
41dc82f056 CDK: provide better user-friendly error messages (#12944)
* initial implementation

* add parse_response_error_message tests

* move error handler to existing try/catch in AbstractSource

* formatting

* var rename

* use isinstance for type checking

* add docstrings

* add more abstract_source and httpstream tests

* fix wrong httperror usage

* more test cases

* bump version, update changelog
2022-05-19 17:28:24 -04:00
Eugene Kulak
d173ce741f CDK: add support for streams with state attribute (#9746)
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2022-02-17 00:20:33 +04:00
Eugene Kulak
f83eca58ea CDK: Fix typing errors (#9037)
* fix typing, drop AirbyteLogger

* format

* bump the version

* use logger instead of fixture logger

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: auganbay <auganenu@gmail.com>
2022-01-14 10:29:34 +06:00
LiRen Tu
c511abd20c Fix master build (#7873)
* Check in format diff

* Remove cache http yaml files

* Ignore tmp files generated by unit tests
2021-11-11 09:47:51 -08:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Davin Chia
9460cb4422 Update all of Pydantic to 1.6.2 per Dependabot. (#3408)
* Update all to 1.6.2.

* Publish new airbyte-cdk version.

* Use repr instead of str for exceptions.

* Use rc.

* Edit test.

* Bump for SAT.

* Format.
2021-05-14 16:44:49 +08:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Sherif A. Nada
184dab77eb CDK: overhaul directory structure (#3295) 2021-05-09 15:27:38 -07:00