1
0
mirror of synced 2026-01-17 12:07:50 -05:00
Commit Graph

12 Commits

Author SHA1 Message Date
Brian Lai
f283652313 [per-stream cdk] Emit per-stream state messages (#16628)
* update abstract source and connector state manager to update shared and stream state

* clean up some extra comments and extra lines

* a few changes based on PR feedback

* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob

* rename a few paramets and add testing for state update via stream.state override

* replace shared_state processing with an explicit error and fix a few comments and pr feedback

* add some polish and additional test cases

* Add create_state_message to manager and emit per-stream state for connectors

* pr feedback and restructuring parts of the connector state manager initialization

* fix unfinished comment

* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* use pytest params to annotate tests better

* change to fix changed class name

* format test params a little better

* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* a few bits of pr feedback

* do not emit namespace in the outbound message if it is None

* fix some issues w/ prior merge from master and pr feedback

* add new tests to ensure we never send None namespace to the platform

* bump CDK version number

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-09-15 23:42:49 -04:00
Brian Lai
db56c75f03 [per-stream cdk] Allow for reading in per-stream state and passing it to connectors (#16505)
* update abstract source and connector state manager to update shared and stream state

* clean up some extra comments and extra lines

* a few changes based on PR feedback

* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob

* rename a few paramets and add testing for state update via stream.state override

* replace shared_state processing with an explicit error and fix a few comments and pr feedback

* add some polish and additional test cases

* pr feedback and restructuring parts of the connector state manager initialization

* fix unfinished comment

* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* use pytest params to annotate tests better

* change to fix changed class name

* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* a few bits of pr feedback

* pr feedback and cleaning up some comments and variable renames

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-09-15 15:55:46 -04:00
Alexandre Girard
79a54a81fd Emit a state message even if no records were read (#15067)
* Emit a state message even if no records were read

* newline

* merge

* comment

* implement logic in the abstract source

* remove logic from declarative source

* comment

* bump cdk version
2022-08-04 12:48:21 -07:00
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Pedro S. Lopez
41dc82f056 CDK: provide better user-friendly error messages (#12944)
* initial implementation

* add parse_response_error_message tests

* move error handler to existing try/catch in AbstractSource

* formatting

* var rename

* use isinstance for type checking

* add docstrings

* add more abstract_source and httpstream tests

* fix wrong httperror usage

* more test cases

* bump version, update changelog
2022-05-19 17:28:24 -04:00
Eugene Kulak
d173ce741f CDK: add support for streams with state attribute (#9746)
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2022-02-17 00:20:33 +04:00
Eugene Kulak
f83eca58ea CDK: Fix typing errors (#9037)
* fix typing, drop AirbyteLogger

* format

* bump the version

* use logger instead of fixture logger

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: auganbay <auganenu@gmail.com>
2022-01-14 10:29:34 +06:00
LiRen Tu
c511abd20c Fix master build (#7873)
* Check in format diff

* Remove cache http yaml files

* Ignore tmp files generated by unit tests
2021-11-11 09:47:51 -08:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Davin Chia
9460cb4422 Update all of Pydantic to 1.6.2 per Dependabot. (#3408)
* Update all to 1.6.2.

* Publish new airbyte-cdk version.

* Use repr instead of str for exceptions.

* Use rc.

* Edit test.

* Bump for SAT.

* Format.
2021-05-14 16:44:49 +08:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Sherif A. Nada
184dab77eb CDK: overhaul directory structure (#3295) 2021-05-09 15:27:38 -07:00