1
0
mirror of synced 2026-01-04 18:04:31 -05:00
Commit Graph

4 Commits

Author SHA1 Message Date
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594)
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
2023-02-08 13:01:16 -08:00
Brian Lai
f283652313 [per-stream cdk] Emit per-stream state messages (#16628)
* update abstract source and connector state manager to update shared and stream state

* clean up some extra comments and extra lines

* a few changes based on PR feedback

* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob

* rename a few paramets and add testing for state update via stream.state override

* replace shared_state processing with an explicit error and fix a few comments and pr feedback

* add some polish and additional test cases

* Add create_state_message to manager and emit per-stream state for connectors

* pr feedback and restructuring parts of the connector state manager initialization

* fix unfinished comment

* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* use pytest params to annotate tests better

* change to fix changed class name

* format test params a little better

* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* a few bits of pr feedback

* do not emit namespace in the outbound message if it is None

* fix some issues w/ prior merge from master and pr feedback

* add new tests to ensure we never send None namespace to the platform

* bump CDK version number

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-09-15 23:42:49 -04:00
Brian Lai
db56c75f03 [per-stream cdk] Allow for reading in per-stream state and passing it to connectors (#16505)
* update abstract source and connector state manager to update shared and stream state

* clean up some extra comments and extra lines

* a few changes based on PR feedback

* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob

* rename a few paramets and add testing for state update via stream.state override

* replace shared_state processing with an explicit error and fix a few comments and pr feedback

* add some polish and additional test cases

* pr feedback and restructuring parts of the connector state manager initialization

* fix unfinished comment

* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* use pytest params to annotate tests better

* change to fix changed class name

* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* a few bits of pr feedback

* pr feedback and cleaning up some comments and variable renames

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-09-15 15:55:46 -04:00
Brian Lai
1d9608cbbe [per-stream cdk] Support deserialization of legacy and per-stream state (#16205)
* interpret legacy and new per-stream format into AirbyteStateMessages

* add ConnectorStateManager stubs for future work

* remove frozen for the time being until we need to hash descriptors

* add validation that AirbyteStateMessage has at least one of stream, global, or data fields

* pr feedback and clean up of the code

* remove changes to airbyte_protocol and perform validation in read_state()

* fix import formatting
2022-09-07 13:20:14 -04:00