1
0
mirror of synced 2025-12-26 05:05:18 -05:00
Commit Graph

43 Commits

Author SHA1 Message Date
Artem Inzhyyants
df34893b63 feat(airbyte-cdk): replace pydantic BaseModel with dataclasses + serpyco-rs in protocol (#44444)
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
2024-09-02 17:48:17 +02:00
Brian Lai
fca0460030 [airbyte-cdk] tech-debt Remove support for parsing legacy state message format (#43459) 2024-08-16 21:06:37 -04:00
Brian Lai
a43b44d776 [RFR for API Sources] fix issue where full refresh state is passed by regression tester (#42986) 2024-08-03 02:10:29 -04:00
Aldo Gonzalez
7a41aebf0d 🐛 bug(Airbyte CDK): fix case where stream wont have a state attribute and needs to resolve get_updated_state (#42083) 2024-07-29 08:57:54 -06:00
Brian Lai
b2d53f552d [RFR for Python Sources] Make it easier for Python sources to automatically use RFR for eligible streams (#39450) 2024-07-18 01:17:14 -04:00
Serhii Lazebnyi
bc60a740a2 [airbyte-cdk] add incomplete stream status to nonexistent stream handling (#40568) 2024-07-10 13:10:30 +02:00
Brian Lai
29d615080a [airbyte-cdk] Fix a bug so that successful Python RFR streams are not synced on subsequent attempts (#38608) 2024-05-24 15:58:18 -04:00
Brian Lai
d74125bf10 [RFR for API Sources] New Python interfaces to support resumable full refresh (#37429) 2024-05-06 18:41:29 -04:00
Ella Rohm-Ensing
b7819d9f6c python: assert actual == expected ordering (#36980) 2024-04-11 15:16:33 +00:00
Christo Grabowski
e74d936350 CDK: updated error message for missing streams (#36833) 2024-04-04 15:39:40 -04:00
Brian Lai
728c92cba1 [airbyte-cdk] Stream should not extract state using legacy get_updated_state if no cursor (#36342) 2024-03-20 23:47:23 -04:00
Brian Lai
ef98194673 Emit final state message for full refresh syncs and consolidate read flows (#35622) 2024-03-05 01:05:06 -05:00
Artem Inzhyyants
0954ad3d3a Airbyte CDK: add interpolation for request options (#35485)
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
2024-02-22 19:40:44 +01:00
Brian Lai
2b87164b89 Emit multiple error trace messages and continue syncs by default (#35129) 2024-02-15 02:16:02 -05:00
Catherine Noll
e8910e427a File-based CDK: make incremental syncs concurrent (#34540) 2024-02-07 20:41:04 -05:00
Brian Lai
60686505f3 Revert "Emit multiple error trace messages and continue syncs by default" (#34990) 2024-02-07 19:47:15 -05:00
Brian Lai
cc2a6e229f Emit multiple error trace messages and continue syncs by default (#34636) 2024-02-07 13:34:43 -05:00
Brian Lai
f8182bc18e airbyte-cdk: failed stream does not end the sync (#33136)
Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
2023-12-07 14:11:39 -05:00
Maxime Carbonneau-Leclerc
32fdd7fd72 [ISSUE #29573] Concurrent CDK: incremental syncs (#31466)
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
2023-11-01 12:00:25 -04:00
Marius Posta
7ae97175a6 gradle: fix repo wide behaviour (#30607) 2023-09-28 05:01:13 -07:00
Yevhenii
8cdafabd82 Airbyte CDK: Change Error message if stream is not found (#30723)
Co-authored-by: Yevhenii Kurochkin <ykurochkin@flyaps.com>
2023-09-25 18:13:19 +03:00
Maxime Carbonneau-Leclerc
f48849fdb4 [ISSUE #26909] adding message repository (#27158)
* [ISSUE #26909] adding message repository

* Automated Commit - Formatting Changes

* [ISSUE #26909] improve entrypoint error handling

* format CDK

* [ISSUE #26909] adding an integration test
2023-06-13 08:40:55 -04:00
Jonathan Pearlin
2ebfa459cf Publish stream status messages in CDK (#24994)
* Publish stream status messages in CDK

* Automated Commit - Formatting Changes

* Convert to StreamDescriptor

* Automated Commit - Formatting Changes

* Bump to latest protocol model

* Automated Commit - Formatting Changes

* Bump protocol version

* Add tests for stream status message creation

* Formatting

* Formatting

* Fix failing test

* Actually emit state message

* Automated Commit - Formatting Changes

* Bump airbyte-protocol

* PR feedback

* Fix parameter input

* Correctly yield status message

* PR feedback

* Formatting

* Fix failing tests

* Automated Commit - Formatting Changes

* Revert accidental change

* Automated Change

* Replace STOPPED with COMPLETE/INCOMPLETE

* Update source-facebook-marketing changelog

* Revert "Update source-facebook-marketing changelog"

This reverts commit 709edb800c.

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
2023-04-26 10:30:36 -04:00
Augustin
bad5bce8ce CDK: remove unexpected error swallowing on abstract source's check method (#24240) 2023-03-23 13:04:51 +00:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594)
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
2023-02-08 13:01:16 -08:00
Ella Rohm-Ensing
b5d3236cca Handle logging non-JSON-serializable classes in stream slices (#22118)
* Add failing test

* handle unserializable classes in stream slices

* format
2023-01-31 16:33:25 +00:00
Maxime Carbonneau-Leclerc
ca8cdc40aa [ISSUE #20771] limiting the number of requests performed to the backe… (#21525)
* [ISSUE #20771] limiting the number of requests performed to the backend without flag

* [ISSUE #20771] code reviewing my own code

* [ISSUE #20771] adding ABC to paginator

* [ISSUE #20771] format code

* [ISSUE #20771] adding slices to connector builder read request (#21605)

* [ISSUE #20771] adding slices to connector builder read request

* [ISSUE #20771] formatting

* [ISSUE #20771] set flag when limit requests reached (#21619)

* [ISSUE #20771] set flag when limit requests reached

* [ISSUE #20771] assert proper value on test read objects __init__

* [ISSUE #20771] code review and fix edge case

* [ISSUE #20771] fix flake8 error

* [ISSUE #20771] code review

* 🤖 Bump minor version of Airbyte CDK

* to run the CI
2023-01-24 15:19:19 +00:00
Alexandre Girard
1a608f846a Stream returns AirbyteMessages (#18572)
* method yielding airbytemessage

* move to Stream

* update abstract source

* reset

* missing file

* add test docker image

* script to run acceptance tests with local cdk

* Update conftest to use a different image

* extract to method

* dont use a different image tag

* Always install local cdk

* break the cdk

* get path from current working directory

* or

* ignore unit test

* debug log

* Revert "AMI change: ami-0f23be2f917510c26 -> ami-005924fb76f7477ce (#18689)"

This reverts commit bf06decf73.

* build from the top

* Update source-acceptance-test

* fix

* copy setup

* some work on the gradle plugin

* reset to master

* delete unused file

* delete unused file

* reset to master

* optional argument

* delete dead code

* use latest cdk with sendgrid

* fix sendgrid dockerfile

* break the cdk

* use local file

* Revert "break the cdk"

This reverts commit 600c195541.

* dont raise an exception

* reset to master

* unit tests

* missing test

* more unit tests

* newline

* reset to master

* remove files

* reset

* Update abstract source

* remove method from stream

* convert to airbytemessage

* unittests

* Update

* unit test

* remove debug logs

* Revert "remove debug logs"

This reverts commit a1a139ef37.

* Revert "Revert "remove debug logs""

This reverts commit b1d62cdb60.

* Revert "reset to master"

This reverts commit 3fa6a004c1.

* Revert "Revert "reset to master""

This reverts commit 5dac7c2804.

* reset to master

* reset to master

* test

* Revert "test"

This reverts commit 2f91b803b0.

* test

* Revert "test"

This reverts commit 62d95ebbb5.

* test

* Revert "test"

This reverts commit 27150ba341.

* format
2022-11-07 21:23:11 -08:00
Alexandre Girard
18da341b91 Revert: #17047 Airbyte CDK: Improve error for returning non-iterable from connectors parse_response (#17707)
* Bump cdk version

* Revert "#17047 Airbyte CDK: Improve error for returning non-iterable from connectors parse_response (#17626)"

This reverts commit d9ad2725b1.

* Bump
2022-10-06 18:43:01 -07:00
Robert
d9ad2725b1 #17047 Airbyte CDK: Improve error for returning non-iterable from connectors parse_response (#17626)
* Improve airbyte cdk invalid message data type error message

* Test cdk invalid message data type custom error is raised

* Fix test to pass stream as a string

* Add valid record message data input type test

* Add object type and value to AirbyteRecordMessage validator message

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
2022-10-06 16:46:33 -07:00
Pedro S. Lopez
fe2238f3f7 🐞 CDK: properly emit state on empty slices when using iterators (#17296)
* fix: emit state on empty slices iterator

* bump version / update changelog

* format
2022-09-28 02:12:04 -04:00
Brian Lai
f283652313 [per-stream cdk] Emit per-stream state messages (#16628)
* update abstract source and connector state manager to update shared and stream state

* clean up some extra comments and extra lines

* a few changes based on PR feedback

* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob

* rename a few paramets and add testing for state update via stream.state override

* replace shared_state processing with an explicit error and fix a few comments and pr feedback

* add some polish and additional test cases

* Add create_state_message to manager and emit per-stream state for connectors

* pr feedback and restructuring parts of the connector state manager initialization

* fix unfinished comment

* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* use pytest params to annotate tests better

* change to fix changed class name

* format test params a little better

* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* a few bits of pr feedback

* do not emit namespace in the outbound message if it is None

* fix some issues w/ prior merge from master and pr feedback

* add new tests to ensure we never send None namespace to the platform

* bump CDK version number

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-09-15 23:42:49 -04:00
Brian Lai
db56c75f03 [per-stream cdk] Allow for reading in per-stream state and passing it to connectors (#16505)
* update abstract source and connector state manager to update shared and stream state

* clean up some extra comments and extra lines

* a few changes based on PR feedback

* remove separate legacy map from state manager and simplify mapping to descriptor -> state blob

* rename a few paramets and add testing for state update via stream.state override

* replace shared_state processing with an explicit error and fix a few comments and pr feedback

* add some polish and additional test cases

* pr feedback and restructuring parts of the connector state manager initialization

* fix unfinished comment

* Update airbyte-cdk/python/unit_tests/sources/test_abstract_source.py

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>

* use pytest params to annotate tests better

* change to fix changed class name

* Update airbyte-cdk/python/airbyte_cdk/sources/connector_state_manager.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* a few bits of pr feedback

* pr feedback and cleaning up some comments and variable renames

Co-authored-by: Augustin <augustin.lafanechere@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-09-15 15:55:46 -04:00
Alexandre Girard
79a54a81fd Emit a state message even if no records were read (#15067)
* Emit a state message even if no records were read

* newline

* merge

* comment

* implement logic in the abstract source

* remove logic from declarative source

* comment

* bump cdk version
2022-08-04 12:48:21 -07:00
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Pedro S. Lopez
41dc82f056 CDK: provide better user-friendly error messages (#12944)
* initial implementation

* add parse_response_error_message tests

* move error handler to existing try/catch in AbstractSource

* formatting

* var rename

* use isinstance for type checking

* add docstrings

* add more abstract_source and httpstream tests

* fix wrong httperror usage

* more test cases

* bump version, update changelog
2022-05-19 17:28:24 -04:00
Eugene Kulak
d173ce741f CDK: add support for streams with state attribute (#9746)
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2022-02-17 00:20:33 +04:00
Eugene Kulak
f83eca58ea CDK: Fix typing errors (#9037)
* fix typing, drop AirbyteLogger

* format

* bump the version

* use logger instead of fixture logger

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: auganbay <auganenu@gmail.com>
2022-01-14 10:29:34 +06:00
LiRen Tu
c511abd20c Fix master build (#7873)
* Check in format diff

* Remove cache http yaml files

* Ignore tmp files generated by unit tests
2021-11-11 09:47:51 -08:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Davin Chia
9460cb4422 Update all of Pydantic to 1.6.2 per Dependabot. (#3408)
* Update all to 1.6.2.

* Publish new airbyte-cdk version.

* Use repr instead of str for exceptions.

* Use rc.

* Edit test.

* Bump for SAT.

* Format.
2021-05-14 16:44:49 +08:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Sherif A. Nada
184dab77eb CDK: overhaul directory structure (#3295) 2021-05-09 15:27:38 -07:00