1
0
mirror of synced 2025-12-30 21:02:43 -05:00
Commit Graph

15 Commits

Author SHA1 Message Date
Serhii Chvaliuk
30d8bb4133 Source Github: implement client-side throttling of requests (#25793)
Signed-off-by: Serhii Chvaliuk <grubberr@gmail.com>
2023-05-17 22:18:37 +03:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594)
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
2023-02-08 13:01:16 -08:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846)
Rename source acceptance test to connector acceptance test
2023-02-02 11:38:19 +01:00
Ella Rohm-Ensing
8bb41282c5 CDK 0.15.0 and source-github 0.3.10 -- revert AvailabillityStrategy changes (#20523)
* Revert "source-github: move known error handling to GithubAvailabilityStrategy (#19978)"

This reverts commit f97db17ccc.

* Revert "🐛 Python CDK: fix `StopIteration` error for `check_availability` (#20429)"

This reverts commit 4e9b014277.

* Revert "CDK: `AbstractSource.read()` skips syncing stream if its unavailable (add `AvailabilityStrategy` concept) (#19977)"

This reverts commit 55a32886a3.

* Restore changelog entries

* bump CDK version

* Bump Github version

* Re-add removed dependencies

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-12-15 14:24:24 -05:00
Ella Rohm-Ensing
f97db17ccc source-github: move known error handling to GithubAvailabilityStrategy (#19978)
* Rough first implememtation of AvailabilityStrategy s

* Basic unit tests for AvailabilityStrategy and ScopedAvailabilityStrategy

* Make availability_strategy a property, separate out tests

* Remove from DeclarativeSource, remove Source parameter from methods, make default no AvailabilityStrategy

* Add skip stream if not available to read()

* Changes to CDK to get source-github working using AvailabilityStrategy, flakecheck

* reorganize cdk class, add HTTPAvailabilityStrategy test

* cleanup, docstrings

* source-github working with availabilitystrategy

* reorganize source-github

* source-github: resolve deprecation warning

* use correct import for HTTPAvailabilitystrategy

* pull out error handling into separate method

* use handle_http_error method

* Pass source and logger to check_connection method

* Add documentation links, handle 403 specifically

* Fix circular import

* Add AvailabilityStrategy to Stream and HTTPStream classes

* Remove AS from abstract_source, add to Stream, HTTPStream, AvailabilityStrategy unit tests passing for per-stream strategies

* Modify MockHttpStream to set no AvailabilityStrategy since source test mocking doesn't support this

* Move AvailabilityStrategy class to sources.streams

* Move HTTPAvailabilityStrategy to http module

* Use pascal case for HttpAvailabilityStrategy

* Remove docs message method :( and default to True availability on unhandled HTTPErrors

* add check_availability method to stream class

* Add optional source parameter

* Add test for connector-specific documentation, small tests refactor

* Add test that performs the read() function for stream with default availability strategy

* Add test for read function behavior when stream is unavailable

* Add 403 info in logger message

* Don't return error for other HTTPErrors

* Split up error handling into methods 'unavailable_error_codes' and 'get_reason_for_error'

* rework overrideable list of status codes to be a dict with reasons, to enforce that users provide reasons for all listed errors

* Fix incorrect typing

* Move HttpAvailability to its own module, fix flake errors

* Fix ScopedAvailabilityStrategy, docstrings and types for streams/availability_strategy.py

* Docstrings and types for core.py and http/availability_strategy.py

* Move _get_stream_slices to a StreamHelper class

* Docstrings + types for stream_helpers.py, cleanup test_availability.py

* Clean up test_source.py

* Move logic of getting the initial record from a stream to StreamHelper class

* Add changelog and bump minor version

* change 'is True' and 'is False' behavior

* use mocker.MagicMock

* Remove ScopedAvailabilityStrategy

* Don't except non-403 errors, check_stream uses availability_strategy if possible

* Move AvailabilityStrategy to stream level, fix tests

* make get_stream_slice public

* Attempt to refactor error code handling into repository-based and organization-based

* split into repository-based availabilitystrategy and organization-based availabilitystrategy

* refactor organization-based availabilitystrategy

* refactor repository-based availabilitystrategy, create separate ones for workflow_runs and projects

* Fix workflow runs availability strategy

* move availability strategies to a different module

* CDK: pass error to reasons_for_error_codes

* make get_stream_slice public

* Revert "make get_stream_slice public"

This reverts commit 9170fe5f59.

* Add tests for raising unhandled errors and retries are handled

* Add tests for CheckStream via AvailabilityStrategy

* Remove moved file

* bump CDK dependency

* Cleanup: Address review comments

* One more fix

* Update changelog and dockerfile version
2022-12-14 16:58:15 -05:00
Brian Lai
3521802545 Update Python Source connectors to use the latest airbyte-cdk minor version (#18261)
* update all connectors currently only adhering to patch versions to pull latest minor version

* also add greenhouse

* fix scaffolds
2022-10-20 16:06:02 -04:00
Serhii Chvaliuk
2e1209e78f Source Github: switch on airbyte-cdk==0.2.0 (#18213)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-10-20 15:16:58 +03:00
Serhii Chvaliuk
62626ee690 🎉 Source Github: use GraphQL for pull_request_stats stream (#13763)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-06-16 20:18:49 +03:00
Marcos Marx
dca2256a7c Bump 2022 license version (#13233)
* Bump year in license short to 2022

* remove protocol from cdk
2022-05-26 15:00:42 -03:00
Serhii Chvaliuk
8c394b3734 🐛 Source Github: add incremental for repositories, workflows, pull_request_comment_reactions, issue_reactions, issue_comment_reactions, commit_comment_reactions (#12294)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-05-13 21:18:24 +03:00
Serhii Chvaliuk
0f475ce6ff 🐛 Source Github: test coverage more than 90% (#10967)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-03-19 13:14:53 +02:00
Dmytro
c0cb8f338a Fix Github check connection for organizations with large number of re… (#8170) 2021-11-22 15:34:42 +02:00
Dmytro
bea9bb5cbb Github: resolve refs for some schemas (#7856) 2021-11-11 14:04:09 +02:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
oleh.zorenko
84ba3e79b3 🎉 Native GitHub connector (#4174)
* GitHub source initial version

All streams are in `full_refresh` mode only.

* Run format command

* Add handling for 404 error in Teams stream

* Update creds naming for github

* Update acceptance tests

* Update check_connection()

* Code review fixes

Remove `/integration_tests/catalog.json` file.
Use `Collaborators` class in `check_connection()` function.
Remove excessive data from GithubStream class.

* Remove unused import

* Implement requested changes

* Add incremental streams

* Remove `supports_incremental`

* Update docs

* Implement requested changes

* Implement requested changes

* Implement requested changes

* Bump connector version
2021-07-06 17:06:34 +03:00