1
0
mirror of synced 2026-01-20 21:06:36 -05:00
Commit Graph

70 Commits

Author SHA1 Message Date
Brian Lai
2aaf33e679 [source-github] Bump cdk version and enable RFR for all non-incremental streams (#42966) 2024-08-09 12:54:01 -04:00
Serhii Lazebnyi
d327ec4960 refactor(source-github): migrate to CDK v3 (#41677) 2024-07-18 20:03:57 +02:00
Christo Grabowski
074bc068f2 Source Github: Refactor incremental state handling (#39513) 2024-06-18 11:59:01 -04:00
Denys Davydov
003fc6ef06 🐛 Source Github: increase discovery time (#36429) 2024-03-26 20:36:36 +02:00
Artem Inzhyyants
2ce32ab24d 🐛 Source Github: Handle rate limit exception as config error (#35986)
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
2024-03-13 10:09:11 +01:00
Artem Inzhyyants
9175956a01 Source GitHub: update CDK (#35915)
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
2024-03-08 17:10:45 +01:00
Artem Inzhyyants
a48924afd9 Source Github: add integration tests (#34933) 2024-02-12 16:53:59 +01:00
Artem Inzhyyants
18cfa24974 🐛 Source GitHub: Continue Sync on Stream failure (#34700) 2024-02-02 17:55:41 +01:00
Artem Inzhyyants
66cd528340 🐛 Source GitHub: Fix MultipleTokenAuthenticator sleep time (#34598) 2024-01-29 11:29:32 +01:00
Artem Inzhyyants
53851609f8 🐛 Source GitHub: Fix MultipleToken rotation logic (#34503) 2024-01-26 15:15:54 +01:00
Daryna Ishchenko
6fce769ce3 🐛 Source Github: return AirbyteMessage if max retry exeeded for 202 status code (#32679)
Co-authored-by: darynaishchenko <darynaishchenko@users.noreply.github.com>
2023-11-23 07:06:38 -06:00
Ella Rohm-Ensing
ac3eb28de2 airbyte-ci: add format commands (#31831)
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: alafanechere <alafanechere@users.noreply.github.com>
2023-11-14 02:17:48 -06:00
Artem Inzhyyants
c7d411061e Source GitHub: handle ContributorActivity continuous ACCEPTED response (#31386) 2023-10-13 18:35:36 +02:00
Artem Inzhyyants
370821d08a Source GitHub: add unit tests (#31285)
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
2023-10-11 18:04:12 +02:00
Artem Inzhyyants
747f1528fa Source GitHub: migrate repo and branches to array in spec (#31056)
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
2023-10-04 23:14:52 +02:00
Daryna Ishchenko
4e44cd03be Source Github: marked start date as optional field (#30971)
Co-authored-by: darynaishchenko <darynaishchenko@users.noreply.github.com>
2023-10-03 18:14:05 +03:00
Artem Inzhyyants
ce08bfd484 Source GitHub: add archived project cards (#30979)
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
2023-10-02 21:45:17 +02:00
Daryna Ishchenko
64edf61cb5 Source Github: updated user error messages (#30927) 2023-09-29 15:40:01 +03:00
Daryna Ishchenko
3c79da9ede Source GitHub: added Issue timeline events stream (#30823)
Co-authored-by: darynaishchenko <darynaishchenko@users.noreply.github.com>
2023-09-29 12:56:19 +03:00
Artem Inzhyyants
bc68d0a442 Source GitHub: ignore 204 error in stats response (#30824)
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
2023-09-28 17:49:53 +02:00
Marius Posta
7ae97175a6 gradle: fix repo wide behaviour (#30607) 2023-09-28 05:01:13 -07:00
Artem Inzhyyants
61a63ec2ff Source GitHub: add stream ProjectsV2 (#30731)
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
2023-09-28 08:46:29 +02:00
Artem Inzhyyants
e763971bf1 Source GitHub: Skip 404 error in TeamMemberShips stream (#30693) 2023-09-25 19:56:36 +02:00
Artem Inzhyyants
32c08d73d5 Source GitHub: support self hosted instances (#30647)
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
2023-09-22 19:36:50 +02:00
Serhii Chvaliuk
df79780cb8 Source Github: deprecate "page size" input parameter (#25838)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2023-05-24 12:20:08 +03:00
Serhii Chvaliuk
627c49c8c5 Source Github: Repository field accept full URL (#25778)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2023-05-23 09:05:07 +03:00
Serhii Chvaliuk
30d8bb4133 Source Github: implement client-side throttling of requests (#25793)
Signed-off-by: Serhii Chvaliuk <grubberr@gmail.com>
2023-05-17 22:18:37 +03:00
Roman Yermilov [GL]
ce9eb8ab4c Source Github: better error explanation for 401 (#26025)
* Source Github: better error explanation for 401

* Source Github: bump version, update changelog

* Source Github: fix test name

* Source Github: fix unittests

* Source Github: bump version

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-05-16 22:35:22 +04:00
Daryna Ishchenko
05990806a1 🐛Source GitHub: added description for no repos case error (#26075)
* added description for no repos case error

* added AirbyteTracedException

* added changelog

* removed handling error in validation

* updated github.md

* added unit test test_streams_no_streams_available_error

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-05-16 18:07:20 +03:00
Serhii Lazebnyi
312c84164a 🐛Source GitHub: Fix invalid validation format of repositories config (#25312)
* Fix repo name validation

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-24 21:36:49 +00:00
Daryna Ishchenko
836e4f4669 Source Github: decrease give up rate (#24457)
* added validation and transformation for repositories config

* added changelog, updated regex and tests

* updated regex, fixed test_check_connection_repos_only

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-30 20:21:41 +03:00
Serhii Chvaliuk
5053b278f5 Source Github: fix caching for get_starting_point in stream "Commits" (#24398)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2023-03-27 13:13:01 +03:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594)
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
2023-02-08 13:01:16 -08:00
Roman Yermilov [GL]
d1688c7287 Source GitHub: #970 handle 502 Bad Gateway (#21481)
* Source Github: handle 502 Bad Gateway with proper log message

* Source Github: bump version

* Source Github: update changelog for new 0.3.12 version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-01-19 19:27:30 +00:00
Artem Inzhyyants
49bee81fc2 Source Github: Raise Error if no organizations or repos are available (#21084)
* Source Github: Raise Error if no organizations or repos are available

* Source GitHub: Update docs; bump version

* Source GitHub: Add test

* Source GitHub: Exclude empty streams

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-01-06 19:50:23 +01:00
Ella Rohm-Ensing
8bb41282c5 CDK 0.15.0 and source-github 0.3.10 -- revert AvailabillityStrategy changes (#20523)
* Revert "source-github: move known error handling to GithubAvailabilityStrategy (#19978)"

This reverts commit f97db17ccc.

* Revert "🐛 Python CDK: fix `StopIteration` error for `check_availability` (#20429)"

This reverts commit 4e9b014277.

* Revert "CDK: `AbstractSource.read()` skips syncing stream if its unavailable (add `AvailabilityStrategy` concept) (#19977)"

This reverts commit 55a32886a3.

* Restore changelog entries

* bump CDK version

* Bump Github version

* Re-add removed dependencies

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-12-15 14:24:24 -05:00
Ella Rohm-Ensing
f97db17ccc source-github: move known error handling to GithubAvailabilityStrategy (#19978)
* Rough first implememtation of AvailabilityStrategy s

* Basic unit tests for AvailabilityStrategy and ScopedAvailabilityStrategy

* Make availability_strategy a property, separate out tests

* Remove from DeclarativeSource, remove Source parameter from methods, make default no AvailabilityStrategy

* Add skip stream if not available to read()

* Changes to CDK to get source-github working using AvailabilityStrategy, flakecheck

* reorganize cdk class, add HTTPAvailabilityStrategy test

* cleanup, docstrings

* source-github working with availabilitystrategy

* reorganize source-github

* source-github: resolve deprecation warning

* use correct import for HTTPAvailabilitystrategy

* pull out error handling into separate method

* use handle_http_error method

* Pass source and logger to check_connection method

* Add documentation links, handle 403 specifically

* Fix circular import

* Add AvailabilityStrategy to Stream and HTTPStream classes

* Remove AS from abstract_source, add to Stream, HTTPStream, AvailabilityStrategy unit tests passing for per-stream strategies

* Modify MockHttpStream to set no AvailabilityStrategy since source test mocking doesn't support this

* Move AvailabilityStrategy class to sources.streams

* Move HTTPAvailabilityStrategy to http module

* Use pascal case for HttpAvailabilityStrategy

* Remove docs message method :( and default to True availability on unhandled HTTPErrors

* add check_availability method to stream class

* Add optional source parameter

* Add test for connector-specific documentation, small tests refactor

* Add test that performs the read() function for stream with default availability strategy

* Add test for read function behavior when stream is unavailable

* Add 403 info in logger message

* Don't return error for other HTTPErrors

* Split up error handling into methods 'unavailable_error_codes' and 'get_reason_for_error'

* rework overrideable list of status codes to be a dict with reasons, to enforce that users provide reasons for all listed errors

* Fix incorrect typing

* Move HttpAvailability to its own module, fix flake errors

* Fix ScopedAvailabilityStrategy, docstrings and types for streams/availability_strategy.py

* Docstrings and types for core.py and http/availability_strategy.py

* Move _get_stream_slices to a StreamHelper class

* Docstrings + types for stream_helpers.py, cleanup test_availability.py

* Clean up test_source.py

* Move logic of getting the initial record from a stream to StreamHelper class

* Add changelog and bump minor version

* change 'is True' and 'is False' behavior

* use mocker.MagicMock

* Remove ScopedAvailabilityStrategy

* Don't except non-403 errors, check_stream uses availability_strategy if possible

* Move AvailabilityStrategy to stream level, fix tests

* make get_stream_slice public

* Attempt to refactor error code handling into repository-based and organization-based

* split into repository-based availabilitystrategy and organization-based availabilitystrategy

* refactor organization-based availabilitystrategy

* refactor repository-based availabilitystrategy, create separate ones for workflow_runs and projects

* Fix workflow runs availability strategy

* move availability strategies to a different module

* CDK: pass error to reasons_for_error_codes

* make get_stream_slice public

* Revert "make get_stream_slice public"

This reverts commit 9170fe5f59.

* Add tests for raising unhandled errors and retries are handled

* Add tests for CheckStream via AvailabilityStrategy

* Remove moved file

* bump CDK dependency

* Cleanup: Address review comments

* One more fix

* Update changelog and dockerfile version
2022-12-14 16:58:15 -05:00
Serhii Chvaliuk
2e1209e78f Source Github: switch on airbyte-cdk==0.2.0 (#18213)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-10-20 15:16:58 +03:00
Serhii Chvaliuk
8e6fb793bb Source Github: Use default behaviour, retry on 429 and all 5XX errors (#17852)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-10-12 11:35:42 +03:00
Serhii Chvaliuk
f4374530e5 Source Github: improve 502 handling for comments stream (#17715)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-10-07 14:58:31 +03:00
Serhii Chvaliuk
a338df1040 🐛 Source Github: bugfix WorkflowJobs stream (#17287)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-09-30 14:45:25 +03:00
Denys Davydov
57c269d9c6 Source github: improve error logging (#16947)
* #410 oncall source github: improve error logging

* source github: upd changelog

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-21 23:16:40 +03:00
Mainara Cavalcanti
97b9c74718 Source Github: add workflow_jobs stream (#16534)
* feat(workflow-jobs)!: new class 'WorkflowJobs' to get job informations from Github;

* feat(workflow-jobs)!: unit test to new class 'WorkflowJobs';

* feat(test_stream/test_stream_workflow_jobs_read_incremental)!: support to method incremetal in unit test to class 'WorkflowJobs';

* feat(test_workflow): unit test incremental and full refresh to class 'WorkflowJobs';

* docs: updated github docs

* docs: updated github docs

* docs: updated github.md

* docs: updated github.md

* fix: add workflow_jobs in abnormal tests, use 'start date' in source and unit test;

* fix: add workflow_jobs in 'abnormal_state.json' and 'sample_state.json' for integration tests;

Co-authored-by: Paulo Roberto <paulo.rb.beserra@gmail.com>
Co-authored-by: Paulo Roberto <71179739+py-paulo@users.noreply.github.com>
2022-09-21 10:19:35 -04:00
Serhii Chvaliuk
255a5bbd5e 🐛 Source Github: validate input organizations and repositories (#15730)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-08-20 00:48:33 +03:00
Oleksandr Sheheda
a16b5d1491 fixed formatting (#15359) 2022-08-05 13:31:21 +03:00
Serhii Chvaliuk
ae58fa52da 🎉 Source Github: PullRequestCommentReactions - re-implemented using GraphQL (#14795)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-08-02 14:57:27 +03:00
Serhii Chvaliuk
38d8d60f02 🎉 Source Github: add Retry for GraphQL API Resource limitations (#14376)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-07-05 00:37:39 +03:00
Serhii Chvaliuk
4072d46a7c 🎉 Source Github: use GraphQL for reviews stream (#13989)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-06-28 18:35:12 +03:00
Serhii Chvaliuk
7a15f2d4a5 🎉 Source Github: secondary rate limits has to retry (#13955)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-06-27 23:36:20 +03:00
Serhii Chvaliuk
f69a78c9a4 🎉 Source Github: break point added for workflows_runs stream (#13926)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-06-22 11:49:04 +03:00