08c42ced3d
## What <!-- * Describe what the change is solving. Link all GitHub issues related to this change. --> OC Issue: https://github.com/airbytehq/oncall/issues/9057 The Google Sheets API is returning a 500 response when requesting the GridData for certain sheets. This is causing us to fail CHECK for some connectors. We confirmed with the user that in these situations, the data in those sheets is unstructured, and they likely don't want to sync it, but still want to sync other sheets in the spreadsheet. Due to this, we should be okay with skipping these sheets altogether. To detect if it is a real 500 error or bad grid data, we can keep the existing backoff for 500's. On the last attempt, we will retry with `"includeGridData=false"` in the parameter. If we get a 200 response, the sheet is highly likely to have bad/corrupted data. If we still get a 500, then there likely are genuine server errors going on with the API, and in that case, we will default to the 500 server error being raised. ## How <!-- * Describe how code changes achieve the solution. --> In the manifest.yaml we need to add a custom error handler that can do this in the `components_resolver_retriever`. The existing error handler is a DefaultErrorHandler, with multiple response_filters: [single_sheet_response_error_filters](663615eee6/airbyte-integrations/connectors/source-google-sheets/manifest.yaml (L328-L331)); which use `ExponentialBackoffStrategy` We cannot refactor this into a CompositeErrorHandler to keep the existing response_filters, and add on the CustomErrrorHandler due to lack of support in the CDK [here](7ab013d412/airbyte_cdk/sources/declarative/declarative_component_schema.yaml (L391-L393)). Due to this, the existing [DefaultErrorHandler](663615eee6/airbyte-integrations/connectors/source-google-sheets/manifest.yaml (L234-L239)), should be refactored into the new custom error handler. This is done in `GridDataErrorHandler`, existing tests are left unchanged, indicating the refactoring was done correctly. We also added some new tests for the 500 error handling. ## Review guide <!-- 1. `x.py` 2. `y.py` --> ## User Impact <!-- * What is the end result perceived by the user? * If there are negative side effects, please list them. --> ## Can this PR be safely reverted and rolled back? <!-- * If unsure, leave it blank. --> - [X ] YES 💚 - [ ] NO ❌ --------- Co-authored-by: maxime.c <maxime@airbyte.io>
22 lines
476 B
TOML
22 lines
476 B
TOML
[build-system]
|
|
requires = ["poetry-core>=1.0.0"]
|
|
build-backend = "poetry.core.masonry.api"
|
|
|
|
[tool.poetry]
|
|
name = "source-google-sheets-tests"
|
|
version = "0.0.0"
|
|
description = "Unit tests for source-google-sheets"
|
|
authors = ["Airbyte <contact@airbyte.io>"]
|
|
|
|
[tool.poetry.dependencies]
|
|
python = "^3.10,<3.13"
|
|
airbyte-cdk = "^7"
|
|
pytest = "^8"
|
|
responses = "^0.25.7"
|
|
requests-mock = "^1.11.0"
|
|
|
|
[tool.pytest.ini_options]
|
|
filterwarnings = [
|
|
"ignore:This class is experimental*"
|
|
]
|