a695b41349
Today we often see HTTP/1.1 header parser received no bytes' during syncs, especially in the Data Plane.
This PR attempts to fix this by adding naive retries.
Add a basic retry wrapper with the unique ability to retry for a much longer period on the last retry. This is particularly useful for us as most of our jobs are long running workflows, and the benefit of not having to restart the entire job outweighs the added wait time.
Alternative solutions I explored:
- Switching the underlying HTTP client to a more fully featured HTTP client. E.g. Apache or OkHttp. Issues with this:
- These clients do not support the ability to configure the retry policy we want.
- These clients do not support the ability to inject application aware logging.
- Most importantly, because this changes the interface, the resulting change set is big and affects many unrelated classes. I do think we eventually want to switch the underlying libraries out. However I don't think we should do this as part of OC work.
- Exploring pairing retry libraries such as https://resilience4j.readme.io/docs with the native http clients. The main issue here is the lack of ability to configure the last retry period.
Since the hand-rolled wrapper is simple + gets the job done, my thoughts are to run with this for the time being and revisit this if additional requirements around the clients come up.
airbyte-api
Defines the OpenApi configuration for the Airbyte Configuration API. It also is responsible for generating the following from the API spec:
- Java API client
- Java API server - this generated code is used in
airbyte-serverto allow us to implement the Configuration API in a type safe way. SeeConfigurationApi.javainairbyte-server - API docs
Key Files
- src/openapi/config.yaml - Defines the config API interface using OpenApi3
- AirbyteApiClient.java - wraps all api clients so that they can be dependency injected together
- PatchedLogsApi.java - fixes generated code for log api.