Some refactoring in preparation for the progress bar persistence changes.
The main change here was to simplify some of the JobPersistence methods by moving the logic to calculate attemptId into the JobPersistence implementation. This logic currently sits outside the class and is duplicated in multiple places. We could expose a helper method to calculate this logic, however that felt unnecessary at this point.
The alternative is further duplicating this logic as the progress bar logic is implemented, so I want to get that out of the way.
The other reason it's cleaner to use jobId and attemptNumber is these concepts/terms are more familiar throughout the rest of the codebase and it feels more intuitive to continue speaking this language (in my opinion).
Some random bits I wanted to clean up on the way as well. I will leave comments in the files as appropriate.
Method interface:
- has generic type that isn't needed. Confusing because no generic is happening here.
- has two existing parameters that are data fields of one of the already existing parameters. Confusing since the two additional parameters aren't actually supposed to be passed in separately from the existing parameter.
Remove both.
We should only force (FULL_REFRESH,OVERWRITE) for the streams to reset.
For the other streams, we only want to replace OVERWRITE with APPEND to
avoid having destination clear the stream. The other cases should be
left as is.
* Pass protocol version into IntegrationLauncherConfig
* Use VersionedStreamStreamFactory in AirbyteSource/Destination
* Add AirbyteMessageBufferedWriter
* Use VersionedBufferedWriter
Implements the webhook operation as part of the sync workflow.
- Introduces the new activity implementation
- Updates the various interfaces that pass input to get the relevant configs to the sync workflow
- Hooks the new activity into the sync workflow
- Passes the webhook configs along into the sync workflow job
* WIP - Add additional sync timing information
* Fixup tests
* fix PMD problem
* send data to segment
* Test JobTracker
* respond to PR suggestions
* fixup test
* formatting
* fix initializer for stats
* Make thread-safe with synchronized
* Don't clobber syncStats on init
* add comments and fix init
* Do what Pedro says
* Extract timeTracker pojo
* query once for all needed models, instead of querying within connections loop
* cleanup and fix failing tests
* pmd fix
* fix query and add test
* return empty if input list is empty
* undo aggressive autoformatting
* don't query for connection operations in a loop, instead query once and group-by connectionID in memory
* try handling operationIds in a single query instead of two
* remove optional
* fix operationIds query
* very annoying, test was failing because operationIds can be listed in a different order. verify operationIds separately from rest of object
* combined queries/functions instead of separate queries for actor and definition
* remove leftover lines that aren't doing anything
* format
* add javadoc
* format
* use leftjoin so that connections that lack operations aren't left out
* clean up comments and format
* Add Airbyte Protocol Range configs
* Refactor metadata read/write
* Add ProtocolVersion Min/Max get/set to JobsPersistence
* Store the supported protocol version range in airbyte_metadata
* Use defaults in EnvConfigs instead of .env
* wip for moving scheduler-persistence to airbyte-persistence
* move main/resources
* move settings include to match existing includes
* fix incorrect import paths
* fix import order
