* Refactoring to improve code re-use
* Add ProtocolVersionChecker
* Add an option to configure if we are automatically upgrading connectors
* Add airbyte version check to pass the fresh install case
* Inject DefinitionsProvider in the BootloaderApp
* Remove AutoUpgradeConnector config
* Improve logging
* Use named argument rather than positional
* Make DefinitionsProvider optional
* Format
Follow up to #18953.
Implement all the DB migrations required for a progress bar.
The main change here is to support saving:
the estimated records/bytes at the sync level
the estimated records/bytes and emitted records/bytes at the stream level
After this, I'll put up a PR for the persistence layer changes, which will writing to and reading from these columns.
Finally, I'll wire this into the API changes, which are currently stubs.
- add the estimated_records and estimated_bytes columns to the SyncStats table.
- create a stream_stats table
- estimated and emitted records/bytes column
- contains attempt_id and stream_name columns. Unique constraints on these two columns.
- foreign key to the attempt_id table.
- this table hopefully sets us up for the parallel sync work.
* Add missing indices for `status` columns
* Remove TODO and delete .class files
* Dump schema
* Run gradle format
* Bump database schema version in Bootloader test
* updated StandardDestinationDefinition.yaml, added normalization and tags to the destination_definition.yaml and added information about normalization and DBT to the ACTOR_DEFINITION table
* updated docs
* updated BootloaderAppTest.java for new migration
* updated schema dump
* Update normalization version and fix bigquery
* Use varchar 255
* Update migration version to the latest
* Update normalized table schema file and add comment
* Revert "Use varchar 255"
This reverts commit e182466e66.
* Use varchar 255
* Add unit test for migration
* Format code
Co-authored-by: Liren Tu <tuliren@gmail.com>
* progress on adding geography throughout api
* fix workspace handler test
* more progress
* implement workspace defaulting and add/update more tests
* fix bootloader tests
* set defaultGeography in missing places
* add Geography column when reading Connection record from DB
* fix pmd
* add more comments/description
* format
* description
* init commit of geography_type and column
* add schema dump
* confirm prod database was meant to be changed
* add AUTO to geography
* set default and not nullable, also edit workspace table
* update version number
* update schema dump
Co-authored-by: pmossman <parker@airbyte.io>
* Add Airbyte Protocol Range configs
* Refactor metadata read/write
* Add ProtocolVersion Min/Max get/set to JobsPersistence
* Store the supported protocol version range in airbyte_metadata
* Use defaults in EnvConfigs instead of .env
* wip for moving scheduler-persistence to airbyte-persistence
* move main/resources
* move settings include to match existing includes
* fix incorrect import paths
* fix import order
Define logic in root build.gradle to create a task in each subproject if the subproject contains dockerImageName in the gradle.properties file.
Some caveats:
- We also define a copyGeneratedTar task that is applied to all subprojects with the dockerImageName property.
- This does not need to be used in each subproject. We leave it up to each subproject to define what task is depends on since some projects have more custom copy logic.
* add migration
* fmt
* test migration
* update expected migration version
* update schema
* allow specifying a target migration version for the baseline
* run previous migrations in test
* fix pmd
* update migration version
