* Migrate airbyte-bootloader to Micronaut
* PR feedback
* More PR feedback
* Rename variable for clarity
* Add properties to support cloud
* Formatting
* Use default values for env vars
* Re-organization to support cloud overrides
* Disable conditional logging
* More singleton cleanup
* test: try CI without fluentbit
* Revert "test: try CI without fluentbit"
This reverts commit 8fa0f74106.
* test: enable SSH on EC2 runner
* Revert "test: enable SSH on EC2 runner"
This reverts commit e4867aae09.
* Avoid early database connection on startup
* Fix compile issues from refactor
* Formatting
Co-authored-by: perangel <perangel@gmail.com>
Implement the persistence layer changes following #19191.
This PR handles writing and reading stats to the new stream stat_table and columns in the existing sync_stats table.
At the same time we introduce upserts of stats records - i.e. merge updates into a single record - in preparation for real time stats updates vs the current approach where a new stat record is always written.
There will be two remaining PRs after this:
- First PR will be to fully wire up and test the API.
- Second PR will be to actually save stats while jobs are running.
* updated IntegrationLauncherConfig.yaml and added to this class suportDBT and normalizationImage fields. Added to the GenerateInputActivityImpl and TemporalClient classes code parts for read destination_definition.yaml and get suportDBT and normalizationImage fields. Added logging and comparing normalization images from NormalizationRunnerFactory and destination_definition.yaml
* updated minor remarks
* updated minor remarks
* fixed minor remarks
* added normalization data to the tests
* fixed minor remarks
* removed NormalizationRunnerFactory
* fixed remarks
* fixed remarks
* fixed remarks
* updated acceptance tests
* updated acceptance tests
* updated check_images_exist.sh script
* updated method for get normalization image name for destination acceptance test
* fixed code style
* fixed code style and removed tests data
* updated JobErrorReporterTest.java
* updated JobErrorReporterTest.java
* fixed remarks
* added integration type field to the dectination_definition file and actor_definition table
* fixed tests
* fixed tests
* fixed minor changes after pulling master changes
* fixed minor changes after pulling master changes
* renamed integrationType to normalizationIntegrationType/ fixed minor remarks
* renamed extra dependencies
* updated docs
* updated docs
* fixed minor remarks
* added NormalizationDestinationDefinitionConfig.yaml for StandardDestinationDefinition.yaml and updated configuration
* updated normalization tag
* updated DestinationAcceptanceTest.java
* updated DestinationAcceptanceTest.java
* updated imports and descriptions
* fix button loading state when creating connector
* use workspace scoped endpoints
* get and update methods
* backfill script
* commit new migration file
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
* Refactoring to improve code re-use
* Add ProtocolVersionChecker
* Add an option to configure if we are automatically upgrading connectors
* Add airbyte version check to pass the fresh install case
* Inject DefinitionsProvider in the BootloaderApp
* Remove AutoUpgradeConnector config
* Improve logging
* Use named argument rather than positional
* Make DefinitionsProvider optional
* Format
Follow up to #18953.
Implement all the DB migrations required for a progress bar.
The main change here is to support saving:
the estimated records/bytes at the sync level
the estimated records/bytes and emitted records/bytes at the stream level
After this, I'll put up a PR for the persistence layer changes, which will writing to and reading from these columns.
Finally, I'll wire this into the API changes, which are currently stubs.
- add the estimated_records and estimated_bytes columns to the SyncStats table.
- create a stream_stats table
- estimated and emitted records/bytes column
- contains attempt_id and stream_name columns. Unique constraints on these two columns.
- foreign key to the attempt_id table.
- this table hopefully sets us up for the parallel sync work.
* Add missing indices for `status` columns
* Remove TODO and delete .class files
* Dump schema
* Run gradle format
* Bump database schema version in Bootloader test
* updated StandardDestinationDefinition.yaml, added normalization and tags to the destination_definition.yaml and added information about normalization and DBT to the ACTOR_DEFINITION table
* updated docs
* updated BootloaderAppTest.java for new migration
* updated schema dump
* Update normalization version and fix bigquery
* Use varchar 255
* Update migration version to the latest
* Update normalized table schema file and add comment
* Revert "Use varchar 255"
This reverts commit e182466e66.
* Use varchar 255
* Add unit test for migration
* Format code
Co-authored-by: Liren Tu <tuliren@gmail.com>
* progress on adding geography throughout api
* fix workspace handler test
* more progress
* implement workspace defaulting and add/update more tests
* fix bootloader tests
* set defaultGeography in missing places
* add Geography column when reading Connection record from DB
* fix pmd
* add more comments/description
* format
* description
* init commit of geography_type and column
* add schema dump
* confirm prod database was meant to be changed
* add AUTO to geography
* set default and not nullable, also edit workspace table
* update version number
* update schema dump
Co-authored-by: pmossman <parker@airbyte.io>
* Add Airbyte Protocol Range configs
* Refactor metadata read/write
* Add ProtocolVersion Min/Max get/set to JobsPersistence
* Store the supported protocol version range in airbyte_metadata
* Use defaults in EnvConfigs instead of .env
