* Migrate airbyte-bootloader to Micronaut
* PR feedback
* More PR feedback
* Rename variable for clarity
* Add properties to support cloud
* Formatting
* Use default values for env vars
* Re-organization to support cloud overrides
* Disable conditional logging
* More singleton cleanup
* test: try CI without fluentbit
* Revert "test: try CI without fluentbit"
This reverts commit 8fa0f74106.
* test: enable SSH on EC2 runner
* Revert "test: enable SSH on EC2 runner"
This reverts commit e4867aae09.
* Avoid early database connection on startup
* Fix compile issues from refactor
* Formatting
Co-authored-by: perangel <perangel@gmail.com>
Implement the persistence layer changes following #19191.
This PR handles writing and reading stats to the new stream stat_table and columns in the existing sync_stats table.
At the same time we introduce upserts of stats records - i.e. merge updates into a single record - in preparation for real time stats updates vs the current approach where a new stat record is always written.
There will be two remaining PRs after this:
- First PR will be to fully wire up and test the API.
- Second PR will be to actually save stats while jobs are running.
* updated IntegrationLauncherConfig.yaml and added to this class suportDBT and normalizationImage fields. Added to the GenerateInputActivityImpl and TemporalClient classes code parts for read destination_definition.yaml and get suportDBT and normalizationImage fields. Added logging and comparing normalization images from NormalizationRunnerFactory and destination_definition.yaml
* updated minor remarks
* updated minor remarks
* fixed minor remarks
* added normalization data to the tests
* fixed minor remarks
* removed NormalizationRunnerFactory
* fixed remarks
* fixed remarks
* fixed remarks
* updated acceptance tests
* updated acceptance tests
* updated check_images_exist.sh script
* updated method for get normalization image name for destination acceptance test
* fixed code style
* fixed code style and removed tests data
* updated JobErrorReporterTest.java
* updated JobErrorReporterTest.java
* fixed remarks
* added integration type field to the dectination_definition file and actor_definition table
* fixed tests
* fixed tests
* fixed minor changes after pulling master changes
* fixed minor changes after pulling master changes
* renamed integrationType to normalizationIntegrationType/ fixed minor remarks
* renamed extra dependencies
* updated docs
* updated docs
* fixed minor remarks
* added NormalizationDestinationDefinitionConfig.yaml for StandardDestinationDefinition.yaml and updated configuration
* updated normalization tag
* updated DestinationAcceptanceTest.java
* updated DestinationAcceptanceTest.java
* updated imports and descriptions
* fix button loading state when creating connector
* use workspace scoped endpoints
* get and update methods
* backfill script
* commit new migration file
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
* Refactoring to improve code re-use
* Add ProtocolVersionChecker
* Add an option to configure if we are automatically upgrading connectors
* Add airbyte version check to pass the fresh install case
* Inject DefinitionsProvider in the BootloaderApp
* Remove AutoUpgradeConnector config
* Improve logging
* Use named argument rather than positional
* Make DefinitionsProvider optional
* Format
Follow up to #18953.
Implement all the DB migrations required for a progress bar.
The main change here is to support saving:
the estimated records/bytes at the sync level
the estimated records/bytes and emitted records/bytes at the stream level
After this, I'll put up a PR for the persistence layer changes, which will writing to and reading from these columns.
Finally, I'll wire this into the API changes, which are currently stubs.
- add the estimated_records and estimated_bytes columns to the SyncStats table.
- create a stream_stats table
- estimated and emitted records/bytes column
- contains attempt_id and stream_name columns. Unique constraints on these two columns.
- foreign key to the attempt_id table.
- this table hopefully sets us up for the parallel sync work.
* Add missing indices for `status` columns
* Remove TODO and delete .class files
* Dump schema
* Run gradle format
* Bump database schema version in Bootloader test
* updated StandardDestinationDefinition.yaml, added normalization and tags to the destination_definition.yaml and added information about normalization and DBT to the ACTOR_DEFINITION table
* updated docs
* updated BootloaderAppTest.java for new migration
* updated schema dump
* Update normalization version and fix bigquery
* Use varchar 255
* Update migration version to the latest
* Update normalized table schema file and add comment
* Revert "Use varchar 255"
This reverts commit e182466e66.
* Use varchar 255
* Add unit test for migration
* Format code
Co-authored-by: Liren Tu <tuliren@gmail.com>
* progress on adding geography throughout api
* fix workspace handler test
* more progress
* implement workspace defaulting and add/update more tests
* fix bootloader tests
* set defaultGeography in missing places
* add Geography column when reading Connection record from DB
* fix pmd
* add more comments/description
* format
* description
* init commit of geography_type and column
* add schema dump
* confirm prod database was meant to be changed
* add AUTO to geography
* set default and not nullable, also edit workspace table
* update version number
* update schema dump
Co-authored-by: pmossman <parker@airbyte.io>
* Add Airbyte Protocol Range configs
* Refactor metadata read/write
* Add ProtocolVersion Min/Max get/set to JobsPersistence
* Store the supported protocol version range in airbyte_metadata
* Use defaults in EnvConfigs instead of .env
* wip for moving scheduler-persistence to airbyte-persistence
* move main/resources
* move settings include to match existing includes
* fix incorrect import paths
* fix import order
* add migration
* fmt
* test migration
* update expected migration version
* update schema
* allow specifying a target migration version for the baseline
* run previous migrations in test
* fix pmd
* update migration version
* Update StateDB to support per Stream states.
* Add `StateType` type
* Add `steam_name`, `namespace` and `type` to `state` table.
* Set the default StateType to LEGACY
* sweep all scheduler application code and new-scheduler conditional logic
* remove airbyte-scheduler from deployments and docs
* format
* remove 'v2' from github actions
* add back scheduler in delete deployment command
* remove scheduler parameters from helm chart values
* add back job cleaner + test and add comment
* remove now-unused env vars from code and docs
* format
* remove feature flags from web backend connection handler as it is no longer needed
* remove feature flags from config api as it is now longer needed
* remove feature flags input from config api test
* format + shorter url
* remove scheduler parameters from helm chart readme
* Adds a new string cloumn to configs for cron
closes#11418
i'm new to this task in Java please be brutal
* Adds airbyte header
* WIP
* Rebase a week of commits
* WIP for davin
* deps update
* Reorganize code for better readability. Also add a schema.
* Update tests.
* Correct bad test.
* Adds note for testing version change
* formatting change
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Better database connection handling for connectors
* Log connection error
* Properly close connection
* Remove unused method
* Close data source
* Use utility to close data source
* Use utility to close data source
* PR feedback
* Add Databricks driver
* Use driver class enum
* Use correct config
* Ensure config created before use
* Fix failing integration test
* Create DSLContext before use
* Address integration test failures
* Ensure DSLContext is closed
* Fix compile error
* Use correct datasource
* Use correct connection properties
* Close DSLContext
* Close DSLContext
* Fix integration test failures
* Properly close datasource
* Fix compilation issues
* Use existing database object
* Wrap close in try/finally
* Update test
* Wrap close in try/finally
* Ensure DSLContext is created
* Revert change to test
* Use correct data source
* Remove unused import
* More cleanup
* Add missing annotation
* Only initialize data source once
* Remove unused import
* Force testcontainers version
* Fix testcontainer issue
* Fix failing test
* Properly close all data sources
* Clear data sources after closing
* Fix compile error
* Fix compilation error
* Add missing method
This is adding a metadata entry in order to make the use of the DB secret store to be the default one.
It will avoid having secret outside of the secret table.
Introduce a migration to a secret manager
If a secret manager is specify, it will go though all the config, save the secret in the configured secret store. If the secret is already in a store, it will not migrate the secret to the secret store.
* implement migration to create workspace_service_account table
* make all columns non nullable
* introduce persistence code for service account table (#11944)
* implement persistence code for workspace_service_account table
* update yaml
* implement secret handling for workspace_service_account table (#11946)
* implement secret handling for workspace_service_account table
* add new line to the mock json
* get rid of file
* address review comments
* update method name and add comment
