* wip: return whether configuration was updated
* updated outputs working
* fix pmd
* update description, format
* add didUpdateConfiguration to metadata, rm unneeded generics
* add didUpdateConfiguration to api response
* update name to fix pmd
* not required
* rename to match api response
* remove unused field
* match naming
* Migrate airbyte-bootloader to Micronaut
* PR feedback
* More PR feedback
* Rename variable for clarity
* Add properties to support cloud
* Formatting
* Use default values for env vars
* Re-organization to support cloud overrides
* Disable conditional logging
* More singleton cleanup
* test: try CI without fluentbit
* Revert "test: try CI without fluentbit"
This reverts commit 8fa0f74106.
* test: enable SSH on EC2 runner
* Revert "test: enable SSH on EC2 runner"
This reverts commit e4867aae09.
* Avoid early database connection on startup
* Fix compile issues from refactor
* Formatting
Co-authored-by: perangel <perangel@gmail.com>
* Upgrade to Micronaut 3.8.0 and related dependencies
* Define hostname/subdomain to be compatible with updated AWS SDK
* Use bucket name for hostname
* Pin dependencies to avoid behavior change
* Add comment
Let's try #20937 again, this time with better test for error cases.
See original PR for description.
This PR adds testing and logic to handle empty/bad job input.
* track latest config message
* pass new config as part of outputs
* persist new config
* persist config as the messages come through, dont set output
* clean up old implementation
* accept control messages for destinations
* get api client from micronaut
* mask instance-wide oauth params when updating configs
* defaultreplicationworker tests
* formatting
* tests for source/destination handlers
* rm todo
* refactor test a bit to fix pmd
* fix pmd
* fix test
* add PersistConfigHelperTest
* update message tracker comment
* fix pmd
* format
* move ApiClientBeanFactory to commons-worker, use in container-orchestrator
* pull out config updating to separate methods
* add jitter
* rename PersistConfigHelper -> UpdateConnectorConfigHelper, docs
* fix exception type
* fmt
* move message type check into runnable
* formatting
* pass api client env vars to container orchestrator
* pass micronaut envs to container orchestrator
* print stacktrace for debugging
* different api host for container orchestrator
* fix default env var
* format
* fix errors after merge
* set source and destination actor id as part of the sync input
* fix: get destination definition
* fix null ptr
* remove "actor" from naming
* fix missing change from rename
* revert ContainerOrchestratorConfigBeanFactory changes
* inject sourceapi/destinationapi directly rather than airbyteapiclient
* UpdateConnectorConfigHelper -> ConnectorConfigUpdater
* rm log
* fix test
* dont fail on config update error
* process control messages for discover jobs
* process control messages for CHECK
* persist config updates on check_connection_for_update
* get last config message rather than first
* fix pmd
* fix failing tests
* add tests
* source id not required for check connection (create case)
* suppress pmd warning for BusyWait literal
* source id not required for checkc onnection (create case) (p2)
* pass id, not full config to runnables/accept control message
* add new config required for api client
* add test file
* remove debugging logs
* rename method (getLast -> getMostRecent)
* rm version check (re-added this in by mistake on merge)
* fix test compatibility
* simplify
* track latest config message
* pass new config as part of outputs
* persist new config
* persist config as the messages come through, dont set output
* clean up old implementation
* accept control messages for destinations
* get api client from micronaut
* mask instance-wide oauth params when updating configs
* defaultreplicationworker tests
* formatting
* tests for source/destination handlers
* rm todo
* refactor test a bit to fix pmd
* fix pmd
* fix test
* add PersistConfigHelperTest
* update message tracker comment
* fix pmd
* format
* move ApiClientBeanFactory to commons-worker, use in container-orchestrator
* pull out config updating to separate methods
* add jitter
* rename PersistConfigHelper -> UpdateConnectorConfigHelper, docs
* fix exception type
* fmt
* move message type check into runnable
* formatting
* pass api client env vars to container orchestrator
* pass micronaut envs to container orchestrator
* print stacktrace for debugging
* different api host for container orchestrator
* fix default env var
* format
* fix errors after merge
* set source and destination actor id as part of the sync input
* fix: get destination definition
* fix null ptr
* remove "actor" from naming
* fix missing change from rename
* revert ContainerOrchestratorConfigBeanFactory changes
* inject sourceapi/destinationapi directly rather than airbyteapiclient
* UpdateConnectorConfigHelper -> ConnectorConfigUpdater
* rm log
* fix test
* dont fail on config update error
* pass id, not full config to runnables/accept control message
* add new config required for api client
* add test file
* fix test compatibility
* mount data plane credentials secret to container orchestrator (#20724)
* mount data plane credentials secret to container orchestrator
* rm copy-pasta
* properly handle empty strings
* set env vars like before
* use the right config vars
Follow up PR to #20787 . Make stats available to the read apis so these are available to the webapp.
After this, all that is left is writing these stats as the job progresses.
Add the required logic in JobHistoryHandler.java.
Took the chance to also rename our internal Attempt models field from id to attemptNumber to better reflect that the field stores not the row's database id, but the job's attempt number. Most of the files changes here are due to that rename.
* pass workspace id to sync workflow and use it to selectively enable field selection
* fix tests around workspace id in job creation
* make sure field selection environment variables get passed through properly
* clean up handling around field selection flags
* debug logging for field selection
* properly handle empty field selection feature flag
* fix pmd
* actually fix pmd
* addlog
* fix applicaion.yml
* remove logging
* var name for boolean
* test setup
* test
* more fix for testing
* self review
* remove unrelated changes
* remove unwanted cdk changes
* more clean ups
* Disable auto detect schema activity bits
* Disable impacted tests
* Disable auto detect schema checks
* Add comment as to why code has been disabled
* Fix PMD warnings
* Fix PMD warning
Co-authored-by: Peter Hu <peter@airbyte.io>
* updated IntegrationLauncherConfig.yaml and added to this class suportDBT and normalizationImage fields. Added to the GenerateInputActivityImpl and TemporalClient classes code parts for read destination_definition.yaml and get suportDBT and normalizationImage fields. Added logging and comparing normalization images from NormalizationRunnerFactory and destination_definition.yaml
* updated minor remarks
* updated minor remarks
* fixed minor remarks
* added normalization data to the tests
* fixed minor remarks
* removed NormalizationRunnerFactory
* fixed remarks
* fixed remarks
* fixed remarks
* updated acceptance tests
* updated acceptance tests
* updated check_images_exist.sh script
* updated method for get normalization image name for destination acceptance test
* fixed code style
* fixed code style and removed tests data
* updated JobErrorReporterTest.java
* updated JobErrorReporterTest.java
* fixed remarks
* added integration type field to the dectination_definition file and actor_definition table
* fixed tests
* fixed tests
* fixed minor changes after pulling master changes
* fixed minor changes after pulling master changes
* renamed integrationType to normalizationIntegrationType/ fixed minor remarks
* renamed extra dependencies
* updated docs
* updated docs
* fixed minor remarks
* added NormalizationDestinationDefinitionConfig.yaml for StandardDestinationDefinition.yaml and updated configuration
* updated normalization tag
* updated DestinationAcceptanceTest.java
* updated DestinationAcceptanceTest.java
* updated imports and descriptions
* Revert "fix: remove unused CONTAINER_ORCHESTRATOR_ENABLED var (#20261)"
This reverts commit ce29361b55.
* docs: add additional commentary on flag usage
* implement column filtering in the replication workflow
* fixes to column selection in replication workflow
* add a basic acceptance test for column selection
* make CI acceptance tests run with new field selection flag enabled
* fix format
* readability improvements around columns selection tests and other small fixes
