* Revert "Revert "Bmoric/temporal cleaning cron (#16414)" (#17192)"
This reverts commit 751667949e.
* Generate an artifact
* Change missing property from value to property
* remove default value from namespaceDefinitionType
* fix test
* switch some tests to use PATCH-style connection update
* add default logic to connection creation
Co-authored-by: Michael Siega <michael@airbyte.io>
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Add the injected temporal client to airbyte-cron
* Update cron to make it work
* Remove useless Bean factory
* PR comments
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Improve conditional bean check
* Match existing Optional functionality
* Add notEquals check
* Add missing env var to Helm chart
* Fix typo
* Mark LogConfigs as Singleton
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Improve conditional bean check
* Match existing Optional functionality
* Add notEquals check
* Add missing env var to Helm chart
* Fix typo
* Mark LogConfigs as Singleton
* Env vars for log/state storage type
* Remove use of Optional in bean declarations
* Fix typo in config property name
* Support Temporal Cloud namespace
* Change to @Value
* Use correct value for conditional check
* Upgrade Micronaut
* Fix merge conflict
* Formatting
* Add missing env var
* Use sync task queue environment variable
* Handle sync task queue as set
* format and force http
* Handle case where sync task queue is empty
* Add correct path to config property
* Remove unused import
* Remove conflict
* Remove unused parameter
* Formatting
* Use pattern for condition process factory beans
* Cleanup
* PR feedback
* Revert hack for testing
* Fix temporal restart by status (#16447)
* Update application.yml
* Re add worker dep
* Add missing env var
* PR comments
* Bmoric/move temporal client (#16778)
* tmp
* tmp
* View diff
* Move part of the temporal client
* tmp
* copy Temporal Utils test
* Uniq Temporal Utils
* Uniq Temporal Workflow Utils
* Move CancellationHandler
* Move commons-temporal to being shared between platform, connector and CLI
* Rm worker dependency from cron
* Fix build
* Update with right value for cron.
* Fix dep conflict
Co-authored-by: jdpgrailsdev <jpearlin1@gmail.com>
Co-authored-by: pmossman <parker@airbyte.io>
Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io>
* remove operationIds from WebBackendConnectionUpdate, just use operations
* refactor connection updates to patch-style update, where null fields remain unchanged
* better comment and arg name
* format
* make sure we are still 'dual-writing' to the old schedule column, even when the patch doesn't specify anything for it
* update acceptance test to update with new schedule syntax
* add catalog sorting to preserve stream order during patch, and more tests
* format
* add description, throw runtime exception for impossible branches, move streamReset to private helper
* PR suggestions
* add nested test classes and write a test for the catalog sorting method
* format
* add comment clarifying that the catalog sort is for UX, and isn't critical
* format
* format
* update acceptance tests to send proper catalog patches instead of whole new catalog
* format
* format
* simplify catalog patching - now, if a catalog is present on the request, replace the entire catalog with it.
Otherwise, if catalog on the request is null, leave the catalog unchanged
* format
* format
* Revert "update acceptance tests to send proper catalog patches instead of whole new catalog"
This reverts commit 71922648b4e070f46ff6c468813b7ab8dd9d6651.
* adjust description
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Improve conditional bean check
* Match existing Optional functionality
* Add notEquals check
* Add missing env var to Helm chart
* Fix typo
* Mark LogConfigs as Singleton
* Env vars for log/state storage type
* Remove use of Optional in bean declarations
* Fix typo in config property name
* Support Temporal Cloud namespace
* Change to @Value
* Use correct value for conditional check
* Upgrade Micronaut
* Fix merge conflict
* Formatting
* Add missing env var
* Use sync task queue environment variable
* Handle sync task queue as set
* format and force http
* Handle case where sync task queue is empty
* Add correct path to config property
* Remove unused import
* Remove unused parameter
* Formatting
* Use pattern for condition process factory beans
* Cleanup
* PR feedback
* Revert hack for testing
Co-authored-by: pmossman <parker@airbyte.io>
* update acceptance tests to use new scheduling format now that frontend has moved over
* readability improvements in sync schedule tests
* fix pmd issues in new acceptance test
* Add is_test to dbConfig
* temp fix pmd just for testing
* remove strict-enforce postgres specifics
* cleanup spacing
* move common vars to test harness
* Add comment
* save
* clean up more usages and remove withRefreshedCatalog
* make webapp use correct endpoint
* add back intercept
* fix acceptance test
* fix log
* remove 'new' from test name
Bug fix from #14310 . The build was broken then from before #15497 so this was hidden.
- Add the new env vars to the integration test yamls.
- Also make sure to properly close the destination test container db.
It's difficult today to parse all the logs from tests. Engineers waste time scrolling through the log outputs and looking for the relevant stack trace.
This PR adds an action to generate a JUnit report so devs can understand test results at a glance. This generates 3 reports for each of the main build jobs when the build completes. We leave the frontend build out since this is aggregated by cypress.
See https://github.com/airbytehq/airbyte/pull/15271/checks?check_run_id=7683783016 for an example of how this works.
Use the https://github.com/dorny/test-reporter action and configure this to look at the Jacoco test report output for top level and second level builds. Note that most of the parameters into the action don't work.
Comment out the trap commands to output logs. Though this can be useful for debugging, there is little practical use in the day-to-day, and results in extremely noisy logs.
* Migrated 'ssl' string to become a constant in JdbcUtils
* Migrated 'database' string to become a constant in JdbcUtils
* Migrated 'username' string to become a constant in JdbcUtils
* Migrated 'password' string to become a constant in JdbcUtils
* Migrated 'schema' string to become a constant in JdbcUtils and updated corresponding config params
* Migrated 'jdbc_url' string to become a constant in JdbcUtils
* Migrated 'connection_properties' string to become a constant in JdbcUtils
* Migrated list of 'host_key' string to become a constant in JdbcUtils
* Migrated list of 'port_key' string to become a constant in JdbcUtils
* Migrated 'host_key' string to become a constant in JdbcUtils and updated corresponding config params
* Migrated 'port_key' string to become a constant in JdbcUtils
* Migrated 'tls' string to become a constant in JdbcUtils
* Migrated 'schemas' string to become a constant in JdbcUtils
* Migrated 'encryption' string to become a constant in JdbcUtils
* Migrated 'jdbc_url_params' string to become a constant in JdbcUtils and fixed uncaught parenthesis
* Fixed linter issues with comments on AcceptanceTest and failed import statement
* set per stream feature flag to true for testing
* add a second table to cdc acceptance tests
* add partial reset test
* format
* add partial reset cdc tests
* test incremental after partial reset
* remove dev image from acceptance test
* fix flag and add comment
* Revert "set per stream feature flag to true for testing"
This reverts commit 164d7da05990268b09e315eb88ff297d3a9f52f4.
* set USE_STREAM_CAPABLE_STATE flag to true in acceptance test script
* call new update endpoint
* use methods in test harness instead
* remove comment
* add env var to worker container
* format
* fix state check in basic acceptance test
* use test info for test name logging
* Re-add acceptance test
* Re-adda acceptance test
* Format
* Fix acceptance test
* Add log
* remove unwanted changes
Co-authored-by: lmossman <lake@airbyte.io>
* Use Debezium Postgres image for CDC tests
* Formatting
* add cdc acceptance tests
* make method public
* format
* add update destination definition version method
* wait for successful job
* add acceptance tests for incremental with old dest, and for delete + some refactoring
* format
* fix assignment of sourceDbConfig
* fix init
* remove logs
* increase timeout on deleteConnection test to prevent transient failures
* TEMPORARY add logs to help with debugging CI failures
* add -i to acceptance test for debugging
* add back destinationPsql start to see if it fixes the issue
* Revert "add -i to acceptance test for debugging"
This reverts commit 693a955041.
* Revert "TEMPORARY add logs to help with debugging CI failures"
This reverts commit a30efd2c54.
* fix timestamp comparison
Co-authored-by: jdpgrailsdev <jpearlin1@gmail.com>
* Refactor Basic acceptance tests and utils
* Refactor Advanced acceptance tests and utils
* Remove unused code
* Clear destination db data during cleanup
* Cleanup comments
* cleanup init code
* test creating new desintation db for each test
* cleanup desintation db init
* Allow to edit api client
* pull in temporal cloud changes
* Rename helper to harness; set some funcs to private; turn init into constructor
* add func to set env vars instead of using static vars and move some functionality out of init into acceptance tests
* update javadoc
Co-authored-by: Davin Chia <davinchia@gmail.com>
* fix javadoc formatting
* fix var naming
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Switch to measure max row byte size
* Reduce fetch size change logs
* Update unit tests
* Determine jdbc buffer size based on max memory
* Bump postgres version
* Bump postgres version
* Bump mysql version
* Bump mssql version
* Format java code
* Increase hikari connection timeout
* Update data source default parameters
* auto-bump connector version
* Mark postgres 0.4.21 as not published
* Revert "Bump mysql version"
This reverts commit ad9135258c.
* Fix unit test
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Part 1 of #13122.
Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.
Explanation for what is happening:
Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
# Summary
- A follow-up PR for #5543.
- This PR separates the `airbyte-db` project to two modules:
- `lib` is the original `airbyte-db`.
- `jooq` is for jOOQ code generation.
- This is necessary because the jOOQ generator requires a custom database implementation that can run Flyway migration. So the code generator logic needs to depend on the compilation of the original `airbyte-db` project.
# Commits
* Separate db to lib and jooq modules
* Update dependencies
* Add jobs db migrator test
* Fix compose build
* Add migration dev center
* Add schema dump task
* Update airbyte-db/lib/README.md
* Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update readme
* Remove bom dependency
* Update readme
* Use jooq code in db config persistence
* Remove AirbyteConfigsTable
Co-authored-by: Davin Chia <davinchia@gmail.com>
