* Add deployment mode to env shared with jobs
* Add adaptive runners
* Migrate postgres source to use adaptive runner
* Add an array of specs in docker image spec definition
* Add copyright
* Parse docker image spec with specs list
* Update spec yaml files
* Pass in DEPLOYMENT_MODE to docker compose file
* Revert "Parse docker image spec with specs list"
This reverts commit 8fe41dd3b7.
* Revert changes in docker image spec
* Read cloud specific spec files based on deployment mode
* Revert "Update spec yaml files"
This reverts commit 059f326432.
* Publish cloud spec file if necessary
* Fix upload script
* Move test files
* Update docker compose file
* Format code
* Add comment about spec filename
* Add unit tests
* Remove redundant jdbc acceptance test
When running `PostgresStrictEncryptJdbcSourceAcceptanceTest`, the `discover` method tests always fail because there are unexpected columns in the catalog:
- `wakeup_at`
- `last_visited_at`
- `last_comment_at`
These columns only exist in `PostgresJdbcSourceAcceptanceTest`. And this failure cannot be reproduced locally.
The hypothesis is that when the JDBC unit tests are run on CI, they are run in parallel, and the same testcontainer is used for both tests. That's why the strict encrypt test can discover columns from the oridinary unit test.
Given that the JDBC strict encrypt test is basically redundant, it is removed.
## What
Part 2 of https://github.com/airbytehq/airbyte/pull/13122.
Follow up to #13476 .
Explanation for what is happening:
Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
* Rename airbyte-config:models to airbyte-config:config-models.
* Rename airbyte-config:persistence to airbyte-config:config-persistence.
Part 1 of #13122.
Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.
Explanation for what is happening:
Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
* added AirbyteLoggedException class
* adding in int runr
* changes
* refactored to AirbyteTracedException to align with python impl.
* added catch for Exceptions that are already AirbyteTracedException
* refactor to static class & catch with UncaughtExceptionHandler
* testing ExceptionHandler
* add tests
* added docs section on using AirbyteTraceMessageUtility
* made AirbyteMessage maker methods more intuitive
* fix spotbugs errors
* format
this is an attempt to merge the main change
from https://github.com/airbytehq/airbyte/pull/11514,
which now has multiple conflicts.
The gist of the change
When creating a Postgres destination connector with SSH tunnel method 'SSH Key Authentication', one is required to provide a RSA key. Creating a rsa-sha2-256 or rsa-sha2-512 key, will result in the error SshException: KeyExchange signature verification failed for key type=ssh-rsa, if you haven't enabled ssh-rsa in the SSH server's host key algorithms.
mina-sshd in version 2.7.0 uses the wrong server key signature algorithm during DH group key exchange. https://issues.apache.org/jira/browse/SSHD-1163.
Bumping mina-sshd to version 2.8.0 addresses this issue. Changelog https://github.com/apache/mina-sshd/blob/master/docs/changes/2.8.0.md.
* Revert "Revert "Build OSS branch for deploying to Cloud env (#11474)""
This reverts commit 55e3c1e051.
* add action to get dev branch tag to OSS project instead of doing it in cloud
* remove dev branch version action, going to do this in cloud afterall
* Rebase bigquery changes to master
* Add comments
* Uncomment test code
* Format code
* Bump versions
* Fix denormalized destination target table name
* Fix avro schema for denormalized destination
* Remove unnecessary params from consumer factory
* Add back previous version
* Add warning about standard mode
* auto-bump connector version
* Bump version for bigquery in seed
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add VERSION buildArg to Dockerfiles, default to current airbyte version but overwritable
* use VERSION env var consistently as Dockerfile buildArg, jar version, and tag
pass version and image_tag into docker build task function
* add github action for building and pushing an OSS branch for Cloud to consume
* allow AirbyteVersion to validate versions containing 'oss-branch' prefix
* change oss-branch prefix to dev for branch-based versions
* better action name
* add docker-compose-cloud.build.yaml to define minimum set of cloud images that are pushed by oss branch action
* update local dev docs to describe optional usage of VERSION env var
* make branch_version_tag input optional, if not provided, generates dev-<commit_hash>
* fix typo
* fix missed merge conflict
* update docker docs
* update integrationRunner isDev check
* Add namespace test for snowflake
* Enable namespace test for bigquery
* Format code
* Capitalize test case id
* Update exception message to point to test case file
* Update snowflake name transformer to prepend underscore
* Override convertStreamName instead of getIdentifier
* Add missing state message
* Remove unused import
* Disable more namespace test cases
We don't want to introduce changes that will affect existing connections for now.
* Dry method that mutates namespace
* Pass through null
* Normalize namespace
* Fix test case
* Revert consumer factory changes
* Normalize namespace in catalog
* Revert catalog normalization
* Enable namespace test for all snowflake destination tests
* Test namespace for both bigquery destination tests
* Add unit test for bigquery name transformer
* Transform bigquery schema name
* Fix avro name transformer
* Normalize avro namespace
* Standardize namespace in gcs utils
* Bump version for snowflake and bigquery
* Enable namespace test for bigquery denormalized
* Dry bigquery denormalized acceptance test
* Revert some of the variable scope change
* Fix unit test
* Bump version
* Introduce getNamespace method
* Implement getNamespace method for bigquery
* Switch to getNamespace methods
* Update comments
* Fix bigquery denormalized acceptance test
* Format code
* Dry bigquery destination test
* Skip partition test for gcs mode
* Bump version
* Refactor Snowflake internal Staging as model to share staging abilities in jdbc destinations
* Switch Snowflake Copy Destination for Staging destination based off Internal Staging
Co-authored-by: LiRen Tu <tuliren.git@outlook.com>
* Bumpversion of destination-snowflake
* Pass worker metadata to connector
* Fix compilation
* Pass in job id and image from worker
* Remove application version
* Add default job environment variables
* Add back removed comment
* Rename env map to job metadata
* Fix env configs
* Read connector from application
* Use empty string
* Remove println
* Fix unit test
* Fix compilation error
* Introduce constants for worker env
* Add worker env to ENV_VARS_TO_TRANSFER
* Pass into getWorkerMetadata map to all constructions
* Format code
* Format octavia cli
* Fix test compilation
* Fix typos
* parse jdbc parameters
* Also fix redshift
* other oracle source acceptance test
* This is & now
* This is & now
* This is & now
* This is & now
* This is & now
* also update nne
* increase sleep to 11 seconds
* Bump to 15 seconds
* gradlew format
* try to reformat
* gradlew format
* Run ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates --scan
* reset to master
* Revert "reset to master"
This reverts commit d6141ed933.
* Interrupt child thread of dying connectors to avoid getting stuck
* Catch and print stacktrace
* Add test on interrupt/kill time outs
* Send message to sentry too
* Avoid redundant adapter construction
* Remove unused logger
* Avoid redundant creation of buffer map
* Decrease max batch byte size to 128 mb
* Format code
* Move data adapter to an instance variable
* Bump version
* Bump version in seed
* Add support to duplicate one stream multiple times
* Update cloud version
* Bump version
* Remove unused logger
* Add trace for each stream in mock source
* Fix cloud version
* Bump version in seed
