#19191 made me realise the DefaultReplicationWorker's metric tracking today has a bug where we aren't accounting for namespace when tracking metrics today. i.e. Streams with the same name and duplicate namespace will merge metrics.
While reading the code to figure out a fix, I realised we don't have a good conceptual representation of stream namespace <> name pairs within the platform today. We use a concatenated string. Though this works, it will become harder and harder to read/track as we do more operations that involve namespace i.e. progress bars and column selection.
This PR introduces the AirbyteStreamNameNamespacePair object into the platform code to make it more convenient to work with Streams in the future. (Especially if we proceed with the project to make streams a first-class citizen!)
The AirbyteStreamNameNamespacePair object was written to deal with the same issue of namespace <> name pair manipulation within the Java destination code. It implements the Comparable interface, which makes it convenient to use for Collections operations.
For an example of how this is consumed, see #19361.
* Extract Operation API
* Extract scheduler API
* Format
* extract source api
* Extract source definition api
* Add path
* Extract State API
* extract webbackend api
* extract webbackend api
* extract workspace api
* Source postgres: encode database name
* Source postgres, mysql: move encoding in util class, apply for mysql
* Source postgres, mysql: make var final
* Source postgres, mysql: bump version
* Source postgres, mysql: format code
* auto-bump connector version
* Source mysql: bump version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Changes method from returning null to throwing Exception to match method structure
* Added javadoc explaining that KeyPair should not be returning null
* Moved ConnectionErrorException to commons package
* Adds error handling for SshWrappedDestinations and markdown to include changes
* Bumps version numbers for Postgres, MySQL, MSSQL
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* 16250 Destination Redis: Add SSH support
* 16250 Resolve port issue
* 11679 Bump version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Add support for using URL in tunneled config
* Add support for SSH tunneling to destination-elasticsearch and destination-elasticsearch-strict-encrypt
* Adds logic to fail upon non-deterministic custom S3 endpoint and documentation for insecure settings
* Reused config factory settings to a single static variable
* Updated error message and example in the spec.json to match expectation of secured endpoint
* Added validation check within the base s3
* Integrated AdaptiveDestinationRunner with S3Destination
* Reduced visibility for testing and fixed AdaptiveDestinationRunner issue
* Adds speicifc secure protocol with S3 and empty endpoint check
* Bumps docker version and adds comments and clearer string methods
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Use LOG4J2 to wrap connectors logs to JSON format
* log connector's stack trace directly as a message
* add stack_trace field to json template
* bump versions
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* revert versions for destinations: postgres, mssql, mysql
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Cleaned error messages factory PR
* Bumped MySQL and Postgres version
* Fixed messages and typos in test
* Fixes the changelog conflict with per-stream state
* Added note for flaky test
* Bumps mysql version to match changelog
* Added exception objects to all LOGGER.error for more visibility
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* new ssl flow for connectors
* Bump versions and update changelogs
* Address review comments
* sanity
* Revert Dockerfile versions
* Fix error in test
* The test certificate we use do not contain a valid hostname so we can only verify ca trust but not hostname match
* verify-ca and verify-full SSL modes are not related to whether a client certificate is used. Make client certificate optional on both modes
* Bump Dockerfile versions and correct change notes.
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* updated mysql source specification and added field for root and clients SSL certificates
* added SSL mode for mysql source
* fixed code style
* updated run process timeout
* updated method for create keystore and updated tests
* updated normalization version for postgres destination
* updated normalization version for postgres destination
* added tests for connection with certificates
* updated tests for connection with full certificates and added tests for CA certificate
* updated tests
* updated source-mysql-strict-encrypt and updated versions
* updated code style
* updated doc
* updated specs
* fixed minor remarks
* fixed minor remarks
* updated tests
* fixed remarks and updated specification
* fixed mysql sources connectors version
* added CDC + SSL Certificates tests
* added property for CDC and added tests for test SSL with CDC together
* fixed MySqlStrictEncryptJdbcSourceAcceptanceTest for work with datetime format
* added property for CDC and added tests for test SSL with CDC together
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* added ssl certificates for postgres source
* added command for remove client private key after transformation to encrypted key with .pk8 extension
* added connection with CA and client certificates for postgres destination
* updated code style
* moved common methods to the common class
* moved common methods to the common class
* fixed remarks
* updated postgres source tests
* added minor changes to spec and added fixes to password mechanism
* updated postgres source tests
* updated strict-encrypt postgres source and destination and added tests for SSL certificates for all postgres connectors
* fixed check style
* updated documentation and versions of connectors
* updated ordrs in test spec
* fixed minor remarks in specs and expected_specs
* fixed minor remarks in specs and expected_specs
* fixed Dockerfile
* fixed remarks
* fixed remarks
* fixed remarks
* fixed remarks
* fixed remarks
* fixed code style
* fixed connectors version in definition file
* updated postgres destination normalization
* updated postgres destination tests
* fixed code style for postgres source and destination
* pulled master changes
* removed allow mode for destination-postgres-strect-encrypt
* updated connectors version
* fixed custom DBT transformation and enabled test for it
* updated normalization version
* updated keystore password generation method
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* added ssl certificates for postgres source
* added command for remove client private key after transformation to encrypted key with .pk8 extension
* added connection with CA and client certificates for postgres destination
* updated code style
* moved common methods to the common class
* moved common methods to the common class
* fixed remarks
* updated postgres source tests
* added minor changes to spec and added fixes to password mechanism
* updated postgres source tests
* updated strict-encrypt postgres source and destination and added tests for SSL certificates for all postgres connectors
* fixed check style
* updated documentation and versions of connectors
* updated ordrs in test spec
* fixed minor remarks in specs and expected_specs
* fixed minor remarks in specs and expected_specs
* fixed Dockerfile
* fixed remarks
* fixed remarks
* fixed remarks
* fixed remarks
* fixed remarks
* rollback changes for postgres destination and move them to a new pull request
* rollback changes for postgres destination and move them to a new pull request
* fixed code style
* fixed code style
* fixed code style
* fixed code style
* fixed code style
* auto-bump connector version
* updated version of postgres strict-encrypt source
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Add deployment mode to env shared with jobs
* Add adaptive runners
* Migrate postgres source to use adaptive runner
* Add an array of specs in docker image spec definition
* Add copyright
* Parse docker image spec with specs list
* Update spec yaml files
* Pass in DEPLOYMENT_MODE to docker compose file
* Revert "Parse docker image spec with specs list"
This reverts commit 8fe41dd3b7.
* Revert changes in docker image spec
* Read cloud specific spec files based on deployment mode
* Revert "Update spec yaml files"
This reverts commit 059f326432.
* Publish cloud spec file if necessary
* Fix upload script
* Move test files
* Update docker compose file
* Format code
* Add comment about spec filename
* Add unit tests
* Remove redundant jdbc acceptance test
When running `PostgresStrictEncryptJdbcSourceAcceptanceTest`, the `discover` method tests always fail because there are unexpected columns in the catalog:
- `wakeup_at`
- `last_visited_at`
- `last_comment_at`
These columns only exist in `PostgresJdbcSourceAcceptanceTest`. And this failure cannot be reproduced locally.
The hypothesis is that when the JDBC unit tests are run on CI, they are run in parallel, and the same testcontainer is used for both tests. That's why the strict encrypt test can discover columns from the oridinary unit test.
Given that the JDBC strict encrypt test is basically redundant, it is removed.
## What
Part 2 of https://github.com/airbytehq/airbyte/pull/13122.
Follow up to #13476 .
Explanation for what is happening:
Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
* Rename airbyte-config:models to airbyte-config:config-models.
* Rename airbyte-config:persistence to airbyte-config:config-persistence.
