Follow up to #18953.
Implement all the DB migrations required for a progress bar.
The main change here is to support saving:
the estimated records/bytes at the sync level
the estimated records/bytes and emitted records/bytes at the stream level
After this, I'll put up a PR for the persistence layer changes, which will writing to and reading from these columns.
Finally, I'll wire this into the API changes, which are currently stubs.
- add the estimated_records and estimated_bytes columns to the SyncStats table.
- create a stream_stats table
- estimated and emitted records/bytes column
- contains attempt_id and stream_name columns. Unique constraints on these two columns.
- foreign key to the attempt_id table.
- this table hopefully sets us up for the parallel sync work.
* Add missing indices for `status` columns
* Remove TODO and delete .class files
* Dump schema
* Run gradle format
* Bump database schema version in Bootloader test
* Postgres/MySQL Source Strict Encrypt: stop enforce SSL if ssl mode disabled
* fixed checkstyle
* updated changelog
* add tests
* replaced MySQL test to mysql-strict-encrypt module
* fixed Connection Refused for mysql test
* replaced Postgres Source strict-encrypt tests into new class
* bump version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Adds TLS check to mongodb destination and migrates util constants
* Migrates MongodbSourceUitls to general purprose Utils file
* Updates expected_spec.json to include SSH tunnel
* Bumps connector version and removes connector from being hidden in UI
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* updated StandardDestinationDefinition.yaml, added normalization and tags to the destination_definition.yaml and added information about normalization and DBT to the ACTOR_DEFINITION table
* updated docs
* updated BootloaderAppTest.java for new migration
* updated schema dump
* Update normalization version and fix bigquery
* Use varchar 255
* Update migration version to the latest
* Update normalized table schema file and add comment
* Revert "Use varchar 255"
This reverts commit e182466e66.
* Use varchar 255
* Add unit test for migration
* Format code
Co-authored-by: Liren Tu <tuliren@gmail.com>
* Changes method from returning null to throwing Exception to match method structure
* Added javadoc explaining that KeyPair should not be returning null
* Moved ConnectionErrorException to commons package
* Adds error handling for SshWrappedDestinations and markdown to include changes
* Bumps version numbers for Postgres, MySQL, MSSQL
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* postgres-source: handle 24:00:00 value for time column
* address review comment
* fix test + use LocalTime.MAX
* bump version
* update alloy db as well
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* init commit of geography_type and column
* add schema dump
* confirm prod database was meant to be changed
* add AUTO to geography
* set default and not nullable, also edit workspace table
* update version number
* update schema dump
Co-authored-by: pmossman <parker@airbyte.io>
* Match entry in client cert keystore to hardcoded postgres driver value
* Match entry in client cert keystore to hardcoded postgres driver value
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add icon for clickhouse in destination folder
* use http port only in clickhouse
* declare driver: http for dbt explicitly
* bump destination clickhouse version
Co-authored-by: restrry <restrry@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Improve conditional bean check
* Match existing Optional functionality
* Add notEquals check
* Add missing env var to Helm chart
* Fix typo
* Mark LogConfigs as Singleton
* Env vars for log/state storage type
* Remove use of Optional in bean declarations
* Fix typo in config property name
* Support Temporal Cloud namespace
* Change to @Value
* Use correct value for conditional check
* Upgrade Micronaut
* Fix merge conflict
* Formatting
* Add missing env var
* Use sync task queue environment variable
* Handle sync task queue as set
* format and force http
* Handle case where sync task queue is empty
* Add correct path to config property
* Remove unused import
* Remove unused parameter
* Formatting
* Use pattern for condition process factory beans
* Cleanup
* PR feedback
* Revert hack for testing
Co-authored-by: pmossman <parker@airbyte.io>
* Cleaned error messages factory PR
* Bumped MySQL and Postgres version
* Fixed messages and typos in test
* Fixes the changelog conflict with per-stream state
* Added note for flaky test
* Bumps mysql version to match changelog
* Added exception objects to all LOGGER.error for more visibility
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Define logic in root build.gradle to create a task in each subproject if the subproject contains dockerImageName in the gradle.properties file.
Some caveats:
- We also define a copyGeneratedTar task that is applied to all subprojects with the dockerImageName property.
- This does not need to be used in each subproject. We leave it up to each subproject to define what task is depends on since some projects have more custom copy logic.
* add migration
* fmt
* test migration
* update expected migration version
* update schema
* allow specifying a target migration version for the baseline
* run previous migrations in test
* fix pmd
* update migration version
* mark libs.platform.testcontainers.postgresql as implementation dep
* use migrations configuration for mig task dependencies
* shuffle things around
* Update build.gradle
* new ssl flow for connectors
* Bump versions and update changelogs
* Address review comments
* sanity
* Revert Dockerfile versions
* Fix error in test
* The test certificate we use do not contain a valid hostname so we can only verify ca trust but not hostname match
* verify-ca and verify-full SSL modes are not related to whether a client certificate is used. Make client certificate optional on both modes
* Bump Dockerfile versions and correct change notes.
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix temporal datatype bug for columns with default in postgres cdc
* fix test
* add test for date and time as well
* add more logs for unknown classes
* review comments
* bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add logic to skip incremental mode for tables which do not have a field types that could not be used as cursor
* added tests for cursor fields incremental updates
* remove not used methods
* fix tests
* fix formatting
* revert RedshiftInsertDestinationAcceptanceTest
* fix ssl clickhouse test
* fix codereview comments
* fix clickhouse strict encrypt test
* fix postgres source after merge from master
* optimize ssl clickhouse test logic
* fix db2 test
* added check for cursor field should be not nullable, fix formatting
* fixed tests, formatting, issue after merge from master
* fix merge issues
* fix test for source-jdbc and remove some code used for local testing
* added description for cockroach db tests
* test changes
* refactoring for cursor field tests after code review comments
* refactoring for cursor field tests after code review comments
* fix code review comments
* remove unused imports
* fix formatting
* fix compilation error
* fix postgres and postgress ssl test
* Fixed bucket naming for S3
* remove CHAR, NCHAR from possible cursor types
* remove SYMBOL from possible cursor types
* removed redundant configs
* fixed mysql-strict-encrypt tests
* fixed mongodb-v2 tests
* minor format changes
* bump version
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: vmaltsev <ihbectop@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* updated mysql source specification and added field for root and clients SSL certificates
* added SSL mode for mysql source
* fixed code style
* updated run process timeout
* updated method for create keystore and updated tests
* updated normalization version for postgres destination
* updated normalization version for postgres destination
* added tests for connection with certificates
* updated tests for connection with full certificates and added tests for CA certificate
* updated tests
* updated source-mysql-strict-encrypt and updated versions
* updated code style
* updated doc
* updated specs
* fixed minor remarks
* fixed minor remarks
* updated tests
* fixed remarks and updated specification
* fixed mysql sources connectors version
* added CDC + SSL Certificates tests
* added property for CDC and added tests for test SSL with CDC together
* fixed MySqlStrictEncryptJdbcSourceAcceptanceTest for work with datetime format
* added property for CDC and added tests for test SSL with CDC together
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* added ssl certificates for postgres source
* added command for remove client private key after transformation to encrypted key with .pk8 extension
* added connection with CA and client certificates for postgres destination
* updated code style
* moved common methods to the common class
* moved common methods to the common class
* fixed remarks
* updated postgres source tests
* added minor changes to spec and added fixes to password mechanism
* updated postgres source tests
* updated strict-encrypt postgres source and destination and added tests for SSL certificates for all postgres connectors
* fixed check style
* updated documentation and versions of connectors
* updated ordrs in test spec
* fixed minor remarks in specs and expected_specs
* fixed minor remarks in specs and expected_specs
* fixed Dockerfile
* fixed remarks
* fixed remarks
* fixed remarks
* fixed remarks
* fixed remarks
* fixed code style
* fixed connectors version in definition file
* updated postgres destination normalization
* updated postgres destination tests
* fixed code style for postgres source and destination
* pulled master changes
* removed allow mode for destination-postgres-strect-encrypt
* updated connectors version
* fixed custom DBT transformation and enabled test for it
* updated normalization version
* updated keystore password generation method
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix postgres data handling from WAL logs in CDC mode
* format
* use formatter for dates also (#15485)
* format
* change test structure
* change log to debug
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
* Extract connectionTimeout from jdbc_url_params along with corresponding tests
* Fixed linter issues
* Reverted createDataSourceWithConnectionTimeout and migrated logic to get operation
* Fixed dangling createDataSourceWithConnectionTimeout and linter issues
* Fixed import to use java standard library
* Bump Postgres Source and Postgres Source Strict Encrypt versions
* Fixed import ordering issues
* Bumped the connector version [CI fix] for definitions not generated
* 13608 & 12026 - align regular and CDC integration tests and data mappers
* format code
* update int handling
* fix build
* fix PR remarks
* revert changes for money type that are broken by #7338
* bump version
* 🐛 Source Postgres: Improve BCE date handling (#15187)
* 13608 & 12026 - align regular and CDC integration tests and data mappers
* format code
* update int handling
* borked merge - re-delete deleted methods
* enable catalog tests for postgres
* fix build
* fix PR remarks
* revert changes for money type that are broken by #7338
* update BCE handling in JDBC
* reuse existing method
* handle bce dates
* inline methods
* fix JDBC BCE year inconsistency
* use correct data type in test
* format
* Update airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/AbstractPostgresSourceDatatypeTest.java
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
* pmd fix
* use class.getname()
* fix pmd
* format
* bump version
* handle incremental mode
* clean up diff
* more comments
* unused imports
* format
* versions+changelog
Co-authored-by: Yurii Bidiuk <yura.bidyuk@gmail.com>
Co-authored-by: Yurii Bidiuk <35812734+yurii-bidiuk@users.noreply.github.com>
* auto-bump connector version [ci skip]
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
