* implement support for per-stream state in mysql source
* enable flag in tests
* activate per stream in tests
* enable per stream state in test
* bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Fixed bucket naming for S3
* removed redundant configs
* MySQL Source : Standardize spec.json for DB connectors that support log-based CDC replication
* fixed strict encrypt tests
* fixed mysql tests
* bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Emit intermediate state messages from mysql source
* Add pr id
* Try ec2 runner with larger disk
* Revert "Try ec2 runner with larger disk"
This reverts commit fb4d99489d.
* enforce ssl in strict mysql and postgres source
* Add test to verify tunnel aware strict SSL behavior
* bump version numbers and changelogs
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* new ssl flow for connectors
* Bump versions and update changelogs
* Address review comments
* sanity
* Revert Dockerfile versions
* Fix error in test
* The test certificate we use do not contain a valid hostname so we can only verify ca trust but not hostname match
* verify-ca and verify-full SSL modes are not related to whether a client certificate is used. Make client certificate optional on both modes
* Bump Dockerfile versions and correct change notes.
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix handling for temporal data type columns with default values
* add tests
* minor NIT comment
* bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add logic to skip incremental mode for tables which do not have a field types that could not be used as cursor
* added tests for cursor fields incremental updates
* remove not used methods
* fix tests
* fix formatting
* revert RedshiftInsertDestinationAcceptanceTest
* fix ssl clickhouse test
* fix codereview comments
* fix clickhouse strict encrypt test
* fix postgres source after merge from master
* optimize ssl clickhouse test logic
* fix db2 test
* added check for cursor field should be not nullable, fix formatting
* fixed tests, formatting, issue after merge from master
* fix merge issues
* fix test for source-jdbc and remove some code used for local testing
* added description for cockroach db tests
* test changes
* refactoring for cursor field tests after code review comments
* refactoring for cursor field tests after code review comments
* fix code review comments
* remove unused imports
* fix formatting
* fix compilation error
* fix postgres and postgress ssl test
* Fixed bucket naming for S3
* remove CHAR, NCHAR from possible cursor types
* remove SYMBOL from possible cursor types
* removed redundant configs
* fixed mysql-strict-encrypt tests
* fixed mongodb-v2 tests
* minor format changes
* bump version
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: vmaltsev <ihbectop@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* updated mysql source specification and added field for root and clients SSL certificates
* added SSL mode for mysql source
* fixed code style
* updated run process timeout
* updated method for create keystore and updated tests
* updated normalization version for postgres destination
* updated normalization version for postgres destination
* added tests for connection with certificates
* updated tests for connection with full certificates and added tests for CA certificate
* updated tests
* updated source-mysql-strict-encrypt and updated versions
* updated code style
* updated doc
* updated specs
* fixed minor remarks
* fixed minor remarks
* updated tests
* fixed remarks and updated specification
* fixed mysql sources connectors version
* added CDC + SSL Certificates tests
* added property for CDC and added tests for test SSL with CDC together
* fixed MySqlStrictEncryptJdbcSourceAcceptanceTest for work with datetime format
* added property for CDC and added tests for test SSL with CDC together
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Allow additional properties in db stream state
* Bump postgres version
* Bump mysql version
* Bump mssql version
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* generate airbyte_type:integer
* normalization accepts `airbyte_type: integer`
* handles ints+longs
* update avro for consistency
* delete long type for now, treat all ints as longs
* update avro type mappings
{type:number, airbyte_type:integer} -> long
{type:number, airbyte_type:big_integer} -> string (i.e. "unbounded integer")
* fix test
* remove long handling
* Revert "remove long handling"
This reverts commit 33ade8d2831e675c3545ac6019d200ec312e54d9.
* Revert "update avro type mappings"
This reverts commit 5b0349badad7545efe8e1191291a628445fe1c84.
* Revert "delete long type for now, treat all ints as longs"
This reverts commit 018efd4a5d0c59f392fd8e3b0d0967c666b72947.
* Revert "update avro for consistency"
This reverts commit bcf47c6799b5906deb4f219d7f6e64ea73b41b74.
* newline@eof
* update test
* slightly better local tests
* fix test
* missed a few cases
* postgres tests use correct hostnames
* fix normalization
* fix int macro
* add test case
* normalization test output
* handle int/long correctly
* fix types for other DBs
* uint32 -> bigint; tests
* add type value assertions
* more test updates
* regenerate output
* reconcile big_integer to match docs
* update comment
* fix type
* fix mysql constructor call
* bigint only has 38 digits
* fix s3 ints, fix DAT test case
* big_integer should be string
* reduce to 28 digit big_ints
* fix test setup, mysql
* kill big_integer tests
* regenerate output
* version bumps
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Default scaffold to use adaptive streaming config
* Switch more connectors to use adaptive streaming config
* Bump version for cockroach db
* Bump version for db2
* Bump mssql version
* Bump mysql version
* Bump oracle version
* Bump postgres version
* Bump redshift version
* Bump snowflake version
* Bump tidb version
* auto-bump connector version
* Fix db2 findbug issue
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
* Fix more findbug issues
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
* Fix findbug issue for mysql-strict-encrypt
* Fix findbugs issue for oracle source
* auto-bump connector version
* Remove suppress warnings annotation
* Fix oracle encrypt tests
* Fix oracle encrypt acceptance test
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Check binlog position on mysql server before run sync job, add error description into log
* fix MySqlStrictEncryptSourceAcceptanceTest
* fix formatting
* fix review comments
* added java docs and fixed few minor comments
* fix formatting
* update versions
* update source_specs.yaml
* Add jdbc compatible layer
* Support routine mysql types
* Format code
* Fix build
* Refactor abstract jdbc source and operation classes
* Update mysql source operations
* Test discover command for mysql
* Remove abstract jdbc compatible source layer
* Format code
* Update template
* Fix more types
* Bump version
* Log original field type
* Update comments
* Bump version in seed
* Source-MySql: do not check cdc required param binlog_row_image for standard replication
* Source-MySql: fix formatting
* Revert "Source-MySql: fix formatting"
This reverts commit 5a94474a52.
* Source-MySql: made a code improvement
* Source-MySql: bump versions
* Source-MySql: fix version in source_specs.yaml
* Source-MySql: bump versions
* Source-MySql:update source_definitions.yaml with new version
* Source-MySql: transform binary data base64 format, add integration tests
* Source-MySql: fix code style
* Source-MySql: bump versions
* Source-MySql: bump versions in source_specs.yaml
* Source-MySql: added test for stream with binary data for DestinationAbstractTest
* Source-MySql: added format
* BUG-5842: Mysql Value '65535' is outside valid range for type java.lang.Short
* BUG-5842: Bumped versions of connector.
* BUG-5842: Fixed unit-test for mysql-encrypt
* Update docs/integrations/sources/mysql.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* move logic from static JdbcUtils to JdbcSourceOperations
* format
* Add methods for all types in order to have possibility rewrite them and use common type mapping.
* Make BigQuery in line impl with JDBC changes
* fix binary type
* add database creations methods with sourceOperations input
* add MySqlSourceOperations
* fix processing
* Fix CDC processing
* format
* add some tests for double and float
* incr version
* #3931#3932 added zero-date converting to null param "zeroDateTimeBehavior=convertToNull"
* remove old tests covered by data type tests + incr ver
* Update airbyte-integrations/connectors/destination-mysql/src/main/java/io/airbyte/integrations/destination/mysql/MySQLDestination.java
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Update docs/integrations/sources/mysql.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* add back comments to the data type mapping
* incr config version
Co-authored-by: Oleksandr Sheheda <alexandrshegeda@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Clarify ssh private key format for ssh tunnels
* Improved SSH Tunnel key generation steps, fixed formatting
* Modified wording 'app' to 'connector' for consistency
* Ran format
