* Add skeleton to support all postgres types
* Consolidate type tests
* Fix corner cases
* Bump postgres version
* Add tests for time and timetz
* Format code
* Revert date to timestamp
* Update comment
* Fix unit tests
* 🐛 Jdbc sources: switch from "string" to "array" schema type for columns with JDBCType.ARRAY (#8749)
* support array for jdbc sources
* fixed PR comments, added test cases
* added more elements for test case
* Fix test case
* add array test case for JdbcSourceOperations
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
* Revert changes to support special number values
Postgres source cannot handle these special values yet
See https://github.com/airbytehq/airbyte/issues/8902
* Revert infinity and nan assertion in unit tests
This reverts commit 3bee7d19ea.
* Update documentation
* Bump postgres source version in seed
Co-authored-by: Yurii Bidiuk <35812734+yurii-bidiuk@users.noreply.github.com>
* updated ci performance test sh script
* updated ci performance test sh script
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* set all params to one arg
* run tests
* update ci creds
* update performance test yaml file
* update performance test yaml file
* update performance test yaml file
* update performance test yaml file
* update performance test sh file
* update performance test sh file
* update performance test sh file
* update performance test sh file
* update java to 17 version in yaml file for performance tests
* update java to 17 version in yaml file for performance tests
* update java to 17 version in yaml file for performance tests
* update java to 17 version in yaml file for performance tests
* update java to 17 version in yaml file for performance tests
* clean test logs
* remove test logs
* remove test logs
* remove test logs
* return performance test logs
* add readme by performance tests
* Added RDS base performance tests for source-postgres, source-mssql and source-mysql
* updated perfomance test with cpu and memory limit
Co-authored-by: andriikorotkov <andrii.h.korotkov@globallogic.com>
* Add jdbc compatible layer
* Support routine mysql types
* Format code
* Fix build
* Refactor abstract jdbc source and operation classes
* Update mysql source operations
* Test discover command for mysql
* Remove abstract jdbc compatible source layer
* Format code
* Update template
* Fix more types
* Bump version
* Log original field type
* Update comments
* Bump version in seed
* Fix postgres money type > 1000
* Convert money to double
* Bump source postgres version
* Do not bump version in seed yet
* Check type ignoring case
* Test insertion with currency symbol
* Test parsing money string method
* Add test case with currency symbol and negative values
* Update version date
* Bump seed version
* airbyte-8254 Source Postgres: views are not listed during schema discovery
* airbyte-8254 Source Postgres: views are not listed during schema discovery
* airbyte-8254 Source Postgres: views are not listed during schema discovery
* airbyte-8254 Remove GRANTEE and PRIVILEGE constants from AbstractJdbcSource.java
* airbyte-8254 Version bumped.
* airbyte-8254 Version bumped.
* airbyte-8254 Version bumped.
* Add option to render fields in HTML format and fix sprint_issue ids
* Bump connector version
* run format
* run format
* run seed generaotr
* run format
Co-authored-by: Chris Wu <chris@faros.ai>
* airbyte-5172 Do not list tables we do not have access to in the connection settings
* airbyte-5172 Do not list tables we do not have access to in the connection settings
* airbyte-5172 Do not list tables we do not have access to in the connection settings
* airbyte-5172 Do not list tables we do not have access to in the connection settings
* airbyte-5172 Added docs, bump dockerfiles
* airbyte-5172 Fix PR comments.
* airbyte-5172 Fixed PR comments.
* airbyte-5172 bumped connector version.
* Fix data type tests in CdcPostgresSourceComprehensiveTest
* update style format
* bump version for postgres source
* bump version for postgres source in json definition
* remove unnecessary comments from test and bump version for postgres strict encrypt source
* resolved potential conflicts with debezium utils in mssql converter implementation
* resolved potential conflicts with debezium utils in mssql converter implementation
* Update notes for money type in postgres.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Update docs/integrations/sources/postgres.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* added test cases for converting data values for postgres cdc, remove time zone utc from test container
* remove redundant void message from test
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* update style format
* fix time zone in DebeziumConverterUtilsTest
* set utc time zone in DataTypeUtils
* set utc time zone for date format
* revert changes regarding timezone in date format, disable tests with number and duration
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* ssh-test
* add authentification via ssh tunnel with bastion docker host and postgres testcontainer
* created SshBastion class in base-java module
* implement Postgres source basic ssh tunneling connection for integration tests
* implement Postgres source ssh tunneling connection and refactoring SshBastion
* generate keys inside a bastion container
* remove throwing Exception from startTestContainers method
* fix checkstyle
* add documentation and changelog for Posthres source and destination
* update documentation for ssh readme.md | update version fo Postgres source and destination to 0.3.12
* update version of Postgres source and destination to 0.3.12
* removed static variables, removed version bump, rename class to SshBastionContainer, removed ci credentials for ssh Postgres Source and Destination
Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
* move logic from static JdbcUtils to JdbcSourceOperations
* format
* Add methods for all types in order to have possibility rewrite them and use common type mapping.
* Make BigQuery in line impl with JDBC changes
* fix binary type
* add database creations methods with sourceOperations input
* Update airbyte-db/lib/src/main/java/io/airbyte/db/SourceOperations.java
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
# Summary
- A follow-up PR for #5543.
- This PR separates the `airbyte-db` project to two modules:
- `lib` is the original `airbyte-db`.
- `jooq` is for jOOQ code generation.
- This is necessary because the jOOQ generator requires a custom database implementation that can run Flyway migration. So the code generator logic needs to depend on the compilation of the original `airbyte-db` project.
# Commits
* Separate db to lib and jooq modules
* Update dependencies
* Add jobs db migrator test
* Fix compose build
* Add migration dev center
* Add schema dump task
* Update airbyte-db/lib/README.md
* Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update readme
* Remove bom dependency
* Update readme
* Use jooq code in db config persistence
* Remove AirbyteConfigsTable
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Add transactionTimestamp field
* Update CDC_timestamp type in integrations and tests
* no jsonschema timestamp type, use string
* Linting
* format
* fix compilation error
* fix compilation error
* fix test output
* upgrade version for mssql,mysql,postgres
* stupid mistake! forgot to upgrade docker file
Co-authored-by: danieldiamond <danieldiamond1@gmail.com>
* 3962: Add oneOf schema rules validation for acceptance test
* Add spec acceptance test for mssql and postgres connectors
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
* first few classes for mssql cdc
* wip
* mssql cdc working against unit tests
* increment version
* add cdc acceptance test
* tweaks
* add file
* working on comprehensive tests
* change isolation from snapshot to read_committed_snapshot
* finalised type tests
* Revert "change isolation from snapshot to read_committed_snapshot"
This reverts commit 20c6768071.
* small docstring fix
* remove unused imports
* stress test fixes
* minor formatting improvements
* mssql cdc docs
* finish off cdc docs
* format fix
* update connector version
* add to changelog
* fix for sql server agent offline failing cdc enable on tables
* final structure
* few more updates
* undo unwanted changes
* add abstract test + more refinement
* remove CDC metadata to debezium
* use new cdc abstraction for mysql
* undo wanted change
* use cdc abstraction for postgres
* add files
* pull in latest changes
* ready
* rename class + add missing property
* use renamed class + move constants to MySqlSource
* use renamed class + move constants to PostgresSource
* move debezium to bases + upgrade debezium version + review comments
* downgrade version + minor fixes
* bring in latest changes from cdc abstraction
* reset to minutes
* bring in the latest changes
* format
* fix build
* address review comments
* bring in latest changes
* bring in latest changes
* use common abstraction for CDC via debezium for sql server
* remove debezium from build
* finalise PR
* should return Optional
* pull in latest changes
* pull in latest changes
* address review comments
* use common abstraction for CDC via debezium for mysql (#4604)
* use new cdc abstraction for mysql
* undo wanted change
* pull in latest changes
* use renamed class + move constants to MySqlSource
* bring in latest changes from cdc abstraction
* format
* bring in latest changes
* pull in latest changes
* use common abstraction for CDC via debezium for postgres (#4607)
* use cdc abstraction for postgres
* add files
* ready
* use renamed class + move constants to PostgresSource
* bring in the latest changes
* bring in latest changes
* pull in latest changes
* lower version for tests to run on CI
* format
* Update docs/integrations/sources/mssql.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* addressing review comments
* fix for testGetTargetPosition
* format changes
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* throw exception if we close engine before snapshot is complete + increase timeout for subsequent records
* add comment + bump postgres version to use new changes
* wip
* add file
* final structure
* few more updates
* undo unwanted changes
* add abstract test + more refinement
* remove CDC metadata to debezium
* rename class + add missing property
* move debezium to bases + upgrade debezium version + review comments
* downgrade version + minor fixes
* reset to minutes
* fix build
* address review comments
* should return Optional
* use common abstraction for CDC via debezium for mysql (#4604)
* use new cdc abstraction for mysql
* undo wanted change
* pull in latest changes
* use renamed class + move constants to MySqlSource
* bring in latest changes from cdc abstraction
* format
* bring in latest changes
* pull in latest changes
* use common abstraction for CDC via debezium for postgres (#4607)
* use cdc abstraction for postgres
* add files
* ready
* use renamed class + move constants to PostgresSource
* bring in the latest changes
* bring in latest changes
* pull in latest changes
* 🐛 Source PostgreSQL: Fixed PostgreSQL CDC numeric handling
* Rm unnecessary tests for non-CDC
* Add more values to test in postgres cdc numeric
* Fix comment so it points to psql docs instead
* Oops, added to the wrong test file
* Add sourceType as well
* Fix precision on tests
* Use CDK to generate source that can be configured to emit a certain number of records and always works.
* Checkpoint: socat works from inside the docker container.
* Override the entry point.
* Clean up and add ReadMe.
* Clean up socat.
* Checkpoint: connect to Kube cluster and list all the pods.
* Checkpoint: Sync worker pod is able to send output to the destination pod.
* Checkpoint: Sync worker creates Dest pod if none existed previously. It also waits for the pod to be ready before doing anything else. Sync worker will also remove the pod on termination.
* update readme
* Checkpoint: Dest pod does nott restart after finishing. Comment out delete command in Sync worker.
* working towards named pipes
* named pipes working
* update readme
* WIP named pipe / socat sidecar kube port forwarding (#3518)
* nearly working sources
* update
* stdin example
* move all kube testing yamls into the airbyte-workers directories. sort the airbyte-workers resource folder; place all the poc yamls together.
* Format.
* Put back the original KubeProcessBuilderFactory.
* Fix slight errors.
* Checkpoint: Worker pod knows its own IP. Successfully starts and writes to Dest pod after refactor.
* remove unused file and update readme
* Dest pod loops back into worker pod. However, the right messages do not seem to be passing in.
* Switch back to worker ip.
* SWEET VICTORY!.
* wrap kube pod in process (#3540)
also clean up kubernetes deploys.
* More clean up. (#3586)
The first 6 points of #3464.
The only interesting thing about this PR is the kube pod shutdown. For whatever reason, the OkHttpPool isn't respecting the evictAll call and 1 idle thread remains. So instead of shutting down immediately, the worker pod shuts down after 5 mins when the idle thread id reaped. There isn't an easy way to modify the pool's idle reap configuration now. I do not think this issue is blocking since it's relatively benign, so I vote we create a ticket and come back to this once we do an e2e test.
* Implements redirecting standard error as well. (#3623)
* Clean up before next implementation.
* kube process launching (#3790)
* processes must handle file mounting
* remove comment
* default to base entrypoint
* use process builder factory / select stdin / use a pool of ports
* fix up
* add super hacky copying example
* Checkpoint: Works end to end!
* Checkpoint: Use API to make sure init container is ready instead of blind sleep. Propagate exception in DefaultCheckConnectionWorker.
* Refactor KubePodProcess. Checked to make sure everything still works.
* Format.
* Clean up code. Begin putting this into variables and breaking up long constructor function.
* Add comments to explain what is happening.
* fix normalization test
* increase timeout for initcontainer
Co-authored-by: Davin Chia <davinchia@gmail.com>
* facepalm moment
* clean up kube poc pr (#3834)
* clean up
* remove source-always-works
* create separate commons-docker
* fix test
* enable kube e2e tests (#3866)
* enable kube e2e tests
* use more generally accepted env definition
* use new runners
* use its own runner and install minikube differently
* update name
* use kubectl alias
* use link instead of alias that doesn't propagate
* start minikube
* use driver=none
* go back to using action
* mess with versions
* revert runner
* install socat
* print logs after run
* also try re-runnining tasks
* always wait for file transfer
* use ports
* increase wait timeout for kube
* use different localhost ips and bump normalization to include an entrypoint
* proposed fix
* all working locally
* revert temporary changes
* revert normalization image change that's happening in a separate pr
* readability
* final comment
* Working Kube Cancel. (#3983)
* Port over the basic changes.
* Add logic to return proper exit code in the event of termination. Add comments to explain why.
* revert envs change and merge master to fix kube acceptance tests (#4012)
* use older env format
* fix build
Co-authored-by: jrhizor <me@jaredrhizor.com>
Co-authored-by: Jared Rhizor <jared@dataline.io>
