* Added RDS base performance tests for source-postgres, source-mssql and source-mysql
* updated perfomance test with cpu and memory limit
Co-authored-by: andriikorotkov <andrii.h.korotkov@globallogic.com>
* Add jdbc compatible layer
* Support routine mysql types
* Format code
* Fix build
* Refactor abstract jdbc source and operation classes
* Update mysql source operations
* Test discover command for mysql
* Remove abstract jdbc compatible source layer
* Format code
* Update template
* Fix more types
* Bump version
* Log original field type
* Update comments
* Bump version in seed
* Source-MySql: do not check cdc required param binlog_row_image for standard replication
* Source-MySql: fix formatting
* Revert "Source-MySql: fix formatting"
This reverts commit 5a94474a52.
* Source-MySql: made a code improvement
* Source-MySql: bump versions
* Source-MySql: fix version in source_specs.yaml
* Source-MySql: bump versions
* Source-MySql:update source_definitions.yaml with new version
* Source-MySql: transform binary data base64 format, add integration tests
* Source-MySql: fix code style
* Source-MySql: bump versions
* Source-MySql: bump versions in source_specs.yaml
* Source-MySql: added test for stream with binary data for DestinationAbstractTest
* Source-MySql: added format
* Fix data type tests Mssql source
* update test for boolean data type in Mysql source
* dump versions for Mssql and Mysql source
* revert changes regarding handling bit data type
* revert changes regarding date and time data type tests Mssql source
* revert changes regarding date data type for Mssql source
* update message in mssql.md
* minor format changes
* format changes
* BUG-5842: Mysql Value '65535' is outside valid range for type java.lang.Short
* BUG-5842: Bumped versions of connector.
* BUG-5842: Fixed unit-test for mysql-encrypt
* Update docs/integrations/sources/mysql.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* move logic from static JdbcUtils to JdbcSourceOperations
* format
* Add methods for all types in order to have possibility rewrite them and use common type mapping.
* Make BigQuery in line impl with JDBC changes
* fix binary type
* add database creations methods with sourceOperations input
* add MySqlSourceOperations
* fix processing
* Fix CDC processing
* format
* add some tests for double and float
* incr version
* #3931#3932 added zero-date converting to null param "zeroDateTimeBehavior=convertToNull"
* remove old tests covered by data type tests + incr ver
* Update airbyte-integrations/connectors/destination-mysql/src/main/java/io/airbyte/integrations/destination/mysql/MySQLDestination.java
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Update docs/integrations/sources/mysql.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* add back comments to the data type mapping
* incr config version
Co-authored-by: Oleksandr Sheheda <alexandrshegeda@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Wrapper around Prometheus lib to interface with Datadog.
We use prometheus because:
- Future-proofing as it's uses the general open metrics format.
- Prometheus makes it's metrics available to a scraper to it lends itself better to the OSS set up.
- Datadog automatically converts promethues metrics into dd metrics so we don't lose much.
# Summary
- A follow-up PR for #5543.
- This PR separates the `airbyte-db` project to two modules:
- `lib` is the original `airbyte-db`.
- `jooq` is for jOOQ code generation.
- This is necessary because the jOOQ generator requires a custom database implementation that can run Flyway migration. So the code generator logic needs to depend on the compilation of the original `airbyte-db` project.
# Commits
* Separate db to lib and jooq modules
* Update dependencies
* Add jobs db migrator test
* Fix compose build
* Add migration dev center
* Add schema dump task
* Update airbyte-db/lib/README.md
* Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update readme
* Remove bom dependency
* Update readme
* Use jooq code in db config persistence
* Remove AirbyteConfigsTable
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Add transactionTimestamp field
* Update CDC_timestamp type in integrations and tests
* no jsonschema timestamp type, use string
* Linting
* format
* fix compilation error
* fix compilation error
* fix test output
* upgrade version for mssql,mysql,postgres
* stupid mistake! forgot to upgrade docker file
Co-authored-by: danieldiamond <danieldiamond1@gmail.com>
* first few classes for mssql cdc
* wip
* mssql cdc working against unit tests
* increment version
* add cdc acceptance test
* tweaks
* add file
* working on comprehensive tests
* change isolation from snapshot to read_committed_snapshot
* finalised type tests
* Revert "change isolation from snapshot to read_committed_snapshot"
This reverts commit 20c6768071.
* small docstring fix
* remove unused imports
* stress test fixes
* minor formatting improvements
* mssql cdc docs
* finish off cdc docs
* format fix
* update connector version
* add to changelog
* fix for sql server agent offline failing cdc enable on tables
* final structure
* few more updates
* undo unwanted changes
* add abstract test + more refinement
* remove CDC metadata to debezium
* use new cdc abstraction for mysql
* undo wanted change
* use cdc abstraction for postgres
* add files
* pull in latest changes
* ready
* rename class + add missing property
* use renamed class + move constants to MySqlSource
* use renamed class + move constants to PostgresSource
* move debezium to bases + upgrade debezium version + review comments
* downgrade version + minor fixes
* bring in latest changes from cdc abstraction
* reset to minutes
* bring in the latest changes
* format
* fix build
* address review comments
* bring in latest changes
* bring in latest changes
* use common abstraction for CDC via debezium for sql server
* remove debezium from build
* finalise PR
* should return Optional
* pull in latest changes
* pull in latest changes
* address review comments
* use common abstraction for CDC via debezium for mysql (#4604)
* use new cdc abstraction for mysql
* undo wanted change
* pull in latest changes
* use renamed class + move constants to MySqlSource
* bring in latest changes from cdc abstraction
* format
* bring in latest changes
* pull in latest changes
* use common abstraction for CDC via debezium for postgres (#4607)
* use cdc abstraction for postgres
* add files
* ready
* use renamed class + move constants to PostgresSource
* bring in the latest changes
* bring in latest changes
* pull in latest changes
* lower version for tests to run on CI
* format
* Update docs/integrations/sources/mssql.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* addressing review comments
* fix for testGetTargetPosition
* format changes
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* wip
* add file
* final structure
* few more updates
* undo unwanted changes
* add abstract test + more refinement
* remove CDC metadata to debezium
* rename class + add missing property
* move debezium to bases + upgrade debezium version + review comments
* downgrade version + minor fixes
* reset to minutes
* fix build
* address review comments
* should return Optional
* use common abstraction for CDC via debezium for mysql (#4604)
* use new cdc abstraction for mysql
* undo wanted change
* pull in latest changes
* use renamed class + move constants to MySqlSource
* bring in latest changes from cdc abstraction
* format
* bring in latest changes
* pull in latest changes
* use common abstraction for CDC via debezium for postgres (#4607)
* use cdc abstraction for postgres
* add files
* ready
* use renamed class + move constants to PostgresSource
* bring in the latest changes
* bring in latest changes
* pull in latest changes
* Use CDK to generate source that can be configured to emit a certain number of records and always works.
* Checkpoint: socat works from inside the docker container.
* Override the entry point.
* Clean up and add ReadMe.
* Clean up socat.
* Checkpoint: connect to Kube cluster and list all the pods.
* Checkpoint: Sync worker pod is able to send output to the destination pod.
* Checkpoint: Sync worker creates Dest pod if none existed previously. It also waits for the pod to be ready before doing anything else. Sync worker will also remove the pod on termination.
* update readme
* Checkpoint: Dest pod does nott restart after finishing. Comment out delete command in Sync worker.
* working towards named pipes
* named pipes working
* update readme
* WIP named pipe / socat sidecar kube port forwarding (#3518)
* nearly working sources
* update
* stdin example
* move all kube testing yamls into the airbyte-workers directories. sort the airbyte-workers resource folder; place all the poc yamls together.
* Format.
* Put back the original KubeProcessBuilderFactory.
* Fix slight errors.
* Checkpoint: Worker pod knows its own IP. Successfully starts and writes to Dest pod after refactor.
* remove unused file and update readme
* Dest pod loops back into worker pod. However, the right messages do not seem to be passing in.
* Switch back to worker ip.
* SWEET VICTORY!.
* wrap kube pod in process (#3540)
also clean up kubernetes deploys.
* More clean up. (#3586)
The first 6 points of #3464.
The only interesting thing about this PR is the kube pod shutdown. For whatever reason, the OkHttpPool isn't respecting the evictAll call and 1 idle thread remains. So instead of shutting down immediately, the worker pod shuts down after 5 mins when the idle thread id reaped. There isn't an easy way to modify the pool's idle reap configuration now. I do not think this issue is blocking since it's relatively benign, so I vote we create a ticket and come back to this once we do an e2e test.
* Implements redirecting standard error as well. (#3623)
* Clean up before next implementation.
* kube process launching (#3790)
* processes must handle file mounting
* remove comment
* default to base entrypoint
* use process builder factory / select stdin / use a pool of ports
* fix up
* add super hacky copying example
* Checkpoint: Works end to end!
* Checkpoint: Use API to make sure init container is ready instead of blind sleep. Propagate exception in DefaultCheckConnectionWorker.
* Refactor KubePodProcess. Checked to make sure everything still works.
* Format.
* Clean up code. Begin putting this into variables and breaking up long constructor function.
* Add comments to explain what is happening.
* fix normalization test
* increase timeout for initcontainer
Co-authored-by: Davin Chia <davinchia@gmail.com>
* facepalm moment
* clean up kube poc pr (#3834)
* clean up
* remove source-always-works
* create separate commons-docker
* fix test
* enable kube e2e tests (#3866)
* enable kube e2e tests
* use more generally accepted env definition
* use new runners
* use its own runner and install minikube differently
* update name
* use kubectl alias
* use link instead of alias that doesn't propagate
* start minikube
* use driver=none
* go back to using action
* mess with versions
* revert runner
* install socat
* print logs after run
* also try re-runnining tasks
* always wait for file transfer
* use ports
* increase wait timeout for kube
* use different localhost ips and bump normalization to include an entrypoint
* proposed fix
* all working locally
* revert temporary changes
* revert normalization image change that's happening in a separate pr
* readability
* final comment
* Working Kube Cancel. (#3983)
* Port over the basic changes.
* Add logic to return proper exit code in the event of termination. Add comments to explain why.
* revert envs change and merge master to fix kube acceptance tests (#4012)
* use older env format
* fix build
Co-authored-by: jrhizor <me@jaredrhizor.com>
Co-authored-by: Jared Rhizor <jared@dataline.io>
* add AIRBYTE_ENTRYPOINT for kubernetes support
* bump versions
* bump version in seed
* Update generic template
* keep scaffold sources at 0.1.0
* add missing newline
* handle python base versions correctly
* re-bump mysql and postgres sources
* re-bump snowflake destination
* add skip tests option
* switch to running tests
* reverse conditional to make it safer
* fix publish to include the test running
* fix iterable version
* fix file generation
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
JDBC sources: Implement an additional handler for the Integer source values in order to cover the case with too large a value (possible for some sources)
* source: implementation for mysql cdc
* add target file and position
* dont want to add file in this PR
* refine tests + add comments
* fix typo
* address review comments
* fix formatting error
* resolve conflicts
* update docs + bump docker minor version
* remove un-necessary new lines + add multiple checks for cdc
* address review comments from Davin
* increase the version in source_definitions.yaml
* rebuild seed
Release all connectors affected by namespace change. Includes all JDBC sources and destinations.
Also add documentation for normalisation. Prerequisite to actually releasing 0.21.0-alpha.
