* Fix data type tests in CdcPostgresSourceComprehensiveTest
* update style format
* bump version for postgres source
* bump version for postgres source in json definition
* remove unnecessary comments from test and bump version for postgres strict encrypt source
* resolved potential conflicts with debezium utils in mssql converter implementation
* resolved potential conflicts with debezium utils in mssql converter implementation
* Update notes for money type in postgres.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Update docs/integrations/sources/postgres.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* added test cases for converting data values for postgres cdc, remove time zone utc from test container
* remove redundant void message from test
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* update style format
* fix time zone in DebeziumConverterUtilsTest
* set utc time zone in DataTypeUtils
* set utc time zone for date format
* revert changes regarding timezone in date format, disable tests with number and duration
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* ssh-test
* add authentification via ssh tunnel with bastion docker host and postgres testcontainer
* created SshBastion class in base-java module
* implement Postgres source basic ssh tunneling connection for integration tests
* implement Postgres source ssh tunneling connection and refactoring SshBastion
* generate keys inside a bastion container
* remove throwing Exception from startTestContainers method
* fix checkstyle
* add documentation and changelog for Posthres source and destination
* update documentation for ssh readme.md | update version fo Postgres source and destination to 0.3.12
* update version of Postgres source and destination to 0.3.12
* removed static variables, removed version bump, rename class to SshBastionContainer, removed ci credentials for ssh Postgres Source and Destination
Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
* move logic from static JdbcUtils to JdbcSourceOperations
* format
* Add methods for all types in order to have possibility rewrite them and use common type mapping.
* Make BigQuery in line impl with JDBC changes
* fix binary type
* add database creations methods with sourceOperations input
* Update airbyte-db/lib/src/main/java/io/airbyte/db/SourceOperations.java
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
# Summary
- A follow-up PR for #5543.
- This PR separates the `airbyte-db` project to two modules:
- `lib` is the original `airbyte-db`.
- `jooq` is for jOOQ code generation.
- This is necessary because the jOOQ generator requires a custom database implementation that can run Flyway migration. So the code generator logic needs to depend on the compilation of the original `airbyte-db` project.
# Commits
* Separate db to lib and jooq modules
* Update dependencies
* Add jobs db migrator test
* Fix compose build
* Add migration dev center
* Add schema dump task
* Update airbyte-db/lib/README.md
* Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update readme
* Remove bom dependency
* Update readme
* Use jooq code in db config persistence
* Remove AirbyteConfigsTable
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Add transactionTimestamp field
* Update CDC_timestamp type in integrations and tests
* no jsonschema timestamp type, use string
* Linting
* format
* fix compilation error
* fix compilation error
* fix test output
* upgrade version for mssql,mysql,postgres
* stupid mistake! forgot to upgrade docker file
Co-authored-by: danieldiamond <danieldiamond1@gmail.com>
* 3962: Add oneOf schema rules validation for acceptance test
* Add spec acceptance test for mssql and postgres connectors
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
* first few classes for mssql cdc
* wip
* mssql cdc working against unit tests
* increment version
* add cdc acceptance test
* tweaks
* add file
* working on comprehensive tests
* change isolation from snapshot to read_committed_snapshot
* finalised type tests
* Revert "change isolation from snapshot to read_committed_snapshot"
This reverts commit 20c6768071.
* small docstring fix
* remove unused imports
* stress test fixes
* minor formatting improvements
* mssql cdc docs
* finish off cdc docs
* format fix
* update connector version
* add to changelog
* fix for sql server agent offline failing cdc enable on tables
* final structure
* few more updates
* undo unwanted changes
* add abstract test + more refinement
* remove CDC metadata to debezium
* use new cdc abstraction for mysql
* undo wanted change
* use cdc abstraction for postgres
* add files
* pull in latest changes
* ready
* rename class + add missing property
* use renamed class + move constants to MySqlSource
* use renamed class + move constants to PostgresSource
* move debezium to bases + upgrade debezium version + review comments
* downgrade version + minor fixes
* bring in latest changes from cdc abstraction
* reset to minutes
* bring in the latest changes
* format
* fix build
* address review comments
* bring in latest changes
* bring in latest changes
* use common abstraction for CDC via debezium for sql server
* remove debezium from build
* finalise PR
* should return Optional
* pull in latest changes
* pull in latest changes
* address review comments
* use common abstraction for CDC via debezium for mysql (#4604)
* use new cdc abstraction for mysql
* undo wanted change
* pull in latest changes
* use renamed class + move constants to MySqlSource
* bring in latest changes from cdc abstraction
* format
* bring in latest changes
* pull in latest changes
* use common abstraction for CDC via debezium for postgres (#4607)
* use cdc abstraction for postgres
* add files
* ready
* use renamed class + move constants to PostgresSource
* bring in the latest changes
* bring in latest changes
* pull in latest changes
* lower version for tests to run on CI
* format
* Update docs/integrations/sources/mssql.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* addressing review comments
* fix for testGetTargetPosition
* format changes
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* throw exception if we close engine before snapshot is complete + increase timeout for subsequent records
* add comment + bump postgres version to use new changes
* wip
* add file
* final structure
* few more updates
* undo unwanted changes
* add abstract test + more refinement
* remove CDC metadata to debezium
* rename class + add missing property
* move debezium to bases + upgrade debezium version + review comments
* downgrade version + minor fixes
* reset to minutes
* fix build
* address review comments
* should return Optional
* use common abstraction for CDC via debezium for mysql (#4604)
* use new cdc abstraction for mysql
* undo wanted change
* pull in latest changes
* use renamed class + move constants to MySqlSource
* bring in latest changes from cdc abstraction
* format
* bring in latest changes
* pull in latest changes
* use common abstraction for CDC via debezium for postgres (#4607)
* use cdc abstraction for postgres
* add files
* ready
* use renamed class + move constants to PostgresSource
* bring in the latest changes
* bring in latest changes
* pull in latest changes
* 🐛 Source PostgreSQL: Fixed PostgreSQL CDC numeric handling
* Rm unnecessary tests for non-CDC
* Add more values to test in postgres cdc numeric
* Fix comment so it points to psql docs instead
* Oops, added to the wrong test file
* Add sourceType as well
* Fix precision on tests
* Use CDK to generate source that can be configured to emit a certain number of records and always works.
* Checkpoint: socat works from inside the docker container.
* Override the entry point.
* Clean up and add ReadMe.
* Clean up socat.
* Checkpoint: connect to Kube cluster and list all the pods.
* Checkpoint: Sync worker pod is able to send output to the destination pod.
* Checkpoint: Sync worker creates Dest pod if none existed previously. It also waits for the pod to be ready before doing anything else. Sync worker will also remove the pod on termination.
* update readme
* Checkpoint: Dest pod does nott restart after finishing. Comment out delete command in Sync worker.
* working towards named pipes
* named pipes working
* update readme
* WIP named pipe / socat sidecar kube port forwarding (#3518)
* nearly working sources
* update
* stdin example
* move all kube testing yamls into the airbyte-workers directories. sort the airbyte-workers resource folder; place all the poc yamls together.
* Format.
* Put back the original KubeProcessBuilderFactory.
* Fix slight errors.
* Checkpoint: Worker pod knows its own IP. Successfully starts and writes to Dest pod after refactor.
* remove unused file and update readme
* Dest pod loops back into worker pod. However, the right messages do not seem to be passing in.
* Switch back to worker ip.
* SWEET VICTORY!.
* wrap kube pod in process (#3540)
also clean up kubernetes deploys.
* More clean up. (#3586)
The first 6 points of #3464.
The only interesting thing about this PR is the kube pod shutdown. For whatever reason, the OkHttpPool isn't respecting the evictAll call and 1 idle thread remains. So instead of shutting down immediately, the worker pod shuts down after 5 mins when the idle thread id reaped. There isn't an easy way to modify the pool's idle reap configuration now. I do not think this issue is blocking since it's relatively benign, so I vote we create a ticket and come back to this once we do an e2e test.
* Implements redirecting standard error as well. (#3623)
* Clean up before next implementation.
* kube process launching (#3790)
* processes must handle file mounting
* remove comment
* default to base entrypoint
* use process builder factory / select stdin / use a pool of ports
* fix up
* add super hacky copying example
* Checkpoint: Works end to end!
* Checkpoint: Use API to make sure init container is ready instead of blind sleep. Propagate exception in DefaultCheckConnectionWorker.
* Refactor KubePodProcess. Checked to make sure everything still works.
* Format.
* Clean up code. Begin putting this into variables and breaking up long constructor function.
* Add comments to explain what is happening.
* fix normalization test
* increase timeout for initcontainer
Co-authored-by: Davin Chia <davinchia@gmail.com>
* facepalm moment
* clean up kube poc pr (#3834)
* clean up
* remove source-always-works
* create separate commons-docker
* fix test
* enable kube e2e tests (#3866)
* enable kube e2e tests
* use more generally accepted env definition
* use new runners
* use its own runner and install minikube differently
* update name
* use kubectl alias
* use link instead of alias that doesn't propagate
* start minikube
* use driver=none
* go back to using action
* mess with versions
* revert runner
* install socat
* print logs after run
* also try re-runnining tasks
* always wait for file transfer
* use ports
* increase wait timeout for kube
* use different localhost ips and bump normalization to include an entrypoint
* proposed fix
* all working locally
* revert temporary changes
* revert normalization image change that's happening in a separate pr
* readability
* final comment
* Working Kube Cancel. (#3983)
* Port over the basic changes.
* Add logic to return proper exit code in the event of termination. Add comments to explain why.
* revert envs change and merge master to fix kube acceptance tests (#4012)
* use older env format
* fix build
Co-authored-by: jrhizor <me@jaredrhizor.com>
Co-authored-by: Jared Rhizor <jared@dataline.io>
* add AIRBYTE_ENTRYPOINT for kubernetes support
* bump versions
* bump version in seed
* Update generic template
* keep scaffold sources at 0.1.0
* add missing newline
* handle python base versions correctly
* re-bump mysql and postgres sources
* re-bump snowflake destination
* add skip tests option
* switch to running tests
* reverse conditional to make it safer
* fix publish to include the test running
* fix iterable version
* fix file generation
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* source: implementation for mysql cdc
* add target file and position
* dont want to add file in this PR
* refine tests + add comments
* fix typo
* address review comments
* fix formatting error
* resolve conflicts
* update docs + bump docker minor version
* remove un-necessary new lines + add multiple checks for cdc
* address review comments from Davin
* increase the version in source_definitions.yaml
* rebuild seed
Release all connectors affected by namespace change. Includes all JDBC sources and destinations.
Also add documentation for normalisation. Prerequisite to actually releasing 0.21.0-alpha.
This PR introduces the following behavior for JDBC sources:
Instead of streamName = schema.tableName, this is now streamName = tableName and namespace = schema. This means that, when replicating from these sources, data will be replicated into a form matching the source. e.g. public.users (postgres source) -> public.users (postgres destination) instead of current behaviour of public.public_users. Since MySQL does not have schemas, the MySQL source uses the database as it's namespace.
To do so:
- Make namespace a field class concept in Airbyte Protocol. This allows the source to propagate namespace and destinations to write to a source-defined namespace. Also sets us up for future namespace related configurability.
- Add an optional namespace field to the AirbyteRecordMessage. This field will be set by sources that support namespace.
- Introduce AirbyteStreamNameNamespacePair as a type-safe manner of identifying streams throughout our code base.
- Modify base_normalisation to better support source defined namespace, specifically allowing normalisation of tables with the same name to different schemas.
* add ssl for source-postgres
* add config in utf8 test
* correct comments from @jrhizor and @sherifnada
* correct config get
* add ssl test postgres
* add sh generate ssl files
* change pg ssl test
* use custom image
* correct spec.json
* correc tests
* remove unecessary config
* add config and correct spec.json
* add ssl to postgres destination
* add tools to generate custom dockers images and correct spec.json
* change how additional parameter is append
* add logic ssl for postgres destination
* remove if for append add params
* gradlew format
Last step (besides documentation) of namespace changes. This is a follow up to #2767 .
After this change, the following JDBC sources will change their behaviour to the behaviour described in the above document.
Namely, instead of streamName = schema.tableName, this will become streamName = tableName and namespace = schema. This means that, when replicating from these sources, data will be replicated into a form matching the source. e.g. public.users (postgres source) -> public.users (postgres destination) instead of current behaviour of public.public_users. Since MySQL does not have schemas, the MySQL source uses the database as it's namespace.
I cleaned up some bits of the CatalogHelpers. This affected the destinations, so I'm also running the destination tests.
* spike
* more
* debezium wip
* use oneof for configuration
* iterator wrapping structure
* push current
* working loop
* move capability into source
* hack it into a sharable state
* debezium test runner (#2617)
* CDC Wait for Values (#2618)
* output actual AirbyteMessages for cdc (#2631)
* message conversion
* fmt
* add lsn extraction and comparison (#2613)
* postgres cdc catalog (#2673)
* update cdc catalog
* A
* table selection for cdc (#2690)
* table selection for cdc
* fix broken merge
* also test double quote in name
* Add state management to CDC (#2718)
* CDC: Fix Producer/Consumer State Machine (#2721)
* CDC Postgres Tests (#2777)
* fix postgres cdc image name and run check before reading data (#2785)
* minor postgres cdc fixes
* add test and fix check behavior
* fix
* improve comment
* remove unused props, remove todos, add some more sanity tests (#2791)
* cdc: add offset store tests (#2793)
* clean (#2798)
* postgres cdc docs (#2784)
* cdc docs
* Update docs/integrations/sources/postgres.md
Co-authored-by: Charles <giardina.charles@gmail.com>
* address gcp
* learn too english
* add link
* add more disk space warnings
* add additional cdc use case
* add information on how to find postgresql.conf
* add how to find the file
Co-authored-by: Charles <giardina.charles@gmail.com>
* various merge conflict fixes (#2799)
* cdc standard tests (#2813)
* require cdc users to create publications & update docs (#2818)
* postgres cdc race condition
* working? but different process
* add additional logging to help debug in the future
* everything done except working config
* remove unintended change
* Use oneOf in PG CDC spec (#2827)
* add oneOf configuration for postgres cdc (#2831)
* add oneof configuration for cdc postgres
* fmt
Co-authored-by: Charles <giardina.charles@gmail.com>
* fix test (#2834)
* fix test
* bump version
* add docs on creating replica identities (#2838)
* add docs on creating replica identities
* emphasize danger
* grammar
* bump pg version in source catalog
* generate seed files
Co-authored-by: cgardens <giardina.charles@gmail.com>
This PR is step 5 of this tech spec - https://docs.google.com/document/d/1qFk4YqnwxE4MCGeJ9M2scGOYej6JnDy9A0zbICP_zjI/edit.
The first of (at least) 2 PRs to implement this on the source side. I made some headway before deciding to break the changes into one PR implementing this for discover schema job, and another PR implementing this for read. The combined PR would have been too big otherwise.
Also refactor MoreResources as the test method was attempting to write to the location classes where loaded out from - the issue is we cannot guarantee where the class is loaded from can be written to. Changing this to write to a random folder in the temp directory.
* Handle destination sync mode in destinations
* Source & Destination sync modes are required (#2500)
* Provide Migration script making sure it is always defined for previous sync configs
