* 8890 Fix large table issue by fetch size
* 8890 Bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* upgrade debezium version to 1.9.6
* test override not required
* bump version
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* mysql source : implement support for snapshot of new tables in cdc mode
* undo unwanted changes
* add more assertions
* format
* fix build
* fix build
* revert acceptance test changes
* bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix postgres data handling from WAL logs in CDC mode
* format
* fix temporal type data handling from in mysql source
* use formatter for dates also (#15485)
* format
* tweak tests
* change test structure
* add test for data read from binlogs
* fix json schema types for data types
* handle cursor parsing
* add more tests
* remove assertion cause its not useful
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
* upgrade debezium version to 1.9.2 for mysql source
* new debezium version doen't require special string handling
* fix log message
* add test tor date time data type
* gracefully handle the case when logs are purged
* fix test
* address review comment
Part 1 of #13122.
Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.
Explanation for what is happening:
Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
* Better database connection handling for connectors
* Log connection error
* Properly close connection
* Remove unused method
* Close data source
* Use utility to close data source
* Use utility to close data source
* PR feedback
* Add Databricks driver
* Use driver class enum
* Use correct config
* Ensure config created before use
* Fix failing integration test
* Create DSLContext before use
* Address integration test failures
* Ensure DSLContext is closed
* Fix compile error
* Use correct datasource
* Use correct connection properties
* Close DSLContext
* Close DSLContext
* Fix integration test failures
* Properly close datasource
* Fix compilation issues
* Use existing database object
* Wrap close in try/finally
* Update test
* Wrap close in try/finally
* Ensure DSLContext is created
* Revert change to test
* Use correct data source
* Remove unused import
* More cleanup
* Add missing annotation
* Only initialize data source once
* Remove unused import
* Force testcontainers version
* Fix testcontainer issue
* Fix failing test
* Properly close all data sources
* Clear data sources after closing
* Fix compile error
* Fix compilation error
* Add missing method
# Summary
- A follow-up PR for #5543.
- This PR separates the `airbyte-db` project to two modules:
- `lib` is the original `airbyte-db`.
- `jooq` is for jOOQ code generation.
- This is necessary because the jOOQ generator requires a custom database implementation that can run Flyway migration. So the code generator logic needs to depend on the compilation of the original `airbyte-db` project.
# Commits
* Separate db to lib and jooq modules
* Update dependencies
* Add jobs db migrator test
* Fix compose build
* Add migration dev center
* Add schema dump task
* Update airbyte-db/lib/README.md
* Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update readme
* Remove bom dependency
* Update readme
* Use jooq code in db config persistence
* Remove AirbyteConfigsTable
Co-authored-by: Davin Chia <davinchia@gmail.com>
* wip
* add file
* final structure
* few more updates
* undo unwanted changes
* add abstract test + more refinement
* remove CDC metadata to debezium
* rename class + add missing property
* move debezium to bases + upgrade debezium version + review comments
* downgrade version + minor fixes
* reset to minutes
* fix build
* address review comments
* should return Optional
* use common abstraction for CDC via debezium for mysql (#4604)
* use new cdc abstraction for mysql
* undo wanted change
* pull in latest changes
* use renamed class + move constants to MySqlSource
* bring in latest changes from cdc abstraction
* format
* bring in latest changes
* pull in latest changes
* use common abstraction for CDC via debezium for postgres (#4607)
* use cdc abstraction for postgres
* add files
* ready
* use renamed class + move constants to PostgresSource
* bring in the latest changes
* bring in latest changes
* pull in latest changes
* source: implementation for mysql cdc
* add target file and position
* dont want to add file in this PR
* refine tests + add comments
* fix typo
* address review comments
* fix formatting error
* resolve conflicts
* update docs + bump docker minor version
* remove un-necessary new lines + add multiple checks for cdc
* address review comments from Davin
* increase the version in source_definitions.yaml
* rebuild seed
* support cached builds for base -> base-java -> snowflake
* use plugin for image building
* fix matching on COPY from
* remove docker.gradle
* tmp commit
* update connectors
* finish rest of build files
* fix ide errors
* more build fixes
* clean up
* clean up for new soruces
* fix spotless
* fix flake problems
* add recommended empty file
* python caching
* fixes upon review
* clean up docker and build test files
* clean up python
* clean up
* fix integration test dependencies
* fix standard tests
* fix
* remove symlink
* re-add requirements to fix normalizatioon build
* fix symlink
* fix dumbest build problem of all
* add missing integration test def
* fix missing dep
* remove class exclusion
* move trim so null source versions are allowed
* rename map
* fix hardcoded value
* remove unnecessary dep
* use dashes for salesforce package name
* fix typo
* DRY and fix test image name
* Fix edit
* assert string is not empty
* build integration test image only for integrationTest
* move code generator to tools and rename docker build tasks
* make source test depend on integration test build, not the other way
* remove guard because the docker build should exist before the integrationtest is applied
* remove comment
* DRY up airbyte-source-test
* fix plugin compilation
* add missing dependency
* rename getTaggedImage to getDevTaggedImage
* fix test vs main docker build bug
