1
0
mirror of synced 2025-12-20 18:39:31 -05:00
Commit Graph

362 Commits

Author SHA1 Message Date
Ben Church
5d8278900f Github Action: Add format.yml workflow (#30604)
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>
2023-09-21 18:11:41 -05:00
Ben Church
5c56ac1d84 Airbyte-ci: Remove gradle task connectorAcceptanceTest (#30326) 2023-09-19 15:16:37 -05:00
Duy Nguyen
f226503bd1 [Source-MySQL] Enhanced Standard sync with PK initial load -> Cursor based switch over (#30270) 2023-09-18 17:17:00 -05:00
Marius Posta
a5eb24e2fc bump testcontainers version to latest (#30493) 2023-09-18 03:50:43 -05:00
Subodh Kant Chaturvedi
5a4cc0d8b0 source-mysql-cdc: introduce logic to compress schema history if greater than 3MB (#30289)
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Prateek Mukhedkar <prateek@airbyte.io>
2023-09-13 12:41:28 -05:00
Marius Posta
51c67d7eaa gradle: remove airbyteDocker.outputs dependencies (#30314) 2023-09-11 17:16:27 -07:00
Marius Posta
ef2849e35e gradle: fix airbyteDocker task inputs (#30187) 2023-09-07 03:46:31 -07:00
Lake Mossman
b19ed6d8b3 Support display_type property for oneOf fields and adjust CDC connectors to use it (#29821)
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2023-08-31 12:19:02 -05:00
Jonathan Pearlin
13ef239ccc Source MongoDB Internal POC: CDC Metadata Injector (#29730)
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
Co-authored-by: Jose Pefaur <jose.pefaur@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
2023-08-28 14:01:32 -05:00
Jonathan Pearlin
0724132a1a Source MongoDB Internal POC: Debezium Configuration Properties (#29727) 2023-08-28 08:44:39 -05:00
Duy Nguyen
f68a828d1e [Source-MsSQL] Set default cursor for cdc mode (#29493) 2023-08-23 14:01:49 -05:00
Augustin
53129c56a6 [skip ci]: alpha connectors should be community (#29550) 2023-08-17 21:50:24 +02:00
Ben Church
40781313da Update Internal Fields: update ql levels to better resemble previous high strictness (#29450) 2023-08-15 18:03:10 -05:00
Akash Kulkarni
807f104414 source-mysql : CDC PK snapshotting (#28757)
* Initial Commit of CDC PK snapshotting

* Added unit tests

* Flip to true

* Refactor flags

* Re-add flag

* Comment cleanup

* Code review comments

* Automated Commit - Format and Process Resources Changes

* minor refactor

* Address comments

---------

Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
Co-authored-by: subodh <subodh1810@gmail.com>
2023-08-04 13:57:51 -05:00
Rodi Reich Zilberman
71f9505a4f Bump versions for mssql strict-encrypt (#28964)
* Bump versions for mssql strict-encrypt

* Fix failing test

* Fix failing test
2023-08-03 14:09:33 -05:00
Ben Church
2f7deaee02 [skip ci] Metadata: Remove leading underscore (#29024)
* DNC

* Add test models

* Add model test

* Remove underscore from metadata files

* Regenerate models

* Add test to check for key transformation

* Allow additional fields on metadata

* Delete transform
2023-08-03 10:56:13 -07:00
Ben Church
e9490e3fb6 Connector Levels: Add new internal metadata fields (#28904)
* Add airbyte internal

* Add tests

* First pass

* Set destinations to same levels as sources

* Best guess at missing statuses

* Best guess at _ql

* Add separate enum class

* Fix support level name

* Update templates

* Add one more test
2023-08-01 18:08:33 -05:00
sh4sh
6df99b24af 🐛 source-mssql: support Read Committed snapshot isolation level (#28545)
* make getSnapshotIsolationConfig grab the correct config

* add object case for getDataToSyncConfig and getSnapshotIsolationConfig

* update tests to use current config parameters

* bump connector version

* add changelog
2023-08-01 16:37:45 -05:00
Duy Nguyen
633c939d46 [Source-postgres] Set default cursor value for cdc mode (#27442)
* use LSN as default cursor for postgres CDC

* Fixed static constant

* Set lsn default cursor value for postgres sync

* Bumped metadata and dockerfile versions

* Disable acceptance backwards compatibility discovery test as this is a breaking change

---------

Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
2023-07-13 11:12:34 -05:00
Evan Tahler
9ce3cc6e79 Java DB Source connector licenses to Elv2 (#27737)
* Java DB Source connector licenses to Elv2

* Bump PR Ids
2023-06-28 18:52:02 -07:00
Akash Kulkarni
2f807440da Fix issue in streaming JDBC database (#27212)
* Fix issue in streaming JDBC database

* Bump + publish

* 🤖 Auto format source-db2 code [skip ci]

* 🤖 Auto format source-mssql code [skip ci]

* 🤖 Auto format source-postgres-strict-encrypt code [skip ci]

* 🤖 Auto format source-postgres code [skip ci]

* 🤖 Auto format source-mssql-strict-encrypt code [skip ci]

* 🤖 Auto format source-oracle code [skip ci]

* 🤖 Auto format source-tidb code [skip ci]

* 🤖 Auto format source-snowflake code [skip ci]

* 🤖 Auto format source-redshift code [skip ci]

* 🤖 Auto format source-mysql code [skip ci]

* 🤖 Auto format source-mysql-strict-encrypt code [skip ci]

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
2023-06-20 12:08:47 -05:00
Akash Kulkarni
d30906d89d Remove noisy logging for LegacyStateManager (#27335)
* Remove noisy logging for legacy state manager

Context : https://airbytehq-team.slack.com/archives/C043JHEEYKG/p1686654588724759

* Bump version + dockerfile
2023-06-16 17:13:33 -05:00
Ryan Fu
7156fb6d89 Ran ./gradlew :spotlessJavaApply so that it's separate from the Snowflake branch (#26869) 2023-05-31 14:39:51 -07:00
Ben Church
1dabc6208e Metadata: add tags field (#26320)
* Add optional tags field

* Remove duplicate icons

* Add programming tags to all

* Update docs

* supportUrl -> documentationUrl

* Ensure one language tag is applied

* Add keyvalue check

* rebase and fix tests

* Format

* Add cache buster

* Improve test

* Automated Commit - Formatting Changes

* Update error

* Fix missing tags

* Fix scaffold

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-05-26 16:13:09 -07:00
Subodh Kant Chaturvedi
9b9809b006 fix(cdc): limit queue size to lower memory consumption (#26473)
* fix(cdc): limit queue size to lower memory consumption

* add queue size attribute in spec

* disable retries

* fix log

* review comments

* add validation test for queue size

* update expected spec

* bump version + changelog

* update metadata files

---------

Co-authored-by: Ben Church <ben@airbyte.io>
2023-05-25 22:34:40 +05:30
Augustin
fd3655707e connectors-ci: add finalize_build logic to handle custom Dockerfiles (#26489) 2023-05-25 09:32:19 +02:00
Ben Church
248bbf94c1 HACKDAY: Icon CDN (#26158)
* Move icons to connector folder

* Delete old icons

* Update upload logic

* Add icon url to definitions

* Update registry model

* Populate cdn url

* DNC butcher the pipeline

* Low hanging fruit fixes

* Fix bucket name

* Merge old and new approaches

* Fix metadata upload step

* Format

* Fix test
2023-05-24 17:25:41 -07:00
Akash Kulkarni
889a5bc50b Run formatter (#25901) 2023-05-09 10:03:22 -07:00
Ben Church
efbdb73e24 Bnchrch/metadata/add missing metadata files (#25899)
* Add source fullstory metadata

* Add source merge metadata

* Add source yotpo metadata

* Update normalization tag

* Update oss diffs

* Add aircall to cloud

* Pin Source sales force

* Ensure strict encrypt matches

* Remove zurora pin

* Make metadata validation mandatory

* Remove top level third party folders
2023-05-08 16:55:11 -05:00
Augustin
7310494846 qa-checks: check metadata version matches dockerfile version (#25661) 2023-05-04 16:08:19 -07:00
Akash Kulkarni
931911dceb 🐛 Source MySQL/MsSQL Disable index logging for MySQL (#25740)
* Disable index logging for MySQL/MsSQL

* Update mysql.md

* Update mysql.md

* auto-bump connector version

* Disable for MsSQL as well

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-05-03 08:44:21 -07:00
Subodh Kant Chaturvedi
94ed6554ea upgrade debezium version to 2.2.0.Final (#25401)
* upgrade debezium version to 2.2.0.Final

* bump version + changelog

* auto-bump connector version

* manual bump version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-27 22:12:20 +05:30
Jonathan Pearlin
a38af089e9 Track stream status in source (#24971)
* WIP Track stream status in source

* Revert formatting

* Revert formatting changes

* Remove unnecessary file

* Automated Change

* Automated Change

* Use new stream status trace message

* Rename class

* Remove unnecessary import

* Formatting

* Add tests

* Fix compile issues

* Automated Commit - Formatting Changes

* Remove TODO

* Fix compilation error

* Split STOPPED into INCOMPLETE and COMPLETE

* Remove unused import

* Changelog updates for source-postgres

* Remove unused import

* auto-bump connector version

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-26 15:14:25 -05:00
Akash Kulkarni
395de77a50 [JDBC sources] : Log database indexes per stream (#25345)
* [JDBC sources] : Log database indexes per stream

* Docker & doc bump

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-20 01:05:15 +00:00
Sergio Ropero
819bba174f postgres-cdc checkpointing: fix LSN parsing bug + refactor for efficiency (#24582)
* Fix LSN parsing from Integer to Long

* rebasing

* Rebase

* Rebase

* Other casting

* Lock the file only when reading, so the file is free when parsing the object.
Increased from 1 to 166 checkpoints, and from skipping hundreds of checkpoints to never skip a state.

* Update load function documentation

* bump mysql and mssql

* cdc: refactor to remove debezium dependency from connector packages

* use gradle's shared dependency

* more refactoring

* upgrade docker version

* resolve master merge conflicts

* Automated Change

* minor changes

* resolve merge conflicts

* avoid deserializing multiple times

* simplify

* enable checkpointing for Postgres

* more improvements

* enable assertions

* changelog + bump version

* auto-bump connector version

* auto-bump connector version

* manual bump

---------

Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-19 21:05:50 +05:30
Ben Church
5563179782 Dagster: rename catalog to registry (#25254)
* rename catalog to registry in metadata service

* rename catalog to registry in metadata files

* Run generate models

* Fix missed renames

* Add github personal access token

* Run black

* Automated Change

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-04-18 22:15:11 +02:00
Akash Kulkarni
f23b3ad88e Add logging functionality to understand Xmin wraparound (#25220)
* Add logging functionality

- Add logging for DB version (all JDBC sources)
- Add logging for Xmin values

* Automated Change

* Renaming + comments

* More comments

* Bump docker version + docs

* auto-bump connector version

* Update failed source_specs.yaml

---------

Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-17 22:39:04 +00:00
Augustin
ebc907cdf7 create metadata files for all connectors (#24964) 2023-04-13 07:45:04 +02:00
Subodh Kant Chaturvedi
b5a2ceb7fe cdc: refactor to remove debezium dependency from connector packages (#24656)
* cdc: refactor to remove debezium dependency from connector packages

* use gradle's shared dependency

* more refactoring

* upgrade docker version

* Automated Change

* auto-bump connector version

* manual bump

---------

Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-11 21:32:17 +05:30
Akash Kulkarni
c0341a3271 Fix emitted intermediate state for initial incremental non-CDC syncs (#24820)
* Initial fix for issue https://github.com/airbytehq/airbyte/issues/24808

* Revert pg

* Adding a more robust unit test for PostgresSource

* Log the query

* Bump dockerfiles + docs

* Bump docs

* auto-bump connector version

* Revert alloydb changes

* Manually update source specs for ones that failed

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-06 22:49:32 +00:00
Subodh Kant Chaturvedi
8748a8d6c7 fix-cdc: errors retry debezium property should be less than max retry (#24833)
* fix-cdc: errors retry debezium property should be less than max retry

* add comment

* version bump + changelog

* auto-bump connector version

* Update source-alloydb versions to match source-postgres

* rebump to 2.0.15

* auto-bump connector version

* definitions + regenerate manually

---------

Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: nguyenaiden <duy@airbyte.io>
2023-04-04 19:23:00 -07:00
Rodi Reich Zilberman
cd928a7844 Fix all tests to pass on local and CI environments (#24683)
* test docker behavior on CI env

* Automated Change

* test docker behavior on CI env

* Make all unit and integration tests in source-postgres pass locally

* Fix mysql ssh integration test

* Fix failing test

* Fix source-mssql build

* source-mssql runss tests locally.
Fix compilation errors

---------

Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-04-03 12:30:11 -07:00
Subodh Kant Chaturvedi
29a7e2ebf5 cdc: fix debezium shutdown interruption bug (#24166)
* cdc: fix debezium shutdown interruption bug

* update retry properties

* add test for queue transfer

* refactor to simplify

* rename parameter

* fix test

* add comments

* add check for engine already closed

* disable checkpointing

* bump version

* auto-bump connector version

* manual bump

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-28 19:47:40 +05:30
Sergio Ropero
80da900d70 feat: Enable CDC checkpointing on Postgres (#24373)
* Fix the error reading offset file meanwhile Debezium is writing.
Enable CDC checkpointing to Postgres.
Minor change in the variable name to fit the type.

* Add final statement on exception ;)

* Add comments to CDC Checkpoint tests.
Clean a bit.

* Bump connector versioning

* Add log message

* Fix changelog

* auto-bump connector version

* Manually generate definitions

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-27 14:00:49 +02:00
VitaliiMaltsev
12eac421de JDBC Sources: remove redundant date-time datatypes formatting (#20760)
* JDBC sources: consolidate date/time values mapping

* fixed Snowflake acceptance tests

* fixed TestJdbcUtils

* refactoring

* update setDate for incremental sync

* fixed CockroachDbJdbcSourceAcceptanceTest

* fixed CockroachDbSourceDatatypeTest

* fixed CockroachDbSourceDatatypeTest

* fixed MsSQLSourceDatatypeTest

* fixed MsSQL cdc datatype test

* refactoring

* fixed mssql cdc test

* fixed Redshift tests

* fixed Redshift tests

* updated db2 datatype tests

* fixed DB2 tests

* fixed DB2 strict-encrypt tests

* fixed Oracle strict-encrypt tests

* refactoring

* updated changelog

* refactoring

* removed redundant overrides

* bump versions

* auto-bump connector version

* JDBC Sources: remove redundant date parsing

* updated Snowflake test

* updated cockroachdb test

* updated date-time datatypes parsing in Postgres Source

* updated formatting in JDBC sources

* updated test cases

* updated Snowflake test

* updated Snowflake, Redshift, Cockroach, DB2 tests

* updated Snowflake, Postgres tests

* updated Redshift tests

* updated MySQL, MsSQL, DB2 tests

* updated MsSQL tests

* updated Oracle tests

* updated Oracle tests

* updated Oracle tests

* fixed some tests

* refactoring

* fixed checkstyle

* fixed TestJdbcUtils

* randomize Reshift test username

* removed redundant test

* updated sanity test

* Automated Change

* format time and bump versions

* auto-bump connector version

* bump versions

* auto-bump connector version

* auto-bump connector version

* manually update definitions

* Automated Change

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: ievgeniit <etsybaev@gmail.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-23 14:31:50 +00:00
Duy Nguyen
c1323f615d Verify catalog schema fieldNames is a subset of DB schema (#24207)
* Update current schema vs catalog validation to omit airbyte metadata fields
* Bumped versions for postgres, mssql, and mySQL + changelog
* Bumped dockerfile version of strict encrypt
* Manually generate definitions for mysql
* Manually generate definitions for mssql

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-22 16:41:15 +00:00
VitaliiMaltsev
f41d8219df MySQL Source : add CDC heartbeat support (#23984)
* MySQL Source : add CDC heartbeat support

* removed logs

* fixed DebeziumRecordIteratorTest

* Automated Change

* use generics to simplify implementation

* use Duration

* more refactoring

* removed redundant null check

* Automated Change

* bump version

* auto-bump connector version

---------

Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-21 16:54:45 +00:00
Sergio Ropero
ca2d9c733f fix: Disable CDC checkpointing using a flag (#24275)
* Disable CDC checkpointing until finished the investigation with reading Offset file during syncs

* Bump connector versions

* auto-bump connector version

* Publish postgres manually

* Definitions manually for mysql

* Generated definitions manually mssql

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-21 15:32:20 +01:00
VitaliiMaltsev
9e94126fd6 MySQL/MsSQL sources fixed NPE during cursor values validation (#24082)
* MySQL/MsSQL sources  fixed NPE during cursor values validation

* added logging

* bump versions

* auto-bump connector version

* manually update mssql definitions

* Automated Change

* Automated Commit - Formatting Changes

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-15 19:27:31 +00:00
VitaliiMaltsev
ace229f971 MySQL/MsSQL Source: stop sync on null cursor value (#23908)
* MySQL Source: stop sync on null cursor value

* updated MsSQL test

* Fixed testTableWithNullCursorValueShouldThrowException test for Mssql

* fixed MsSQL and MySQL tests

* modified MsSQL query

* fixed typo

* fixed typo

* simplified Mysql query logic

* simplified Mssql query logic

* add quotes to Mssql query

* add mysql view test

* MySQL and MsSQL source throw a warning if a cursor column contains null value

* disabled tests; log warn if null cursor values found

* removed logging in AbstractDbSource

* bump version

* bump version

* auto-bump connector version

* updated changelog

* Automated Change

* updated definitions

* Automated Change

---------

Co-authored-by: ievgeniit <etsybaev@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-14 22:34:58 +02:00