1
0
mirror of synced 2025-12-20 18:39:31 -05:00
Commit Graph

192 Commits

Author SHA1 Message Date
Marius Posta
f8edc18039 airbyte-ci,gradle: replace airbyte-docker with airbyte-ci (#30743) 2023-10-04 08:38:17 -07:00
Jonathan Pearlin
5d6597e5b6 Source MongoDB V2: Fix nested object filtering (#31034) 2023-10-03 18:31:00 -05:00
Jonathan Pearlin
26787a82e5 Source MongoDB V2: Support incremental syncs via Debezium (#29969)
Co-authored-by: Cole Snodgrass <cole@airbyte.io>
Co-authored-by: colesnodgrass <colesnodgrass@users.noreply.github.com>
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
Co-authored-by: Jose Pefaur <jose.pefaur@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Ben Church <ben@airbyte.io>
2023-10-03 12:37:10 -05:00
Marius Posta
7ae97175a6 gradle: fix repo wide behaviour (#30607) 2023-09-28 05:01:13 -07:00
Aaron ("AJ") Steers
da32fc86bf 🚀 Launch Java CDK v0.1.0 (PR #30383)
- Launch Java CDK `v0.1.0`. 🚀
- Remove `airbyte-docker` gradle plugin from java connectors.
- Add `airbyte-java-connector` gradle plugin.
- Remove `airbyte-commons` from global build.gradle.
- Add `/publish-java-cdk` slash command.
- Initial launch of `/airbyte-ci` slash command to test connectors.
2023-09-27 15:46:02 -07:00
Ben Church
5c56ac1d84 Airbyte-ci: Remove gradle task connectorAcceptanceTest (#30326) 2023-09-19 15:16:37 -05:00
Marius Posta
a5eb24e2fc bump testcontainers version to latest (#30493) 2023-09-18 03:50:43 -05:00
Marius Posta
51c67d7eaa gradle: remove airbyteDocker.outputs dependencies (#30314) 2023-09-11 17:16:27 -07:00
Marius Posta
ef2849e35e gradle: fix airbyteDocker task inputs (#30187) 2023-09-07 03:46:31 -07:00
Marius Posta
f5c7c1c0b8 chore: get ./gradlew format to pass for the whole repo (same java style) (#29786) 2023-08-24 05:09:42 -05:00
Augustin
53129c56a6 [skip ci]: alpha connectors should be community (#29550) 2023-08-17 21:50:24 +02:00
Ben Church
40781313da Update Internal Fields: update ql levels to better resemble previous high strictness (#29450) 2023-08-15 18:03:10 -05:00
Ben Church
2f7deaee02 [skip ci] Metadata: Remove leading underscore (#29024)
* DNC

* Add test models

* Add model test

* Remove underscore from metadata files

* Regenerate models

* Add test to check for key transformation

* Allow additional fields on metadata

* Delete transform
2023-08-03 10:56:13 -07:00
Jonathan Pearlin
4540eab27d 🐛 Source Mongo: Fix failing acceptance tests (#28816)
* Fix failing acceptance tests

* Fix failing strict acceptance tests
2023-08-03 09:13:37 -05:00
Ben Church
e9490e3fb6 Connector Levels: Add new internal metadata fields (#28904)
* Add airbyte internal

* Add tests

* First pass

* Set destinations to same levels as sources

* Best guess at missing statuses

* Best guess at _ql

* Add separate enum class

* Fix support level name

* Update templates

* Add one more test
2023-08-01 18:08:33 -05:00
Jonathan Pearlin
9ca8cdece6 🐛 Source Mongo: Revert recent changes (#28815)
* revert changes

* revert changes

* revert changes

* revert changes

* revert changes

* Prepare release

* Add pull request to release notes

* Automated Commit - Formatting Changes

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
2023-07-27 14:47:47 -05:00
Cole Snodgrass
0622860baf 🐛 Source Mongo: fix potential npe when collecting stats (#28760)
* avoid NPE in getCollectionStats

* remove accidental whitespace

* update mongo changelog
2023-07-27 08:18:58 -05:00
Jonathan Pearlin
3eb222fc89 Source MongoDB V2: Clean fieldPath before use in projection (#28733)
* Clean fieldPath before use in projection

* Update release notes

* Automated Commit - Format and Process Resources Changes

* Remove prefix $ from field name

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
2023-07-26 12:11:46 -05:00
Jonathan Pearlin
28bedf9675 Source MongoDB V2: Handle statistics collection failure gracefully (#28692)
* Fix bug introduced by statistics collection

* Fix syntax

* Formatting

* Fix failing test

* Fix failing test

* Update release notes
2023-07-25 15:22:02 -05:00
Jonathan Pearlin
496678df5f Source MongoDB V2: Record statistics about MongoDB source connection (#28527)
* Record statistics about source database/collection

* Formatting

* Remove unnecessary log message

* Only record statistics during sync

* Emit structured Google log data

* Revert use of System.out.println

* Try using Log4j ObjectMessage

* Revert "Try using Log4j ObjectMessage"

This reverts commit d518c87aa8.

* Prepare release

* Update changelog

* Correct release version
2023-07-24 12:41:40 -05:00
Jonathan Pearlin
b017ee67f9 Test MongoDB Replica Set/Oplog via Testcontainers (#28400)
* Update dependencies

* Use RawBsonDocument

* Add tests

* Formatting

* Incremental refresh test

* Formatting

* Fix NPE

* Test replica set/oplog via testcontainers

* Formatting

* PR feedback

* Automated Commit - Formatting Changes

* Add logging to debug integration test failures

* Fix failing test

* Remove credentials conversion logic

* Update documentation to include correct config format

* Use database from credentials

* Make expected catalog match test data

* Fix formatting

* Cleanup

* Improve logging

* Revert test catalog change

* Use test database

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
2023-07-20 11:27:07 -05:00
Evan Tahler
9ce3cc6e79 Java DB Source connector licenses to Elv2 (#27737)
* Java DB Source connector licenses to Elv2

* Bump PR Ids
2023-06-28 18:52:02 -07:00
Ben Church
1dabc6208e Metadata: add tags field (#26320)
* Add optional tags field

* Remove duplicate icons

* Add programming tags to all

* Update docs

* supportUrl -> documentationUrl

* Ensure one language tag is applied

* Add keyvalue check

* rebase and fix tests

* Format

* Add cache buster

* Improve test

* Automated Commit - Formatting Changes

* Update error

* Fix missing tags

* Fix scaffold

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-05-26 16:13:09 -07:00
Augustin
fd3655707e connectors-ci: add finalize_build logic to handle custom Dockerfiles (#26489) 2023-05-25 09:32:19 +02:00
Ben Church
248bbf94c1 HACKDAY: Icon CDN (#26158)
* Move icons to connector folder

* Delete old icons

* Update upload logic

* Add icon url to definitions

* Update registry model

* Populate cdn url

* DNC butcher the pipeline

* Low hanging fruit fixes

* Fix bucket name

* Merge old and new approaches

* Fix metadata upload step

* Format

* Fix test
2023-05-24 17:25:41 -07:00
Augustin
7310494846 qa-checks: check metadata version matches dockerfile version (#25661) 2023-05-04 16:08:19 -07:00
Jonathan Pearlin
a38af089e9 Track stream status in source (#24971)
* WIP Track stream status in source

* Revert formatting

* Revert formatting changes

* Remove unnecessary file

* Automated Change

* Automated Change

* Use new stream status trace message

* Rename class

* Remove unnecessary import

* Formatting

* Add tests

* Fix compile issues

* Automated Commit - Formatting Changes

* Remove TODO

* Fix compilation error

* Split STOPPED into INCOMPLETE and COMPLETE

* Remove unused import

* Changelog updates for source-postgres

* Remove unused import

* auto-bump connector version

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-26 15:14:25 -05:00
Ben Church
5563179782 Dagster: rename catalog to registry (#25254)
* rename catalog to registry in metadata service

* rename catalog to registry in metadata files

* Run generate models

* Fix missed renames

* Add github personal access token

* Run black

* Automated Change

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-04-18 22:15:11 +02:00
Augustin
ebc907cdf7 create metadata files for all connectors (#24964) 2023-04-13 07:45:04 +02:00
Akash Kulkarni
c0341a3271 Fix emitted intermediate state for initial incremental non-CDC syncs (#24820)
* Initial fix for issue https://github.com/airbytehq/airbyte/issues/24808

* Revert pg

* Adding a more robust unit test for PostgresSource

* Log the query

* Bump dockerfiles + docs

* Bump docs

* auto-bump connector version

* Revert alloydb changes

* Manually update source specs for ones that failed

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-06 22:49:32 +00:00
Akash Kulkarni
6996ae9949 Remove createDatabaseInternal() (#23689)
* Remove createDatabaseInternal()

* Fix test

* fix format
2023-03-06 16:50:40 -06:00
Catherine Noll
7da6a3bb77 Run CATs with local CDK (#23084)
Scripts to 

* Run CATs against the local CDK for one connector

* Run CATs against the local CDK for multiple connectors

* Create a connecter image with the local CDK

---------

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2023-02-24 16:13:42 -05:00
Peter Hu
e5eac0a4cc use published protocol models jar (#22498)
* add airbyte-protocol to deps.toml

* use published protocol jar for platform

* use published protocol jar for connectors

* point at published jar

* fix dep

* bump gcs storage

* fix build failures in standard-source-test

* fix deps

* downgrade alloy db because it is missing strictness tests

* Revert "downgrade alloy db because it is missing strictness tests"

This reverts commit cc6089d053.

---------

Co-authored-by: cgardens <charles@airbyte.io>
2023-02-13 12:50:43 -06:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594)
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
2023-02-08 13:01:16 -08:00
Sergio Ropero
b87647df62 Revert "feat: Filter out system views out of system namespaces (#22221)" (#22425)
This reverts commit 3bf87a992e.
2023-02-06 23:54:28 +05:30
Sergio Ropero
3bf87a992e feat: Filter out system views out of system namespaces (#22221)
* This changes allows to filter out system views created out of system namespaces

* Add extra view

* Fix issue

* Bump Postgres source version
2023-02-06 19:10:30 +01:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846)
Rename source acceptance test to connector acceptance test
2023-02-02 11:38:19 +01:00
Augustin
0c8d6d3a30 remove supportsNormalization + supportsDbt from source spec (#21317) 2023-01-14 18:40:57 +00:00
Akash Kulkarni
bac789e2cb Code cleanup in SourceOperations (#20874)
* Refactor SourceOperations class

* More cleanup

* Addressing comments

* Formatting
2023-01-03 03:09:48 -08:00
Yevhen Sukhomud
2a5d0344e7 Fix formatting (#20570)
* Fix formatting

* try fixing quote escapes?

* unignore sql files

Co-authored-by: Edward Gao <edward.gao@airbyte.io>
2022-12-16 13:43:42 -08:00
Jimmy Ma
094aff1520 Java connectors use protocol v0 objects (#20404)
* Dupl v0 objects

* v0 namespace swap

* Introduce some object conversions in the tests

* Update source-java-jdbc template

* Use v0 objects in db-lib

* Fix more protocol object references

* Fix arg capture in tests
2022-12-15 13:39:50 -08:00
Augustin
061e269f23 source-mongodb-v2: enable SAT (#20484) 2022-12-15 20:54:51 +01:00
Andrii Leonets
58e4ef6912 Enable SPEC SAT for Java sources (#18779)
* enable py spec compatibility tests

* add missing properties

* use expected spec file instead of the source spec.json + use dummy config file

* add missing files for the clickhouse

* move test files to the integration-test from the unit test folder

* add missing files to cockroachdb

* add missing files to db2 + fix spec format

* add missing files to elasticsearch + fix spec format

* add missing files to jdbc

* add missing files to mongodb_v2 + fix spec format

* add missing files to mssql + fix spec format

* add missing files to mysql + fix spec format

* add missing files to postgres + fix spec format

* add missing files to oracle + fix spec format

* add missing files to redshift

* add missing files to sftp

* add missing files to snowflake + fix spec format

* add missing files to tidb

* add missing files to kafka - fix spec format

* airbyte-source-acceptance-test added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* add missing import

* Delete acceptance-test-config.yml

* Delete acceptance-test-docker.sh

* Update build.gradle

* Update build.gradle

* format

* revert changes

* manual .sh files format

* upd expected spec

* format

* fix SAT after master merge

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-11-09 21:45:46 +02:00
Ryan Fu
804af5d8ed Adds TLS check to mongodb destination and migrates util constants (#18892)
* Adds TLS check to mongodb destination and migrates util constants

* Migrates MongodbSourceUitls to general purprose Utils file

* Updates expected_spec.json to include SSH tunnel

* Bumps connector version and removes connector from being hidden in UI

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-11-08 13:21:15 -08:00
Ryan Fu
cc41385f50 Updates getPrivateKeyPair to throw exception when private key pair does not exist (#18263)
* Changes method from returning null to throwing Exception to match method structure

* Added javadoc explaining that KeyPair should not be returning null

* Moved ConnectionErrorException to commons package

* Adds error handling for SshWrappedDestinations and markdown to include changes

* Bumps version numbers for Postgres, MySQL, MSSQL

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-21 16:04:39 -07:00
Liren Tu
888347a0d0 🎉 JDBC sources: store cursor record count in db state (#15535)
* Add cursor_record_count to db stream state

* Add cursor record count to cursor info

* Emit max cursor record count

* Add original cursor record count

* Unify logging format

* Add backward compatible methods

* Update unit tests for state decorating iterator

* Update test (not done yet)

* Fix one more unit test

* Change where clause operator according to record count

* Add branch for null cursor

* Skip saving record count when it is 0

* Fix log wording

* Set mock record count in test

* Check cursor value instead of cursor info

* Fix source jdbc test

* Read record count from state

* Fix tests

* Add an acceptance test case

* Fix npe

* Change record count from int to long to avoid type conversion

* Fix references

* Fix oracle container

* Use uppercase for snowflake

* Use uppercase for db2

* Fix and use uppercase

* Update test case to include the edge case

* Format code

* Remove extra assertion in clickhouse

* Merge ms sql incremental query method

* Log query for debugging

* Clean up name_and_timestamp table

* Fix db2 tests

* Fix mssql tests

* Fix oracle tests

* Fix oracle tests

* Fix cockroachdb tests

* Fix snowflake tests

* Add changelog

* Fix mssql tests

* Fix db2-strict-encrypt tests

* Fix oracle-strict-encrypt tests

* Bump postgres version

* Fix oracle-strict-encrypt tests

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-14 01:09:12 -07:00
Eugene
33dfd5d6be Fixed "Ensure no file change" build issue. Only checkstyle changes, no logic chnages (#17885) 2022-10-12 15:06:28 +03:00
Pedro S. Lopez
938436bcc9 update connector specs and definitions with new .com documentation urls (#17585)
* update definitions with new .com docs urls

* update docs urls in specs

* update generators

* regenerate scaffold connectors

* remove unrelated changes

* update more urls

* update specs

* fix tests

* run `:airbyte-config:specs:generateSeedConnectorSpecs` to fix formatting

* revert docs changes to make pr more reviewable

* revert generator readme changes to make more reviewable

* fix mysql strict encrypt expected spec

* fix postgres expected spec
2022-10-11 11:04:23 -04:00
VitaliiMaltsev
84f02995ba MongoDb Source: Increase performance of discover (#17614)
* MongoDb Source: Increase performance of discover

* bump version

* fixed tests

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-07 14:18:29 +03:00
Akash Kulkarni
4b990d0797 🚨 Add ability to enforce SSL in MongoDB source connector (#17590)
* Check and test for if we are disabling SSL/TLS in source-mongodb-strict-encrypt

* Update docs

* Address comments

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-06 11:57:36 -07:00