1
0
mirror of synced 2026-01-06 06:04:16 -05:00
Commit Graph

118 Commits

Author SHA1 Message Date
Ryan Br...
f127d7ada9 chore: add channel between file aggregation and load steps (#48865)
Co-authored-by: Johnny Schmidt <john.schmidt@airbyte.io>
2024-12-12 18:07:20 -08:00
Edward Gao
58d01c1825 Bulk load CDK: test runner not micronaut, fix concurrent execution (#47006) 2024-10-18 22:45:16 +03:00
Edward Gao
a2ead855ab Bulk load CDK: set up dockerized/non-dockerized integration test tasks (#46691) 2024-10-11 10:38:37 -07:00
Edward Gao
fb4a43f603 Bulk load cdk: Implement the docker-capable DestinationProcess (#46363) 2024-10-10 23:22:05 +03:00
Edward Gao
25944cad4a Bulk CDK: remove redundant source set? (#46690) 2024-10-09 13:50:54 -07:00
Johnny Schmidt
7cdd998aa5 Load CDK E2ETest Destination w/o tests (#45147) 2024-09-26 14:24:49 -07:00
Johnny Schmidt
a6b721f82f Bulk CDK Plugin: typo in build.gradle (#45933) 2024-09-26 19:51:36 +00:00
Johnny Schmidt
1f7158fa2c Bulk CDK Plugin: use required integration test task name (#45931) 2024-09-26 12:02:04 -07:00
Johnny Schmidt
2852238d94 Bulk CDK: airbyte-bulk-connector plugin supports integration tests (#45886) 2024-09-25 10:26:39 -07:00
Marius Posta
619b7571e8 gradle: fix broken plugin source (#44835) 2024-08-27 16:23:24 -05:00
Marius Posta
ff31b44a41 gradle: add airbyte-bulk-connector plugin (#44834) 2024-08-27 15:59:23 -05:00
Edward Gao
df2bb71fed Java connectors: Fix kotlin compilation (#37346) 2024-04-16 17:38:52 +00:00
Edward Gao
d70b3a9bcc jvm connectors: allow integration tests in kotlin (#37114) 2024-04-12 17:20:37 -07:00
Marius Posta
2495575795 java-cdk: re-export airbyte-api dependency (#36759) 2024-04-03 10:43:05 -07:00
Stephane Geneix
56c35cc845 add lombok dependencies to all java connectors (#36733) 2024-04-01 13:03:07 -07:00
Marius Posta
f47db9051b delete bad or useless README files (#36196) 2024-03-15 12:02:23 -07:00
Marius Posta
f93c0df331 gradle: overall simplification (#35307) 2024-02-16 12:46:32 +00:00
Marius Posta
dc088bc3a9 gradle: split off python cdk (#35306) 2024-02-16 04:39:14 -08:00
Marius Posta
b9ab3d1a90 migrate everything from java 17 to java 21 (#35103) 2024-02-09 19:26:19 -06:00
Marius Posta
796b2e8dad java CDK: clean up dependencies, refactor modules (#34745) 2024-02-08 19:46:51 -06:00
Marius Posta
d01bb65dcb java CDK: build no longer downloads files from connector registry (#34441) 2024-01-25 11:44:48 -08:00
Edward Gao
670ffdd32c fix java cdk utility tasks (#34174)
these tasks were only checking within the cdk directory, so they didn't actually accomplish anything. Fix them to walk the entire tree.

... afaict we don't actually use these tasks for anything, but might as well make them work?
2024-01-16 06:04:29 -08:00
Marius Posta
30a5127168 enable groovy formatting in airbyte-ci format config, remove spotless from gradle build (#33013)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-12-01 09:27:16 -08:00
Marius Posta
d97a399a24 source-mysql, source-mssql: parallelize test execution (#32772)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-11-27 12:49:06 -06:00
Marius Posta
6813ed60ba gradle: speed up all java connector integration tests (#32318) 2023-11-08 12:57:03 -08:00
Marius Posta
04ae91ff96 java CDK: spin off s3 destinations (#32050)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-11-03 20:31:58 +00:00
Marius Posta
ef6dbd0cb9 gradle: test task configuration changes (#32108)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-11-03 12:54:09 -05:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-10-30 12:03:06 -07:00
Marius Posta
1394a19ab1 gradle: repo cleanup (#31460) 2023-10-18 13:43:41 -05:00
Edward Gao
f1baf2a2f2 Destinations snowflake+bigquery: Improve performance by filtering raw table on extracted_at (#31191)
Co-authored-by: edgao <edgao@users.noreply.github.com>
2023-10-17 17:48:18 +00:00
Marius Posta
696118de6a gradle: remove broken mypy task (#31468) 2023-10-16 12:05:56 -07:00
Marius Posta
ae2ca40113 gradle: search for python3.11 binary for AL2023 support (#31227) 2023-10-10 13:15:02 -07:00
Marius Posta
f8edc18039 airbyte-ci,gradle: replace airbyte-docker with airbyte-ci (#30743) 2023-10-04 08:38:17 -07:00
Marius Posta
7ae97175a6 gradle: fix repo wide behaviour (#30607) 2023-09-28 05:01:13 -07:00
Aaron ("AJ") Steers
da32fc86bf 🚀 Launch Java CDK v0.1.0 (PR #30383)
- Launch Java CDK `v0.1.0`. 🚀
- Remove `airbyte-docker` gradle plugin from java connectors.
- Add `airbyte-java-connector` gradle plugin.
- Remove `airbyte-commons` from global build.gradle.
- Add `/publish-java-cdk` slash command.
- Initial launch of `/airbyte-ci` slash command to test connectors.
2023-09-27 15:46:02 -07:00
Ben Church
5c56ac1d84 Airbyte-ci: Remove gradle task connectorAcceptanceTest (#30326) 2023-09-19 15:16:37 -05:00
Marius Posta
532ef097eb airbyte-ci: fix integration test log buffering (#30438) 2023-09-14 11:06:19 -05:00
Ben Church
59f803e323 gradle docker: skip if not dev image (#30371) 2023-09-12 16:02:44 -07:00
Marius Posta
51c67d7eaa gradle: remove airbyteDocker.outputs dependencies (#30314) 2023-09-11 17:16:27 -07:00
Marius Posta
e6e62c65d1 gradle: airbyte-docker improvements (#30262)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-09-08 10:25:17 -07:00
Marius Posta
ef2849e35e gradle: fix airbyteDocker task inputs (#30187) 2023-09-07 03:46:31 -07:00
Marius Posta
be1e1adabd gradle: cleanup (#30060) 2023-09-05 14:05:40 -05:00
Edward Gao
934acaa137 Destination bigquery: rerelease 1s1t behind gate (#27936)
* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)""

This reverts commit 348c577dbb.

* version bumps+changelog

* Speed up BQ by having 2 queries, and not an OR (#27981)

* 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (#27856)

* only run t+d code if it's enabled

* dockerfile+changelog

* remove changelog entry

* Destinations V2: handle optional fields for `object` and `array` types (#27898)

* catch null schema

* fix null properties

* clean up

* consolidate + add more tests

* try catch

* empty json test

* Automated Commit - Formatting Changes

* remove todo

* destination bigquery: misc updates to 1s1t code (#28057)

* switch to checkedconsumer

* add unit test for buildColumnId

* use flag

* restructure prefix check

* fix build

* more type-parsing fixes (#28100)

* more type-parsing fixes

* handle duplicates

* Automated Commit - Format and Process Resources Changes

* add tests for asColumns

* Automated Commit - Format and Process Resources Changes

* log warnings instead of throwing exception

* better log message

* error level

---------

Co-authored-by: edgao <edgao@users.noreply.github.com>

* Automated Commit - Formatting Changes

* Improve protocol type parsing (#28126)

* Automated Commit - Formatting Changes

* Change from T&D every 10k records to an increasing time based interval (#28130)

* fifteen minute t&d

* add typing and deduping operation valve for increased intervals of typing and deduping

* Automated Commit - Format and Process Resources Changes

* resolve bizarre merge conflict

* Automated Commit - Format and Process Resources Changes

---------

Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>

* Simplify and speed up CDC delete support [DestinationsV2] (#28029)

* Simplify and speed up CDC delete support [DestinationsV2]

* better QUOTE

* spotbugs?

* recompile dbt image for local arch and use that when building images

* things compile, but tests fail

* tests working-ish

* comment

* fix logic to re-insert deleted records for cursor comparison.

tests pass!

* remove comment

* Skip CDC re-include logic if there are no CDC columns

* stop hardcoding pk (#28092)

* wip

* remove TODOs

---------

Co-authored-by: Edward Gao <edward.gao@airbyte.io>

* update method name

* Automated Commit - Formatting Changes

* depend on pinned normalization version

* implement 1s1t DATs for destination-bigquery (#27852)

* intiial implementation

* Automated Commit - Formatting Changes

* add second sync to test

* do concurrent things

* Automated Commit - Formatting Changes

* clarify comment

* minor tweaks

* more stuff

* Automated Commit - Formatting Changes

* minor cleanup

* lots of fixes

* handle sql vs json null better
* verify extra columns
* only check deleted_at if in DEDUP mode and the column exists
* add full refresh append test case

* Automated Commit - Formatting Changes

* add tests for the remaining sync modes

* Automated Commit - Formatting Changes

* readability stuff

* Automated Commit - Formatting Changes

* add test for gcs mode

* remove static fields

* Automated Commit - Formatting Changes

* add more test cases, tweak test scaffold

* cleanup

* Automated Commit - Formatting Changes

* extract recorddiffer

* and use it in the sql generator test

* fix

* comment

* naming+comment

* one more comment

* better assert

* remove unnecessary thing

* one last thing

* Automated Commit - Formatting Changes

* enable concurrent execution on all java integration tests

* add test for default namespace

* Automated Commit - Formatting Changes

* implement a 2-stream test

* Automated Commit - Formatting Changes

* extract methods

* invert jsonNodesNotEquivalent

* Automated Commit - Formatting Changes

* fix conditional

* pull out diffSingleRecord

* Automated Commit - Formatting Changes

* handle nulls correctly

* remove raw-specific handling; break up methods

* Automated Commit - Formatting Changes

---------

Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>

* Destinations V2: move create raw tables earlier (#28255)

* move create raw tables

* better log message

* stop building normalization (#28256)

* fix ability to run tests

* disable incremental t+d for now

* Automated Commit - Formatting Changes

---------

Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Cynthia Yin <cynthia@airbyte.io>
Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>
2023-07-14 09:34:56 -05:00
Alexandre Girard
97a353d5c5 Run mypy on airbyte-cdk as part of the build pipeline and fix typing issues in the file-based module (#27790)
* Try running only on modified files

* make a change

* return something with the wrong type

* Revert "return something with the wrong type"

This reverts commit 23b828371e.

* fix typing in file-based

* format

* Mypy

* fix

* leave as Mapping

* Revert "leave as Mapping"

This reverts commit 908f063f70.

* Use Dict

* update

* move dict()

* Revert "move dict()"

This reverts commit fa347a8236.

* Revert "Revert "move dict()""

This reverts commit c9237df2e4.

* Revert "Revert "Revert "move dict()"""

This reverts commit 5ac1616414.

* use Mapping

* point to config file

* comment

* strict = False

* remove --

* Revert "comment"

This reverts commit 6000814a82.

* install types

* install types in same command as mypy runs

* non-interactive

* freeze version

* pydantic plugin

* plugins

* update

* ignore missing import

* Revert "ignore missing import"

This reverts commit 1da7930fb7.

* Install pydantic instead

* fix

* this passes locally

* strict = true

* format

* explicitly import models

* Update

* remove old mypy.ini config

* temporarily disable mypy

* format

* any

* format

* fix tests

* format

* Automated Commit - Formatting Changes

* Revert "temporarily disable mypy"

This reverts commit eb8470fa3f.

* implicit reexport

* update test

* fix mypy

* Automated Commit - Formatting Changes

* fix some errors in tests

* more type fixes

* more fixes

* more

* .

* done with tests

* fix last files

* format

* Update gradle

* change source-stripe

* only run mypy on cdk

* remove strict

* Add more rules

* update

* ignore missing imports

* cast to string

* Allow untyped decorator

* reset to master

* move to the cdk

* derp

* move explicit imports around

* Automated Commit - Formatting Changes

* Revert "move explicit imports around"

This reverts commit 56e306b72f.

* move explicit imports around

* Upgrade mypy version

* point to config file

* Update readme

* Ignore errors in the models module

* Automated Commit - Formatting Changes

* move check to gradle build

* Any

* try checking out master too

* Revert "try checking out master too"

This reverts commit 8a8f3e373c.

* fetch master

* install mypy

* try without origin

* fetch from the script

* checkout master

* ls the branches

* remotes/origin/master

* remove some cruft

* comment

* remove pydantic types

* unpin mypy

* fetch from the script

* Update connectors base too

* modify a non-cdk file to confirm it doesn't get checked by mypy

* run mypy after generateComponentManifestClassFiles

* run from the venv

* pass files as arguments

* update

* fix when running without args

* with subdir

* path

* try without /

* ./

* remove filter

* try resetting

* Revert "try resetting"

This reverts commit 3a54c424de.

* exclude autogen file

* do not use the github action

* works locally

* remove extra fetch

* run on connectors base

* try bad  typing

* Revert "try bad  typing"

This reverts commit 33b512a3e4.

* reset stripe

* Revert "reset stripe"

This reverts commit 28f23fc6dd.

* Revert "Revert "reset stripe""

This reverts commit 5bf5dee371.

* missing return type

* do not ignore the autogen file

* remove extra installs

* run from venv

* Only check files modified on current branch

* Revert "Only check files modified on current branch"

This reverts commit b4b728e654.

* use merge-base

* Revert "use merge-base"

This reverts commit 3136670cbf.

* try with updated mypy

* bump

* run other steps after mypy

* reset task ordering

* run mypy though

* looser config

* tests pass

* fix mypy issues

* type: ignore

* optional

* this is always a bool

* ignore

* fix typing issues

* remove ignore

* remove mapping

* Automated Commit - Formatting Changes

* Revert "remove ignore"

This reverts commit 9ffeeb6cb1.

* update config

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
2023-07-13 16:55:48 -07:00
Catherine Noll
fbe4ecc5be Fix: only build docker images with the local CDK for connectors (#24259) 2023-03-20 19:41:18 -04:00
Sergio Ropero
7fa8508999 fix: Cast the number of threads to integer in the Java connector tests 😵‍💫 (#23917)
* reduce threads

* Fix real problem between CHAR and INTEGER
2023-03-09 19:12:33 +00:00
Sergio Ropero
3db452dff9 Property to limit test parallelisation (#23870)
* Adds new functionality on integration test pipeline and test pipeline in order to reduce parallelization.
It's done by creating a `gradle.properties` file in the connector root folder.

Extra: removes test concatenation for performance tests.

* rename parameter

* fix names

* Reduce threads
2023-03-08 18:48:20 +01:00
Sergio Ropero
f0909f6fa2 feat: Set number of forks to execute tests in parallel. (#23506)
* Set number of forks to execute tests in parallel.

* Add back `upToDateWhen`
2023-03-01 15:27:31 +01:00
Catherine Noll
d4ec2d952e Update /test to optionally test against the local CDK (#23434) 2023-02-28 11:53:51 -05:00
Joe Reuter
053d352a1c Make connector acceptance test gradle plugin work for destinations (#22609)
Fix connector acceptance test gradle plugin
2023-02-24 18:10:44 +01:00