1
0
mirror of synced 2025-12-19 10:00:34 -05:00

138 Commits

Author SHA1 Message Date
Edward Gao
0bb3665f90 Upgrade develocity plugin; disable failOnSkippedAfterRetry (#69221) 2025-11-06 08:07:30 -08:00
Wenqi Hu
00259fc881 Publish docker image to dockerRepository from metadata.yaml (#67072)
## What
Currently the connector image publishing flow use project name/connector
directory to construct the repo name in dockerhub and ignoring the repo
defined in metadata.yml file. It creates inconsistency between
publishing the pulling image flow. This PR is to fix the issue

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO 
2025-10-16 14:33:00 -07:00
Edward Gao
1d0ac26943 Bulk CDK: better test retry config (#68087)
## What
Correctly ignore previous failed test attempts if we eventually
succeeded. See e.g.
https://github.com/airbytehq/airbyte/runs/52648135211 for an example
incorrect failure - the test ran twice and succeeded on the second
attempt, but the overall PR check was still marked as failed.

## How
Enable the flag to tell gradle to do a smarter thing.

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [x] YES 💚
- [ ] NO 
2025-10-14 09:15:29 -07:00
Edward Gao
eba9c379e1 integration test configurations extend other configurations (#66694) 2025-09-26 10:07:25 -07:00
Edward Gao
e08e849be6 add timeout+retries to JVM connector integration tests (#66670) 2025-09-24 21:18:53 +00:00
Jose Pefaur
9266402079 ci: add gradle tasks to manage connector cdk version (#66224)
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
2025-09-22 14:58:08 -05:00
Edward Gao
335acd4052 Bulk CDK: move connector cdk version decl into property file (#66484) 2025-09-22 10:14:55 -07:00
Ryan Br...
373e73b1d8 Pin these destinations with flakey testing to latest CDK. (#65625)
Co-authored-by: davinchia <davinchia@gmail.com>
2025-08-29 16:07:55 -07:00
Edward Gao
6baa9c0e09 Destination Bigquery: Even more handling for billing errors (#63700) 2025-07-24 09:13:00 -07:00
Edward Gao
e137a10160 Destination bigquery: Implement direct load (#59752)
Co-authored-by: Francis Genet <francis.genet@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Ian Alton <ian.alton@airbyte.io>
2025-06-25 20:09:17 +00:00
Jimmy Ma
71733f28dc chore: enable cdk toolkit version pinning (#61430) 2025-06-06 14:20:39 -07:00
Davin Chia
7cef78e9f6 chore: fix mssql build (#61017)
- the bulk connector plugin hardcodes integration tests set up today instead of inheriting the same properties from the general test set up. Remove this.
- the Dest MSSQL uses a custom network to spin up test containers that is no longer required since we are no longer in Dagger. Remove this. This also simplifies things.
2025-05-30 23:09:16 +00:00
Edward Gao
8f447197ec Destination bigquery: use new cdk (#56980)
Co-authored-by: Francis Genet <francis.genet@airbyte.io>
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
2025-05-15 15:34:06 +00:00
Davin Chia
ad538f4bab chore: move dest-dev-null to new non-dagger flow. (#59759)
Prep to transition dev-null onto the new flow set up in #59716 .
2025-05-09 21:47:21 +00:00
Davin Chia
18e6370fc1 chore: non-dagger builds. (#59716)
Introduce the airbyte gradle docker plugin to build JVM connectors.
Add logic to allow us to slowly migrate from one build set up to the other.

The majority of the work is taking in the metadata.yaml file, and converting that to buildArgs to be injected into the airbyte gradle docker plugin.

In particular, we extract the base image from the metadata file. We also add the connector name by referencing the module. This outputs a file buildArgs.properties into the build folder. This is done by the DockerGenerateConnectorBuildArgs task.

The AirbyteDockerConventionPlugin then takes in the generated buildArgs.properties file and feeds it into the Platform Docker plugin. It also feeds in the top-level Dockerfile, so the Docker plugin can copy it into the build folder.

I tried to migrate this to Kotlin, however the repo needs to be upgraded to the same Kotlin and KSP version as the gradle plugin repos for this to be worth while. Otherwise we end up writing kotlin plugins with reflection, which is worse than the current groovy plugins.
2025-05-08 20:04:22 -04:00
Edward Gao
cd1f957764 Java connectors: echo junit test stdout to console (#53677) 2025-02-13 23:14:50 +00:00
Edward Gao
1bebecf25c Bulk Load CDK: always run integration tests (#52569) 2025-02-12 22:06:22 +00:00
Marius Posta
b309a7c309 gradle: bulk-cdk connector junit tests should respect withSlowTests (#52661) 2025-01-30 15:31:37 -06:00
Johnny Schmidt
ba92625c6b Destination S3-V2: Rc 9 with bug fixes (#51048) 2025-01-12 22:02:56 +02:00
Edward Gao
526c159758 Bulk load CDK: integration test should log full exception (#49466) 2024-12-13 22:08:30 +00:00
Ryan Br...
f127d7ada9 chore: add channel between file aggregation and load steps (#48865)
Co-authored-by: Johnny Schmidt <john.schmidt@airbyte.io>
2024-12-12 18:07:20 -08:00
Edward Gao
58d01c1825 Bulk load CDK: test runner not micronaut, fix concurrent execution (#47006) 2024-10-18 22:45:16 +03:00
Edward Gao
a2ead855ab Bulk load CDK: set up dockerized/non-dockerized integration test tasks (#46691) 2024-10-11 10:38:37 -07:00
Edward Gao
fb4a43f603 Bulk load cdk: Implement the docker-capable DestinationProcess (#46363) 2024-10-10 23:22:05 +03:00
Edward Gao
25944cad4a Bulk CDK: remove redundant source set? (#46690) 2024-10-09 13:50:54 -07:00
Johnny Schmidt
7cdd998aa5 Load CDK E2ETest Destination w/o tests (#45147) 2024-09-26 14:24:49 -07:00
Johnny Schmidt
a6b721f82f Bulk CDK Plugin: typo in build.gradle (#45933) 2024-09-26 19:51:36 +00:00
Johnny Schmidt
1f7158fa2c Bulk CDK Plugin: use required integration test task name (#45931) 2024-09-26 12:02:04 -07:00
Johnny Schmidt
2852238d94 Bulk CDK: airbyte-bulk-connector plugin supports integration tests (#45886) 2024-09-25 10:26:39 -07:00
Marius Posta
619b7571e8 gradle: fix broken plugin source (#44835) 2024-08-27 16:23:24 -05:00
Marius Posta
ff31b44a41 gradle: add airbyte-bulk-connector plugin (#44834) 2024-08-27 15:59:23 -05:00
Edward Gao
df2bb71fed Java connectors: Fix kotlin compilation (#37346) 2024-04-16 17:38:52 +00:00
Edward Gao
d70b3a9bcc jvm connectors: allow integration tests in kotlin (#37114) 2024-04-12 17:20:37 -07:00
Marius Posta
2495575795 java-cdk: re-export airbyte-api dependency (#36759) 2024-04-03 10:43:05 -07:00
Stephane Geneix
56c35cc845 add lombok dependencies to all java connectors (#36733) 2024-04-01 13:03:07 -07:00
Marius Posta
f47db9051b delete bad or useless README files (#36196) 2024-03-15 12:02:23 -07:00
Marius Posta
f93c0df331 gradle: overall simplification (#35307) 2024-02-16 12:46:32 +00:00
Marius Posta
dc088bc3a9 gradle: split off python cdk (#35306) 2024-02-16 04:39:14 -08:00
Marius Posta
b9ab3d1a90 migrate everything from java 17 to java 21 (#35103) 2024-02-09 19:26:19 -06:00
Marius Posta
796b2e8dad java CDK: clean up dependencies, refactor modules (#34745) 2024-02-08 19:46:51 -06:00
Marius Posta
d01bb65dcb java CDK: build no longer downloads files from connector registry (#34441) 2024-01-25 11:44:48 -08:00
Edward Gao
670ffdd32c fix java cdk utility tasks (#34174)
these tasks were only checking within the cdk directory, so they didn't actually accomplish anything. Fix them to walk the entire tree.

... afaict we don't actually use these tasks for anything, but might as well make them work?
2024-01-16 06:04:29 -08:00
Marius Posta
30a5127168 enable groovy formatting in airbyte-ci format config, remove spotless from gradle build (#33013)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-12-01 09:27:16 -08:00
Marius Posta
d97a399a24 source-mysql, source-mssql: parallelize test execution (#32772)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-11-27 12:49:06 -06:00
Marius Posta
6813ed60ba gradle: speed up all java connector integration tests (#32318) 2023-11-08 12:57:03 -08:00
Marius Posta
04ae91ff96 java CDK: spin off s3 destinations (#32050)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-11-03 20:31:58 +00:00
Marius Posta
ef6dbd0cb9 gradle: test task configuration changes (#32108)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-11-03 12:54:09 -05:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-10-30 12:03:06 -07:00
Marius Posta
1394a19ab1 gradle: repo cleanup (#31460) 2023-10-18 13:43:41 -05:00
Edward Gao
f1baf2a2f2 Destinations snowflake+bigquery: Improve performance by filtering raw table on extracted_at (#31191)
Co-authored-by: edgao <edgao@users.noreply.github.com>
2023-10-17 17:48:18 +00:00