1
0
mirror of synced 2026-01-05 03:04:38 -05:00
Commit Graph

647 Commits

Author SHA1 Message Date
Octavia Squidington III
8c96a5e7f5 Bump Airbyte version from 0.39.22-alpha to 0.39.23-alpha (#13984)
Co-authored-by: pmossman <pmossman@users.noreply.github.com>
2022-06-21 12:46:58 -07:00
Parker Mossman
9403c28b50 Parker/temporal cloud (#13243)
* switch to temporal cloud client for now

* format

* use client cert/key env secret instead of path to secret

* add TODO comments

* format

* add logging to debug timeout issue

* add more logging

* change workflow task timeout

* PR feedback: consolidate as much as possible, add missing javadoc

* fix acceptance test, needs to specify localhost

* add internal-use only comments

* format

* refactor to clean up TemporalClient and prepare it for future dependency injection framework

* remove extraneous log statements

* PR feedback

* fix test

* return isInitialized true in test
2022-06-21 11:37:27 -07:00
Octavia Squidington III
5105c00a5d Bump Airbyte version from 0.39.21-alpha to 0.39.22-alpha (#13979)
Co-authored-by: Phlair <Phlair@users.noreply.github.com>
2022-06-21 19:37:05 +01:00
Octavia Squidington III
689064e373 Bump Airbyte version from 0.39.20-alpha to 0.39.21-alpha (#13938)
Co-authored-by: alafanechere <alafanechere@users.noreply.github.com>
2022-06-20 18:04:15 +02:00
Serhii Chvaliuk
4b116f4889 0.2.5 -> 0.2.6 (#13924)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-06-20 09:01:39 +03:00
Benoit Moriceau
74d16ccb09 Do not hide unexpected errors in the check connection (#13903)
* Do not hide unexpected errors in the check connection

* Fix test
2022-06-17 16:46:42 -07:00
Lake Mossman
3bab4c9d1b Persist state to reset connection job config (#13867)
* add current connection state to JobResetConnectionConfig

* pass state from reset connection config to job sync config in generate input activity

* format
2022-06-16 17:49:14 -07:00
Davin Chia
18b023fbab Tweak schema validation code for readability. (#13721)
* Refactor schema validation for better reading.

* Better comments.

* Format.
2022-06-16 10:26:17 -07:00
Octavia Squidington III
aed9244696 Bump Airbyte version from 0.39.19-alpha to 0.39.20-alpha (#13829)
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2022-06-15 17:19:28 -07:00
Lake Mossman
fc5ba66e1c Restart workflow after activity failure instead of quarantining (#13779)
* use SHORT_ACTIVITY_OPTIONS on check connection activity so that it has retries

* retry workflow after delay instead of quarantining

* allow activity env vars to be configured in docker and kube

* add env var for workflow restart delay and refactor slightly

* update tests to handle new restart behavior

* update test name

* add empty env var values to .env files

* fail attempt before job in cleanJobState to prevent state machine failure

* change default value of max activity attempt retries from 10 to 5
2022-06-15 16:07:47 -07:00
Augustin
e8146e5ec2 Normalization: Upgrade MySQL to dbt 1.0.0 (#11470) 2022-06-15 15:05:49 -07:00
Octavia Squidington III
7d9f8c97c5 Bump Airbyte version from 0.39.18-alpha to 0.39.19-alpha (#13820)
Co-authored-by: edgao <edgao@users.noreply.github.com>
2022-06-15 13:14:50 -07:00
Edward Gao
61ce03a436 🐛 Normalization correctly propagates deletions to the final tables (#12846) 2022-06-14 14:56:18 -07:00
Octavia Squidington III
50f2a340d6 Bump Airbyte version from 0.39.17-alpha to 0.39.18-alpha (#13759)
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
Co-authored-by: Benoit Moriceau <benoit@airbyte.io>
2022-06-14 11:17:28 -07:00
Benoit Moriceau
aa3a1b4347 Fail the connection state instead of throwing an exception (#13728)
* Fail the connection state instead of throwin exception

* Fix test

* rm import

* Update airbyte-workers/src/main/java/io/airbyte/workers/general/DefaultCheckConnectionWorker.java

Co-authored-by: Evan Tahler <evan@airbyte.io>

Co-authored-by: Evan Tahler <evan@airbyte.io>
2022-06-13 13:26:04 -10:00
Lake Mossman
e7f8128111 Save streams to reset in job config when creating reset job (#13703)
* save streams to reset in job config when creating reset job

* change streamDescriptors to streamsToReset
2022-06-13 13:57:14 -07:00
Lake Mossman
7134625540 add ResetSourceConfiguration to JobResetConnectionConfig, and retriev… (#13696)
* add ResetSourceConfiguration to JobResetConnectionConfig, and retrieve in GenerateInputActivity

* add newline

* add comment explaining null check

* format

* more formatting
2022-06-10 15:17:18 -07:00
hugoJuhel
df3ebf3158 🎉 Destination mssql : adding ssh tunnel for normalisation (#11204) 2022-06-10 13:54:26 -07:00
Serhii Chvaliuk
2daaf5b4c3 Normalization - BigQuery use json_extract_string_array for array of simple types (#13289)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: andrii.leonets <aleonets@gmail.com>
Co-authored-by: Andrii Leonets <30464745+DoNotPanicUA@users.noreply.github.com>
2022-06-10 23:31:32 +03:00
Jonathan Pearlin
2b31011bce Separate platform and connector testcontainer versions (#13642)
* Separate platform and connector testcontainer versions

* Fix dependency

* Fix dependency

* Fix dependency usage

* Prevent leaking testcontainer dependencies
2022-06-10 09:34:31 -04:00
Octavia Squidington III
a9c5c4ec2e Bump Airbyte version from 0.39.16-alpha to 0.39.17-alpha (#13641)
Co-authored-by: pedroslopez <pedroslopez@users.noreply.github.com>
2022-06-09 11:37:26 -04:00
Octavia Squidington III
cd4f445d49 Bump Airbyte version from 0.39.15-alpha to 0.39.16-alpha (#13627)
Co-authored-by: xiaohansong <xiaohansong@users.noreply.github.com>
2022-06-08 16:43:45 -07:00
Lake Mossman
c53b46a932 properly handle correcting quarantined workflows when performing update (#13567) 2022-06-08 14:48:36 -07:00
Octavia Squidington III
35e65f444f Bump Airbyte version from 0.39.14-alpha to 0.39.15-alpha (#13611)
Co-authored-by: girarda <girarda@users.noreply.github.com>
2022-06-08 11:04:21 -07:00
Octavia Squidington III
8d57144670 Bump Airbyte version from 0.39.13-alpha to 0.39.14-alpha (#13578)
Co-authored-by: girarda <girarda@users.noreply.github.com>
2022-06-07 15:39:31 -07:00
Xiaohan Song
eea964f993 Create interface, factory for metric client (#13296)
* Create interface, factory for metric client

* remove unused func

* change count val to use long

* PR fix

* datadog statsd client name fix

* remove schedulerapp because it has been refactored out
2022-06-07 13:38:13 -07:00
Octavia Squidington III
53c95bfdaa Bump Airbyte version from 0.39.12-alpha to 0.39.13-alpha (#13513)
Co-authored-by: pmossman <pmossman@users.noreply.github.com>
2022-06-06 13:59:05 -07:00
Octavia Squidington III
574e9cf472 Bump Airbyte version from 0.39.11-alpha to 0.39.12-alpha (#13509)
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2022-06-06 12:32:08 -07:00
Octavia Squidington III
0aa21ddb87 Bump Airbyte version from 0.39.10-alpha to 0.39.11-alpha (#13490)
Co-authored-by: davinchia <davinchia@users.noreply.github.com>
2022-06-06 19:21:52 +08:00
Davin Chia
7788594e22 Start publishing proper artifacts. (#13484)
## What
Finale of https://github.com/airbytehq/airbyte/pull/13122.

We've renamed all directories in previous PRs. Here we remove the fat jar configuration and add publishing to all subprojects.

Explanation for what is happening:

Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.

## How
* Remove fat jar configuration.
* Add publishing to all subprojects.
2022-06-06 17:15:25 +08:00
Davin Chia
e9a94396e5 Fat Jar: Rename Dir Part 3 (#13480)
## What
Part 3 of https://github.com/airbytehq/airbyte/pull/13122.
Follow up to #13478  .

Explanation for what is happening:

Identically named subprojects have the following issues:

* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.

## How
Rename airbyte-scheduler:models to airbyte-scheduler:scheduler-models.
Rename airbyte-scheduler:persistence to airbyte-scheduler:scheduler-persistence.
2022-06-06 03:13:36 +08:00
Davin Chia
eb99f47746 Fat Jar: Rename Dir Part 2 (#13478)
## What
Part 2 of https://github.com/airbytehq/airbyte/pull/13122.

Follow up to #13476 .

Explanation for what is happening:

Identically named subprojects have the following issues:

* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.

* Rename airbyte-config:models to airbyte-config:config-models.
* Rename airbyte-config:persistence to airbyte-config:config-persistence.
2022-06-06 02:21:54 +08:00
Davin Chia
83a89aa843 Fat Jar: Rename Dir Part 1 (#13476)
Part 1 of #13122.

Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.

Explanation for what is happening:

Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
2022-06-06 00:35:43 +08:00
Octavia Squidington III
12f51b7d4f Bump Airbyte version from 0.39.9-alpha to 0.39.10-alpha (#13470)
Co-authored-by: alovew <alovew@users.noreply.github.com>
2022-06-03 17:13:19 -07:00
Anne
d2921925b5 Check for invalidFields size before accessing (#13457)
* Check for invalidFields size before accessing
2022-06-03 11:56:29 -07:00
Octavia Squidington III
ac01bd383d Bump Airbyte version from 0.39.8-alpha to 0.39.9-alpha (#13454)
Co-authored-by: girarda <girarda@users.noreply.github.com>
2022-06-03 10:14:28 -07:00
Octavia Squidington III
39de3318e5 Bump Airbyte version from 0.39.7-alpha to 0.39.8-alpha (#13402)
* Bump Airbyte version from 0.39.7-alpha to 0.39.8-alpha

* update date

* format

Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
2022-06-02 06:26:44 -07:00
Octavia Squidington III
8e173e6e96 Bump Airbyte version from 0.39.6-alpha to 0.39.7-alpha (#13358)
Co-authored-by: alovew <alovew@users.noreply.github.com>
2022-05-31 18:20:06 -07:00
Lake Mossman
fcbffe0495 Implement job state machine validations (#13213)
* add job state machine validations to persistence and tests

* update state machine diagram

* format

* remove terminal state transitions

* fail attempt before cancelling job, to respect job state machine

* update job status state machine diagram

* verify order of db calls
2022-05-31 17:21:20 -07:00
Anne
5f0f106cb6 Limit the number of record schema validations performed (#13351)
* Better error messages for record schema validations, and validate a maximum of 10 records per stream
2022-05-31 16:58:09 -07:00
Octavia Squidington III
417bdc1cb2 Bump Airbyte version from 0.39.5-alpha to 0.39.6-alpha (#13350)
Co-authored-by: git-phu <git-phu@users.noreply.github.com>
2022-05-31 13:44:46 -07:00
Octavia Squidington III
3d9a972d28 Bump Airbyte version from 0.39.4-alpha to 0.39.5-alpha (#13288)
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
2022-05-27 14:34:55 -04:00
Octavia Squidington III
85eb7483d1 Bump Airbyte version from 0.39.3-alpha to 0.39.4-alpha (#13279)
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
2022-05-27 11:25:07 -04:00
Jonathan Pearlin
e06a9de60f Use Database Availability/Initialization Check (#13178)
* Use isolated database initialization logic

* Address PMD warnings

* Use test provider where possible

* Initialize database on bootloader load

* Combine availability and migration checks

* Ensure env vars are set

* Fix typo

* Avoid duplicate literals

* Add log message

* Use correct data source

* Revert change

* Update copyright

* Remove redundant exception catch/throw
2022-05-27 09:47:33 -04:00
LiRen Tu
3dcda7ae52 Use cheaper operation to estimate json data byte size (#13240)
* Simplify byte size estimation

* Format code

* Update comment
2022-05-26 19:42:54 -07:00
Benoit Moriceau
df782f79b4 Fix flaky test (#13244) 2022-05-26 16:15:45 -07:00
Octavia Squidington III
296b5c588a Bump Airbyte version from 0.39.2-alpha to 0.39.3-alpha (#13239)
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
2022-05-26 16:48:58 -04:00
Jonathan Pearlin
880c759cac Use "generated" in generated code package names (#13183) 2022-05-26 11:11:23 -04:00
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Octavia Squidington III
ddf529cb78 Bump Airbyte version from 0.39.1-alpha to 0.39.2-alpha (#13188)
Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
2022-05-25 18:48:09 -04:00