1
0
mirror of synced 2026-01-10 00:03:08 -05:00
Commit Graph

863 Commits

Author SHA1 Message Date
Benoit Moriceau
a586537adb Specify namespace when creating pod (#19399)
* Specify namespace when creating pof

* PR comments

* rm new line

* Fix micronaut injection
2022-11-15 15:52:42 -08:00
Benoit Moriceau
e8368696ff Make sure that long running activity are not retried (#19452) 2022-11-15 14:35:18 -08:00
Benoit Moriceau
63ccc32788 Bmoric/remove unused code (#19188)
* Tmp

* Move when the deletion is performed

* Re-enable disable test

* PR comments

* Use cancel

* rename

* Fix test and version check position

* remove unused temporal deletion code

* Remove false todo

* Rm repeated test

* Rm unused import
2022-11-15 14:21:05 -08:00
Anne
fdb96d0a63 Notifications Workflow (#18735)
* notification workflow
2022-11-15 13:58:30 -08:00
Jonathan Pearlin
6b92151dc9 Filter out Temporal errors from SyncWorkflowImpl (#19293) 2022-11-10 13:40:57 -05:00
Xiaohan Song
bde7fe9616 Metric fixes on multi cloud (#19268)
* fix numpending secs

* fix worker
2022-11-10 09:05:35 -08:00
Jonathan Pearlin
0e036dbb24 Add attempt number to trace tags (#19241)
* Add attempt number to trace tags

* Use correct tag key for error suppression
2022-11-09 16:27:53 -05:00
Benoit Moriceau
b3db914265 Change where a connection is deleted (#19096)
* Tmp

* Move when the deletion is performed

* Re-enable disable test

* PR comments

* Use cancel

* rename

* Fix test and version check position

* Log exception
2022-11-09 13:05:27 -08:00
Jonathan Pearlin
c8a7cb333b Handle null JobSyncConfig (#18969)
* Handle null JobSyncConfig

* Add unit test

* Fix PMD warning

* Do not update jobs in terminal state

* Fix failing tests

* Fix compile error
2022-11-04 12:51:29 -04:00
Jonathan Pearlin
418d0ca7c8 Update ReplicationActivityImpl.java (#18919) 2022-11-03 12:25:33 -04:00
Jonathan Pearlin
fad7ff2fdb Record replication stats as part of trace (#18909)
* Record replication stats as part of trace

* PR feedback
2022-11-03 12:02:04 -04:00
Jonathan Pearlin
fcf47584e8 Filter DestroyWorkflowThreadError from trace (#18907) 2022-11-03 11:50:17 -04:00
Jonathan Pearlin
87d79f7f36 Filter exit errors by operation name (#18850) 2022-11-02 12:51:22 -04:00
Jonathan Pearlin
0cd09f9dce Avoid NPE when adding connection ID to trace (#18856) 2022-11-02 12:49:18 -04:00
Jonathan Pearlin
7c2c701098 Add connection ID to span (#18852) 2022-11-02 11:58:11 -04:00
Octavia Squidington III
b299688f3b Bump Airbyte version from 0.40.17 to 0.40.18 (#18827)
Co-authored-by: grishick <grishick@users.noreply.github.com>
2022-11-01 19:35:23 -07:00
Charles
87a0a2a27c hide ConfigPersistence inside ConfigRepository to discourage use (#18803) 2022-11-01 16:51:09 -07:00
Charles
20ab047ae0 Remove the bulk actions from ConfigPersistence (#18800) 2022-11-01 16:50:03 -07:00
Jonathan Pearlin
b14bd05e91 Use equalsIgnoreCase (#18810) 2022-11-01 17:04:54 -04:00
Jonathan Pearlin
354db219da Filter Temporal exit errors from traces (#18777)
* Filter Temporal exit errors from traces

* Remove unnecessary PMD suppression

* Formatting
2022-11-01 12:31:52 -04:00
Davin Chia
d0521e7f33 Simplify the job persistence method. (#18698)
Method interface:
- has generic type that isn't needed. Confusing because no generic is happening here.
- has two existing parameters that are data fields of one of the already existing parameters. Confusing since the two additional parameters aren't actually supposed to be passed in separately from the existing parameter.

Remove both.
2022-10-31 08:40:23 -07:00
Benoit Moriceau
85e6792e36 More comprehensive temporal error message (#18608)
* To be remove

* Remove the signal for waiting after a failed activity and ensure we are waiting the expected time

* Revert "To be remove"

This reverts commit 3a5f7b4f72.

* Remove unused and move failure reason to the helper

* Avoid repetitive new Set()
2022-10-28 15:18:23 -07:00
Jonathan Pearlin
e3aadbf47e Remove workflow version check (#18613) 2022-10-28 14:11:48 -04:00
Jimmy Ma
e32a641cbe Use Protocol Versions in Syncs (#18324)
* Pass protocol version into IntegrationLauncherConfig

* Use VersionedStreamStreamFactory in AirbyteSource/Destination

* Add AirbyteMessageBufferedWriter

* Use VersionedBufferedWriter
2022-10-27 12:57:33 -07:00
Jonathan Pearlin
d2911e5ec0 Ensure that restart delay is set before any other activity is executed (#18547)
* Ensure that restart delay is set before any other activity is executed

* Add version check

* Formatting
2022-10-27 12:50:27 -04:00
Jonathan Pearlin
657f997aae Update to latest Temporal SDK release (#18492) 2022-10-26 12:54:00 -04:00
Anne
5361f14322 RefreshSchema method on RefreshSchema activity (#18460)
* Add refreshSchema method to RefreshSchema activity
2022-10-25 19:47:37 -07:00
Anne
30a8d1767e RefreshSchema Activity (#18388)
* Add RefreshSchema Activity
2022-10-25 14:56:06 -07:00
Xiaohan Song
21431f6297 use api to do jobpersistence query (#18308)
* use api to do jobpersistence query

* renaming some variables

* fix test
2022-10-25 14:27:42 -07:00
Benoit Moriceau
434a556dd8 Fix the tag in openAPI (#18445)
* Fix the tag

* remove unused
2022-10-25 14:23:47 -07:00
Jonathan Pearlin
ef08d88455 Trace errors in orchestrator workers (#18381)
* Trace errors in orchestrator workers

* Trace cancel method of workers

* Formatting

* Restore final modifier
2022-10-24 14:36:22 -04:00
Octavia Squidington III
73336ff4de Bump Airbyte version from 0.40.16 to 0.40.17 (#18316)
Co-authored-by: edgao <edgao@users.noreply.github.com>
2022-10-21 14:00:39 -07:00
Jimmy Ma
f71fe5ac15 Add message translation to GetSpec (#18130)
* Update SpecActivityImpl to build a VersionedStreamFactory

* Enable Protocol Version detection from Stream for SPEC

* Print log before the action for better debugging

* Fix buffer size for protocol detection

* Improve detectVersion error handling

* extract constan

* Rename attribute for clarity
2022-10-21 10:27:34 -07:00
Edward Gao
b347829b85 upgrade commons-text to 1.10.0 (#18275)
* test

* fix

* fix other modules

* version bumps + changelog

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-21 09:42:34 -07:00
Jimmy Ma
cf28f7a01f Add message migration to discover schema (#18205) 2022-10-21 09:32:42 -07:00
Parker Mossman
4e236b5b42 Add Geography support to RouterService (#17902)
* router service uses geography in database instead of env var

move geography map to a helper that can be overridden with a separate implementation in cloud

format

pmd

fix import

move geography mapper interface to airbyte-commons-temporal

* add DefaultGeographyMapper back in

* remove all args constructor and extranneous import

* rename GeographyMapper to TaskQueueMapper
2022-10-20 14:48:27 -07:00
Octavia Squidington III
ce0bbf3f75 Bump Airbyte version from 0.40.15 to 0.40.16 (#18268)
Co-authored-by: evantahler <evantahler@users.noreply.github.com>
2022-10-20 13:29:54 -07:00
Jonathan Pearlin
48a6c56a2a Add APM tracing to Container Orchestrator (#18252)
* Add env vars to support APM

* Formatting

* Fix property name

* Add custom tracing to orchestrator

* Use correct constant

* PR feedback

* Formatting

* Add custom trace to dbt job orchestrator
2022-10-20 15:38:57 -04:00
andriikorotkov
e62c3c3cb7 Skip CHECKs steps if previous sync success (#17999)
* Disabled CHECKs for source and destination, when previous sync or previous attempt is failure

* added tests and added minor changes to the isLastJobOrAttemptFailure method

* updated remarks

* updated code style

* updated code style

* fixed remarks

* fixed remarks
2022-10-20 12:26:15 +03:00
Cole Snodgrass
d75604adb4 add a span interceptor to treat expected errors as non-errors (#18192)
* wip; tracer interceptor

* fix span check

* update logs for testing

* more debugging

* more debugging

* more debugging

* more debugging

* remove debugging; add test

* address one pmd error and ignore another
2022-10-19 17:19:42 -07:00
Xiaohan Song
6b1c5ee384 save queue name into Attempts table (#17921)
* save queue name

* make input nullable because we changed signature

* PR Comments fix
2022-10-19 17:10:37 -07:00
Michael Siega
b18f95021e Fix minor DBT Cloud Errors. (#18147)
Fixes URL formatting for dbt cloud integration. Note, without a trailing / the dbt Cloud API returns a 308.

Some other small fixes:
- Retries around the dbt Cloud invocation
- Updates the regex to handle the new URL format.
2022-10-18 19:52:47 -07:00
Benoit Moriceau
adf2c5ff8e Bmoric/remove dep connector worker (#17977)
* test [ci skip]

* Autogenerated files

* Add missing annotation

* Remove unused json2Schema block from worker

* Move tess

* Missing deps and format

* Fix test build

* TMP

* Add missing dependencies

* PR comments

* Tmp

* [ci skip] Tmp

* Fix acceptance test and add the seed dependency

* Fix build

* For diff

* tmp

* Build pass

* make the worker to be  on the platform only

* fix setting.yaml

* remove dependency

* Fix pmd

* Fix Cron

* Add chart

* Fix cron

* Fix server build.gradle

* Fix jar conflict

* PR comments

* Add cron micronaut environemnt

* Update required build path

* remove all dependencies on worker

* Typo

* Add gradle exclusions

* Add exclusion to avoid conflict

* more exclusion

* Move more class

* Exclude worker from the general build

* Fix connector build
2022-10-18 11:51:25 -07:00
Jonathan Pearlin
b4c16bfdbb Custom APM Tracing (#17947)
* Add APM tracing to airbyte-workers

* Add custom traces to airbyte-cron

* Formatting

* Refactor tag constants

* Handle potential null object

* Handle null tag values
2022-10-18 13:14:29 -04:00
Michael Siega
71967885e4 Implement webhook operation in the sync workflow (#18022)
Implements the webhook operation as part of the sync workflow.

- Introduces the new activity implementation
- Updates the various interfaces that pass input to get the relevant configs to the sync workflow
- Hooks the new activity into the sync workflow
- Passes the webhook configs along into the sync workflow job
2022-10-17 18:36:53 -07:00
Octavia Squidington III
a5a0409ad5 Bump Airbyte version from 0.40.14 to 0.40.15 (#17970)
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
2022-10-17 13:33:20 -07:00
Benoit Moriceau
7a71c55c1f Bmoric/remove dep server worker (#17894)
* test [ci skip]

* Autogenerated files

* Add missing annotation

* Remove unused json2Schema block from worker

* Move tess

* Missing deps and format

* Fix test build

* TMP

* Add missing dependencies

* PR comments

* Tmp

* [ci skip] Tmp

* Fix acceptance test and add the seed dependency

* Fix build

* For diff

* tmp

* Build pass

* make the worker to be  on the platform only

* fix setting.yaml

* Fix pmd

* Fix Cron

* Add chart

* Fix cron

* Fix server build.gradle

* Fix jar conflict

* PR comments

* Add cron micronaut environemnt
2022-10-17 12:35:04 -07:00
Evan Tahler
f338e47c2f Add additional sync timing information (#17643)
* WIP - Add additional sync timing information

* Fixup tests

* fix PMD problem

* send data to segment

* Test JobTracker

* respond to PR suggestions

* fixup test

* formatting

* fix initializer for stats

* Make thread-safe with synchronized

* Don't clobber syncStats on init

* add comments and fix init

* Do what Pedro says

* Extract timeTracker pojo
2022-10-13 16:16:06 -07:00
Nataly Merezhuk
97912d53fc 🐛 Correct kube annotations variable as per the docs. (#17972)
* Correct kube annotations variable as per the docs.

* Requested changes.
2022-10-13 16:07:11 -04:00
Jimmy Ma
f267746a18 Use MessageMigration for Source Connection Check. (#17656)
* More AirbyteVersion references fix

* Propagate protocol version from sourceDef to SchedulerClient

* Propagate protocol version to LauncherConfig

* Add VersionedMigratorFactory

* Update VersionedAirbyteStreamFactory

* Fix Version Json serialization/deserialization

* Plug message migration in CheckConnection for Sources
2022-10-13 08:11:31 -07:00