1
0
mirror of synced 2026-01-03 06:02:23 -05:00
Commit Graph

155 Commits

Author SHA1 Message Date
Edward Gao
934acaa137 Destination bigquery: rerelease 1s1t behind gate (#27936)
* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)""

This reverts commit 348c577dbb.

* version bumps+changelog

* Speed up BQ by having 2 queries, and not an OR (#27981)

* 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (#27856)

* only run t+d code if it's enabled

* dockerfile+changelog

* remove changelog entry

* Destinations V2: handle optional fields for `object` and `array` types (#27898)

* catch null schema

* fix null properties

* clean up

* consolidate + add more tests

* try catch

* empty json test

* Automated Commit - Formatting Changes

* remove todo

* destination bigquery: misc updates to 1s1t code (#28057)

* switch to checkedconsumer

* add unit test for buildColumnId

* use flag

* restructure prefix check

* fix build

* more type-parsing fixes (#28100)

* more type-parsing fixes

* handle duplicates

* Automated Commit - Format and Process Resources Changes

* add tests for asColumns

* Automated Commit - Format and Process Resources Changes

* log warnings instead of throwing exception

* better log message

* error level

---------

Co-authored-by: edgao <edgao@users.noreply.github.com>

* Automated Commit - Formatting Changes

* Improve protocol type parsing (#28126)

* Automated Commit - Formatting Changes

* Change from T&D every 10k records to an increasing time based interval (#28130)

* fifteen minute t&d

* add typing and deduping operation valve for increased intervals of typing and deduping

* Automated Commit - Format and Process Resources Changes

* resolve bizarre merge conflict

* Automated Commit - Format and Process Resources Changes

---------

Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>

* Simplify and speed up CDC delete support [DestinationsV2] (#28029)

* Simplify and speed up CDC delete support [DestinationsV2]

* better QUOTE

* spotbugs?

* recompile dbt image for local arch and use that when building images

* things compile, but tests fail

* tests working-ish

* comment

* fix logic to re-insert deleted records for cursor comparison.

tests pass!

* remove comment

* Skip CDC re-include logic if there are no CDC columns

* stop hardcoding pk (#28092)

* wip

* remove TODOs

---------

Co-authored-by: Edward Gao <edward.gao@airbyte.io>

* update method name

* Automated Commit - Formatting Changes

* depend on pinned normalization version

* implement 1s1t DATs for destination-bigquery (#27852)

* intiial implementation

* Automated Commit - Formatting Changes

* add second sync to test

* do concurrent things

* Automated Commit - Formatting Changes

* clarify comment

* minor tweaks

* more stuff

* Automated Commit - Formatting Changes

* minor cleanup

* lots of fixes

* handle sql vs json null better
* verify extra columns
* only check deleted_at if in DEDUP mode and the column exists
* add full refresh append test case

* Automated Commit - Formatting Changes

* add tests for the remaining sync modes

* Automated Commit - Formatting Changes

* readability stuff

* Automated Commit - Formatting Changes

* add test for gcs mode

* remove static fields

* Automated Commit - Formatting Changes

* add more test cases, tweak test scaffold

* cleanup

* Automated Commit - Formatting Changes

* extract recorddiffer

* and use it in the sql generator test

* fix

* comment

* naming+comment

* one more comment

* better assert

* remove unnecessary thing

* one last thing

* Automated Commit - Formatting Changes

* enable concurrent execution on all java integration tests

* add test for default namespace

* Automated Commit - Formatting Changes

* implement a 2-stream test

* Automated Commit - Formatting Changes

* extract methods

* invert jsonNodesNotEquivalent

* Automated Commit - Formatting Changes

* fix conditional

* pull out diffSingleRecord

* Automated Commit - Formatting Changes

* handle nulls correctly

* remove raw-specific handling; break up methods

* Automated Commit - Formatting Changes

---------

Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>

* Destinations V2: move create raw tables earlier (#28255)

* move create raw tables

* better log message

* stop building normalization (#28256)

* fix ability to run tests

* disable incremental t+d for now

* Automated Commit - Formatting Changes

---------

Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Cynthia Yin <cynthia@airbyte.io>
Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>
2023-07-14 09:34:56 -05:00
Edward Gao
52b8cbe39d Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)" (#27891)
* Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)"

This reverts commit ba3e39bb0c.

* bump versions to 1.5.1 everywhere
2023-06-30 20:26:48 -04:00
Edward Gao
ba3e39bb0c Destination Bigquery: Scaffolding for destinations v2 (#27268)
* copy files from edgao branch

* start writing create table statement

* add basic unit test setup

* create a table, probably

* remove outdated todo

* derp, one more column

* ugh

* add partitioning+clustering

* use StringSubstitutor

* substitutions in updateTable

* wip generate update/insert statement

* split up into smaller methods

* handle json types correctly

* rename stuff

* more json_query vs _value stuff

* minor tweak

* super basic test setup

* laying foundation for type parsing

* more stuff

* tweaks

* more progress on type parsing

* fix json_value stuff?

* misc fixes in insert

* fix dedupFinalTable

* add testDedupRaw

* full e2e test

* type parsing: gave up and mirrored the dbt code structure to avoid bugs

* type parsing - more cleanup

* handle column name collisions

* handle tablename collisions...?

* comments

* remove original ns/name from quotedstream

* also javadoc

* remove redundant method

* fix table rename

* add incremental append test

* add full refresh append test

* comment

* call T+D sql in a reasonable location for standard inserts

* add config option

* use config option here

* type parsing - fix fromJsonSchema

* gate everything

* log query + runtime

* add spec option temporarily

* Raw Table Updates

* fix more stuff

* first big pass at toDialectType

* no quotes

* wrap everything in quotes

* resolve some TODOs

* log sql statement in tests

* overwriteFinalTable returns optional

* minor clean up

* add raw dataset override

* try to preserve the original namespace for t+d?

* write to the raw table correctly

* update todos

* write directly to raw table

this is kind of dumb because we're still trying to do tmp table operations,
and we still don't ack state until the end of the entire sync.

* standard inserts write to raw table correctly

* imports + log statements

* move logs + add comment

* explicitly create raw table

* move comment to better place

* Typing issues

* bash attempt

* formatting updates

* formatting updates

* write to the airbyte schema by default unless overriden by config options

* standard inserts truncate raw table at start of sync

* full refresh overwrite will overwrite correctly!

* fix avro record schema parsing

* better raw table recreate

* rename raw table to match standard inserts

* full refresh overwrite does tmp table things

* small clean up

* small clean up

* remove errors entry if no errors

* pull out destination config into singleton

* clean up singleton stuff

* make sure dest config exists when trying to do lookups

* avoid stringifying null

* quick thoughts on alter table

* add basic cdc testcase

* tweak cdc test setup

* rename raw table to match standard inserts

* minor tweak

* delete exact sql string assertions

* switch to JSON type

* minor cleanup

* sql whitespace changes

* explain cdc deletions

* GCS Staging Full Refresh create temp table

* assert schema

* first out of order cdc test

* add another cdc test case (currently failing)

* better test structure

* make this work

* oops, fix test

* stop trying to delete deletion records

* minor improvements to code+test

* enable concurrent test runs on integration test

* move stuff to static initializer

* extract utility method

* formatting

* Move conditional to the base java package, replace conditionals which did not use the typing and deduping flag but should have been.

* 🤖 Auto format destination-bigquery code [skip ci]

* 🤖 Auto format destination-gcs code [skip ci]

* switch back to empty list; write big assert

* minor wording tweaks

* 🤖 Auto format destination-bigquery code [skip ci]

* 🤖 Auto format destination-gcs code [skip ci]

* DestinationConfigTest

* 🤖 Auto format destination-bigquery code [skip ci]

* 🤖 Auto format destination-gcs code [skip ci]

* formatting

* remove ParsedType

* 🤖 Auto format destination-gcs code [skip ci]

* 🤖 Auto format destination-bigquery code [skip ci]

* tests verify every data type

* 🤖 Auto format destination-bigquery code [skip ci]

* 🤖 Auto format destination-gcs code [skip ci]

* full update with all data types

* 🤖 Auto format destination-bigquery code [skip ci]

* 🤖 Auto format destination-gcs code [skip ci]

* move stuff to new base lib

* 🤖 Auto format destination-gcs code [skip ci]

* Automated Commit - Formatting Changes

* 🤖 Auto format destination-bigquery code [skip ci]

* fix test

* 🤖 Auto format destination-bigquery code [skip ci]

* 🤖 Auto format destination-bigquery code [skip ci]

* 🤖 Auto format destination-gcs code [skip ci]

* asserts in dedupFinalTable

* better asserts in dedupRawTable

* [wip] test case for all data types

* 🤖 Auto format destination-gcs code [skip ci]

* 🤖 Auto format destination-bigquery code [skip ci]

* AirbyteTypeTest

* Automated Commit - Formatting Changes

* remove comments

* test chooseOneOf

* slightly better test output

* Automated Commit - Formatting Changes

* add some awful pretty print code

* more comment

* minor tweaks

* verify array/object type

* fix test

* handle deletions more correctly

* test toDialectType

* Destinations v2: better namespace handling (#27682)

* [wip] better namespace handling

* 🤖 Auto format destination-bigquery code [skip ci]

* wip also implement in gcs

* get gcs working (?)

* 🤖 Auto format destination-bigquery code [skip ci]

* remove duplicate method

* 🤖 Auto format destination-bigquery code [skip ci]

* fixed my code style settings

* make ci happy?

* 🤖 Auto format destination-bigquery code [skip ci]

* make ci happy?

* remove incorrect test

* blank line change

* initialize singleton

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>

* reset args correctly

* Automated Commit - Formatting Changes

* more bash stuff

* parse implicit structs

* initialize singleton in more tests

* Automated Commit - Formatting Changes

* I missed this namespace handling thing

* test more schemas

* fix singular types specified in arrays

* Automated Commit - Formatting Changes

* disable test for unimplemented feature

* initialize singleton

* remove spec options; changelogs+metadata

* randomize namespace

* also bump dockerfile

* unremove namespace sanitizing in legacy mode

* ... disable the correct test

* even more unit test fixes!

* move integration test to integration tests

---------

Co-authored-by: Cynthia Yin <cynthia@airbyte.io>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com>
2023-06-29 08:44:37 -07:00
Ben Church
dcec1b236b Registry: Refactor and Remove registry download from git. (#26217) 2023-05-19 09:05:09 -07:00
Rodi Reich Zilberman
6374f4780f destination harness (#25030)
* destination harness

* Automated Commit - Formatting Changes

* refactor to support multiple connectors

* better byte counting

* Automated Change

* fix main cp

* harness type

* harness type

* harness type

* harness type

* harness type

* harness type

* harness type

* speed up harness build

* harness type

* harness type

* harness type

* harness type

* 1m catalog

* 1m catalog

* Automated Commit - Formatting Changes

* 1m catalog

* Automated Commit - Formatting Changes

* 1m catalog

* Automated Commit - Formatting Changes

* 1m catalog

* 1m catalog

* Automated Commit - Formatting Changes

* 1m catalog

* Automated Commit - Formatting Changes

* Automated Change

* add some additional logging for debugging

* test

* test

* debugging code

* Automated Commit - Formatting Changes

* debugging code

* debugging code

* debugging code

* Automated Commit - Formatting Changes

* Checkpoint: This is working!.

* Automated Commit - Formatting Changes

* Updates plumbing for Snowflake with static records - Working

* Automated Commit - Formatting Changes

* working 10m

* Automated Commit - Formatting Changes

* add destination datasource

* Automated Commit - Formatting Changes

* sanity

* sanity

* Delete 1m_catalog.json

* sanity

* Update airbyte-integrations/connectors-performance/destination-harness/README.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* sanity

* Update airbyte-integrations/connectors/destination-e2e-test/src/main/java/io/airbyte/integrations/destination/e2e_test/TestingDestinations.java

Co-authored-by: Davin Chia <davinchia@gmail.com>

* Splits performance harness into separated methods

* Automated Commit - Formatting Changes

* Updates test description to force Github builds

---------

Co-authored-by: rodireich <rodireich@users.noreply.github.com>
Co-authored-by: Jeff Cowan (Airbyte) <4992320+jcowanpdx@users.noreply.github.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: davinchia <davinchia@users.noreply.github.com>
Co-authored-by: ryankfu <ryan.fu@airbyte.io>
Co-authored-by: ryankfu <ryankfu@users.noreply.github.com>
2023-05-16 06:37:00 +00:00
Ryan Fu
95dd113a5f Revert "Splits bases and updates build.gradle files (#25649)" (#26081)
* Revert "Splits bases and updates build.gradle files (#25649)"

This reverts commit c673b0a692.

* Bumps branch to prevent a conflict with publishing

* Forward fixes Snowflake to use singular base-java and develop within a new package within the same module

* Forcing automated change to merge changes
2023-05-15 21:56:40 +00:00
Ryan Fu
c673b0a692 Splits bases and updates build.gradle files (#25649)
* Splits bases and updates build.gradle files

* Fixed changelog out of sync

* Bumps version number and metadata files

* auto-bump connector version

* Downgraded untouched connector bumps

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-05-06 00:24:40 +00:00
Ryan Fu
0dcc653bd1 Shuffled destination-jdbc into bases folder (#25120)
* Testing changes for reshuffle

* Resolves dependency/race-condition with base classes and docker creation

* Removes dockerfile

* Updates java connectors gradle dependencies
2023-04-17 12:29:03 -07:00
Rodi Reich Zilberman
0bab1756b8 Rename airbyte-config module (#24885)
* rename airbyte-config module

* Automated Commit - Formatting Changes

* sanity

* update import

* update import

* update script

* update script

* update script

* update script

* Automated Change

* Automated Change

* Automated Change

* Automated Change

* update awsdatalake icon

* point slash commands to new path

* sanity

* Automated Commit - Formatting Changes

* sanity

* Automated Change

* Automated Change

* sanity

---------

Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-04-06 10:47:30 -07:00
Rodi Reich Zilberman
0a38d4e1ca Source connector performance harness (#23786)
* remove arguments

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* Source Performance harness (#23922)

* test

* initial commit

* Working dev

* src-platform-dst

* Integrate secrets

* Update .dockerignore

* Update build.gradle

* Update base.sh

* Build harness module and use it to measure

* Build harness module and use it to measure

* Build harness module and use it to measure

* Automated Change

* Integrate secrets

* Integrate secrets

* Integrate secrets

* Integrate secrets

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* Automated Change

* test

* Automated Change

* better error handling

* temp

* Automated Change

* use connector image name

* use connector image name

* use connector image name

* use connector image name

* use connector image name

* use connector image name

* use connector image name

* use connector image name

* add test name argument

* add test name argument

* add test name argument

* add test name argument

* add test name argument

* add test name argument

* Automated Change

* add bottleneck_stream1 dataset

* Automated Change

* fix dependency

* fix dependency

* test downgrade of commons to 41.1

* Automated Change

* test downgrade of commons to 41.0

* test

* test

* test

* test

* test

* exit condition

* exit condition

* exit condition

* test

* test

* test

* test

* test

* test

* sanity

* readme

* change command name

* update gradle settings

* sanity

* architecture aware docker build

* sanity

* put run harness step in a script

* sanity

* sanity

---------

Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-03-30 10:26:31 -07:00
Subodh Kant Chaturvedi
f6ec8769a3 DB sources cdc : upgrade Debezium version to 2.1.2 (#23112)
* WIP

* remove wal2json

* revert test change

* working version for MySQL

* cleanup

* mssql changes

* rename module

* format

* undo unwanted change

* disable backward compatibility spec test

* fix acceptance-test-config.yml

* fix acceptance-test-config.yml

* update doc link

* fix mssql

* review comments

* master merge

* review comments

* disable few mssql tests

* temp commit for mssql tests experiment

* another temp commit

* another temp commit

* revert temp commits

* update test

* ssl support added for mssql cdc

* improve tests

* fix strict encrypt test

* make sql-server state backward compatible

* missed this one

* add error properties

* upgrade version and docs

* remove wal2json reference from docs

* auto-bump connector version

* add extra safety validation

* increase wait time

* auto-bump connector version

* update spec

---------

Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-07 18:12:07 +05:30
Maxime Carbonneau-Leclerc
bb753f950b Removing unused gradle task for common build (#23627) 2023-03-01 12:08:02 -05:00
Maxime Carbonneau-Leclerc
ee1ae2d4a2 Isolate our build so that the feedback loop is faster (#23411)
* Isolate our build so that the feedback loop is faster

* Improve gradle build condition

* Update documentation

* Adding auto-commit of formatting changes (as https://github.com/airbytehq/airbyte/pull/23415)
2023-03-01 10:05:50 -05:00
Charles
f83ef9eea7 Remove workers (#23422) 2023-02-24 17:45:44 -08:00
Conor
f82d3e4e8b Remove featureflag and notification modules (#23365)
* remote airbyte-featureflag

* remove airbyte-notifications
2023-02-22 21:19:54 -06:00
Conor
bdf493fcce Remove airbyte persistence, analytics, and metrics (#23263)
* remove airbyte-metrics

* format

* fix pmd failures

* remove airbyte-persistence

* remove airbyte-analytics

* format
2023-02-20 22:05:51 -06:00
Conor
e37b721fdc remove airbyte-oauth (#23258)
* remove airbyte-oauth

* remove oauth usages

* delete the oauth folder
2023-02-20 13:55:10 -06:00
Conor
f1bab07d25 remove airbyte-commons-temporal/worker-models (#23239)
* remove airbyte-commons-temporal

* remove airbyte-worker-models
2023-02-20 08:29:48 -06:00
Conor
7953c45a09 delete platform code and workflows (#23214)
* delete platform code and workflows

* delete more platform workflows

* sunset kustomize (#23187)

* sunset kustomize

* remove build steps that use kustomize

* remove k8s kustomize gke-kube-acceptance-tests

* delete docker-compose files

* remove platform from check_images_exist.sh

* Working replacement script for docker

* rename start script and add download files to gitignore

* add args and download only option

* integration test downloads docker compose files

* remove helm charts

* remove more platform files

* add flags.yml

* use version from gradle.properties instead of .env

* Working replacement script for docker (#23227)

* Working replacement script for docker

* Working debugging line

* rename start script and add download files to gitignore

* add args and download only option

* Made flags cleaner and more cleanup

i've never used this flag pattern, its much
easier than my usual way

* warn about a sharp edge

* Don't run when we don't know a command

* better help spacing

* add flags.yml

---------

Co-authored-by: cpdeethree <conor@airbyte.io>
Co-authored-by: Conor <cpdeethree@users.noreply.github.com>

* remove more-secrets env

---------

Co-authored-by: Charles <charles@airbyte.io>
Co-authored-by: Topher Lubaway <asimplechris@gmail.com>
2023-02-19 09:42:08 -06:00
Conor
903d95d2ef Download latest oss connector catalog on build and use as reference (#22376)
* download latest connector catalog on build

* format

* Add small todos

* Add skeleton for local provider change

* Load catalog from local

* Remove seed class from localdefprovider

* Get catalog path from config

* refactor arguments out of download catalog

* Move to a constant file method

* Fix arguments

* Rename to CatalogDefinitionsConfig

* Add todos

* Update todo

* Apply formatting

* remove unnessessary try catch

* Refactor IconValidationTask

* Fix null value issue

* Only build oss catalog via github action

* Add LOCAL_CONNECTOR_CATALOG_PATH env var override

* Format

* Add logger

* Copy file directly to avoid circular dep

* Format

* Ensure run before tests

* Move airbyte:init:spec to general dependency

* LOCAL_CATALOG_PATH -> LOCAL_CONNECTOR_CATALOG_PATH

---------

Co-authored-by: Ben Church <ben@airbyte.io>
2023-02-16 16:09:59 -06:00
Peter Hu
1fe811d272 remove airbyte protocol (#22519)
* remove airbyte-protocol

moved to a separate repository

* delete references to airbyte-protocol dir

* ensure generateWellKnownTypes task works

extracct well_known_types.yaml from airbyte-protocol jar

* update url to new airbyte-protocol repo

* add back treenode

* use published jar for commons-converters

* format

* resolve bad merge conflict

* resolve bad merge conflict

---------

Co-authored-by: cpdeethree <conor@airbyte.io>
Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
2023-02-16 11:05:54 -06:00
Davin Chia
5578c8fd8f Revert "Use Micronaut with airbyte-server-wrapped (#22669)" (#23109)
This reverts commit d8ae36fe54 as this wasn't passing Airbyte Prod tess.
2023-02-16 07:45:20 -08:00
Jonathan Pearlin
d8ae36fe54 Use Micronaut with airbyte-server-wrapped (#22669)
Support conversion of the airbyte-server-wrapped Cloud module to Micronaut

- Remove @Requires rule from controllers so they will be used in both OSS and Cloud
- Remove airbyte-commons-worker dependency from airbyte-server by moving shared code to new airbyte-commons-converters module

Co-authored-by: Davin Chia <davinchia@gmail.com>
2023-02-14 13:21:45 -08:00
Simon Späti
2bbc4f6f83 🎉 New Destination: DuckDB (#17494)
This is the first version of the DuckDB destination. There are potential edge cases that still need to be taken care of. But looking forward to your feedback.
2023-02-07 11:33:10 +01:00
Cole Snodgrass
f09828779f Revert "Revert "remove airbyte-commons-docker (#21756)" (#22237)" (#22239)
This reverts commit 6d65070008.
2023-02-06 11:55:51 -08:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846)
Rename source acceptance test to connector acceptance test
2023-02-02 11:38:19 +01:00
Cole Snodgrass
6d65070008 Revert "remove airbyte-commons-docker (#21756)" (#22237)
This reverts commit 9c4100b6ba.
2023-02-01 12:43:05 -08:00
Cole Snodgrass
9c4100b6ba remove airbyte-commons-docker (#21756)
* remove references to airbyte-commons-docker

* remove commons-docker

* update missed DockerUtils reference

* remove commons-docker that was re-added

* remove final references to airbyte-commons-docker

* add bouncycastle test

* add bouncycastle test

* add bouncycastle to acceptance-tests
2023-02-01 11:05:09 -08:00
Cole Snodgrass
e32215f7f3 implement PerfBackgroundJsonValidation feature-flag (#21569)
builds on add perf feature-flag; ensure compatability with LaunchDarkly (LDv5) users #21534
add support for checking the PerfBackgroundJsonValidation feature-flag within the RecordSchemaValidator class

I spent a lot of time with trying to get the Kotlin beans injectable into the java code. What should just work doesn't and it well documented on the Micronaut side as to what exactly is required (or at least I never found the documentation). After many attempts I and looking at the auto-generated kotlin Micronaut projects I landed on kapt being necessary in kotlin projects. Even this isn't entirely true.

If the Kotlin project is defined with the following gradle setup, the beans will not be visible to java projects (and possibly other projects as well):

plugins {
    kotlin("jvm") version "1.8.0"
}

dependencies {
    annotationProcessor(platform(libs.micronaut.bom))
    annotationProcessor(libs.bundles.micronaut.annotation.processor)
}
But they will be visible with the following setup:

plugins {
    kotlin("jvm") version "1.8.0"
    id("io.micronaut.minimal.application") version "3.7.0"
}

dependencies {
    annotationProcessor(platform(libs.micronaut.bom))
    annotationProcessor(libs.bundles.micronaut.annotation.processor)
}

micronaut {
  version("3.8.2")
}
and also visible with the following setup:

plugins {
    kotlin("jvm") version "1.8.0"
    kotlin("kapt") version "1.8.0"
}

dependencies {
    kapt(platform(libs.micronaut.bom))
    kapt(libs.bundles.micronaut.annotation.processor)
}
I went with the kapt solution.

It's worth noting that kapt is deprecated and is being replaced with kps in Micronaut 4 (this is a good thing).

---------

Co-authored-by: Davin Chia <davinchia@gmail.com>
2023-01-30 17:56:09 -08:00
Benoit Moriceau
0b97ce3d6c Extract split the server into 2 modules (#21736)
* tmp

* For diff

* Extract common server

* Format

* Better way to get resources

* Extract the old mapper

* Move test as well

* Format
2023-01-24 07:51:20 -08:00
Cole Snodgrass
4731a9cb0f add feature-flag client (#21091)
* feature-flag client; wip

* feature-flag; wip

* updated wip/poc

* make account truly optional

* add temp flag example

* wip of feature-flag client; upgrade to kotlin 1.8

* inject LDClient

* feature-flag client wip; add tests; finalize classes

* add cloud test with mockk

* add EnvVar tests

* remove account from workspace, add user

* comment out test

* doc typo

* move flags to top of file

* change LogConnectorMessages to EnvVar

* add @JvmOverloads to Workspace context

* rename clients; add TestClient (with tests)

* rename teams

* rename Client -> FeatureFlagClient

* doc typo

* change parameter ordering in flags

* rename files

* typo

* update docs to match re-org'd params

* rename PlatformClient -> ConfigFileClient; CloudClient -> LaunchDarklyClient

* pr feedback; comment updates

* pr feedback - remove star imports

* pr feedback; update to use version catalog

* pr feedback; add Connection, Source, Destination contexts

* pr feedback - add multicontext support

* replace Team idea with attributes map

* pr feedback

* explicit types

* rename data classes to match client

* pass context to envvar flags; make enabled open

* change fetcher to an internal property

* add UUID secondary constructors to context classes

* make values optional for TestClient

* add anonymous support

* pr-feedback; add newlines to each file
2023-01-17 09:40:32 -08:00
Lake Mossman
684fbff12f Enable the Connector Builder server in the platform (#19269)
* init

* bad copy/paste

* move to top level

* Revert "move to top level"

This reverts commit aca3534d38.

* attempt to wire up connector builder frontend to server

* copy from octaviacli

* fix connection to builder server

* update

* delete

* Update

* delete python-version

* Revert "delete python-version"

This reverts commit f9258a7755.

* setup python

* install python

* rename

* kube stuff

* Install python

* missing kube file

* rename

* Update files

* Update bumpversion

* install python

* try with different entrypoint

* rename container

* point to docker-compose.yaml file

* derp

* copy acceptance_test.sh

* copy from acceptance tests

* delete cruft

* update

* remove application env

* reset

* reset to master

* update

* skip comprehensive incremental tests

* Revert "skip comprehensive incremental tests"

This reverts commit 9cee657596.

* reset to master

* remove cruft

* readme

* readme

* save

* buncha stuff

* it works

* improvements

* comment

* codesnippet

* delete superfluous steps

* update port to 8003

* reset to master

* Update publish docker

* move openapi spec to airbyte-connector-builder

* point to openapi spec

* dont expose the connector builder to localhost

* reset FE components to master

* Don't deploy the connector-builder

* Revert "Don't deploy the connector-builder"

This reverts commit 3d157494cf.

* Revert "Revert "Don't deploy the connector-builder""

This reverts commit beac3d48f0.

* comment out more things related to connector builder server

* more attempts at removing the connector builder

* comment out more things

* Update airbyte-connector-builder/README.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* fixes

* rename gradle task

* save

* copyright header

* Apply suggestions from code review

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update airbyte-webapp/src/config/configProviders.ts

Co-authored-by: Lake Mossman <lake@airbyte.io>

* update

* rename

* indent

* Revert "move openapi spec to airbyte-connector-builder"

This reverts commit 57dda04723.

* Revert "rename"

This reverts commit b2d802b8fa.

* Revert "Revert "rename""

This reverts commit 91db24fd4a.

* point to wrong file in case it fixes the build

* point to right openapi file

* Revert "Revert "move openapi spec to airbyte-connector-builder""

This reverts commit e46a837454.

* point to moved file

* fix path

* Update from master

* newline

* Add failing test

* Revert "Add failing test"

This reverts commit ed9fea09b5.

* comment

* update commented requires

* Add a comment

* comments

* save

* formatting is for losers anyways

* pyproject

* toml

* toml

* saved

* nice

* anothaone

* fmt

* fixmypy

* fix autogen notice header location

* improvecomment

* save working branch

* fmt

* delete license and disable caching

* removeyaml

* fix merge and uncomment flakeCheck

* undo css change

* fix typo

* format

* enable CORS

* move template to backend server

* try adding connector builder server to 'generate-docker' task to see if it fixes CI

* try adding buildDockerImage task

* try adding gradle.properties

* Revert "try adding buildDockerImage task"

This reverts commit 2102009ddb.

* Revert "try adding connector builder server to 'generate-docker' task to see if it fixes CI"

This reverts commit 40b5ae0fcd.

* process detail in apiOverride

* fix gradle

* fix scrollbars

* Run connector builder server tests and static analysis from a docker container (#19495)

* Revert "try adding buildDockerImage task"

This reverts commit 2102009ddb.

* Revert "try adding connector builder server to 'generate-docker' task to see if it fixes CI"

This reverts commit 40b5ae0fcd.

* missing file

* build airbyte-connector-builder-server image

* Add .

* comment

* build connector builder server as part of octavia_cli

* build connector builder too

* patch

* missing file

* comment

* comment

* run unit tests in a container

* Run integration tests

* run

* move to platform build

* format

* rootdir

* get the path from the project

* Update

* pyproject

* delete a bunch of code

* parameterize

* run tests

* update

* rename

* comment

* reset

* uncomment

* dont format

* run everything except isort

* delete dead code

* Revert "delete dead code"

This reverts commit b11df716a0.

* comment

* Update module

* reset to master

* mount in home

* commit formatted files

* try running format

* Better comment

* reset

* Revert "reset"

This reverts commit 20808f8262.

* Update buildSrc/src/main/groovy/airbyte-python-docker.gradle

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* comment

* Add everything except build and hidden directories

* symlink

* Update airbyte-connector-builder-server/run_format.sh

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-connector-builder-server/run_tests.sh

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: sherifnada <snadalive@gmail.com>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
2022-11-17 19:44:45 -08:00
Conor
9bf5dac807 feat: add remote build caching (#19390)
* feat: add remote build caching

* ci: retrigger actions
2022-11-14 15:10:47 -06:00
Alexandre Girard
0373497fe2 disable connector builder server (#19335) 2022-11-10 20:18:04 -08:00
Alexandre Girard
64b50adfdd Connector-builder server stub (#18410)
* init

* bad copy/paste

* move to top level

* Revert "move to top level"

This reverts commit aca3534d38.

* attempt to wire up connector builder frontend to server

* copy from octaviacli

* fix connection to builder server

* update

* delete

* Update

* delete python-version

* Revert "delete python-version"

This reverts commit f9258a7755.

* setup python

* install python

* rename

* kube stuff

* Install python

* missing kube file

* rename

* Update files

* Update bumpversion

* install python

* try with different entrypoint

* rename container

* point to docker-compose.yaml file

* derp

* copy acceptance_test.sh

* copy from acceptance tests

* delete cruft

* update

* remove application env

* reset

* reset to master

* update

* skip comprehensive incremental tests

* Revert "skip comprehensive incremental tests"

This reverts commit 9cee657596.

* reset to master

* remove cruft

* delete superfluous steps

* update port to 8003

* reset to master

* Update publish docker

* move openapi spec to airbyte-connector-builder

* point to openapi spec

* dont expose the connector builder to localhost

* reset FE components to master

* Don't deploy the connector-builder

* Revert "Don't deploy the connector-builder"

This reverts commit 3d157494cf.

* Revert "Revert "Don't deploy the connector-builder""

This reverts commit beac3d48f0.

* comment out more things related to connector builder server

* more attempts at removing the connector builder

* comment out more things

* Apply suggestions from code review

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update airbyte-webapp/src/config/configProviders.ts

Co-authored-by: Lake Mossman <lake@airbyte.io>

* update

* rename

* indent

* Revert "move openapi spec to airbyte-connector-builder"

This reverts commit 57dda04723.

* Revert "rename"

This reverts commit b2d802b8fa.

* Revert "Revert "rename""

This reverts commit 91db24fd4a.

* point to wrong file in case it fixes the build

* point to right openapi file

* Revert "Revert "move openapi spec to airbyte-connector-builder""

This reverts commit e46a837454.

* point to moved file

* fix path

* Update from master

* newline

* Add failing test

* Revert "Add failing test"

This reverts commit ed9fea09b5.

* comment

* update commented requires

* Add a comment

* 2022

* rename to connector-builder-server

* typo

Co-authored-by: lmossman <lake@airbyte.io>
2022-11-09 13:07:26 -08:00
Subodh Kant Chaturvedi
3468516a2b cleanup: delete debezium 1-4-2 module (#18733)
* mssql-source:upgrade debezium version to 1.9.6

* delete debezium 1-4-2

* more improvements
2022-11-05 00:24:26 +05:30
Evan Tahler
7fcadf433c Remove airbyte-queue (#18250) 2022-10-20 16:02:31 -07:00
Evan Tahler
3d053e32ee Use Nginx + Basic Auth to secure OSS Airbyte (#17694)
* Use Nginx + Basic Auth to secure OSS Airbyte

* use local passwords

* Use gradle builds

* K8s setup and source values from ENV

* note about disabling

* add back defaults

* custom 401 page

* update http message

* update docs

* remove kube files

* additional doc updates

* Add a test suite

* fix failure exit codes

* doc updates

* Add docs

* bump to re-test

* add more sleep in tests for CI

* better sleep in test

* Update docs/operator-guides/security.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* PR updates

* test comment

* change test host on CI

* update tests and nginx to boot without backend

* proxy updates for docker DNS

* simpler test for uptime

* acceptance test skips PWs

* remove resolver madness

* fixup tests

* more proxy_pass revert

* update acceptance test exit codes

* relax test expectations

* add temporal mount back for testing

* Update docs/operator-guides/security.md

Co-authored-by: swyx <shawnthe1@gmail.com>

* Update airbyte-proxy/401.html

Co-authored-by: swyx <shawnthe1@gmail.com>

* more doc updates

* Octavia CLI uses Basic Auth  (#17982)

* [WIP] Octavia CLI uses Basic Auth

* readme

* augustin: add basic auth headers to clien

* augustin: add basic auth headers to client

* tests passing

* lint

* docs

* Move monkey patch to test

* coerce headers into strings

* monkey patch get_basic_auth_token

Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>

* fix launch permissions

* Keep worker port internal

* more readme

Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: swyx <shawnthe1@gmail.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
2022-10-19 15:52:01 -07:00
Benoit Moriceau
adf2c5ff8e Bmoric/remove dep connector worker (#17977)
* test [ci skip]

* Autogenerated files

* Add missing annotation

* Remove unused json2Schema block from worker

* Move tess

* Missing deps and format

* Fix test build

* TMP

* Add missing dependencies

* PR comments

* Tmp

* [ci skip] Tmp

* Fix acceptance test and add the seed dependency

* Fix build

* For diff

* tmp

* Build pass

* make the worker to be  on the platform only

* fix setting.yaml

* remove dependency

* Fix pmd

* Fix Cron

* Add chart

* Fix cron

* Fix server build.gradle

* Fix jar conflict

* PR comments

* Add cron micronaut environemnt

* Update required build path

* remove all dependencies on worker

* Typo

* Add gradle exclusions

* Add exclusion to avoid conflict

* more exclusion

* Move more class

* Exclude worker from the general build

* Fix connector build
2022-10-18 11:51:25 -07:00
Benoit Moriceau
7a71c55c1f Bmoric/remove dep server worker (#17894)
* test [ci skip]

* Autogenerated files

* Add missing annotation

* Remove unused json2Schema block from worker

* Move tess

* Missing deps and format

* Fix test build

* TMP

* Add missing dependencies

* PR comments

* Tmp

* [ci skip] Tmp

* Fix acceptance test and add the seed dependency

* Fix build

* For diff

* tmp

* Build pass

* make the worker to be  on the platform only

* fix setting.yaml

* Fix pmd

* Fix Cron

* Add chart

* Fix cron

* Fix server build.gradle

* Fix jar conflict

* PR comments

* Add cron micronaut environemnt
2022-10-17 12:35:04 -07:00
Evan Tahler
792fa7cead Remove unused airbyte-cli (#18009) 2022-10-17 10:28:53 -07:00
Benoit Moriceau
f650005f2a Remove the dependency between the orchestrator container and the worker app (#17570)
* test [ci skip]

* Autogenerated files

* Add missing annotation

* Remove unused json2Schema block from worker

* Move tess

* Missing deps and format

* Fix test build

* Add missing dependencies

* PR comments

* Fix acceptance test and add the seed dependency

* Fix build

* Fix build again
2022-10-11 17:12:00 -07:00
Jonathan Pearlin
6dde52187f Add dependency required by airbyte-workers (#17702) 2022-10-06 17:20:25 -04:00
Jimmy Ma
38ea570f77 Fix deps (#17518) 2022-10-03 14:00:38 -07:00
Subodh Kant Chaturvedi
0c1823d9aa upgrade debezium version to 1.9.6 (#17459)
* upgrade debezium version to 1.9.6

* test override not required

* bump version

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-01 15:08:42 +05:30
Jimmy Ma
42800f7b37 Add AirbyteProtocol Migration and SerDe framework (#17114)
* Introduce AirbyteProtocol Migration and SerDe framework

* Update serde

* Add demo migration test

* Enable Micronaut

* Do not expose internals

* Naming consistency

* More reordering and visibility fixes

* Use jakarta instead of javax

* Rename Old->Previous New->Current

* Add comments

* Lombok.@Getter
2022-09-28 15:08:28 -07:00
Benoit Moriceau
3c457e051d Bmoric/restore temporal cleaning cron (#17258)
* Revert "Revert "Bmoric/temporal cleaning cron (#16414)" (#17192)"

This reverts commit 751667949e.

* Generate an artifact

* Change missing property from value to property
2022-09-27 10:59:39 -07:00
Benoit Moriceau
751667949e Revert "Bmoric/temporal cleaning cron (#16414)" (#17192)
This reverts commit bbb59d8675.
2022-09-26 16:58:23 -07:00
Benoit Moriceau
bbb59d8675 Bmoric/temporal cleaning cron (#16414)
* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* Add the injected temporal client to airbyte-cron

* Update cron to make it work

* Remove useless Bean factory

* PR comments

* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* Improve conditional bean check

* Match existing Optional functionality

* Add notEquals check

* Add missing env var to Helm chart

* Fix typo

* Mark LogConfigs as Singleton

* WIP Convert airbyte-workers to Micronaut framework

* Rebase cleanup

* Fix broken tests

* Simplify code

* Support control vs data plane configuration

* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests

* Formatting

* Pairing on Worker Micronaut (#16364)

* add RouteToSyncTaskQueue activity

* use new route activity in connection manager workflow

* format

* call router service for task queue

* Revert temporal proxy changes

* Formatting

* Fix default value

* register new route activity in test

* fix SyncWorkflowTest now that it isn't doing any routing

* Update dependencies

* More dependency updates

* Update dependencies

* Improve conditional bean check

* Match existing Optional functionality

* Add notEquals check

* Add missing env var to Helm chart

* Fix typo

* Mark LogConfigs as Singleton

* Env vars for log/state storage type

* Remove use of Optional in bean declarations

* Fix typo in config property name

* Support Temporal Cloud namespace

* Change to @Value

* Use correct value for conditional check

* Upgrade Micronaut

* Fix merge conflict

* Formatting

* Add missing env var

* Use sync task queue environment variable

* Handle sync task queue as set

* format and force http

* Handle case where sync task queue is empty

* Add correct path to config property

* Remove unused import

* Remove conflict

* Remove unused parameter

* Formatting

* Use pattern for condition process factory beans

* Cleanup

* PR feedback

* Revert hack for testing

* Fix temporal restart by status (#16447)

* Update application.yml

* Re add worker dep

* Add missing env var

* PR comments

* Bmoric/move temporal client (#16778)

* tmp

* tmp

* View diff

* Move part of the temporal client

* tmp

* copy Temporal Utils test

* Uniq Temporal Utils

* Uniq Temporal Workflow Utils

* Move CancellationHandler

* Move commons-temporal to being shared between platform, connector and CLI

* Rm worker dependency from cron

* Fix build

* Update with right value for cron.

* Fix dep conflict

Co-authored-by: jdpgrailsdev <jpearlin1@gmail.com>
Co-authored-by: pmossman <parker@airbyte.io>
Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io>
2022-09-26 15:14:35 -07:00
Liren Tu
d14b4423ea Publish R2 destination (#17122)
* Format code

* Update doc

* Add r2 to build doc

* Update readme

* Add r2 to destination definitions

* Move test resource to base-java-s3

* Remove redundant dependency

* Add missing dependency

* Format json

* Remove s3-destination-base-integration-test setting.gradle root

* Add missing lzo dependency

* Fix import in destination jdbc
2022-09-25 21:45:00 -07:00