This workflow does too much today so it's long, flaky and expensive.
Instead, scope this down to only run CDK tests. Existing workflows already handle per-connector tests so this is fine.
- Edit the workflow to only run on changes to the CDK modules.
- Modify CDK so there is a convenient ./gradlew :airbyte-cdk:check method
- Use convenience method instead.
- Remove all the build failure publishing errors.
Introduce the airbyte gradle docker plugin to build JVM connectors.
Add logic to allow us to slowly migrate from one build set up to the other.
The majority of the work is taking in the metadata.yaml file, and converting that to buildArgs to be injected into the airbyte gradle docker plugin.
In particular, we extract the base image from the metadata file. We also add the connector name by referencing the module. This outputs a file buildArgs.properties into the build folder. This is done by the DockerGenerateConnectorBuildArgs task.
The AirbyteDockerConventionPlugin then takes in the generated buildArgs.properties file and feeds it into the Platform Docker plugin. It also feeds in the top-level Dockerfile, so the Docker plugin can copy it into the build folder.
I tried to migrate this to Kotlin, however the repo needs to be upgraded to the same Kotlin and KSP version as the gradle plugin repos for this to be worth while. Otherwise we end up writing kotlin plugins with reflection, which is worse than the current groovy plugins.
* copy files from edgao branch
* start writing create table statement
* add basic unit test setup
* create a table, probably
* remove outdated todo
* derp, one more column
* ugh
* add partitioning+clustering
* use StringSubstitutor
* substitutions in updateTable
* wip generate update/insert statement
* split up into smaller methods
* handle json types correctly
* rename stuff
* more json_query vs _value stuff
* minor tweak
* super basic test setup
* laying foundation for type parsing
* more stuff
* tweaks
* more progress on type parsing
* fix json_value stuff?
* misc fixes in insert
* fix dedupFinalTable
* add testDedupRaw
* full e2e test
* type parsing: gave up and mirrored the dbt code structure to avoid bugs
* type parsing - more cleanup
* handle column name collisions
* handle tablename collisions...?
* comments
* remove original ns/name from quotedstream
* also javadoc
* remove redundant method
* fix table rename
* add incremental append test
* add full refresh append test
* comment
* call T+D sql in a reasonable location for standard inserts
* add config option
* use config option here
* type parsing - fix fromJsonSchema
* gate everything
* log query + runtime
* add spec option temporarily
* Raw Table Updates
* fix more stuff
* first big pass at toDialectType
* no quotes
* wrap everything in quotes
* resolve some TODOs
* log sql statement in tests
* overwriteFinalTable returns optional
* minor clean up
* add raw dataset override
* try to preserve the original namespace for t+d?
* write to the raw table correctly
* update todos
* write directly to raw table
this is kind of dumb because we're still trying to do tmp table operations,
and we still don't ack state until the end of the entire sync.
* standard inserts write to raw table correctly
* imports + log statements
* move logs + add comment
* explicitly create raw table
* move comment to better place
* Typing issues
* bash attempt
* formatting updates
* formatting updates
* write to the airbyte schema by default unless overriden by config options
* standard inserts truncate raw table at start of sync
* full refresh overwrite will overwrite correctly!
* fix avro record schema parsing
* better raw table recreate
* rename raw table to match standard inserts
* full refresh overwrite does tmp table things
* small clean up
* small clean up
* remove errors entry if no errors
* pull out destination config into singleton
* clean up singleton stuff
* make sure dest config exists when trying to do lookups
* avoid stringifying null
* quick thoughts on alter table
* add basic cdc testcase
* tweak cdc test setup
* rename raw table to match standard inserts
* minor tweak
* delete exact sql string assertions
* switch to JSON type
* minor cleanup
* sql whitespace changes
* explain cdc deletions
* GCS Staging Full Refresh create temp table
* assert schema
* first out of order cdc test
* add another cdc test case (currently failing)
* better test structure
* make this work
* oops, fix test
* stop trying to delete deletion records
* minor improvements to code+test
* enable concurrent test runs on integration test
* move stuff to static initializer
* extract utility method
* formatting
* Move conditional to the base java package, replace conditionals which did not use the typing and deduping flag but should have been.
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* switch back to empty list; write big assert
* minor wording tweaks
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* DestinationConfigTest
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* formatting
* remove ParsedType
* 🤖 Auto format destination-gcs code [skip ci]
* 🤖 Auto format destination-bigquery code [skip ci]
* tests verify every data type
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* full update with all data types
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* move stuff to new base lib
* 🤖 Auto format destination-gcs code [skip ci]
* Automated Commit - Formatting Changes
* 🤖 Auto format destination-bigquery code [skip ci]
* fix test
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* asserts in dedupFinalTable
* better asserts in dedupRawTable
* [wip] test case for all data types
* 🤖 Auto format destination-gcs code [skip ci]
* 🤖 Auto format destination-bigquery code [skip ci]
* AirbyteTypeTest
* Automated Commit - Formatting Changes
* remove comments
* test chooseOneOf
* slightly better test output
* Automated Commit - Formatting Changes
* add some awful pretty print code
* more comment
* minor tweaks
* verify array/object type
* fix test
* handle deletions more correctly
* test toDialectType
* Destinations v2: better namespace handling (#27682)
* [wip] better namespace handling
* 🤖 Auto format destination-bigquery code [skip ci]
* wip also implement in gcs
* get gcs working (?)
* 🤖 Auto format destination-bigquery code [skip ci]
* remove duplicate method
* 🤖 Auto format destination-bigquery code [skip ci]
* fixed my code style settings
* make ci happy?
* 🤖 Auto format destination-bigquery code [skip ci]
* make ci happy?
* remove incorrect test
* blank line change
* initialize singleton
---------
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
* reset args correctly
* Automated Commit - Formatting Changes
* more bash stuff
* parse implicit structs
* initialize singleton in more tests
* Automated Commit - Formatting Changes
* I missed this namespace handling thing
* test more schemas
* fix singular types specified in arrays
* Automated Commit - Formatting Changes
* disable test for unimplemented feature
* initialize singleton
* remove spec options; changelogs+metadata
* randomize namespace
* also bump dockerfile
* unremove namespace sanitizing in legacy mode
* ... disable the correct test
* even more unit test fixes!
* move integration test to integration tests
---------
Co-authored-by: Cynthia Yin <cynthia@airbyte.io>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com>
* Revert "Splits bases and updates build.gradle files (#25649)"
This reverts commit c673b0a692.
* Bumps branch to prevent a conflict with publishing
* Forward fixes Snowflake to use singular base-java and develop within a new package within the same module
* Forcing automated change to merge changes
* Splits bases and updates build.gradle files
* Fixed changelog out of sync
* Bumps version number and metadata files
* auto-bump connector version
* Downgraded untouched connector bumps
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* remove arguments
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* Source Performance harness (#23922)
* test
* initial commit
* Working dev
* src-platform-dst
* Integrate secrets
* Update .dockerignore
* Update build.gradle
* Update base.sh
* Build harness module and use it to measure
* Build harness module and use it to measure
* Build harness module and use it to measure
* Automated Change
* Integrate secrets
* Integrate secrets
* Integrate secrets
* Integrate secrets
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* Automated Change
* test
* Automated Change
* better error handling
* temp
* Automated Change
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* add test name argument
* add test name argument
* add test name argument
* add test name argument
* add test name argument
* add test name argument
* Automated Change
* add bottleneck_stream1 dataset
* Automated Change
* fix dependency
* fix dependency
* test downgrade of commons to 41.1
* Automated Change
* test downgrade of commons to 41.0
* test
* test
* test
* test
* test
* exit condition
* exit condition
* exit condition
* test
* test
* test
* test
* test
* test
* sanity
* readme
* change command name
* update gradle settings
* sanity
* architecture aware docker build
* sanity
* put run harness step in a script
* sanity
* sanity
---------
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
