* copy files from edgao branch
* start writing create table statement
* add basic unit test setup
* create a table, probably
* remove outdated todo
* derp, one more column
* ugh
* add partitioning+clustering
* use StringSubstitutor
* substitutions in updateTable
* wip generate update/insert statement
* split up into smaller methods
* handle json types correctly
* rename stuff
* more json_query vs _value stuff
* minor tweak
* super basic test setup
* laying foundation for type parsing
* more stuff
* tweaks
* more progress on type parsing
* fix json_value stuff?
* misc fixes in insert
* fix dedupFinalTable
* add testDedupRaw
* full e2e test
* type parsing: gave up and mirrored the dbt code structure to avoid bugs
* type parsing - more cleanup
* handle column name collisions
* handle tablename collisions...?
* comments
* remove original ns/name from quotedstream
* also javadoc
* remove redundant method
* fix table rename
* add incremental append test
* add full refresh append test
* comment
* call T+D sql in a reasonable location for standard inserts
* add config option
* use config option here
* type parsing - fix fromJsonSchema
* gate everything
* log query + runtime
* add spec option temporarily
* Raw Table Updates
* fix more stuff
* first big pass at toDialectType
* no quotes
* wrap everything in quotes
* resolve some TODOs
* log sql statement in tests
* overwriteFinalTable returns optional
* minor clean up
* add raw dataset override
* try to preserve the original namespace for t+d?
* write to the raw table correctly
* update todos
* write directly to raw table
this is kind of dumb because we're still trying to do tmp table operations,
and we still don't ack state until the end of the entire sync.
* standard inserts write to raw table correctly
* imports + log statements
* move logs + add comment
* explicitly create raw table
* move comment to better place
* Typing issues
* bash attempt
* formatting updates
* formatting updates
* write to the airbyte schema by default unless overriden by config options
* standard inserts truncate raw table at start of sync
* full refresh overwrite will overwrite correctly!
* fix avro record schema parsing
* better raw table recreate
* rename raw table to match standard inserts
* full refresh overwrite does tmp table things
* small clean up
* small clean up
* remove errors entry if no errors
* pull out destination config into singleton
* clean up singleton stuff
* make sure dest config exists when trying to do lookups
* avoid stringifying null
* quick thoughts on alter table
* add basic cdc testcase
* tweak cdc test setup
* rename raw table to match standard inserts
* minor tweak
* delete exact sql string assertions
* switch to JSON type
* minor cleanup
* sql whitespace changes
* explain cdc deletions
* GCS Staging Full Refresh create temp table
* assert schema
* first out of order cdc test
* add another cdc test case (currently failing)
* better test structure
* make this work
* oops, fix test
* stop trying to delete deletion records
* minor improvements to code+test
* enable concurrent test runs on integration test
* move stuff to static initializer
* extract utility method
* formatting
* Move conditional to the base java package, replace conditionals which did not use the typing and deduping flag but should have been.
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* switch back to empty list; write big assert
* minor wording tweaks
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* DestinationConfigTest
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* formatting
* remove ParsedType
* 🤖 Auto format destination-gcs code [skip ci]
* 🤖 Auto format destination-bigquery code [skip ci]
* tests verify every data type
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* full update with all data types
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* move stuff to new base lib
* 🤖 Auto format destination-gcs code [skip ci]
* Automated Commit - Formatting Changes
* 🤖 Auto format destination-bigquery code [skip ci]
* fix test
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* asserts in dedupFinalTable
* better asserts in dedupRawTable
* [wip] test case for all data types
* 🤖 Auto format destination-gcs code [skip ci]
* 🤖 Auto format destination-bigquery code [skip ci]
* AirbyteTypeTest
* Automated Commit - Formatting Changes
* remove comments
* test chooseOneOf
* slightly better test output
* Automated Commit - Formatting Changes
* add some awful pretty print code
* more comment
* minor tweaks
* verify array/object type
* fix test
* handle deletions more correctly
* test toDialectType
* Destinations v2: better namespace handling (#27682)
* [wip] better namespace handling
* 🤖 Auto format destination-bigquery code [skip ci]
* wip also implement in gcs
* get gcs working (?)
* 🤖 Auto format destination-bigquery code [skip ci]
* remove duplicate method
* 🤖 Auto format destination-bigquery code [skip ci]
* fixed my code style settings
* make ci happy?
* 🤖 Auto format destination-bigquery code [skip ci]
* make ci happy?
* remove incorrect test
* blank line change
* initialize singleton
---------
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
* reset args correctly
* Automated Commit - Formatting Changes
* more bash stuff
* parse implicit structs
* initialize singleton in more tests
* Automated Commit - Formatting Changes
* I missed this namespace handling thing
* test more schemas
* fix singular types specified in arrays
* Automated Commit - Formatting Changes
* disable test for unimplemented feature
* initialize singleton
* remove spec options; changelogs+metadata
* randomize namespace
* also bump dockerfile
* unremove namespace sanitizing in legacy mode
* ... disable the correct test
* even more unit test fixes!
* move integration test to integration tests
---------
Co-authored-by: Cynthia Yin <cynthia@airbyte.io>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com>
* Revert "Splits bases and updates build.gradle files (#25649)"
This reverts commit c673b0a692.
* Bumps branch to prevent a conflict with publishing
* Forward fixes Snowflake to use singular base-java and develop within a new package within the same module
* Forcing automated change to merge changes
* Splits bases and updates build.gradle files
* Fixed changelog out of sync
* Bumps version number and metadata files
* auto-bump connector version
* Downgraded untouched connector bumps
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* remove arguments
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* Source Performance harness (#23922)
* test
* initial commit
* Working dev
* src-platform-dst
* Integrate secrets
* Update .dockerignore
* Update build.gradle
* Update base.sh
* Build harness module and use it to measure
* Build harness module and use it to measure
* Build harness module and use it to measure
* Automated Change
* Integrate secrets
* Integrate secrets
* Integrate secrets
* Integrate secrets
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* Automated Change
* test
* Automated Change
* better error handling
* temp
* Automated Change
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* add test name argument
* add test name argument
* add test name argument
* add test name argument
* add test name argument
* add test name argument
* Automated Change
* add bottleneck_stream1 dataset
* Automated Change
* fix dependency
* fix dependency
* test downgrade of commons to 41.1
* Automated Change
* test downgrade of commons to 41.0
* test
* test
* test
* test
* test
* exit condition
* exit condition
* exit condition
* test
* test
* test
* test
* test
* test
* sanity
* readme
* change command name
* update gradle settings
* sanity
* architecture aware docker build
* sanity
* put run harness step in a script
* sanity
* sanity
---------
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
* delete platform code and workflows
* delete more platform workflows
* sunset kustomize (#23187)
* sunset kustomize
* remove build steps that use kustomize
* remove k8s kustomize gke-kube-acceptance-tests
* delete docker-compose files
* remove platform from check_images_exist.sh
* Working replacement script for docker
* rename start script and add download files to gitignore
* add args and download only option
* integration test downloads docker compose files
* remove helm charts
* remove more platform files
* add flags.yml
* use version from gradle.properties instead of .env
* Working replacement script for docker (#23227)
* Working replacement script for docker
* Working debugging line
* rename start script and add download files to gitignore
* add args and download only option
* Made flags cleaner and more cleanup
i've never used this flag pattern, its much
easier than my usual way
* warn about a sharp edge
* Don't run when we don't know a command
* better help spacing
* add flags.yml
---------
Co-authored-by: cpdeethree <conor@airbyte.io>
Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
* remove more-secrets env
---------
Co-authored-by: Charles <charles@airbyte.io>
Co-authored-by: Topher Lubaway <asimplechris@gmail.com>
* download latest connector catalog on build
* format
* Add small todos
* Add skeleton for local provider change
* Load catalog from local
* Remove seed class from localdefprovider
* Get catalog path from config
* refactor arguments out of download catalog
* Move to a constant file method
* Fix arguments
* Rename to CatalogDefinitionsConfig
* Add todos
* Update todo
* Apply formatting
* remove unnessessary try catch
* Refactor IconValidationTask
* Fix null value issue
* Only build oss catalog via github action
* Add LOCAL_CONNECTOR_CATALOG_PATH env var override
* Format
* Add logger
* Copy file directly to avoid circular dep
* Format
* Ensure run before tests
* Move airbyte:init:spec to general dependency
* LOCAL_CATALOG_PATH -> LOCAL_CONNECTOR_CATALOG_PATH
---------
Co-authored-by: Ben Church <ben@airbyte.io>
* remove airbyte-protocol
moved to a separate repository
* delete references to airbyte-protocol dir
* ensure generateWellKnownTypes task works
extracct well_known_types.yaml from airbyte-protocol jar
* update url to new airbyte-protocol repo
* add back treenode
* use published jar for commons-converters
* format
* resolve bad merge conflict
* resolve bad merge conflict
---------
Co-authored-by: cpdeethree <conor@airbyte.io>
Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
Support conversion of the airbyte-server-wrapped Cloud module to Micronaut
- Remove @Requires rule from controllers so they will be used in both OSS and Cloud
- Remove airbyte-commons-worker dependency from airbyte-server by moving shared code to new airbyte-commons-converters module
Co-authored-by: Davin Chia <davinchia@gmail.com>
This is the first version of the DuckDB destination. There are potential edge cases that still need to be taken care of. But looking forward to your feedback.
