* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)""
This reverts commit 348c577dbb.
* version bumps+changelog
* Speed up BQ by having 2 queries, and not an OR (#27981)
* 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (#27856)
* only run t+d code if it's enabled
* dockerfile+changelog
* remove changelog entry
* Destinations V2: handle optional fields for `object` and `array` types (#27898)
* catch null schema
* fix null properties
* clean up
* consolidate + add more tests
* try catch
* empty json test
* Automated Commit - Formatting Changes
* remove todo
* destination bigquery: misc updates to 1s1t code (#28057)
* switch to checkedconsumer
* add unit test for buildColumnId
* use flag
* restructure prefix check
* fix build
* more type-parsing fixes (#28100)
* more type-parsing fixes
* handle duplicates
* Automated Commit - Format and Process Resources Changes
* add tests for asColumns
* Automated Commit - Format and Process Resources Changes
* log warnings instead of throwing exception
* better log message
* error level
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
* Automated Commit - Formatting Changes
* Improve protocol type parsing (#28126)
* Automated Commit - Formatting Changes
* Change from T&D every 10k records to an increasing time based interval (#28130)
* fifteen minute t&d
* add typing and deduping operation valve for increased intervals of typing and deduping
* Automated Commit - Format and Process Resources Changes
* resolve bizarre merge conflict
* Automated Commit - Format and Process Resources Changes
---------
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
* Simplify and speed up CDC delete support [DestinationsV2] (#28029)
* Simplify and speed up CDC delete support [DestinationsV2]
* better QUOTE
* spotbugs?
* recompile dbt image for local arch and use that when building images
* things compile, but tests fail
* tests working-ish
* comment
* fix logic to re-insert deleted records for cursor comparison.
tests pass!
* remove comment
* Skip CDC re-include logic if there are no CDC columns
* stop hardcoding pk (#28092)
* wip
* remove TODOs
---------
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
* update method name
* Automated Commit - Formatting Changes
* depend on pinned normalization version
* implement 1s1t DATs for destination-bigquery (#27852)
* intiial implementation
* Automated Commit - Formatting Changes
* add second sync to test
* do concurrent things
* Automated Commit - Formatting Changes
* clarify comment
* minor tweaks
* more stuff
* Automated Commit - Formatting Changes
* minor cleanup
* lots of fixes
* handle sql vs json null better
* verify extra columns
* only check deleted_at if in DEDUP mode and the column exists
* add full refresh append test case
* Automated Commit - Formatting Changes
* add tests for the remaining sync modes
* Automated Commit - Formatting Changes
* readability stuff
* Automated Commit - Formatting Changes
* add test for gcs mode
* remove static fields
* Automated Commit - Formatting Changes
* add more test cases, tweak test scaffold
* cleanup
* Automated Commit - Formatting Changes
* extract recorddiffer
* and use it in the sql generator test
* fix
* comment
* naming+comment
* one more comment
* better assert
* remove unnecessary thing
* one last thing
* Automated Commit - Formatting Changes
* enable concurrent execution on all java integration tests
* add test for default namespace
* Automated Commit - Formatting Changes
* implement a 2-stream test
* Automated Commit - Formatting Changes
* extract methods
* invert jsonNodesNotEquivalent
* Automated Commit - Formatting Changes
* fix conditional
* pull out diffSingleRecord
* Automated Commit - Formatting Changes
* handle nulls correctly
* remove raw-specific handling; break up methods
* Automated Commit - Formatting Changes
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>
* Destinations V2: move create raw tables earlier (#28255)
* move create raw tables
* better log message
* stop building normalization (#28256)
* fix ability to run tests
* disable incremental t+d for now
* Automated Commit - Formatting Changes
---------
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Cynthia Yin <cynthia@airbyte.io>
Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>
58 lines
1.7 KiB
Bash
Executable File
58 lines
1.7 KiB
Bash
Executable File
#!/bin/bash
|
|
# Intentionally no set -e, because we want to run normalization even if the destination fails
|
|
set -o pipefail
|
|
|
|
/airbyte/base.sh $@
|
|
destination_exit_code=$?
|
|
echo '{"type": "LOG","log":{"level":"INFO","message":"Destination process done (exit code '"$destination_exit_code"')"}}'
|
|
|
|
# store original args
|
|
args=$@
|
|
|
|
while [ $# -ne 0 ]; do
|
|
case "$1" in
|
|
--config)
|
|
CONFIG_FILE="$2"
|
|
shift 2
|
|
;;
|
|
*)
|
|
# move on
|
|
shift
|
|
;;
|
|
esac
|
|
done
|
|
|
|
# restore original args after shifts
|
|
set -- $args
|
|
|
|
USE_1S1T_FORMAT="false"
|
|
if [[ -s "$CONFIG_FILE" ]]; then
|
|
USE_1S1T_FORMAT=$(jq -r '.use_1s1t_format' "$CONFIG_FILE")
|
|
fi
|
|
|
|
if test "$1" != 'write'
|
|
then
|
|
normalization_exit_code=0
|
|
elif test "$NORMALIZATION_TECHNIQUE" = 'LEGACY' && test "$USE_1S1T_FORMAT" != "true"
|
|
then
|
|
echo '{"type": "LOG","log":{"level":"INFO","message":"Starting in-connector normalization"}}'
|
|
# the args in a write command are `write --catalog foo.json --config bar.json`
|
|
# so if we remove the `write`, we can just pass the rest directly into normalization
|
|
/airbyte/entrypoint.sh run ${@:2} --integration-type $AIRBYTE_NORMALIZATION_INTEGRATION | java -cp "/airbyte/lib/*" io.airbyte.integrations.destination.normalization.NormalizationLogParser
|
|
normalization_exit_code=$?
|
|
echo '{"type": "LOG","log":{"level":"INFO","message":"In-connector normalization done (exit code '"$normalization_exit_code"')"}}'
|
|
else
|
|
echo '{"type": "LOG","log":{"level":"INFO","message":"Skipping in-connector normalization"}}'
|
|
normalization_exit_code=0
|
|
fi
|
|
|
|
if test $destination_exit_code -ne 0
|
|
then
|
|
exit $destination_exit_code
|
|
elif test $normalization_exit_code -ne 0
|
|
then
|
|
exit $normalization_exit_code
|
|
else
|
|
exit 0
|
|
fi
|