Destination bigquery: rerelease 1s1t behind gate (#27936)
* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)""
This reverts commit 348c577dbb.
* version bumps+changelog
* Speed up BQ by having 2 queries, and not an OR (#27981)
* 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (#27856)
* only run t+d code if it's enabled
* dockerfile+changelog
* remove changelog entry
* Destinations V2: handle optional fields for `object` and `array` types (#27898)
* catch null schema
* fix null properties
* clean up
* consolidate + add more tests
* try catch
* empty json test
* Automated Commit - Formatting Changes
* remove todo
* destination bigquery: misc updates to 1s1t code (#28057)
* switch to checkedconsumer
* add unit test for buildColumnId
* use flag
* restructure prefix check
* fix build
* more type-parsing fixes (#28100)
* more type-parsing fixes
* handle duplicates
* Automated Commit - Format and Process Resources Changes
* add tests for asColumns
* Automated Commit - Format and Process Resources Changes
* log warnings instead of throwing exception
* better log message
* error level
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
* Automated Commit - Formatting Changes
* Improve protocol type parsing (#28126)
* Automated Commit - Formatting Changes
* Change from T&D every 10k records to an increasing time based interval (#28130)
* fifteen minute t&d
* add typing and deduping operation valve for increased intervals of typing and deduping
* Automated Commit - Format and Process Resources Changes
* resolve bizarre merge conflict
* Automated Commit - Format and Process Resources Changes
---------
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
* Simplify and speed up CDC delete support [DestinationsV2] (#28029)
* Simplify and speed up CDC delete support [DestinationsV2]
* better QUOTE
* spotbugs?
* recompile dbt image for local arch and use that when building images
* things compile, but tests fail
* tests working-ish
* comment
* fix logic to re-insert deleted records for cursor comparison.
tests pass!
* remove comment
* Skip CDC re-include logic if there are no CDC columns
* stop hardcoding pk (#28092)
* wip
* remove TODOs
---------
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
* update method name
* Automated Commit - Formatting Changes
* depend on pinned normalization version
* implement 1s1t DATs for destination-bigquery (#27852)
* intiial implementation
* Automated Commit - Formatting Changes
* add second sync to test
* do concurrent things
* Automated Commit - Formatting Changes
* clarify comment
* minor tweaks
* more stuff
* Automated Commit - Formatting Changes
* minor cleanup
* lots of fixes
* handle sql vs json null better
* verify extra columns
* only check deleted_at if in DEDUP mode and the column exists
* add full refresh append test case
* Automated Commit - Formatting Changes
* add tests for the remaining sync modes
* Automated Commit - Formatting Changes
* readability stuff
* Automated Commit - Formatting Changes
* add test for gcs mode
* remove static fields
* Automated Commit - Formatting Changes
* add more test cases, tweak test scaffold
* cleanup
* Automated Commit - Formatting Changes
* extract recorddiffer
* and use it in the sql generator test
* fix
* comment
* naming+comment
* one more comment
* better assert
* remove unnecessary thing
* one last thing
* Automated Commit - Formatting Changes
* enable concurrent execution on all java integration tests
* add test for default namespace
* Automated Commit - Formatting Changes
* implement a 2-stream test
* Automated Commit - Formatting Changes
* extract methods
* invert jsonNodesNotEquivalent
* Automated Commit - Formatting Changes
* fix conditional
* pull out diffSingleRecord
* Automated Commit - Formatting Changes
* handle nulls correctly
* remove raw-specific handling; break up methods
* Automated Commit - Formatting Changes
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>
* Destinations V2: move create raw tables earlier (#28255)
* move create raw tables
* better log message
* stop building normalization (#28256)
* fix ability to run tests
* disable incremental t+d for now
* Automated Commit - Formatting Changes
---------
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Cynthia Yin <cynthia@airbyte.io>
Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>
This commit is contained in:
Edward Gao
@@ -84,7 +84,7 @@ In the `docker-compose.debug.yaml` file you should see an entry for the `worker`
|
||||
Similar to the previous debugging example, we want to pass an environment variable to the `docker compose` command. This time we're setting the
|
||||
`DEBUG_CONTAINER_IMAGE` environment variable to the name of the container we're targeting. For our example that is `destination-postgres` so run the command:
|
||||
```bash
|
||||
DEBUG_CONTAINER_IMAGE="destination-postgres" VERSION="dev" docker compose -f docker-compose.yaml -f docker-compose.debug.yaml up
|
||||
DEBUG_CONTAINER_IMAGE="destination-postgres:5005" VERSION="dev" docker compose -f docker-compose.yaml -f docker-compose.debug.yaml up
|
||||
```
|
||||
The `worker` container now has an environment variable `DEBUG_CONTAINER_IMAGE` with a value of `destination-postgres` which when it compares when it is
|
||||
spawning containers. If the container name matches the environment variable, it will set the `JAVA_TOOL_OPTIONS` environment variable in the container to
|
||||
|
||||
Reference in New Issue
Block a user