Instead of inserts, we write the data to S3 and issue a COPY command copying from S3 into redshift.
Use a single file as its sufficiently performant and we do not want to introduce file destination related operations yet.
Use an open source library for uploads as AWS does not natively support streaming loads.
My intention with this PR is to first implement the meat of the write-and-copy strategy. This is mainly centered around the RedshiftCopier class. I plan hook up the RedshiftCopier to the actual Destination class, and implement all the plumbing, in a follow up PR.
Co-authored-by: Davin Chia <davinchia@Davins-MacBook-Pro.local>
* gives better visibility into the progress of phase of both the source and destination.
* 1 log per every 10K records in the source
* destination logs each step in destination including setting up tmp tables, creating final tables, cleaning up tmp tables
* support cached builds for base -> base-java -> snowflake
* use plugin for image building
* fix matching on COPY from
* remove docker.gradle
* tmp commit
* update connectors
* finish rest of build files
* fix ide errors
* more build fixes
* clean up
* clean up for new soruces
* fix spotless
* fix flake problems
* add recommended empty file
* python caching
* fixes upon review
* clean up docker and build test files
* clean up python
* clean up
* fix integration test dependencies
* fix standard tests
* fix
* remove symlink
* re-add requirements to fix normalizatioon build
* fix symlink
* fix dumbest build problem of all
* add missing integration test def
* fix missing dep
* remove class exclusion
* move trim so null source versions are allowed
* rename map
* fix hardcoded value
* remove unnecessary dep
* use dashes for salesforce package name
* fix typo
* DRY and fix test image name
* Fix edit
* assert string is not empty
* build integration test image only for integrationTest
* move code generator to tools and rename docker build tasks
* make source test depend on integration test build, not the other way
* remove guard because the docker build should exist before the integrationtest is applied
* remove comment
* DRY up airbyte-source-test
* fix plugin compilation
* add missing dependency
* rename getTaggedImage to getDevTaggedImage
* fix test vs main docker build bug
