Instead of inserts, we write the data to S3 and issue a COPY command copying from S3 into redshift.
Use a single file as its sufficiently performant and we do not want to introduce file destination related operations yet.
Use an open source library for uploads as AWS does not natively support streaming loads.
My intention with this PR is to first implement the meat of the write-and-copy strategy. This is mainly centered around the RedshiftCopier class. I plan hook up the RedshiftCopier to the actual Destination class, and implement all the plumbing, in a follow up PR.
Co-authored-by: Davin Chia <davinchia@Davins-MacBook-Pro.local>
* Google directory source #2110 - creating new source
* Google Directory #2110 - implementing new source
* Google directory #2110 - handling rate limit
* Google Directory #2110 - handling errors and rate limits
* Google Directory #2110 - reformat
* Google Directory #2110 - adding CI credentials
* Google Directory #2110 - adding to the source definition registry
* Google Directory #2110 - adding to the source definition registry(fix)
* Google Directory #2110 - injecting the config into the build environment
* Update google-directory.md
* Update google-directory.md
* Google directory #2110 - rename max_results to results_per_page and increase it to 100, fixing setup.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
