jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-03 15:04:01 -05:00
Code Issues Packages Projects Releases Wiki Activity
6,265 Commits 5,149 Branches 518 Tags
e4d3d60ca8b683c301dfd5408917e1bb5e261cc0
Commit Graph

49 Commits

Author SHA1 Message Date
Tuhai Maksym
aa28d448d8 12708: Add an option to use encryption with staging in Redshift Destination (#13675) 
* 12708: Add an option to use encryption with staging in Redshift Destination

* 12708: docs/docker configs updated

* 12708: merge with master

* 12708: merge fix

* 12708: code review implementation

* 12708: fix for older configs

* 12708: fix for older configs in check

* 12708: merge from master (consolidation issue)

* 12708: versions updated
 2022-06-22 14:31:04 +03:00
VitaliiMaltsev
c283d9d159 Deprecate PART_SIZE_MB in connectors using S3/GCS storage (#13753) 
* Removed part_size from connectors that use StreamTransferManager

* fixed S3DestinationConfigTest

* fixed S3JsonlFormatConfigTest

* upadate changelog and bump version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* upadate changelog and bump version for Redshift and Snowflake destinations

* auto-bump connector version

* fix GCS staging test

* fix GCS staging test

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-06-20 14:56:33 +03:00
Alexander Tsukanov
b45014da6b 🎉 Destination Redshift: Add "Loading Method" option to Redshift Destination spec and UI (#13415) 
* airbyte-12709 Add "Loading Method" option to Redshift Destination spec.

* airbyte-12709: Fixed unit tests.

* airbyte-12709: Updated README.md.

* airbyte-12709: change the number of PR in redshift.md.

* airbyte-12709: Added backward compatibility for old json schema.

* airbyte-12709: Fix PR comments.

* airbyte-12709: Removed throwing an exception. Fixed PR comments.

* airbyte-12709: Bump the airbyte version.

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-06-10 14:47:03 +03:00
Marcos Marx
9aa0220820 revert redshift and json operations (#13465) 2022-06-03 19:44:40 -03:00
Adam
071a8e9dc8 Destination Redshift: fixed array contents verification for SUPER (#13069) 
* add array handling to json flatten

* bump version, add changelog entry

* test: add flatten unit tests

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-06-03 15:46:16 -03:00
Alexander Tsukanov
5371f87ee6 🐛 Destination Redshift: Fix DATs blind spot - connection check (#13090) 
* airbyte-12459: Removed RedshiftDataTmpTableMode.java as not needed enum. Code refactoring and clean-up.

* airbyte-12459: Removed RedshiftDataTmpTableMode.java as not needed enum. Code refactoring and clean-up.

* airbyte-12459: Merged master.

* airbyte-12459: Merged master.

* auto-bump connector version

Co-authored-by: Oleksandr Sheheda <alexandr-shegeda@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-25 19:14:10 +03:00
Oleksandr Sheheda
44a43f0338 Improve check connection performance for all JDBC-related DBs (#12820) 
* removed redundant read permission check for destination

* bump docker image versions, update definitions and changelog

* bump docker image versions, update definitions and changelog

* update changelog

* fix for MSSQL and Oracle tests

* fix for MSSQL strict encrypt

* fixed conflicts

* revert versions that were not published
 2022-05-24 16:06:01 +03:00
Adam
9e31d026c8 Destination Redshift: Fixed maximum record size for SUPER type (#12940) 
* fix #12680: use correct max size for SUPER type

* address review comments

* docs: add changelog entry

* bump version

* test: make large record actually too big

* fix: use a method instead of member variable for correct override

* fix: if overall object is invalid, don't check individual elements

* feat: add unit test for isValidData

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-20 10:32:21 -04:00
VitaliiMaltsev
4804f746dc Destination Redshift fixed NPE in S3 staging (#12869) 
* Destination Snowflake fixed npe in S3 staging

* removed star imports

* updated changelog

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-16 16:37:28 +03:00
VitaliiMaltsev
4b5bd2b5a4 Redshift Destination Apply buffering strategy (#12601) 
* Redshift Destination: Apply buffering strategy

* add manifest uploading

* refactoring

* fixed checkstyle

* updated CHANGELOG

* removed redundant static

* airbyte-12265: Added  stagingOperations.onDestinationCloseOperations() to StagingConsumerFactory.java.

* airbyte-12265: Created operations and copiers java packages.

* safe delete of RedshiftCopyS3Destination.java

* rename tests

* bump version

* auto-bump connector version

Co-authored-by: Oleksandr Tsukanov <alexander.tsukanovvv@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-12 12:30:54 +03:00
VitaliiMaltsev
b16e13e9cf Redshift Destination: update spec (#12100) 
* Redshift Destination: update spec

* update spec.json

* update links in spec.json

* added more links to spec.json | refactoring

* updated docs with stadard connector template

* added hyperlink to documentation for part_size field
 2022-04-27 14:01:26 +03:00
Yurii Bidiuk
785bcc4a9a 🐛 Destination Redshift: fix switching mode (#12085) 
* fix switching mode for redshift

* bump version

* format code

* update spec
 2022-04-20 16:57:15 +03:00
Serhii Chvaliuk
7023fbd48e Redshift SUPER type (#12064) 
* 🎉 Destination Redshift: Use SUPER data type on Redshift destination for raw JSON data (#9407)

Co-authored-by: Oleksandr Tsukanov <alexander.tsukanovvv@gmail.com>
Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-04-20 15:11:22 +03:00
Edward Gao
c1381cde2c Revert Redshift SUPER PRs (#12041) 2022-04-14 12:36:26 -07:00
Marcos Marx
538dec4a37 Docs Redshift: remove duplicate section (#11990) 2022-04-14 13:53:00 -03:00
Jared Rhizor
14dfc1d417 fix parentheses and info/warning/danger formatting in docs (#11981) 
* fix parens

* fix formatting for hints

* remove hint from non-docusaurus or gitbook doc
 2022-04-13 12:08:34 -07:00
Alexander Tsukanov
674221c07a 🎉 Destination Redshift: Use SUPER data type on Redshift destination for raw JSON data (#9407) 
* airbyte-5050: Added support of SUPER datatype for destination-redshift on the Java side

Co-authored-by: Oleksandr Tsukanov <alexander.tsukanovvv@gmail.com>
Co-authored-by: user <user@HRK1-LMC-A13537.local>
Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
 2022-04-12 16:33:26 +03:00
VitaliiMaltsev
a0b7dd0289 🐛 Destination Redshift and Postgres handle custom namespace with '-' (#11729) 
* fix for jdk 17

* add CHANGELOG

* JdbcBufferedConsumerFactory resolve namespace with dash

* add CHANGELOG

* bump versions

* bump postgres version

* update destination-jdbc version

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2022-04-07 13:08:57 +03:00
VitaliiMaltsev
9b7b46e321 🐛 Destination Snowflake and Redshift : missing records when using S3 staging (#11254) 
* fix for jdk 17

* fixed bug with missing records during S3 staging

* test

* add CHANGELOG

* add assertion using all staging files

* bump redshift version

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2022-03-18 17:02:23 +02:00
girarda
5d8cb41150 Refactor to enable support for optional JDBC parameters for all JDBC destinations (#10421) 
* refactoring to allow testing

* MySQLDestination uses connection property map instead of url arguments

* Update jdbc destinations

* A little more generic

* reset to master

* reset to master

* move to jdbcutils

* Align when multiline

* Align when multiline

* Update postgres to use property map

* Move tests to AbstractJdbcDestinationTest

* clean

* Align when multiline

* return property map

* Add postgres tests

* update clickhouse

* reformat

* reset

* reformat

* fix test

* reformat

* fix bug

* Add mssql tests

* refactor test

* fix oracle destination test

* oracle tests

* fix redshift acceptance test

* Pass string

* Revert "Pass string"

This reverts commit 697821738c.

* Double deserialization

* Revert "Double deserialization"

This reverts commit ee8d75245b.

* try updating json_operations

* Revert "try updating json_operations"

This reverts commit c8022c2994.

* json parse

* Revert "json parse"

This reverts commit 11a6725eaa.

* Revert "Revert "Double deserialization""

This reverts commit 213f47acc4.

* Revert "Revert "Revert "Double deserialization"""

This reverts commit 66822454af.

* move to constant

* Add comment

* map can be constant

* Add comment

* move map

* hide in method

* no need to create new map

* no need to create new map

* no need to create new map

* enably mysql test

* Update changelogs

* Update changelog

* update changelog

* Bump versions

* bump version

* disable dbt support

* update spec

* update other oracle tests

* update doc

* bump seed

* fix source test

* update seed spec file

* fix expected spec
 2022-02-28 09:25:04 -08:00
bmcgilli
3386cbe9ce Update redshift.md (#10567) 2022-02-22 17:00:31 -08:00
andriikorotkov
b3916c987a 🐛 Snowflake Destination: use better file size with S3 staging files (#9920) 
* split s3 staging files to files by 100 Mb and removed legacyS3StreamCopier

* split s3 staging files to files by 100 Mb and removed legacyS3StreamCopier

* updated code style

* fix remarks

* fix remarks

* fix code style

* fix remarks

* fix remarks

* fix remarks

* updated documentations and images versions

* updated documentation
 2022-02-15 22:20:22 +02:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00
Edward Gao
86e08d0fa4 🎉 Destination Redshift: add option to enable/disable deleting staging data (#8855) 2021-12-17 12:42:09 -08:00
Edward Gao
7038533ae0 🎉 Destination Redshift (copy): accept bucket path for staging data (#8607) 2021-12-16 16:41:22 -08:00
Edward Gao
0b8c0f5dff Bump connector versions for publishing (#8717) 
version bumps for https://github.com/airbytehq/airbyte/pull/8562
 2021-12-10 19:27:26 -08:00
Sherif A. Nada
efb5151011 🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering (#7719) 2021-11-08 19:26:32 -08:00
VitaliiMaltsev
0c932749d4 🎉 Redshift Source and Destination set SSL as default option (#7234) 
* Redshift Source and Restination set SSL as default option

* add changelog

* remove SSL test| add more documentation

* bump new version

* bump new version

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2021-10-22 12:28:54 +03:00
VitaliiMaltsev
ae9048cfaa 🎉 Redshift Source/Destination SSL Support (#6965) 
* add tls option to spec

* Redshift Source add acceptance test

* Redshift Destination add ssl field to spec

* add RedshiftDestinationAcceptanceTestSSL

* fix checkstyle

* added changelog

* update docs

* bump versions of Redshift Source and Destination \ changed default tls to true

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2021-10-14 12:29:00 +03:00
andriikorotkov
80c9ed2b02 🐛 AWS S3 Staging COPY is writing records from different table in the same raw table (#6949) 
* updated jdbc destination

* updated snowflake and redshift destination version

* updated documentation

* updated documentation

* updated prefix length for snowflake and redshift streams

* fixed remarks

* updated new redshift version
 2021-10-14 01:04:54 +03:00
Abhi Vaidyanatha
ae32ecbb27 GitBook: [master] 186 pages and 77 assets modified 2021-10-08 21:17:47 +00:00
Sherif A. Nada
b050f65a92 fix redshift docs formatting 2021-09-23 19:07:28 -07:00
Abhi Vaidyanatha
d732eaf80e Add information on which destinations support Incremental - Deduped History in their docs (#6031) 
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-09-16 16:40:12 -07:00
andriikorotkov
71f51e0c36 🐛 Snowflake destination: snowflake s3 destination COPY is writing records from different table in the same raw table fix (#5924) 
* snowflake s3 destination COPY is writing records from different table in the same raw table fix

* updated snowflake s3 file name

* updated snowflake documentation

* updated snowflake documentation

* updated snowflake documentation

* updated code style

* updated code style

* updated redshift destination
 2021-09-13 10:31:18 +03:00
Sherif A. Nada
f3f12b2a14 publish pr 5745 (#5825) 2021-09-02 18:50:23 -07:00
Sherif A. Nada
356ca18b67 🐛 Fix Oracle spec to declare sid instead of database param, Redshift to allow additionalProperties, MSSQL test and spec to declare spec type correctly (#4874) 2021-07-20 17:04:36 -07:00
Abhi Vaidyanatha
3b2009723e Add note about collocating Redshift cluster and Airbyte server in the same security group (#4490) 
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-07-02 12:42:35 -07:00
Marcos Marx
7c70eb02cd GitBook: [master] 84 pages and 72 assets modified 2021-05-03 12:14:16 +00:00
Christophe Duong
77ffd74b32 Ignore records that are too big in Redshift destinations (instead of failing) (#2988) 
* Abort sync if one of the part fails to copy to temp table

* Check for record size when copying data from s3 to redshift

* Handle big record in RedshiftInsertDestination too
 2021-04-30 21:04:03 +02:00
Davin Chia
e63ab8490a Make Redshift part size configurable. (#3053) 
The Redshift Copy strategy currently has it's part size set to 10 MB. Since S3 allows a file to be broken into max of 10k parts, this results in a 100GB table limit. A user is trying to sync a table of 115GB and running into this issue.

This makes the part size configurable so users can increase this size if needed.
 2021-04-27 13:35:35 +08:00
Davin Chia
42686add8a Release connectors with namespace change. (#2990) 
Release all connectors affected by namespace change. Includes all JDBC sources and destinations.

Also add documentation for normalisation. Prerequisite to actually releasing 0.21.0-alpha.
 2021-04-21 11:35:08 +08:00
Michel Tricot
e4326aa21e Revert "GitBook: [master] 25 pages and 46 assets modified" 
This reverts commit 0b4eddc7cf.
 2021-04-16 14:20:42 -07:00
Michel Tricot
0b4eddc7cf GitBook: [master] 25 pages and 46 assets modified 2021-04-16 20:29:16 +00:00
Davin Chia
e8c13cf0fe Update documentation to include Redshift COPY strategy. (#2728) 2021-04-06 09:10:19 +08:00
Christophe Duong
194d6cb5d0 Add metadata prefix to destination internal columns (#1708) 
* Add metadata prefix to destination internal columns

* Bumpversion all destinations for new normalization behavior

* Fix acceptance tests
 2021-01-19 17:21:15 +01:00
Charles
01cdeafdb1 release v0.8.0-alpha (#1339) 2020-12-16 14:01:57 -08:00
John Lafleur
6668a514f4 GitBook: [master] 10 pages and 335 assets modified 2020-12-04 06:42:08 +00:00
Christophe Duong
d06392e900 Redshift Destination & refactoring to introduce destination-jdbc 
Closes #193
Closes #1126
 2020-12-03 18:07:46 +01:00
Christophe Duong
206d3cbea8 Naming conventions managed in destinations (#1060) 
Handled field and table identifier conversions as necessary in order to successfully in destinations
 2020-11-25 18:53:23 +01:00