jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 11:01:41 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

163 Commits

Author SHA1 Message Date
Edward Gao
f90ffb0a12 Destination Redshift: handle empty s3 bucket_path (#18434) 
* make bucket_path required

* check for nonempty path

* not required

* handle null/empty path

* version bump + changelog

* fix comment

* auto-bump connector version

* format

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-26 11:04:10 -07:00
Yurii Bidiuk
10e4aae891 Use LOG4J2 to wrap connectors logs to JSON format (#15668) 
* Use LOG4J2 to wrap connectors logs to JSON format

* log connector's stack trace directly as a message

* add stack_trace field to json template

* bump versions

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* revert versions for destinations: postgres, mssql, mysql

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-09-14 23:27:13 +03:00
Liren Tu
fd547c64d7 🐛 Avro related destinations: fix field name clashing with combined restrict (#16243) 
* Add test case for allOf and oneOf

* Bump version

* Add pr id

* Bump gcs version

* Bump dest jdbc

* Bump redshift

* Bump snowflake

* Bump databricks

* Bump bigquery

* Revert "Bump dest jdbc"

This reverts commit f10497e96a.

* Use a new pat to avoid api rate limit

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* Revert databricks bump

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-09-01 18:06:27 -07:00
Yevhen Sukhomud
5773c224bc Destination Redshift: custom JDBC parameters (#16208) 2022-09-01 18:09:04 +07:00
Jagruti Tiwari
8288c16485 fix: replace airbyte oss with airbyte open source (#15885) 
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2022-08-24 01:01:53 -03:00
Alexander Tsukanov
82297a9b18 🎉Redshift, Databricks, Snowflake, S3 Destinations: Make S3 output filename configurable (#14494) 
airbyte-4184: Make S3 output filename configurable
 2022-07-15 15:56:21 +03:00
Topher Lubaway
9c6c092a22 Revert "Improving docusaurus sidebar generation (#1927) (#14369)" (#14596) 
This reverts commit a2c194a11f.
 2022-07-11 15:27:14 -05:00
Mykyta Serbynevskiy
a2c194a11f Improving docusaurus sidebar generation (#1927) (#14369) 
* Improving docusaurus sidebar generation (#1927)

* Added "Career & open positions" folder to sidebar, adjusted "Project overview" folder

* Deleted "career-and-open-positions" folder from sidebar
 2022-07-08 14:18:27 -05:00
Adam
79f7d060db 🐛 destination-redshift: use s3 bucket path for s3 cleanup (#14190) 2022-06-27 23:19:26 +02:00
Adam
16d99b9b0c 🐛 Destination Redshift: use s3 bucket path for s3 staging operations (#13916) 2022-06-27 20:02:45 +02:00
VitaliiMaltsev
11ea066f47 Remove "additionalProperties": false from specs for connectors with staging (#14114) 
* Remove "additionalProperties": false from spec for connectors with staging

* Remove "additionalProperties": false from spec for Redshift destination

* bump versions

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-06-24 20:04:04 +03:00
Alexander Tsukanov
01a366ca57 🐛 Destination Redshift: Improved discovery for redshift-destination not SUPER streams (#13690) 
airbyte-12843: Improved discovery for redshift-destination not SUPER tables, excluded views from discovery.
 2022-06-24 17:50:17 +03:00
Tuhai Maksym
6ca0e53469 12708: Add an option to use encryption with staging in Redshift Desti… (#14013) 
* 12708: Add an option to use encryption with staging in Redshift Destination (#13675)

* 12708: Add an option to use encryption with staging in Redshift Destination

* 12708: docs/docker configs updated

* 12708: merge with master

* 12708: merge fix

* 12708: code review implementation

* 12708: fix for older configs

* 12708: fix for older configs in check

* 12708: merge from master (consolidation issue)

* 12708: versions updated

* 12708: specs updated

* 12708: specs updated

* 12708: removing autogenerated files from PR

* 12708: changelog updated

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-06-24 16:07:28 +03:00
Tuhai Maksym
3a179a822b Revert "12708: Add an option to use encryption with staging in Redshift Destination (#13675)" (#14010) 
This reverts commit aa28d448d8.
 2022-06-22 17:02:18 +03:00
Tuhai Maksym
aa28d448d8 12708: Add an option to use encryption with staging in Redshift Destination (#13675) 
* 12708: Add an option to use encryption with staging in Redshift Destination

* 12708: docs/docker configs updated

* 12708: merge with master

* 12708: merge fix

* 12708: code review implementation

* 12708: fix for older configs

* 12708: fix for older configs in check

* 12708: merge from master (consolidation issue)

* 12708: versions updated
 2022-06-22 14:31:04 +03:00
VitaliiMaltsev
c283d9d159 Deprecate PART_SIZE_MB in connectors using S3/GCS storage (#13753) 
* Removed part_size from connectors that use StreamTransferManager

* fixed S3DestinationConfigTest

* fixed S3JsonlFormatConfigTest

* upadate changelog and bump version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* upadate changelog and bump version for Redshift and Snowflake destinations

* auto-bump connector version

* fix GCS staging test

* fix GCS staging test

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-06-20 14:56:33 +03:00
Alexander Tsukanov
b45014da6b 🎉 Destination Redshift: Add "Loading Method" option to Redshift Destination spec and UI (#13415) 
* airbyte-12709 Add "Loading Method" option to Redshift Destination spec.

* airbyte-12709: Fixed unit tests.

* airbyte-12709: Updated README.md.

* airbyte-12709: change the number of PR in redshift.md.

* airbyte-12709: Added backward compatibility for old json schema.

* airbyte-12709: Fix PR comments.

* airbyte-12709: Removed throwing an exception. Fixed PR comments.

* airbyte-12709: Bump the airbyte version.

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-06-10 14:47:03 +03:00
Marcos Marx
9aa0220820 revert redshift and json operations (#13465) 2022-06-03 19:44:40 -03:00
Adam
071a8e9dc8 Destination Redshift: fixed array contents verification for SUPER (#13069) 
* add array handling to json flatten

* bump version, add changelog entry

* test: add flatten unit tests

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-06-03 15:46:16 -03:00
Alexander Tsukanov
5371f87ee6 🐛 Destination Redshift: Fix DATs blind spot - connection check (#13090) 
* airbyte-12459: Removed RedshiftDataTmpTableMode.java as not needed enum. Code refactoring and clean-up.

* airbyte-12459: Removed RedshiftDataTmpTableMode.java as not needed enum. Code refactoring and clean-up.

* airbyte-12459: Merged master.

* airbyte-12459: Merged master.

* auto-bump connector version

Co-authored-by: Oleksandr Sheheda <alexandr-shegeda@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-25 19:14:10 +03:00
Oleksandr Sheheda
44a43f0338 Improve check connection performance for all JDBC-related DBs (#12820) 
* removed redundant read permission check for destination

* bump docker image versions, update definitions and changelog

* bump docker image versions, update definitions and changelog

* update changelog

* fix for MSSQL and Oracle tests

* fix for MSSQL strict encrypt

* fixed conflicts

* revert versions that were not published
 2022-05-24 16:06:01 +03:00
Adam
9e31d026c8 Destination Redshift: Fixed maximum record size for SUPER type (#12940) 
* fix #12680: use correct max size for SUPER type

* address review comments

* docs: add changelog entry

* bump version

* test: make large record actually too big

* fix: use a method instead of member variable for correct override

* fix: if overall object is invalid, don't check individual elements

* feat: add unit test for isValidData

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-20 10:32:21 -04:00
VitaliiMaltsev
4804f746dc Destination Redshift fixed NPE in S3 staging (#12869) 
* Destination Snowflake fixed npe in S3 staging

* removed star imports

* updated changelog

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-16 16:37:28 +03:00
VitaliiMaltsev
4b5bd2b5a4 Redshift Destination Apply buffering strategy (#12601) 
* Redshift Destination: Apply buffering strategy

* add manifest uploading

* refactoring

* fixed checkstyle

* updated CHANGELOG

* removed redundant static

* airbyte-12265: Added  stagingOperations.onDestinationCloseOperations() to StagingConsumerFactory.java.

* airbyte-12265: Created operations and copiers java packages.

* safe delete of RedshiftCopyS3Destination.java

* rename tests

* bump version

* auto-bump connector version

Co-authored-by: Oleksandr Tsukanov <alexander.tsukanovvv@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-12 12:30:54 +03:00
VitaliiMaltsev
b16e13e9cf Redshift Destination: update spec (#12100) 
* Redshift Destination: update spec

* update spec.json

* update links in spec.json

* added more links to spec.json | refactoring

* updated docs with stadard connector template

* added hyperlink to documentation for part_size field
 2022-04-27 14:01:26 +03:00
Yurii Bidiuk
785bcc4a9a 🐛 Destination Redshift: fix switching mode (#12085) 
* fix switching mode for redshift

* bump version

* format code

* update spec
 2022-04-20 16:57:15 +03:00
Serhii Chvaliuk
7023fbd48e Redshift SUPER type (#12064) 
* 🎉 Destination Redshift: Use SUPER data type on Redshift destination for raw JSON data (#9407)

Co-authored-by: Oleksandr Tsukanov <alexander.tsukanovvv@gmail.com>
Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-04-20 15:11:22 +03:00
Edward Gao
c1381cde2c Revert Redshift SUPER PRs (#12041) 2022-04-14 12:36:26 -07:00
Marcos Marx
538dec4a37 Docs Redshift: remove duplicate section (#11990) 2022-04-14 13:53:00 -03:00
Jared Rhizor
14dfc1d417 fix parentheses and info/warning/danger formatting in docs (#11981) 
* fix parens

* fix formatting for hints

* remove hint from non-docusaurus or gitbook doc
 2022-04-13 12:08:34 -07:00
Alexander Tsukanov
674221c07a 🎉 Destination Redshift: Use SUPER data type on Redshift destination for raw JSON data (#9407) 
* airbyte-5050: Added support of SUPER datatype for destination-redshift on the Java side

Co-authored-by: Oleksandr Tsukanov <alexander.tsukanovvv@gmail.com>
Co-authored-by: user <user@HRK1-LMC-A13537.local>
Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
 2022-04-12 16:33:26 +03:00
VitaliiMaltsev
a0b7dd0289 🐛 Destination Redshift and Postgres handle custom namespace with '-' (#11729) 
* fix for jdk 17

* add CHANGELOG

* JdbcBufferedConsumerFactory resolve namespace with dash

* add CHANGELOG

* bump versions

* bump postgres version

* update destination-jdbc version

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2022-04-07 13:08:57 +03:00
VitaliiMaltsev
9b7b46e321 🐛 Destination Snowflake and Redshift : missing records when using S3 staging (#11254) 
* fix for jdk 17

* fixed bug with missing records during S3 staging

* test

* add CHANGELOG

* add assertion using all staging files

* bump redshift version

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2022-03-18 17:02:23 +02:00
girarda
5d8cb41150 Refactor to enable support for optional JDBC parameters for all JDBC destinations (#10421) 
* refactoring to allow testing

* MySQLDestination uses connection property map instead of url arguments

* Update jdbc destinations

* A little more generic

* reset to master

* reset to master

* move to jdbcutils

* Align when multiline

* Align when multiline

* Update postgres to use property map

* Move tests to AbstractJdbcDestinationTest

* clean

* Align when multiline

* return property map

* Add postgres tests

* update clickhouse

* reformat

* reset

* reformat

* fix test

* reformat

* fix bug

* Add mssql tests

* refactor test

* fix oracle destination test

* oracle tests

* fix redshift acceptance test

* Pass string

* Revert "Pass string"

This reverts commit 697821738c.

* Double deserialization

* Revert "Double deserialization"

This reverts commit ee8d75245b.

* try updating json_operations

* Revert "try updating json_operations"

This reverts commit c8022c2994.

* json parse

* Revert "json parse"

This reverts commit 11a6725eaa.

* Revert "Revert "Double deserialization""

This reverts commit 213f47acc4.

* Revert "Revert "Revert "Double deserialization"""

This reverts commit 66822454af.

* move to constant

* Add comment

* map can be constant

* Add comment

* move map

* hide in method

* no need to create new map

* no need to create new map

* no need to create new map

* enably mysql test

* Update changelogs

* Update changelog

* update changelog

* Bump versions

* bump version

* disable dbt support

* update spec

* update other oracle tests

* update doc

* bump seed

* fix source test

* update seed spec file

* fix expected spec
 2022-02-28 09:25:04 -08:00
bmcgilli
3386cbe9ce Update redshift.md (#10567) 2022-02-22 17:00:31 -08:00
andriikorotkov
b3916c987a 🐛 Snowflake Destination: use better file size with S3 staging files (#9920) 
* split s3 staging files to files by 100 Mb and removed legacyS3StreamCopier

* split s3 staging files to files by 100 Mb and removed legacyS3StreamCopier

* updated code style

* fix remarks

* fix remarks

* fix code style

* fix remarks

* fix remarks

* fix remarks

* updated documentations and images versions

* updated documentation
 2022-02-15 22:20:22 +02:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00
Edward Gao
86e08d0fa4 🎉 Destination Redshift: add option to enable/disable deleting staging data (#8855) 2021-12-17 12:42:09 -08:00
Edward Gao
7038533ae0 🎉 Destination Redshift (copy): accept bucket path for staging data (#8607) 2021-12-16 16:41:22 -08:00
Edward Gao
0b8c0f5dff Bump connector versions for publishing (#8717) 
version bumps for https://github.com/airbytehq/airbyte/pull/8562
 2021-12-10 19:27:26 -08:00
Sherif A. Nada
efb5151011 🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering (#7719) 2021-11-08 19:26:32 -08:00
VitaliiMaltsev
0c932749d4 🎉 Redshift Source and Destination set SSL as default option (#7234) 
* Redshift Source and Restination set SSL as default option

* add changelog

* remove SSL test| add more documentation

* bump new version

* bump new version

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2021-10-22 12:28:54 +03:00
VitaliiMaltsev
ae9048cfaa 🎉 Redshift Source/Destination SSL Support (#6965) 
* add tls option to spec

* Redshift Source add acceptance test

* Redshift Destination add ssl field to spec

* add RedshiftDestinationAcceptanceTestSSL

* fix checkstyle

* added changelog

* update docs

* bump versions of Redshift Source and Destination \ changed default tls to true

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2021-10-14 12:29:00 +03:00
andriikorotkov
80c9ed2b02 🐛 AWS S3 Staging COPY is writing records from different table in the same raw table (#6949) 
* updated jdbc destination

* updated snowflake and redshift destination version

* updated documentation

* updated documentation

* updated prefix length for snowflake and redshift streams

* fixed remarks

* updated new redshift version
 2021-10-14 01:04:54 +03:00
Abhi Vaidyanatha
ae32ecbb27 GitBook: [master] 186 pages and 77 assets modified 2021-10-08 21:17:47 +00:00
Sherif A. Nada
b050f65a92 fix redshift docs formatting 2021-09-23 19:07:28 -07:00
Abhi Vaidyanatha
d732eaf80e Add information on which destinations support Incremental - Deduped History in their docs (#6031) 
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-09-16 16:40:12 -07:00
andriikorotkov
71f51e0c36 🐛 Snowflake destination: snowflake s3 destination COPY is writing records from different table in the same raw table fix (#5924) 
* snowflake s3 destination COPY is writing records from different table in the same raw table fix

* updated snowflake s3 file name

* updated snowflake documentation

* updated snowflake documentation

* updated snowflake documentation

* updated code style

* updated code style

* updated redshift destination
 2021-09-13 10:31:18 +03:00
Sherif A. Nada
f3f12b2a14 publish pr 5745 (#5825) 2021-09-02 18:50:23 -07:00
Sherif A. Nada
356ca18b67 🐛 Fix Oracle spec to declare sid instead of database param, Redshift to allow additionalProperties, MSSQL test and spec to declare spec type correctly (#4874) 2021-07-20 17:04:36 -07:00