jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-22 03:21:25 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

2421 Commits

Author SHA1 Message Date
LiRen Tu
e00f487d98 📝 Destination bigquery: mark service account as required for cloud (#12768) 
* Update spec

* Update doc

* Bump version and update changelog

* Modify wording

* Add sample service account key json

* Add screenshots and common permission issues

* Refactor service account helper method

* Update log message

* Update version date in changelog

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-15 01:28:36 -07:00
Andy
a3658ba13a Update spec to replace markdown (#12797) 
* Update spec to replace markdown

* Update version

* S3 Destination: Update change log

* auto-bump connector version

Co-authored-by: oneshcheret <sashkaliife@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-12 19:16:00 +03:00
George Claireaux
440f3db8bb Destinations BigQuery & Snowflake: use latest base-java to emit AirbyteTraceMessage on errros (#12805) 
* bumping BQ and Snowflake dests to get AirbyteTraceMessage from base-java

* add PR link to changelogs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-12 14:23:29 +01:00
VitaliiMaltsev
4b5bd2b5a4 Redshift Destination Apply buffering strategy (#12601) 
* Redshift Destination: Apply buffering strategy

* add manifest uploading

* refactoring

* fixed checkstyle

* updated CHANGELOG

* removed redundant static

* airbyte-12265: Added  stagingOperations.onDestinationCloseOperations() to StagingConsumerFactory.java.

* airbyte-12265: Created operations and copiers java packages.

* safe delete of RedshiftCopyS3Destination.java

* rename tests

* bump version

* auto-bump connector version

Co-authored-by: Oleksandr Tsukanov <alexander.tsukanovvv@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-12 12:30:54 +03:00
Baz
feb0d2f378 🎉 New Destination: Implement Destination Google Sheets using CDK (#12135) 2022-05-12 01:10:28 +03:00
Stéphane Kattoor
66d760955d Fix typo in spec.json for destination AWS Datalake & fix python license plugin setup (#11811) 2022-05-06 01:28:00 -07:00
LiRen Tu
bd59726668 Publish new connectors to log offending json fields in json to avro conversion for debugging (#12578) 
* Publish new connectors to log debugging info in json to avro conversion

* Add pull request id

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-04 11:47:07 -07:00
Edward Gao
43470a2a87 🎉 Destination Snowflake: Add option to stage encrypted files via S3; fix purge_staging_data (#12452) 2022-05-04 07:31:53 -07:00
Alexandre Girard
1021428998 Destination Bigquery: Update Dataset location field description (#12528) 
* Update field description

* reset

* Bump

* reset

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-02 17:45:36 -07:00
Alexandre Girard
553024780f Destination Bigquery: Dataset location is a required field (#12477) 
* Dataset location is required

* Update doc

* Bump versions

* reset

* Bump

* downgrade

* Revert "downgrade"

This reverts commit b482c54d6f.

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-04-29 15:34:43 -07:00
oneshcheret
fd7f21e071 S3 destination: updating docs regarding certification (#11966) 
* S3 destination: updating docs regarding certification

* S3 destination: updating docs by new template

* Apply suggestions from code review

Co-authored-by: Andy <andy@airbyte.io>

* S3 destination: updating docs by new template

Co-authored-by: Andy <andy@airbyte.io>
 2022-04-29 18:17:13 +03:00
Eugene
6565315a04 Destination-Postgres: updated postgres.md documentation as per BETA template (#12279) 
* Updated postgres.md documentation as per BETA template
 2022-04-29 09:55:11 +03:00
VitaliiMaltsev
b16e13e9cf Redshift Destination: update spec (#12100) 
* Redshift Destination: update spec

* update spec.json

* update links in spec.json

* added more links to spec.json | refactoring

* updated docs with stadard connector template

* added hyperlink to documentation for part_size field
 2022-04-27 14:01:26 +03:00
Greg Solovyev
597d6ccbb8 Reconcile versions in git with versions in docker hub (#12368) 2022-04-26 11:41:59 -07:00
Greg Solovyev
cce3b21506 Manually bump destination-mssql version (#12291) 2022-04-26 08:52:46 -07:00
Greg Solovyev
bcf4f89ae8 Manually bump destination-oracle version (#12292) 2022-04-26 08:50:50 -07:00
noahkawasaki-airbyte
d46058e456 🎉 Destination Postgres: Add jdbc_url_params input (#12195) 
* Add jdbc_url_params input for Destination Postgres
 2022-04-25 08:12:31 -07:00
Greg Solovyev
63f6fc5712 Update destination-postgres version in Dockerfile and docs (#12290) 
* Update destination-postgres version in Dockerfile and docs

* Manually bump destination-postgres version
 2022-04-22 18:44:52 -07:00
LiRen Tu
9a0442c143 🎉 Destination s3 / gcs: add option for uncompressed csv and jsonl format (#12167) 
* Add gzip compression option

* Add file extension method to s3 format config

* Pass gzip compression to serialized buffer

* Add unit test

* Format code

* Update integration test

* Bump version and update doc

* Fix unit test

* Add extra gzip tests for csv and jsonl

* Make compression an oneOf param

* Migrate csv config to new compression spec

* Migrate jsonl config to new compression spec

* Update docs

* Fix unit test

* Fix integration tests

* Format code

* Bump version

* auto-bump connector version

* Bump gcs version in seed

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-04-22 15:38:28 -07:00
George Claireaux
e57ac9e4bb Destination GCS & S3: bump version (#12274) 
* bump version for GCS & S3 Destination

* auto-bump connector version

* gcs version bump

* Update docs

Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
 2022-04-22 15:10:53 -07:00
Yurii Bidiuk
785bcc4a9a 🐛 Destination Redshift: fix switching mode (#12085) 
* fix switching mode for redshift

* bump version

* format code

* update spec
 2022-04-20 16:57:15 +03:00
Serhii Chvaliuk
7023fbd48e Redshift SUPER type (#12064) 
* 🎉 Destination Redshift: Use SUPER data type on Redshift destination for raw JSON data (#9407)

Co-authored-by: Oleksandr Tsukanov <alexander.tsukanovvv@gmail.com>
Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-04-20 15:11:22 +03:00
tirth7777777
079e8b09b7 Source/Destination Kafka: added support for PLAIN authentication (#12134) 
* added support for PLAIN authentication for source-kafka and destination-kafka

* update doc

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
 2022-04-19 16:02:12 -03:00
Sreeram Venkitesh
893b4e4c7a Docs: Fixed typo in postgres destination doc (#12106) 2022-04-18 18:24:42 -03:00
VitaliiMaltsev
0f8e11e243 🐛 BigQuery Destination failed conditional binding (#12068) 
* BigQuery Destination check conditional binding

* fixed checkstyle

* BigQuery Destination fixed bug with contitional binding

* update CHANGELOG

* update CHANGELOG

* add explaining comment

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-04-15 19:17:46 +03:00
Eugene
281e0eec35 🎉Destination-bigquery-denorm: fixed emittedAt timestamp (#11978) 
* [11067] Destination-bigquery-denorm: fixed emittedAt timestamp
 2022-04-15 17:39:51 +03:00
andriikorotkov
4ac5d3c8e4 updated docs (#11840) 2022-04-15 17:06:48 +03:00
Edward Gao
c1381cde2c Revert Redshift SUPER PRs (#12041) 2022-04-14 12:36:26 -07:00
Marcos Marx
538dec4a37 Docs Redshift: remove duplicate section (#11990) 2022-04-14 13:53:00 -03:00
Jared Rhizor
14dfc1d417 fix parentheses and info/warning/danger formatting in docs (#11981) 
* fix parens

* fix formatting for hints

* remove hint from non-docusaurus or gitbook doc
 2022-04-13 12:08:34 -07:00
Alexander Tsukanov
674221c07a 🎉 Destination Redshift: Use SUPER data type on Redshift destination for raw JSON data (#9407) 
* airbyte-5050: Added support of SUPER datatype for destination-redshift on the Java side

Co-authored-by: Oleksandr Tsukanov <alexander.tsukanovvv@gmail.com>
Co-authored-by: user <user@HRK1-LMC-A13537.local>
Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
 2022-04-12 16:33:26 +03:00
LiRen Tu
8bd2d9b518 🎉 BigQuery destination: use serialized buffer for gcs staging (#11776) 
* Rebase bigquery changes to master

* Add comments

* Uncomment test code

* Format code

* Bump versions

* Fix denormalized destination target table name

* Fix avro schema for denormalized destination

* Remove unnecessary params from consumer factory

* Add back previous version

* Add warning about standard mode

* auto-bump connector version

* Bump version for bigquery in seed

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-04-07 16:59:19 -07:00
Topher Lubaway
c8337ea3ba Add docs sidebar (#11817) 
* Brings pairity for doc checks from cloud

* WIP Sidebar config

It's sketchy JSON let's commit before we break i

* MVP sidebars

summary.md is no longer functional
it's higherarchy was translated into sidebar.js
as a JSON format that docusaurus likes

* Removes another unused readme
 2022-04-07 14:16:35 -07:00
VitaliiMaltsev
a0b7dd0289 🐛 Destination Redshift and Postgres handle custom namespace with '-' (#11729) 
* fix for jdk 17

* add CHANGELOG

* JdbcBufferedConsumerFactory resolve namespace with dash

* add CHANGELOG

* bump versions

* bump postgres version

* update destination-jdbc version

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2022-04-07 13:08:57 +03:00
Topher Lubaway
17407da1e6 Add docusaurus tool not pipeline logic (#11716) 
* Adds docusaurs tooling

tooling works
broken links still exist

needs the pipeline to work with GH pages
merging sooner means that we won't have forked problems from
the relatively frequently updated docs folder

* Updates readme

* Update docusaurus/README.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* Update docusaurus/README.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* Update docusaurus/README.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* Update docusaurus/README.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* Update docusaurus/docusaurus.config.js

Co-authored-by: Davin Chia <davinchia@gmail.com>

Co-authored-by: Davin Chia <davinchia@gmail.com>
 2022-04-06 10:32:36 -07:00
Marcos Marx
d67e32b5d1 Documentation: add snowflake dest note (#11709) 2022-04-05 20:01:28 -03:00
andriikorotkov
72dcd8126a 🐛 Destination bigquery denormalize "allOf" and "anyOf" fix (#11166) 
* fix bug with anyOf and allOf json blocks

* updated tests and formatter

* clean code

* updated tests

* updated destination_specs yaml file

* updated version of connector and docs
 2022-04-05 21:52:13 +03:00
Christophe Duong
393ba35626 🐛 fix s3/gcs bucket cleanup (#11728) 
* Restrict bucket clean up

* bumpversion

* Fix GCS acceptance tests
 2022-04-05 18:46:21 +02:00
andriikorotkov
99875c42c1 GCS Destination: MLP documentation (Beta) (#11499) 
* updated gcs destination spec end documentation

* fixed remarks

* updated version

* updated destination_specs yaml file
 2022-04-05 16:19:55 +03:00
Yurii Bidiuk
2f850b98ac DAT: verify that a destination is able to write any ISO8601-compliant date string (#9816) 
* added format_reference.txt, edited edge_case_messages.txt, edge_case_catalog.json

* updated due to comments

* fix message order for destiantion acceptance tests

* remove unused import

* handle date-time messages for destiantion acceptance tests

* refactoring

* format code

* remove non iso8601 date

* add comments

* fixed pubsub tests

* add test date with microseconds

* fix comments

* fixed PR remarks

* fix assertion for nested objects

* format code

* fix remarks

* format code

* add additional check to convert only datetime values

* changed key structure for saved date-time fieds

* format code

* add comments

* refactored iteration

* format code

* pubsub bump version

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
 2022-04-05 14:04:02 +03:00
Stéphane Kattoor
e3523e0a70 🎉 New Destination: AWS Datalake (#10760) 
* Implementation of a destination connector for AWS Datalake (governed tables)

* refactor

* state message

* correct source config

* add connector spec config

* update docs

* modify spec json

* run seed config

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
 2022-04-04 19:17:34 -03:00
LiRen Tu
c2b5ddd883 Publish GCS destination 0.2.0 (#11706) 
* Remove unpublished tag in changelog

* Bump version in seed
 2022-04-04 13:31:53 -07:00
LiRen Tu
4eb6d060f1 🎉 GCS destination: use serialized buffer; compress csv & jsonl (#11686) 
* Update gcs destination to s3 consumer factory

* Fix typo

* Update gcs config

* Remove gcs s3 helper

* Rename enum

* Clean up s3 config constructors

* Move path format to s3 config

* Fix acceptance test

* Fix unit test

* Format code

* Remove deprecated files

* Update comment

* Fix jsonl acceptance test

* Update serialized buffer factory

* Fix redshift stream copier

* Create gcs name transformer

* Add constants for suffix

* Bump gcs destination version

* Fix typo and unit test

* Add unit test for s3 dest config

* Fix acceptance test due to bucket object prefix

* Update doc

* Mark version as unpublished
 2022-04-04 12:55:56 -07:00
Christophe Duong
18043c8448 🐛 Make s3 file name backward compatible with <0.2.12 (#11666) 
* fix s3 file name
 2022-04-04 14:18:14 +02:00
Oleksandr Sheheda
24d7ab1f58 10148 BigQuery Destination/Denormalized Destination spec updates (#11620) 
* 10148 BigQuery Destination/Denormalized Destination spec updates

* 10148 updated changelog

* 10148 updated docker image version

* 10148 updated docker image version
 2022-04-02 20:24:39 +03:00
Eugene
2008970c7c 🎉Destination-Bigquery-denormalized: added tests for schema selection (#11636) 
* [8895] Destination-Bigquery-denormalized: added tests for schema selection
 2022-04-01 20:34:30 +03:00
Topher Lubaway
03ed894917 Updated GA connector docs and connector (#11640) 
pipeline trigger because gitbook is sad

This reverts commit 230134068f.
 2022-03-31 15:29:34 -05:00
Topher Lubaway
230134068f Revert "Updated GA connector docs and connector " (#11639) 
gitbook pipelines failed.  reverting to trigger gitbook pipeline again
This reverts commit 6aa7e4ce79.
 2022-03-31 15:24:17 -05:00
Amruta Ranade
6aa7e4ce79 Updated GA connector docs and connector stages (#11628) 
* updated GA connector docs and README

* minor edit
 2022-03-31 10:07:03 -04:00
Eugene
69841cee73 🎉BigQuery denormalized Destination: Integration test enhancement for picking test-data and schemas (#11454) 
* [8870] BigQuery denormalized Destination: Integration test enhancement for picking test-data and schemas
 2022-03-30 21:35:37 +03:00