jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 19:11:14 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

2421 Commits

Author SHA1 Message Date
Mario Molina
5b0b5840d6 🎉 New Destination: MQTT (#7718) 
* MQTT destination

* Reformat code

* Updating readme

* run seed

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
 2021-11-21 14:20:42 -03:00
Ivica Taseski
8a957caf14 🎉 New destination: Redis (#7653) 
* add redis destination connector

* add redis destination connector

* refactor & docs

* format

* refactor code

* debug checkpoint

* refactor and code cleanup

* format code

* add redis docs

* bump version, docs and run seed:

* correct uuid

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
 2021-11-20 11:26:37 -03:00
Ivica Taseski
fcba5263e1 🎉 New destination: ScyllaDB (#7468) 
* add scylla destination connector

* minor fixes

Co-authored-by: itaseski <ivica.taseski@seavus.com>
Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
 2021-11-16 13:28:46 -03:00
Sherif A. Nada
cf126ced9b 🐛 Destination BQ Denormalized: handle null values in fields described by a $ref schema (#7804) 2021-11-09 21:53:41 -08:00
Alexander Tsukanov
48d825046f 🐛 Destination BigQuery-Denormalized: Fix JSON with $ref Definition keys (#7736) 
* BUG-6638: Fix fields with $ref definition.

* BUG-6638: Added integration tests.

* BUG-6638: Added integration tests.

* BUG-6638: Added integration tests.

* BUG-6638: Added doc and pumped Dockerfile version.

* BUG-6638: Added doc and pumped Dockerfile version.

* BUG-6638: replaced for with forEach

* BUG-6638: Bumped specification
 2021-11-09 13:39:39 +02:00
Sherif A. Nada
efb5151011 🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering (#7719) 2021-11-08 19:26:32 -08:00
Mario Molina
16d9d15f5a 🎉 New Destination: Pulsar (#7315) 
* Adding Pulsar destination

* Add enable_chunking property

* Add dependency for DNS resolve for Mac

* Updating build.gradle

* Adding definition

* Refactor and fix tests

* Adding new config properties

* Reformat code

* Close resources when checking connection

* Updating doc

* Adding more test to build the producer map

* Rename brokers property

* Avoid emitting null states to the output collector

* fix config file

* run seed source/destination

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
 2021-11-05 20:03:27 -03:00
itaseskii
f53fd5e66b 🎉 New destination: Cassandra (#7186) 
* add cassandra destination connector

* refactor and docs.

* delete test dockerfile

* revert Dockerfile rm change

* refactor & fix acceptance tests & format

* revert stream peek

* remove get pip

* add address example

* improved copy and code refactor

* add docker-compose and improved docs

Co-authored-by: itaseski <ivica.taseski@seavus.com>
 2021-11-05 19:02:01 -03:00
LiRen Tu
b6a93c4e09 Update Databricks doc (#7618) 2021-11-03 20:34:08 -07:00
LiRen Tu
c9c41dcd08 🎉 Destination S3 & GCS: support additional properties (#7288) 
* Log json schema

* Use patched json avro converter

* Rename schema

* Update unit test cases

* Fix ab ap field schema conversion

* Rename files

* Add unit test cases

* Fix dependency for databricks

* Bump versions

* Update documentations

* Update gcs doc

* Set additional properties field name

* Revert s3 and gcs version

* Specify extra props fields

* Refactor json avro conversion doc

* Update connector doc

* Fix databricks spec typo

* Bump connector versions in seed
 2021-11-03 11:37:24 -07:00
Alexander Tsukanov
a915034954 🐛 Destination BigQuery-denormalized: Added conversion from JSON Datetime to BigQuery format (#7413) 2021-11-03 13:29:45 +02:00
Jeremy
fb313a6d26 🎉 New Destination: Elasticsearch (#7005) 
* feat: adds destination-elasticsearch

* feat: adds destination-elasticsearch es server container

* refactor: header configuration

* update: only call createIndex when preparing the writes

* update: reuse container

* fix: make index names valid and use namespace

* refactor: use bulk process and buffered consumer

* refactor: fix bulk process and buffered consumer

* chore: update documentation

* update: remove ssl reference

* fix: bulk indexing

adds test logging config to inspect http wire
begins work for overrwriting existing records

* docs: update for authentication

* refactor: simplify config

* refactor: cleanup indices, implement auth

* update: cleanup equals/toString in Elasticsearch ConnectionConfiguration

* chore: use conventions and remove unused code

* update: close underlying rest connection

* update: enable `supportsNormalization`

* refactor: better encapsulate index naming

* update: allow upserting

* update: use oneOf for auth method

* refactor: use encapsulated auth object

* chore: pretty

* update: simplify auth header creation

* chore: remove unused class

* update: use boolean as field type

* adds: elasticsearch example server

* fix: api secret test
 2021-11-01 22:52:01 -03:00
Sherif A. Nada
10889d8338 Add user creation instructions to Postgres destination 2021-10-28 15:44:03 -07:00
Christophe Duong
27df558a89 🎉 BigQuery destinations with partitionned/clustered keys (#7240) 
* [ #5959 ][ #2579 ] Add support of partitioned tables by _airbyte_emitted_at field (#7141)

Co-authored-by: Andrés Bravo <andresbravog@gmail.com>
 2021-10-25 12:41:04 +02:00
Brian Bugh
a9e01db8b3 Fix incorrect code for Snowflake GCS (#7287) 
This was confusing, as `io_airbyte_test_staging` looks like a literal string and is not mentioned in the explanation above. Slack support context: https://app.slack.com/client/T01AB4DDR2N/C01MFR03D5W/thread/C01MFR03D5W-1634852461.058400
 2021-10-22 16:22:29 -03:00
VitaliiMaltsev
0c932749d4 🎉 Redshift Source and Destination set SSL as default option (#7234) 
* Redshift Source and Restination set SSL as default option

* add changelog

* remove SSL test| add more documentation

* bump new version

* bump new version

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2021-10-22 12:28:54 +03:00
irynakruk
265986b5bc Create a secure-only MongoDb destination (#6945) 
* Added mongodb destination strict encrypt
 2021-10-20 09:04:30 -04:00
VitaliiMaltsev
ae9048cfaa 🎉 Redshift Source/Destination SSL Support (#6965) 
* add tls option to spec

* Redshift Source add acceptance test

* Redshift Destination add ssl field to spec

* add RedshiftDestinationAcceptanceTestSSL

* fix checkstyle

* added changelog

* update docs

* bump versions of Redshift Source and Destination \ changed default tls to true

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2021-10-14 12:29:00 +03:00
Eugene
05e09aff7d 🎉Destination MySQl - added ssl strict encrypt connector (#6763) 
* [6423] Destination MySQl - added ssl strict encrypt connector
 2021-10-14 08:28:43 +03:00
andriikorotkov
80c9ed2b02 🐛 AWS S3 Staging COPY is writing records from different table in the same raw table (#6949) 
* updated jdbc destination

* updated snowflake and redshift destination version

* updated documentation

* updated documentation

* updated prefix length for snowflake and redshift streams

* fixed remarks

* updated new redshift version
 2021-10-14 01:04:54 +03:00
maciej-nedza
cb448681b9 Update Keen documentation (#6948) 2021-10-13 12:23:40 -07:00
Oleksandr Sheheda
2b39c11425 🎉 Destination Oracle: implemented connection encryption using N… (#6893) 
* #6013 🎉 Destination Oracle: implemented connection encryption using NNE and TLS
 2021-10-12 18:20:09 +03:00
Baz
6fb989feb7 🎉Destination MSSQL: Fix the Acceptance Tests based on Normalization (#6877) 
* Enabled Normalization for destination-mssql

* added changelog

* fixed normalization for mssql source

Co-authored-by: ievgeniit <etsybaev@gmail.com>
 2021-10-11 21:58:52 +03:00
Abhi Vaidyanatha
ae32ecbb27 GitBook: [master] 186 pages and 77 assets modified 2021-10-08 21:17:47 +00:00
Baz
e5abaeccef 🎉 Base-normalization: Implement normalization for MSSQL-destination (#6079) 
See the attached PR (https://github.com/airbytehq/airbyte/pull/6079)
 2021-10-07 18:46:27 +03:00
Abhi Vaidyanatha
276d914a57 Direct to Discourse tagged list instead of specific issues on Troubleshooting. (#6848) 2021-10-07 00:05:56 -07:00
Abhi Vaidyanatha
acff636e4b Fix list formatting in SSH tunnel sections (#6847) 2021-10-06 15:49:31 -07:00
Marcos Marx
118492071c Destination BigQuery: add support to dataset starting with numbers (#6733) 
* change check function

* make test harder

* change tests

* add tests˜
 2021-10-06 13:00:33 -03:00
VitaliiMaltsev
6e53a574e3 🐛 Destination Oracle: maxStringLength should be 128 (#6611) 
* revert back to 128 characters in OracleNameTransformer | Create Custom OracleContainer

* update to 0.1.9

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2021-10-06 13:50:07 +03:00
Sherif A. Nada
1bf339e78a Update oracle.md 2021-10-06 00:27:15 -07:00
Sherif A. Nada
d5ed24b65e Update oracle.md 2021-10-06 00:26:58 -07:00
LiRen Tu
914531cc21 Add accept-terms property to databricks spec (#6792) 
* Add accept-terms property to databricks spec

* Update PR id

* Update version

* Update sample config
 2021-10-05 12:38:29 -07:00
Abhi Vaidyanatha
fd2f96e5ff Docs Refresh: S3 Destination (#6704) 2021-10-04 00:36:58 -07:00
Abhi Vaidyanatha
ecbc3f78ed Add emitted field error Troubleshooting for BigQuery Destination (#6702) 2021-10-04 00:26:18 -07:00
Abhi Vaidyanatha
770bbf9e14 📚 Docs Refresh: Jira Source (#6699) 
* Clean up MSSQL Troubleshooting

* Docs Refresh: Jira Source

* Update BQ troubleshooting

* Add discussion
 2021-10-04 00:05:52 -07:00
Abhi Vaidyanatha
68f1b2cf13 Add Troubleshooting for BigQuery Destination. (#6698) 2021-10-03 23:55:42 -07:00
Abhi Vaidyanatha
e824cad57f Docs Refresh: Oracle Destination (#6692) 2021-10-03 23:17:07 -07:00
Abhi Vaidyanatha
87b035f26d Docs Rework: MSSQL Destination (#6677) 2021-10-03 23:07:47 -07:00
Abhi Vaidyanatha
0f0bb85a1f 📚 Docs Refresh: MySQL Destination (#6676) 
* Docs Refresh: MySQL Destination

* Add SSH Tunnel.
 2021-10-03 23:07:20 -07:00
Abhi Vaidyanatha
c8c736b575 📚 Docs Refresh: Postgres Destination (#6675) 
* Docs Refresh: Postgres Destination

* Add SSL note.
 2021-10-03 23:05:56 -07:00
Abhi Vaidyanatha
2a651847ef 📚 Docs Refresh: MongoDB Destination (#6673) 
* Docs Refresh: MongoDB Destination

* Revert changes to Getting Started guide.

* Update TLS message.
 2021-10-03 23:05:24 -07:00
Abhi Vaidyanatha
bee03dc50f 📚 Docs Refresh: BigQuery Destination (#6665) 
* Docs Refresh: BigQuery Destination

* Add language for Cloud.

* Update docs/integrations/destinations/bigquery.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/destinations/bigquery.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/destinations/bigquery.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* Update docs/integrations/destinations/bigquery.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/destinations/bigquery.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* Update docs/integrations/destinations/bigquery.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* Add link to GCS.

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2021-10-03 22:54:48 -07:00
irynakruk
992590a501 Destination MongoDb: added support via TLS/SSL (#6536) 
* Updated MongoDatabase

* Destination MongoDb: added support via TLS/SSL

* Updated doc and fixed test

* Updated changelog

* Code review enhancements

* Update airbyte-integrations/connectors/destination-mongodb/src/main/java/io/airbyte/integrations/destination/mongodb/MongodbDestination.java

Co-authored-by: LiRen Tu <tuliren@gmail.com>

* fixed test

Co-authored-by: LiRen Tu <tuliren@gmail.com>
 2021-10-01 00:43:49 -04:00
andriikorotkov
bbf098a753 🎉 Destination MySQl - Added support for MySQL destination via TLS/SSL (#6506) 
* add support SSL for MySQL destination

* updated ssl tests and add documentation

* updated code style

* changed default ssl value as true
 2021-09-29 15:23:54 +03:00
oneshcheret
a595c753e3 🎉 Destination MSSQL: Added support for connection via SSH tunnels (#5… (#6503) 
* 🎉 Destination MSSQL: Added support for connection via SSH tunnels (#5970)

* 🎉 Destination MSSQL: bumb image version

*  🎉 Destination MSSQL: update mssql.md file
 2021-09-29 13:12:15 +03:00
Nicolas MOREAU
0105ca91b4 🎉 BigQuery Denormalized Destination: Support for more bigquery types through the format annotation (#6145) 
* destination-bigquery-denormalized: introduce json spec format key handling

* destination-bigquery-denormalized: Bump version
 2021-09-28 21:06:14 -03:00
VitaliiMaltsev
c2561141fc 🎉 Destination Oracle - Added support for connection via ssh tunnel (#6370) 
* Oracle destination with ssh tunneling

* add ssh key integration testing for Oracle destination

* fix checkstyle

* add container approach to Oracle Destination Integration Tests

* add container approach to Oracle Destination Integration Tests

* add container approach to Oracle Source Integration Tests/ delete secrets for Oracle db

* Oracle Destination - bump version to 0.1.8

* remove oracle ssh secrets from ci_credentials.sh

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
 2021-09-28 14:25:33 +03:00
andriikorotkov
8fa15713c3 🎉 Destination MySQl - Added support for connection via ssh (aka bastion server) (#6317) 
* updated mysql tests

* updated mysql tests

* added mysql ssh tunnel tests by key

* fixed remarks

* fixed remarks

* updated DatabricksStreamCopier

* switch to custom file for ssh config in normalization

* updated MySQL SSH tests

* bump version

* get local port properly

* updated assertSameValue for MySQL ssh tunnel

* updated image version and documentation

* updated code style

* updated CI credentials

* updated normalization documentation

Co-authored-by: George Claireaux <george@claireaux.co.uk>
 2021-09-28 13:11:32 +03:00
Sherif A. Nada
b050f65a92 fix redshift docs formatting 2021-09-23 19:07:28 -07:00
Abhi Vaidyanatha
bf665cb280 Fix language consistency in Databricks page (#6360) 
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-09-21 15:52:25 -07:00