1
0
mirror of synced 2025-12-21 19:11:14 -05:00
Commit Graph

28471 Commits

Author SHA1 Message Date
Abhi Vaidyanatha
15971e89b1 0.27.1 Connector Patch Notes (#4646)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-09 00:15:28 -07:00
vovavovavovavova
db223a4d06 🎉 New Source: Zendesk Sunshine (#4359)
* pre-PR

* add git config

* format

* Update airbyte-integrations/connectors/source-zendesk-sunshine/requirements.txt

upd requirements.txt remove extra

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Update airbyte-integrations/connectors/source-zendesk-sunshine/source_zendesk_sunshine/streams.py

backoff time int to float (btw real return type in headers is integer)

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* requested changes

* fix newline absence && rm unnecessary temp file

* url_base to property

* rm extra var coming property

* rm extra var coming property

* save

* finishing updating the documentation

* forgotten definition

* add nullable to pass the test

* fix date in the log

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-07-09 09:55:40 +03:00
midavadim
ca3befee14 🎉 New Source: Paypal Transaction (#4240)
* Added spec.json

* Initialization

* added oauth2 autorization

* added spec, check, discover + catalogs/configurared_catalogs

* updated request_params

* added paging, slicing (1d)

* Use oath2 for paypal

* incremental sync, acceptance test

* incremental sync, acceptance test

* Added spec.json

* Initialization

* added oauth2 autorization

* added spec, check, discover + catalogs/configurared_catalogs

* updated request_params

* added paging, slicing (1d)

* Use oath2 for paypal

* incremental sync, acceptance test

* incremental sync, acceptance test

* Added spec.json

* Initialization

* added oauth2 autorization

* added spec, check, discover + catalogs/configurared_catalogs

* updated request_params

* added paging, slicing (1d)

* Use oath2 for paypal

* incremental sync, acceptance test

* updated slices and api limits, added validation for input dates

* added tests, fixed cursor related information in schemas and configured catalogs, removed old comments, re-arranged Base PaypalTransactionStream class

* added input param 'env' to support production and sandbox envs

* added support for sandbox option, updated pattern for optional end date option

* added github secrets

* added support for sandbox option, updated pattern for optional end date option

* fixed Copyright date, removed debug mesages

* added docs

* fix for test failure - The sync should produce at least one STATE message

* removed optional parameter 'end_date'

* removed detailed info about balances schema

* Delete employees.json

* Delete customers.json

* Added requests_per_minute rate limit

* added unit tests, added custom backoff

* added test for stream slices with stream state

* removed comments

* updated docs pages

* fixed format for json files

* fixed types in schemas and link to the schema. fixed primary key for Transactions stream

* updated stream slices

* Updated tests, unified stream_slices for both streams, all instance variables instantiated directly in __init__ method

* added CHANGELOG.md

* Added build seeds

* fixed closing double quotation mark

* added paypal entry in builds.md

* add fixture helper

* added paypal transaction generator script

* fixed styling

* maximum allowed start_date is extracted from API response now.

* fixed schemas

* fixed schemas - removed datetime

* now maximum_allowed_start_date is identified by last_refreshed_datetime attr in API response.

* added possibility to specify additional properties

Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-07-08 18:47:00 +03:00
vovavovavovavova
6fdf1ef6ae Mailchimp fix url-base (#4621)
* minimal change to show acceptance test failure

* exactly fix

* bump version and readme

* upd
2021-07-08 18:15:54 +03:00
LiRen Tu
646aae1ac8 🎉 Destination S3: support anyOf allOf and oneOf (#4613)
* Support combined restrictions in json schema

* Bump s3 version

* Add more test cases

* Update changelog

* Add more test cases

* Update documentation

* Format code
2021-07-07 15:41:36 -07:00
Yevhenii
c8b8b526f2 🎉 Instagram Source: Migrate to use CDK, upgrade API version and fix failures with HTTP status 400 (#4210)
* handling 400 HTTP error with Backoff
* migrate to CDK

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-07-07 14:11:27 -04:00
vovavovavovavova
4774edf2ca add source-definitions (#4601)
* add source-definitions

* tmp: add docs

* upd docs

* add changelog

* changes

* upd docs/integrations/readme
2021-07-07 20:48:20 +03:00
Dmytro
b8b205a3ff 🎉 New source: Square (#4439) 2021-07-07 20:20:18 +03:00
oleh.zorenko
3e37e904b1 Source GitHub: few updates (#4598)
* Few updates for GitHub connector

Add `AIRBYTE_ENTRYPOINT` to `Dockerfile`.
Add `Changelog` section to `docs/integrations/sources/github.md`.
Remove `source-github/CHANGELOG.md` file.

* Bump GitHub connector version
2021-07-07 15:48:47 +03:00
Marcos Marx
6201100e72 Add entrypoint and bump version for remaining connectors. (#4539)
See the PR descriptions for all the connectors published.
2021-07-07 17:37:22 +08:00
Christophe Duong
af57170d21 Deploy a sweeper pod to clear completed pod histories (#4500)
* Deploy a sweeper pod to clear completed pod histories
2021-07-07 10:32:09 +02:00
Marcos Marx
31555bab4d bump stripe version (#4587) 2021-07-06 21:13:24 -03:00
Abhi Vaidyanatha
3e3cf7b544 Fix broken link in Local CSV doc (#4578)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-06 11:24:26 -07:00
oleh.zorenko
84ba3e79b3 🎉 Native GitHub connector (#4174)
* GitHub source initial version

All streams are in `full_refresh` mode only.

* Run format command

* Add handling for 404 error in Teams stream

* Update creds naming for github

* Update acceptance tests

* Update check_connection()

* Code review fixes

Remove `/integration_tests/catalog.json` file.
Use `Collaborators` class in `check_connection()` function.
Remove excessive data from GithubStream class.

* Remove unused import

* Implement requested changes

* Add incremental streams

* Remove `supports_incremental`

* Update docs

* Implement requested changes

* Implement requested changes

* Implement requested changes

* Bump connector version
2021-07-06 17:06:34 +03:00
jaimefr
2efac85c1f add deprecation info for google adwords (#4561) 2021-07-06 09:27:04 -03:00
Yevhenii
94f1d217b8 🎉 Native Gitlab connector (#4027)
* Create Airbyte-Native connector for Gitlab

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-07-06 15:25:01 +03:00
Oleksandr
9517fae702 🎉 Source Shopify - change the default stream cursor_field to 'updated_at' where possible (#4472)
#4472 - Source Shopify - change the default stream cursor_field to 'updated_at' where possible

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-07-05 15:42:35 +03:00
LiRen Tu
3d3988b1e3 Destination MySQL: bump version (#4531)
* Bump mysql destination version to pick up normalization

* Also change the publish command to run on ec2-runners to try and avoid build errors with gradle unable to find the right volume.

Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-07-04 21:54:01 +08:00
Vadym
9531c4d4ec 🎉 New source: airbyte-native twilio connector (#4070)
* Add Twilio CDK source

* Implement all streams full refresh support.

* Complete all streams and configs.

* Apply code format

* Fix streams data response.
Update streams which should not be incremental.
Fix configs.

* Fix incremental streams.
Fix configs.

* Fix acceptance tests config.

* Fix stream schemas.

* Fix messages stream schema.

* Catalogs fixed

* Update setup requirements.

* Add standard CI creds

* Update build.gradle and sample config files to remove unused references

* Updated sample config files json format

* Update streams.py

* Update incremental streams cursor_field formatting

* Update streams and test config

* Add no_empty_streams_no_usage_records_catalog.json

* Update streams code style.
Add twilio.md docs.
Other PR style fixes.

* Update twilio.md docs.

* Odd docstring removed.

* Uncomment tests.

* Add Docker AIRBYTE_ENTRYPOINT
2021-07-03 20:59:38 +03:00
Abhi Vaidyanatha
d725aa9fbe Add standalone getting started guides structure. (#4492)
* Add standalone getting started guides.

* Add GitHub source

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-02 13:49:26 -07:00
Sherif A. Nada
9079cbdf78 Publish PR (#4121) (#4495)
Co-authored-by: Rodrigo Parra <rodpar07@gmail.com>
2021-07-02 13:34:57 -07:00
Rodrigo Parra
4b0c3494ae 🎉 Source Shopify: Add support for draft orders (#4121) 2021-07-02 13:33:47 -07:00
Abhi Vaidyanatha
3b2009723e Add note about collocating Redshift cluster and Airbyte server in the same security group (#4490)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-02 12:42:35 -07:00
Abhi Vaidyanatha
d985fe73f4 Add note about rate limits with the HubSpot API (#4491)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-01 23:04:35 -07:00
Sherif A. Nada
9708e4c8de 🐛 bugfix Source Okta: Fix endless loop while syncing data from logs stream (#4464)
Co-authored-by: sabifranjo <sabifranjo@gmail.com>
Co-authored-by: Sabolc Franjo <sabolc.franjo@ev-box.com>
2021-07-01 17:39:51 -07:00
Abhi Vaidyanatha
b94d2eb4c4 Fix pubsub description YAML frontmatter (#4475)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-01 14:32:02 -07:00
Eugene
4399656ec9 🎉 Source Cockroachdb: Added new source for Cockroachdb (#4382)
* [1705] Added cockroachdb source connector
2021-07-01 22:04:21 +03:00
Abhi Vaidyanatha
da36e22272 Add missing Sources and Destinations to Catalog Table. (#4423)
* Add missing Sources and Destinations to Catalog Table.

* Add Snowflake Source.

* Address review comments

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-30 13:27:26 -07:00
Sherif A. Nada
71c7cecd74 Source postgres: document that replicating views is supported (#4008) 2021-06-28 00:11:37 -05:00
Kirill Chernov
40aed2249d 🎉 fix Google Adwords CI & publish new version (#4205)
* added SAT tests, deleted incremental test sfrom pipeline

* pr changes and tests uncommented

* deleted facebook file from pr

* deleted facebook file from pr

* removed tap adwords from install_requires

* Update airbyte-integrations/connectors/source-google-adwords-singer/acceptance-test-config.yml

* added changelog info

* version bump

* version bump

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-06-26 01:45:22 +03:00
Sherif Nada
ea975027bc Merge branch 'master' of github.com:airbytehq/airbyte 2021-06-25 12:45:04 -07:00
Sherif Nada
10d1168a03 update docs 2021-06-25 12:44:57 -07:00
LiRen Tu
8954a99290 Bump source looker connector version (#4350)
* Bump looker connector version

* Update changelog

* Format code
2021-06-25 11:23:16 -07:00
Prasanna Ram Venkatachalam
0c25e1cca1 🎉 New Destination: Google PubSub (#4339)
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-06-25 11:09:07 -07:00
Oleksandr
0e0d0a102e 🐛 Bugfix for pagination in Shopify Source Connector #4284 (#4290)
#4284 - Bugfix for pagination in Shopify Source Connector

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-06-25 19:45:45 +03:00
Michael Irvine
842f10bd83 🎉 Source Looker: add run-look endpoint (#3911)
* looker run_looks method works with normalization

* update catalogs

* validate that look(s) entered actually exist at config time

* field type lookup works, though it is a bit messy

* don't re-run explore query if we have already run it

* add more type mappings

* add some docstrings and comments

* add title for ease of use

* Revert "update catalogs"

This reverts commit 7ecaaa4d0d.

* fix configuration file formatting

* fix some issues from code review and reorder methods

* update docs to include run_look endpoint
2021-06-25 08:31:56 -07:00
Yaroslav Dudar
2ccb1c2de0 🎉 New source: AWS CloudTrail (#4122)
AWS CloudTrail connector initial release
2021-06-25 11:10:07 +03:00
Marcos Marx
77f7096d14 🐛 Source Mailchimp: use datacenter parameter from apikey info (#4285)
* replace datacenter parameter using apikey info

* change to use dynamic property

* add dynamic property

* bump version

* change config versino
2021-06-24 21:55:13 -03:00
Sherif A. Nada
c130465048 Release Shopify source 0.1.6 (#4332) 2021-06-24 16:39:35 -07:00
jacqueskpoty
e01576d461 🎉 Source Shopify: add pages, price rules and discount codes streams (#4009)
Co-authored-by: jsewawilson <jwilson@pebblepost.com>
2021-06-24 16:18:15 -07:00
Harshith Mullapudi
f4ddf4042d 🐛 Bugfix incorrectly failing check connection operations in Google Ads and Amazon AP API connectors (#4288) 2021-06-24 15:58:37 -07:00
LiRen Tu
e2c5b1d4c7 🎉 S3 destination: Avro & Jsonl output (#4227)
* Add jsonl format to spec.json

* Implement jsonl writer

* Add documentation

* Add acceptance test

* Update document

* Bump version

* Update document example

* Implement avro writer

* Implement compression codec

* Update documentation

* Revise documentation

* Add more tests

* Add acceptance test

* Format code

* Create helper method for name updater

* Update csv doc with normalization

* Update version date
2021-06-23 14:54:39 -07:00
irynakruk
0a4712cbe7 🎉 New Source: IBM DB2 (#4197)
* Db2 source: implementation and acceptance tests

* Db2 source: added documentation, build info, Db2SourceComprehensiveTest

* Db2 source: improved Db2SourceComprehensiveTest

* Db2 source: improved Db2SourceComprehensiveTest

* Db2 source: updated docs

* Db2 source: added test for the special values in Decfloat

* Formatter changes

Co-authored-by: Iryna Kruk <iryna.o.kruk@globallogic.com>
2021-06-23 18:06:31 +03:00
Christophe Duong
75a1dda07e 🎉 New BigQuery destination with Structured/Repeated Records (#4176) 2021-06-23 16:19:36 +02:00
Eugene Kulak
2f7c15a540 Source Facebook: migrate to CDK (#3743)
* move to CDK implementation
* make insights streams configurable
* improve Async implementation using slices
* minimize set of streams in configured_catalog.json
* apply suggestions from @sherifnada
* improve async jobs handling even more

Co-authored-by: Ka-Ping Yee <zestyping@gmail.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-06-22 18:48:17 -04:00
Vadym
4c2fc3ba00 Bump harvest version (#4277)
* Bump source Harvest docker version

* Update harvest.md docs
2021-06-22 20:22:02 +03:00
Marcos Marx
b57736b0ce Documentation correct markdown list in csv and json destination (#4165)
* correct list in markdown csv and json

* add copy cmd and change variable name to used in dest settings
2021-06-21 19:53:26 -03:00
Marcos Marx
156629ca95 Bump Harvest Source connector 0.1.2 (#4223)
* fixes #4221. Fix broken spec file property and point to correct docs page

* bump harvest connector

* add changelog for harvest

* bump dockerfile version

Co-authored-by: Tyler Schroeder <tyler@tylerschroeder.com>
2021-06-21 15:22:48 -03:00
Marcos Marx
88d7953a18 Add CHANGELOG for Sources (#4215)
* add changelog for postres source

* add changelog for mysql source

* add changelog for mssql source

* add changelog for file source

* add changelog for google sheets

* add full changelog for file source

* add full changelog for google sheet source

* add full changelog for mssql source

* add changelog for http requests

* add changelog for salesforce source

* add changelog for facebook marketing source

* add changelog for google analytics source

* add changelog for shopify native source connector
2021-06-18 17:45:09 -03:00
Marcos Marx
f365c0c430 destination mssql correct spec doc info (#4185) 2021-06-17 19:51:36 -03:00