1
0
mirror of synced 2025-12-21 11:01:41 -05:00
Commit Graph

26035 Commits

Author SHA1 Message Date
Sherif A. Nada
d1f6a7a6c9 Correct some links and add notes about finding DB params in db-backed connector docs (#5017) 2021-07-26 22:41:17 -07:00
Sherif A. Nada
e528e79726 save (#5007) 2021-07-27 15:48:00 +11:00
Marcos Marx
d91ff7dcf1 Documentation add data types to databases (#5008)
* add postgres data types

* add mysql data types

* add mssql data types

* add oracle data type
2021-07-26 23:02:46 -03:00
vitaliizazmic
5700645946 🎉 New Source: PrestaShop (#4465)
* Source PrestaShop #2980 - init commit

* Source PrestaShop #2980 - schemas

* Source PrestaShop #2980 - full refresh sync

* Source PrestaShop #2980 - incremental sync, acceptance tests

* Source PrestaShop #2980 - incremental acceptance tests

* Source PrestaShop #2980 - incremental acceptance tests (remove configurations from incremental)

* Source PrestaShop #2980 - change Configurations stream to full refresh

* Source PrestaShop #2980 - docs

* format

* fix schemas, fill streams with data

* fix tests and future_state

* rename presta-shop -> prestashop

* fix mounting volumes inside a nested container

* special handle for NULL dates (MySQL)

* fix build URL

* fix URL and polishing

* fix header

* fix template

* format, update pre-commit to execute in the right order

* fix scaffolding

* bump and changelog for SAT

* register as the default source

* fix test timeout

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-07-26 21:55:07 -04:00
Eugene Kulak
76508b88fa Source PostHog: incremental streams read only relevant pages (#4001)
* fix init state that cause loading all pages

* fix init state that caused read of all pages

* update changelog and apply suggestion from @sherifnada

* fix null state

* Fix SAT and clean up code a bit

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-07-26 21:52:47 -04:00
Dmytro
718bd75c00 🐛 4836: Stripe connector: Fix incorect data field type (#4878)
* 4836: Stripe connector: Fix incorect data field type

   - Change type of percent_off field to number according to reference:
   https://stripe.com/docs/api/coupons/object#coupon_object-percent_off
   - Change invoice/discounts field type to array:
   https://stripe.com/docs/api/invoices/object#invoice_object-discounts

* Bump stripe connector version to 0.1.15

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-07-26 17:44:00 +03:00
Oleksandr
34317cdfb1 🐛 Source Shopify: fix the products schema according to API 2021-07 (#4943)
#4927 - Shopify integration schema discrepancy (products.json)

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-07-25 00:12:08 +03:00
Arthur Galuza
743a054140 Source Pipedrive: update money type to support floating point (#4912)
* Upd money type: prices can be with floating point

* Upd version in source definitions

* Add a stub integration test to bypass CI/CD test stage

* Remove validate_output_from_all_streams param from acceptance-test-config

* Upd pipedrive changelog
2021-07-23 17:28:45 +03:00
Yevhenii
3e8f12da17 🚀Source Google Ads: support main streams (#4788)
* Source Google Ads: support new streams: Campaigns, Ads, Ad Groups, Accounts

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-07-23 14:25:32 +03:00
vovavovavovavova
c403fc230e New Source Cart: Base (#4754)
* pushing auto-generated files for cart source

* New Source Cart: Implementation (#4785)

* save the setup

* preparing the streams::in progress

* adjust new source streams.py

* upd spec

* save schemas without nullable

* fixing with schemas

* save before test

* successful check

* fix discover

* rename wrong name

* read+

* upd configured_catalog

* format

* fixing schemas (doc mismatch a bit with responses also with fields presense) + fix abnormal json

* modify configured_catalog

* gradle jsons

* requested changes

* upd wording

* increase timeouts

* if Cart store name is incorrect: improve user hint && upd failure timeout to be able to check wrong stor name (not only wrong token which is fast)

* cleanup spec.json

* latest upd

* New Source Cart (#4588): Documentation (#4859)

* save docs

* forgotten file name

* finish docs?

* Update docs/integrations/sources/cart.md

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Update docs/integrations/sources/cart.md

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Update docs/integrations/sources/cart.md

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Update docs/integrations/sources/cart.md

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* merge && resolve conflict

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* apply add abs_property

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* rm outdated in acceptance.yaml

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-07-23 12:37:27 +03:00
Serhii Lazebnyi
8488e9ff5d 🐛 Source Slack: Fix reading threads issue (#4860)
* Fixed reading threads issue

* Added filtering for threads stream

* Added acceptance test for threads stream

* Added integration dummy test

* Added requirements file

* Deleted version of airbyte-cdk

* Deleted validate_output_from_all_streams flag

* Deleted condition status code in backoff
2021-07-23 12:24:11 +03:00
Sherif A. Nada
e5719e0b3e Fix typo in GA docs 2021-07-21 10:51:48 -07:00
Sherif A. Nada
05bd157e50 📚 improve mongo docs and param descriptions (#4870) 2021-07-20 13:51:01 -07:00
Oleksandr
7d538a7ce6 🐛 Source JIRA: Fix DBT failing normalization on Labels schema. (#4817)
(#4817) 🐛 Source JIRA: Fix DBT failing normalization on `Labels` schema. 

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-07-20 10:26:58 +03:00
Daniel Mateus Pires
c4de62e77a 🎉 New source: US census (#4228)
Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-07-20 00:17:53 -07:00
Sherif A. Nada
1ba4304581 🐛 Sendgrid source: Gracefully handle malformed responses from sendgrid API (#4839) 2021-07-19 19:55:33 -07:00
midavadim
472857d672 🎉 Source mixpanel: migration to CDK (#4566)
* Mixpanel initiation

* copied schemas and specs file from singer connector

* authentication and a few streams

* Added Funnels + FunnelsList

* Added example of funnel response

* added incremental Funnels stream with tests

* added Annotations, CohortMembers, Engage, Cohorts, Funnels

* added Revenue

* fixed formatting

* fixed variable names

* fixed cohort_members and updated export streams

* moved start_date and date checks into SourceMixpanel class

* added error handling

* added unit test, update docs and ci creds

* fix url base for export stream

* added full and incremental read for export stream

* updated acceptance tests, added limit correction based on number of streams, export cursor is stored in datatime string

* Funnel stream - added complex state which contains state for each funnel

* added attribution windows support and project timezone config

* fixed formatting

* added default timezone

* added dynamic schema generation for Engage and Export streams

* fixed formatting

* fixed ability to pass start_date in datetime format as well

* fixed ability to pass start_date in datetime format as well

* added additional_properties field for dynamic schemas. updates regex for start_date matching to support old config file

* fixed formatting

* export stream - convert all values to default type - string

* added schema ref

* added new properties for funnel stream

* fixed formatting in funnel schema

* added build related files

* update changelog

* fixed and added comments, renamed rate_limit variable

* fixed formatting

* changed normalization for reserved mixpanel attributes like $browser

* alphabetise spec fields

* added description about API limit handling

* updated comment
2021-07-20 02:31:13 +03:00
Sherif A. Nada
c755c8f244 Update GAds docs to indicate incremental support 2021-07-19 16:06:22 -07:00
Eugene Kulak
b05ab17520 🐛 Source Instagram: Read previous state format and upgrade it (#4805)
* few fixes for user_insights state

* support old state format

* format

* bump

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-07-19 18:26:59 -04:00
Oleksandr
84aad359aa 📚 Source Shopify: migrate to new sandbox, update API version to 2021-07 (#4830)
(#4830) Source Shopify: migrate to new sandbox, update API version to 2021-07

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-07-19 19:57:11 +03:00
Vladimir remar
219389b2d1 🐛 Source Facebook: Improve rate limit management (#4820)
* Improve rate limit management

* bump version

* facebook-marketing.md update the changelog
2021-07-19 09:08:10 -04:00
Vadym
24be682163 🎉 New source: Pipedrive connector (#4686)
* Add pipedrive source initial

* Add initial schemas.
Add MVP source implementation.

* Implement MVP streams

* Complete MVP streams implementation

* Apply schema format

* Add test creds

* Update streams.py
Fix schemas

* Update replication_start_date format.
Add extra pagination condition

* Refactor streams, remove unused classes.

* Add pipedrive.md docs file.
Add Pipedrive source definitions.

* Add json source definition.

* Update spec.json

* Add docs mentions throughout the project files
2021-07-19 15:08:23 +03:00
Serhii Lazebnyi
9b2388cd31 🎉 Source intercom: migration to CDK (#4676)
* Added Intercom implementation

* Updated segments docs

* Updated _send_request method to new airbyte-cdk version

* Updated cursor field to datetime string

* Added filtering by state for incremental sync

* Updated cursor paths for test incremental sync

* Added dict type validation to get_data method

* Updated catalog

* Updated typing for start_date

* Updated singer seed to cdk seed

* Updated connector docs

* Updated sample config file

* Sorted streams alphabetically

* Removed placeholder comments

* Renamed rate_limit to queries_per_hour

* Updated common sleep time to backoff_time method
2021-07-19 14:14:51 +03:00
Anna Lvova
28239a87a1 🎉 Stripe Source: Fix subscriptions stream to return all kinds of subscriptions (including expired and canceled) (#4669)
#4669 Stripe Source: Fix subscriptions stream to return all kinds of subscriptions (including expired and canceled)
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-07-16 20:49:57 +03:00
vovavovavovavova
a10c8051d5 Source PostHog: Use account information for checking the connection (#4692)
* this should fix the check if no records in annotations stream

* update schemas for new SAT requirements && apply user hint upgrade on wrong api key

* save schema upd

* upd insights schema

* upd insights schema2

* upd insights schema3

* upd insights schema4

* upd insights schema5 (null is joking)

* upd insights schema6 (null is joking)

* upd insights schema7

* upd insights schema8

* upd insights schema8

* bump version && docs
2021-07-15 10:52:26 +03:00
oleh.zorenko
b405c9c0be 🐛 Source GitHub: fix bug with IssueEvents stream and add handling for rate limiting (#4708)
* Few updates for GitHub source

Set correct `cursor_field` for `IssueEvents` stream.
Add rate limit handling.
Add handling for 403 error.
Add handling for 502 error.

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-07-14 20:49:01 -04:00
Marcos Marx
b1107381b1 🐛 Source Slack: add float_ts field (#4683)
* rename float_ts to ts cursor_field

* add float_ts

* change float_ts to number

* change channel_msg

* bump version

* increase default timeout_seconds slack acc test

* timeout_seconds to 1750

* timeout_seconds to 3600 :p

* add changelog for slack connector
2021-07-14 18:24:08 -03:00
George Claireaux
43d86114a3 fixed broken links and styling (#4747) 2021-07-14 18:35:54 +01:00
Subodh Kant Chaturvedi
cc7b376044 🎉 Source MSSQL: implementation for CDC (#4689)
* first few classes for mssql cdc

* wip

* mssql cdc working against unit tests

* increment version

* add cdc acceptance test

* tweaks

* add file

* working on comprehensive tests

* change isolation from snapshot to read_committed_snapshot

* finalised type tests

* Revert "change isolation from snapshot to read_committed_snapshot"

This reverts commit 20c6768071.

* small docstring fix

* remove unused imports

* stress test fixes

* minor formatting improvements

* mssql cdc docs

* finish off cdc docs

* format fix

* update connector version

* add to changelog

* fix for sql server agent offline failing cdc enable on tables

* final structure

* few more updates

* undo unwanted changes

* add abstract test + more refinement

* remove CDC metadata to debezium

* use new cdc abstraction for mysql

* undo wanted change

* use cdc abstraction for postgres

* add files

* pull in latest changes

* ready

* rename class + add missing property

* use renamed class + move constants to MySqlSource

* use renamed class + move constants to PostgresSource

* move debezium to bases + upgrade debezium version + review comments

* downgrade version + minor fixes

* bring in latest changes from cdc abstraction

* reset to minutes

* bring in the latest changes

* format

* fix build

* address review comments

* bring in latest changes

* bring in latest changes

* use common abstraction for CDC via debezium for sql server

* remove debezium from build

* finalise PR

* should return Optional

* pull in latest changes

* pull in latest changes

* address review comments

* use common abstraction for CDC via debezium for mysql (#4604)

* use new cdc abstraction for mysql

* undo wanted change

* pull in latest changes

* use renamed class + move constants to MySqlSource

* bring in latest changes from cdc abstraction

* format

* bring in latest changes

* pull in latest changes

* use common abstraction for CDC via debezium for postgres (#4607)

* use cdc abstraction for postgres

* add files

* ready

* use renamed class + move constants to PostgresSource

* bring in the latest changes

* bring in latest changes

* pull in latest changes

* lower version for tests to run on CI

* format

* Update docs/integrations/sources/mssql.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* addressing review comments

* fix for testGetTargetPosition

* format changes

Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-07-14 17:15:55 +01:00
Abhi Vaidyanatha
1f3acff869 Remove statement about Postgres connector being based on Singer (#4740)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-13 23:22:21 -07:00
Abhi Vaidyanatha
4f0170bd7d Update docs on GitHub connector now that its Airbyte native (#4739)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-13 23:19:37 -07:00
Yaroslav Dudar
20f9c2da62 🎉 New source: Typeform (#4541)
Typeform source: Forms and Responses streams
2021-07-13 13:06:50 +03:00
Sherif A. Nada
258a905469 🎉 Destination Snowflake: tag snowflake traffic with airbyte ID to enable optimizations from Snowflake (#4713) 2021-07-13 00:44:57 -07:00
Dmytro
3add914ada 🐛 Source Square: Update _send_request method due to changes in Airbyte CDK (#4645) 2021-07-13 10:27:34 +03:00
Oliver Meyer
21c961e104 🎉 New source: Dixa (#4358) 2021-07-09 09:06:04 -07:00
Abhi Vaidyanatha
15971e89b1 0.27.1 Connector Patch Notes (#4646)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-09 00:15:28 -07:00
vovavovavovavova
db223a4d06 🎉 New Source: Zendesk Sunshine (#4359)
* pre-PR

* add git config

* format

* Update airbyte-integrations/connectors/source-zendesk-sunshine/requirements.txt

upd requirements.txt remove extra

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Update airbyte-integrations/connectors/source-zendesk-sunshine/source_zendesk_sunshine/streams.py

backoff time int to float (btw real return type in headers is integer)

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* requested changes

* fix newline absence && rm unnecessary temp file

* url_base to property

* rm extra var coming property

* rm extra var coming property

* save

* finishing updating the documentation

* forgotten definition

* add nullable to pass the test

* fix date in the log

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-07-09 09:55:40 +03:00
midavadim
ca3befee14 🎉 New Source: Paypal Transaction (#4240)
* Added spec.json

* Initialization

* added oauth2 autorization

* added spec, check, discover + catalogs/configurared_catalogs

* updated request_params

* added paging, slicing (1d)

* Use oath2 for paypal

* incremental sync, acceptance test

* incremental sync, acceptance test

* Added spec.json

* Initialization

* added oauth2 autorization

* added spec, check, discover + catalogs/configurared_catalogs

* updated request_params

* added paging, slicing (1d)

* Use oath2 for paypal

* incremental sync, acceptance test

* incremental sync, acceptance test

* Added spec.json

* Initialization

* added oauth2 autorization

* added spec, check, discover + catalogs/configurared_catalogs

* updated request_params

* added paging, slicing (1d)

* Use oath2 for paypal

* incremental sync, acceptance test

* updated slices and api limits, added validation for input dates

* added tests, fixed cursor related information in schemas and configured catalogs, removed old comments, re-arranged Base PaypalTransactionStream class

* added input param 'env' to support production and sandbox envs

* added support for sandbox option, updated pattern for optional end date option

* added github secrets

* added support for sandbox option, updated pattern for optional end date option

* fixed Copyright date, removed debug mesages

* added docs

* fix for test failure - The sync should produce at least one STATE message

* removed optional parameter 'end_date'

* removed detailed info about balances schema

* Delete employees.json

* Delete customers.json

* Added requests_per_minute rate limit

* added unit tests, added custom backoff

* added test for stream slices with stream state

* removed comments

* updated docs pages

* fixed format for json files

* fixed types in schemas and link to the schema. fixed primary key for Transactions stream

* updated stream slices

* Updated tests, unified stream_slices for both streams, all instance variables instantiated directly in __init__ method

* added CHANGELOG.md

* Added build seeds

* fixed closing double quotation mark

* added paypal entry in builds.md

* add fixture helper

* added paypal transaction generator script

* fixed styling

* maximum allowed start_date is extracted from API response now.

* fixed schemas

* fixed schemas - removed datetime

* now maximum_allowed_start_date is identified by last_refreshed_datetime attr in API response.

* added possibility to specify additional properties

Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-07-08 18:47:00 +03:00
vovavovavovavova
6fdf1ef6ae Mailchimp fix url-base (#4621)
* minimal change to show acceptance test failure

* exactly fix

* bump version and readme

* upd
2021-07-08 18:15:54 +03:00
Yevhenii
c8b8b526f2 🎉 Instagram Source: Migrate to use CDK, upgrade API version and fix failures with HTTP status 400 (#4210)
* handling 400 HTTP error with Backoff
* migrate to CDK

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-07-07 14:11:27 -04:00
vovavovavovavova
4774edf2ca add source-definitions (#4601)
* add source-definitions

* tmp: add docs

* upd docs

* add changelog

* changes

* upd docs/integrations/readme
2021-07-07 20:48:20 +03:00
Dmytro
b8b205a3ff 🎉 New source: Square (#4439) 2021-07-07 20:20:18 +03:00
oleh.zorenko
3e37e904b1 Source GitHub: few updates (#4598)
* Few updates for GitHub connector

Add `AIRBYTE_ENTRYPOINT` to `Dockerfile`.
Add `Changelog` section to `docs/integrations/sources/github.md`.
Remove `source-github/CHANGELOG.md` file.

* Bump GitHub connector version
2021-07-07 15:48:47 +03:00
Marcos Marx
6201100e72 Add entrypoint and bump version for remaining connectors. (#4539)
See the PR descriptions for all the connectors published.
2021-07-07 17:37:22 +08:00
Marcos Marx
31555bab4d bump stripe version (#4587) 2021-07-06 21:13:24 -03:00
oleh.zorenko
84ba3e79b3 🎉 Native GitHub connector (#4174)
* GitHub source initial version

All streams are in `full_refresh` mode only.

* Run format command

* Add handling for 404 error in Teams stream

* Update creds naming for github

* Update acceptance tests

* Update check_connection()

* Code review fixes

Remove `/integration_tests/catalog.json` file.
Use `Collaborators` class in `check_connection()` function.
Remove excessive data from GithubStream class.

* Remove unused import

* Implement requested changes

* Add incremental streams

* Remove `supports_incremental`

* Update docs

* Implement requested changes

* Implement requested changes

* Implement requested changes

* Bump connector version
2021-07-06 17:06:34 +03:00
jaimefr
2efac85c1f add deprecation info for google adwords (#4561) 2021-07-06 09:27:04 -03:00
Yevhenii
94f1d217b8 🎉 Native Gitlab connector (#4027)
* Create Airbyte-Native connector for Gitlab

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-07-06 15:25:01 +03:00
Oleksandr
9517fae702 🎉 Source Shopify - change the default stream cursor_field to 'updated_at' where possible (#4472)
#4472 - Source Shopify - change the default stream cursor_field to 'updated_at' where possible

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-07-05 15:42:35 +03:00
Vadym
9531c4d4ec 🎉 New source: airbyte-native twilio connector (#4070)
* Add Twilio CDK source

* Implement all streams full refresh support.

* Complete all streams and configs.

* Apply code format

* Fix streams data response.
Update streams which should not be incremental.
Fix configs.

* Fix incremental streams.
Fix configs.

* Fix acceptance tests config.

* Fix stream schemas.

* Fix messages stream schema.

* Catalogs fixed

* Update setup requirements.

* Add standard CI creds

* Update build.gradle and sample config files to remove unused references

* Updated sample config files json format

* Update streams.py

* Update incremental streams cursor_field formatting

* Update streams and test config

* Add no_empty_streams_no_usage_records_catalog.json

* Update streams code style.
Add twilio.md docs.
Other PR style fixes.

* Update twilio.md docs.

* Odd docstring removed.

* Uncomment tests.

* Add Docker AIRBYTE_ENTRYPOINT
2021-07-03 20:59:38 +03:00