1
0
mirror of synced 2026-01-05 21:02:13 -05:00
Commit Graph

274 Commits

Author SHA1 Message Date
Sherif A. Nada
e5719e0b3e Fix typo in GA docs 2021-07-21 10:51:48 -07:00
Sherif A. Nada
05bd157e50 📚 improve mongo docs and param descriptions (#4870) 2021-07-20 13:51:01 -07:00
Oleksandr
7d538a7ce6 🐛 Source JIRA: Fix DBT failing normalization on Labels schema. (#4817)
(#4817) 🐛 Source JIRA: Fix DBT failing normalization on `Labels` schema. 

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-07-20 10:26:58 +03:00
Daniel Mateus Pires
c4de62e77a 🎉 New source: US census (#4228)
Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-07-20 00:17:53 -07:00
Sherif A. Nada
1ba4304581 🐛 Sendgrid source: Gracefully handle malformed responses from sendgrid API (#4839) 2021-07-19 19:55:33 -07:00
midavadim
472857d672 🎉 Source mixpanel: migration to CDK (#4566)
* Mixpanel initiation

* copied schemas and specs file from singer connector

* authentication and a few streams

* Added Funnels + FunnelsList

* Added example of funnel response

* added incremental Funnels stream with tests

* added Annotations, CohortMembers, Engage, Cohorts, Funnels

* added Revenue

* fixed formatting

* fixed variable names

* fixed cohort_members and updated export streams

* moved start_date and date checks into SourceMixpanel class

* added error handling

* added unit test, update docs and ci creds

* fix url base for export stream

* added full and incremental read for export stream

* updated acceptance tests, added limit correction based on number of streams, export cursor is stored in datatime string

* Funnel stream - added complex state which contains state for each funnel

* added attribution windows support and project timezone config

* fixed formatting

* added default timezone

* added dynamic schema generation for Engage and Export streams

* fixed formatting

* fixed ability to pass start_date in datetime format as well

* fixed ability to pass start_date in datetime format as well

* added additional_properties field for dynamic schemas. updates regex for start_date matching to support old config file

* fixed formatting

* export stream - convert all values to default type - string

* added schema ref

* added new properties for funnel stream

* fixed formatting in funnel schema

* added build related files

* update changelog

* fixed and added comments, renamed rate_limit variable

* fixed formatting

* changed normalization for reserved mixpanel attributes like $browser

* alphabetise spec fields

* added description about API limit handling

* updated comment
2021-07-20 02:31:13 +03:00
Sherif A. Nada
c755c8f244 Update GAds docs to indicate incremental support 2021-07-19 16:06:22 -07:00
Eugene Kulak
b05ab17520 🐛 Source Instagram: Read previous state format and upgrade it (#4805)
* few fixes for user_insights state

* support old state format

* format

* bump

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-07-19 18:26:59 -04:00
Oleksandr
84aad359aa 📚 Source Shopify: migrate to new sandbox, update API version to 2021-07 (#4830)
(#4830) Source Shopify: migrate to new sandbox, update API version to 2021-07

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-07-19 19:57:11 +03:00
Vladimir remar
219389b2d1 🐛 Source Facebook: Improve rate limit management (#4820)
* Improve rate limit management

* bump version

* facebook-marketing.md update the changelog
2021-07-19 09:08:10 -04:00
Vadym
24be682163 🎉 New source: Pipedrive connector (#4686)
* Add pipedrive source initial

* Add initial schemas.
Add MVP source implementation.

* Implement MVP streams

* Complete MVP streams implementation

* Apply schema format

* Add test creds

* Update streams.py
Fix schemas

* Update replication_start_date format.
Add extra pagination condition

* Refactor streams, remove unused classes.

* Add pipedrive.md docs file.
Add Pipedrive source definitions.

* Add json source definition.

* Update spec.json

* Add docs mentions throughout the project files
2021-07-19 15:08:23 +03:00
Serhii Lazebnyi
9b2388cd31 🎉 Source intercom: migration to CDK (#4676)
* Added Intercom implementation

* Updated segments docs

* Updated _send_request method to new airbyte-cdk version

* Updated cursor field to datetime string

* Added filtering by state for incremental sync

* Updated cursor paths for test incremental sync

* Added dict type validation to get_data method

* Updated catalog

* Updated typing for start_date

* Updated singer seed to cdk seed

* Updated connector docs

* Updated sample config file

* Sorted streams alphabetically

* Removed placeholder comments

* Renamed rate_limit to queries_per_hour

* Updated common sleep time to backoff_time method
2021-07-19 14:14:51 +03:00
Anna Lvova
28239a87a1 🎉 Stripe Source: Fix subscriptions stream to return all kinds of subscriptions (including expired and canceled) (#4669)
#4669 Stripe Source: Fix subscriptions stream to return all kinds of subscriptions (including expired and canceled)
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-07-16 20:49:57 +03:00
vovavovavovavova
a10c8051d5 Source PostHog: Use account information for checking the connection (#4692)
* this should fix the check if no records in annotations stream

* update schemas for new SAT requirements && apply user hint upgrade on wrong api key

* save schema upd

* upd insights schema

* upd insights schema2

* upd insights schema3

* upd insights schema4

* upd insights schema5 (null is joking)

* upd insights schema6 (null is joking)

* upd insights schema7

* upd insights schema8

* upd insights schema8

* bump version && docs
2021-07-15 10:52:26 +03:00
oleh.zorenko
b405c9c0be 🐛 Source GitHub: fix bug with IssueEvents stream and add handling for rate limiting (#4708)
* Few updates for GitHub source

Set correct `cursor_field` for `IssueEvents` stream.
Add rate limit handling.
Add handling for 403 error.
Add handling for 502 error.

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-07-14 20:49:01 -04:00
Marcos Marx
b1107381b1 🐛 Source Slack: add float_ts field (#4683)
* rename float_ts to ts cursor_field

* add float_ts

* change float_ts to number

* change channel_msg

* bump version

* increase default timeout_seconds slack acc test

* timeout_seconds to 1750

* timeout_seconds to 3600 :p

* add changelog for slack connector
2021-07-14 18:24:08 -03:00
George Claireaux
43d86114a3 fixed broken links and styling (#4747) 2021-07-14 18:35:54 +01:00
Subodh Kant Chaturvedi
cc7b376044 🎉 Source MSSQL: implementation for CDC (#4689)
* first few classes for mssql cdc

* wip

* mssql cdc working against unit tests

* increment version

* add cdc acceptance test

* tweaks

* add file

* working on comprehensive tests

* change isolation from snapshot to read_committed_snapshot

* finalised type tests

* Revert "change isolation from snapshot to read_committed_snapshot"

This reverts commit 20c6768071.

* small docstring fix

* remove unused imports

* stress test fixes

* minor formatting improvements

* mssql cdc docs

* finish off cdc docs

* format fix

* update connector version

* add to changelog

* fix for sql server agent offline failing cdc enable on tables

* final structure

* few more updates

* undo unwanted changes

* add abstract test + more refinement

* remove CDC metadata to debezium

* use new cdc abstraction for mysql

* undo wanted change

* use cdc abstraction for postgres

* add files

* pull in latest changes

* ready

* rename class + add missing property

* use renamed class + move constants to MySqlSource

* use renamed class + move constants to PostgresSource

* move debezium to bases + upgrade debezium version + review comments

* downgrade version + minor fixes

* bring in latest changes from cdc abstraction

* reset to minutes

* bring in the latest changes

* format

* fix build

* address review comments

* bring in latest changes

* bring in latest changes

* use common abstraction for CDC via debezium for sql server

* remove debezium from build

* finalise PR

* should return Optional

* pull in latest changes

* pull in latest changes

* address review comments

* use common abstraction for CDC via debezium for mysql (#4604)

* use new cdc abstraction for mysql

* undo wanted change

* pull in latest changes

* use renamed class + move constants to MySqlSource

* bring in latest changes from cdc abstraction

* format

* bring in latest changes

* pull in latest changes

* use common abstraction for CDC via debezium for postgres (#4607)

* use cdc abstraction for postgres

* add files

* ready

* use renamed class + move constants to PostgresSource

* bring in the latest changes

* bring in latest changes

* pull in latest changes

* lower version for tests to run on CI

* format

* Update docs/integrations/sources/mssql.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* addressing review comments

* fix for testGetTargetPosition

* format changes

Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-07-14 17:15:55 +01:00
Abhi Vaidyanatha
1f3acff869 Remove statement about Postgres connector being based on Singer (#4740)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-13 23:22:21 -07:00
Abhi Vaidyanatha
4f0170bd7d Update docs on GitHub connector now that its Airbyte native (#4739)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-13 23:19:37 -07:00
Yaroslav Dudar
20f9c2da62 🎉 New source: Typeform (#4541)
Typeform source: Forms and Responses streams
2021-07-13 13:06:50 +03:00
Sherif A. Nada
258a905469 🎉 Destination Snowflake: tag snowflake traffic with airbyte ID to enable optimizations from Snowflake (#4713) 2021-07-13 00:44:57 -07:00
Dmytro
3add914ada 🐛 Source Square: Update _send_request method due to changes in Airbyte CDK (#4645) 2021-07-13 10:27:34 +03:00
Oliver Meyer
21c961e104 🎉 New source: Dixa (#4358) 2021-07-09 09:06:04 -07:00
Abhi Vaidyanatha
15971e89b1 0.27.1 Connector Patch Notes (#4646)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-09 00:15:28 -07:00
vovavovavovavova
db223a4d06 🎉 New Source: Zendesk Sunshine (#4359)
* pre-PR

* add git config

* format

* Update airbyte-integrations/connectors/source-zendesk-sunshine/requirements.txt

upd requirements.txt remove extra

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Update airbyte-integrations/connectors/source-zendesk-sunshine/source_zendesk_sunshine/streams.py

backoff time int to float (btw real return type in headers is integer)

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* requested changes

* fix newline absence && rm unnecessary temp file

* url_base to property

* rm extra var coming property

* rm extra var coming property

* save

* finishing updating the documentation

* forgotten definition

* add nullable to pass the test

* fix date in the log

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-07-09 09:55:40 +03:00
midavadim
ca3befee14 🎉 New Source: Paypal Transaction (#4240)
* Added spec.json

* Initialization

* added oauth2 autorization

* added spec, check, discover + catalogs/configurared_catalogs

* updated request_params

* added paging, slicing (1d)

* Use oath2 for paypal

* incremental sync, acceptance test

* incremental sync, acceptance test

* Added spec.json

* Initialization

* added oauth2 autorization

* added spec, check, discover + catalogs/configurared_catalogs

* updated request_params

* added paging, slicing (1d)

* Use oath2 for paypal

* incremental sync, acceptance test

* incremental sync, acceptance test

* Added spec.json

* Initialization

* added oauth2 autorization

* added spec, check, discover + catalogs/configurared_catalogs

* updated request_params

* added paging, slicing (1d)

* Use oath2 for paypal

* incremental sync, acceptance test

* updated slices and api limits, added validation for input dates

* added tests, fixed cursor related information in schemas and configured catalogs, removed old comments, re-arranged Base PaypalTransactionStream class

* added input param 'env' to support production and sandbox envs

* added support for sandbox option, updated pattern for optional end date option

* added github secrets

* added support for sandbox option, updated pattern for optional end date option

* fixed Copyright date, removed debug mesages

* added docs

* fix for test failure - The sync should produce at least one STATE message

* removed optional parameter 'end_date'

* removed detailed info about balances schema

* Delete employees.json

* Delete customers.json

* Added requests_per_minute rate limit

* added unit tests, added custom backoff

* added test for stream slices with stream state

* removed comments

* updated docs pages

* fixed format for json files

* fixed types in schemas and link to the schema. fixed primary key for Transactions stream

* updated stream slices

* Updated tests, unified stream_slices for both streams, all instance variables instantiated directly in __init__ method

* added CHANGELOG.md

* Added build seeds

* fixed closing double quotation mark

* added paypal entry in builds.md

* add fixture helper

* added paypal transaction generator script

* fixed styling

* maximum allowed start_date is extracted from API response now.

* fixed schemas

* fixed schemas - removed datetime

* now maximum_allowed_start_date is identified by last_refreshed_datetime attr in API response.

* added possibility to specify additional properties

Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-07-08 18:47:00 +03:00
vovavovavovavova
6fdf1ef6ae Mailchimp fix url-base (#4621)
* minimal change to show acceptance test failure

* exactly fix

* bump version and readme

* upd
2021-07-08 18:15:54 +03:00
Yevhenii
c8b8b526f2 🎉 Instagram Source: Migrate to use CDK, upgrade API version and fix failures with HTTP status 400 (#4210)
* handling 400 HTTP error with Backoff
* migrate to CDK

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-07-07 14:11:27 -04:00
vovavovavovavova
4774edf2ca add source-definitions (#4601)
* add source-definitions

* tmp: add docs

* upd docs

* add changelog

* changes

* upd docs/integrations/readme
2021-07-07 20:48:20 +03:00
Dmytro
b8b205a3ff 🎉 New source: Square (#4439) 2021-07-07 20:20:18 +03:00
oleh.zorenko
3e37e904b1 Source GitHub: few updates (#4598)
* Few updates for GitHub connector

Add `AIRBYTE_ENTRYPOINT` to `Dockerfile`.
Add `Changelog` section to `docs/integrations/sources/github.md`.
Remove `source-github/CHANGELOG.md` file.

* Bump GitHub connector version
2021-07-07 15:48:47 +03:00
Marcos Marx
6201100e72 Add entrypoint and bump version for remaining connectors. (#4539)
See the PR descriptions for all the connectors published.
2021-07-07 17:37:22 +08:00
Marcos Marx
31555bab4d bump stripe version (#4587) 2021-07-06 21:13:24 -03:00
oleh.zorenko
84ba3e79b3 🎉 Native GitHub connector (#4174)
* GitHub source initial version

All streams are in `full_refresh` mode only.

* Run format command

* Add handling for 404 error in Teams stream

* Update creds naming for github

* Update acceptance tests

* Update check_connection()

* Code review fixes

Remove `/integration_tests/catalog.json` file.
Use `Collaborators` class in `check_connection()` function.
Remove excessive data from GithubStream class.

* Remove unused import

* Implement requested changes

* Add incremental streams

* Remove `supports_incremental`

* Update docs

* Implement requested changes

* Implement requested changes

* Implement requested changes

* Bump connector version
2021-07-06 17:06:34 +03:00
jaimefr
2efac85c1f add deprecation info for google adwords (#4561) 2021-07-06 09:27:04 -03:00
Yevhenii
94f1d217b8 🎉 Native Gitlab connector (#4027)
* Create Airbyte-Native connector for Gitlab

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-07-06 15:25:01 +03:00
Oleksandr
9517fae702 🎉 Source Shopify - change the default stream cursor_field to 'updated_at' where possible (#4472)
#4472 - Source Shopify - change the default stream cursor_field to 'updated_at' where possible

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-07-05 15:42:35 +03:00
Vadym
9531c4d4ec 🎉 New source: airbyte-native twilio connector (#4070)
* Add Twilio CDK source

* Implement all streams full refresh support.

* Complete all streams and configs.

* Apply code format

* Fix streams data response.
Update streams which should not be incremental.
Fix configs.

* Fix incremental streams.
Fix configs.

* Fix acceptance tests config.

* Fix stream schemas.

* Fix messages stream schema.

* Catalogs fixed

* Update setup requirements.

* Add standard CI creds

* Update build.gradle and sample config files to remove unused references

* Updated sample config files json format

* Update streams.py

* Update incremental streams cursor_field formatting

* Update streams and test config

* Add no_empty_streams_no_usage_records_catalog.json

* Update streams code style.
Add twilio.md docs.
Other PR style fixes.

* Update twilio.md docs.

* Odd docstring removed.

* Uncomment tests.

* Add Docker AIRBYTE_ENTRYPOINT
2021-07-03 20:59:38 +03:00
Sherif A. Nada
9079cbdf78 Publish PR (#4121) (#4495)
Co-authored-by: Rodrigo Parra <rodpar07@gmail.com>
2021-07-02 13:34:57 -07:00
Rodrigo Parra
4b0c3494ae 🎉 Source Shopify: Add support for draft orders (#4121) 2021-07-02 13:33:47 -07:00
Abhi Vaidyanatha
d985fe73f4 Add note about rate limits with the HubSpot API (#4491)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-01 23:04:35 -07:00
Sherif A. Nada
9708e4c8de 🐛 bugfix Source Okta: Fix endless loop while syncing data from logs stream (#4464)
Co-authored-by: sabifranjo <sabifranjo@gmail.com>
Co-authored-by: Sabolc Franjo <sabolc.franjo@ev-box.com>
2021-07-01 17:39:51 -07:00
Eugene
4399656ec9 🎉 Source Cockroachdb: Added new source for Cockroachdb (#4382)
* [1705] Added cockroachdb source connector
2021-07-01 22:04:21 +03:00
Sherif A. Nada
71c7cecd74 Source postgres: document that replicating views is supported (#4008) 2021-06-28 00:11:37 -05:00
Kirill Chernov
40aed2249d 🎉 fix Google Adwords CI & publish new version (#4205)
* added SAT tests, deleted incremental test sfrom pipeline

* pr changes and tests uncommented

* deleted facebook file from pr

* deleted facebook file from pr

* removed tap adwords from install_requires

* Update airbyte-integrations/connectors/source-google-adwords-singer/acceptance-test-config.yml

* added changelog info

* version bump

* version bump

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-06-26 01:45:22 +03:00
LiRen Tu
8954a99290 Bump source looker connector version (#4350)
* Bump looker connector version

* Update changelog

* Format code
2021-06-25 11:23:16 -07:00
Oleksandr
0e0d0a102e 🐛 Bugfix for pagination in Shopify Source Connector #4284 (#4290)
#4284 - Bugfix for pagination in Shopify Source Connector

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-06-25 19:45:45 +03:00
Michael Irvine
842f10bd83 🎉 Source Looker: add run-look endpoint (#3911)
* looker run_looks method works with normalization

* update catalogs

* validate that look(s) entered actually exist at config time

* field type lookup works, though it is a bit messy

* don't re-run explore query if we have already run it

* add more type mappings

* add some docstrings and comments

* add title for ease of use

* Revert "update catalogs"

This reverts commit 7ecaaa4d0d.

* fix configuration file formatting

* fix some issues from code review and reorder methods

* update docs to include run_look endpoint
2021-06-25 08:31:56 -07:00
Yaroslav Dudar
2ccb1c2de0 🎉 New source: AWS CloudTrail (#4122)
AWS CloudTrail connector initial release
2021-06-25 11:10:07 +03:00