1
0
mirror of synced 2026-01-04 09:04:47 -05:00
Commit Graph

152 Commits

Author SHA1 Message Date
Maksym Pavlenok
c0cc288f97 🎉 CI secrets: migration to GSM (#7528)
* fix 404 responses for the ticket_comments stream

* add unit test

* add unit test

* add oauth2 access token

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* update spec file

* update CI secrets logic

* update CI secrets logic

* remove debug data

* add a debug message

* fix json convertation

* fix json convertation

* support one secret by several connectors

* Update tools/bin/ci_credentials.sh

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update tools/bin/ci_credentials.sh

Co-authored-by: LiRen Tu <tuliren@gmail.com>

* update function names

* update docs

* reset failed changes

* update json set value

* update secrets json key

* Update tools/bin/ci_credentials.sh


* add JSON validation

* Update docs/connector-development/README.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* update doc

* Update README.md

* rename the function write_standard_creds

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: LiRen Tu <tuliren@gmail.com>
2021-11-10 10:49:44 +02:00
midavadim
c3c3e318c6 🎉 intercom added oauth support (#7060)
* Added oauth support

* fixed description of access_token field in Oauth authentication

* test selenium

* add java implementation

* remove commented dependencies| remove star import

* remove System.out.println

* added support for old format of spec.json files.

* added support for old format of spec.json files.

* added support for old format of spec.json files

* added credential with new spec format

* bumped connector version

* Reverted format of spec.json file

* Override extractRefreshToken method

* java oauth test fixes, minor updates, cloud spec version update

* Fixed docs strings

* Added HttpClient argument

* revert changes in seed folder to follow new dev flow

* Updated image version in seed

* Fixed import of IntercomOAuthFlow

* updated IntercomOAuthFlow to match base class changes

* Change image version to 0.1.8

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
2021-11-09 22:13:46 +02:00
Serhii Lazebnyi
2d2965b879 🎉 New Source: Pinterest (#7493)
* Add Pinterest implementation

* Updated to review

* Added rate limits and PR number to doc

* Fixed unit tests and data slicing bug

* Update airbyte-integrations/connectors/source-pinterest/source_pinterest/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-pinterest/build.gradle

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-pinterest/source_pinterest/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Updated to review

* Updated window in days

* Fixed unit test

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-11-09 12:54:12 +02:00
James Sewa Wilson
46fa44ab80 🎉 New Source: Commerce Tools (#5957)
* adding new source connector Commercetools

* adding new source connector Commercetools

* quick fix

* quick config

* changes

* format files

* add ci credentials run

* correct bin creds

* add publish cmd

* add spec host enum

* correct itnegration tests

* run seed file

Co-authored-by: jsewawilson <jwilson@pebblepost.com>
Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2021-11-08 23:59:29 -03:00
Tuan Nguyen
0b9b8ba472 🎉 New Source: Monday (#7168)
* add items stream

* add boards stream

* add remaining streams

* check point

* lint

* fix unit test & linting

* Delete launch.json

* fix schema

* add retry logic

* Update airbyte-integrations/connectors/source-monday/source_monday/source.py

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* small fix + add creds

* bump version and config file

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2021-11-06 17:32:13 -03:00
vitaliizazmic
1f295f24e6 🎉 Source Google Directory: support oauth
* Source Google Directory #6265 - add oauth support

* Source Google Directory #6265 - update credentials

* Source Google Directory #6265 - fixing according to PR

* Source Google directory #6265 - update docs

* Source Google directory #5190 - update doc

* Source Google Directory #6265 - resolve merge conflict

* Source Google Directory #6265 - SAT for oauth

* Source Google Directory #6265 - bump version and update changelog

* Source Google Directory #6265 - bump version and update changelog (update publish)
2021-11-05 21:40:56 +02:00
Harshith Mullapudi
af1c62f0c6 Publish new source Confluence #7241 (#7666)
* initial commit

* add remaining streams

* fix warning and unit test

* clean up

* revert unrelated files

* remove state files

* lint

* add documentation strings

* added confluence credentials to ci

* fix: removed audit from configured catalog as it cannot be pulled in free account

* fix: format

Co-authored-by: Tuan Nguyen <anhtuan.nguyen@me.com>
2021-11-05 15:17:26 +05:30
Marcos Marx
0e09ac3032 Publish PR 6963: new source freshsales (#7585)
* initial commit

* finish implementing full_refresh

* add other tables

* cleaning up

* add docs and use requests_native_auth

* fix function return different number of values

* change package author

* fix schema

* fix schema bug

* linting

* fix unit test

* clean up

* add null for schemas

* remove fc_widget_collaboration col

* change accpt test

* add creds

* run format

* add config files

* run format

Co-authored-by: Tuan Nguyen <anhtuan.nguyen@me.com>
2021-11-03 01:06:38 -03:00
Harshith Mullapudi
8ddf686e9e added credentials to CI for new source sentry
* add credentials to CI for new source sentry

* add ci cred
2021-11-02 19:55:58 +05:30
Marcos Marx
c846188772 Publish PR 6966: new source retently (#7552)
* retently

* PR comments

* PR comments

* mask api keys

* README

* keep gradle happy

* add github creds

* run format

Co-authored-by: Subhash <subhash@kloud.io>
2021-11-01 23:58:16 -03:00
Marcos Marx
fe118e676e Publish PR 4551: new source delighted correct format (#7540)
* Add Delighted source with people, unsubscribes and bounces streams

* Add survey responses stream

* Add documentation for Delighted source

* publish delighted source

* correct acceptance test

Co-authored-by: Rodrigo Parra <rodpar07@gmail.com>
2021-11-01 22:09:13 -03:00
Harshith Mullapudi
bdaec1cf3f add credentials in for New Source Paystack: #7214 (#7529)
* added credentials to ci for new source paystack

* format
2021-11-01 23:49:53 +05:30
Marcos Marx
2c8151fef2 Add creds for strava, lemlist, amazon sqs and freshservice (#7388)
* add creds for strava, lemlist, amazon sqs and freshservice

* correct source name in ci_creds
2021-10-26 11:42:36 -03:00
Marcos Marx
c83cafc5c1 add salesloft credentials github wf and ci script (#7275) 2021-10-22 01:28:46 -03:00
Marcos Marx
4efadb228c add onesignal credentials + gradlew format (#7270) 2021-10-21 23:55:06 -03:00
Maksym Pavlenok
059645fad4 🎉 Source File: separate secure fork (#6768)
* create a fork for the source-file connector

* update docs and tests

* update docs and tests

* add secrets

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* refactoring after reviews

* correction of doc

* update spec file



Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
2021-10-20 12:28:26 +03:00
George Claireaux
1d3a17a8fb 🎉 Source S3 - memory & performance optimisations + advanced CSV options (#6615)
* memory & performance optimisations

* address comments

* version bump

* added advanced_options for reading csv without header, and more custom pyarrow ReadOptions

* updated to use the latest airbyte-cdk

* updated docs

* bump source-s3 to 0.1.6

* remove unneeded lines

* Use the all dep ami for python builds.

* ec2-instance-id should be ec2-image-id

* ec2-instance-id should be ec2-image-id

Co-authored-by: Jingkun Zhuang <Jingkun.Zhuang@icims.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-19 16:50:51 +01:00
Davin Chia
214b158bde Add labels to EC2 build machines. Refactor actions to reduce duplicate start ec2 runner actions. (#7119)
As the title.

I also took the chance to compose the actions, instead of injecting it everywhere.
2021-10-19 18:02:18 +08:00
Maksym Pavlenok
592f7fe2b2 🐛 Source ZenDesk Support: fix 404 responses for the ticket_comments stream (#6513)
* fix 404 responses for the ticket_comments stream

* add unit test

* add unit test

* add oauth2 access token

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* Update docs/integrations/sources/zendesk-support.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* add log message for 404 errors

* Update docs/integrations/sources/zendesk-support.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* remove oauth logic from this PR

* remove oauth logic from this PR

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-16 01:20:41 +03:00
Maksym Pavlenok
713fe61efd 🎉 New Source: TikTok Marketing (#5887)
* create a new connector folder

* add base classes

* add schemas

* add ads/ad_groups streams

* update tests and docs

* add a bootstrap file

* update the base domain name for API

* update the base domain name for API

* update Dockerfile libs

* remove unused comments

* Update airbyte-integrations/connectors/source-tiktok-marketing/bootstrap.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* Update airbyte-integrations/connectors/source-tiktok-marketing/bootstrap.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* fix UI error with input parameters

* fix problem with updated state

* Update airbyte-integrations/connectors/source-tiktok-marketing/source_tiktok_marketing/spec.py

Co-authored-by: Davin Chia <davinchia@gmail.com>

* add an unit test

* update README.md

* bump version

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-15 19:24:14 +03:00
Baz
cfae84cbb0 🎉 Source Shopify: support oauth flow (#6951) 2021-10-14 22:37:30 +03:00
Arthur Galuza
bfe865aebf 🎉 Source Google Sheets: Support connecting via oAuth webflow (#6354)
* Add service account support

* Upd oauth support

* Upd auth creds selector
2021-10-13 23:02:22 +03:00
Arthur Galuza
99b1e91c65 🎉 Source Pipedrive: add oAuth support (#6821)
* Add oauth support

* Upd changelog

* Update airbyte-integrations/connectors/source-pipedrive/source_pipedrive/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-10-12 09:45:46 +03:00
Arthur Galuza
bf3189cd12 🎉 Source Google Analytics v4: Declare oauth parameters in google sources (#6414)
* Upd auth: oauth support

* Rename authorization/credentials, upd spec, refactor

* Add backward compatibility

* Upd CI

* Bump version

* Upd changelog
2021-10-08 12:38:40 +03:00
Yevhenii
00d56e7711 🎉 Source Slack: Implement OAuth support with OAuth authenticator (#6570)
* Source Slack: Implement OAuth support with OAuth authenticator
2021-10-07 14:01:15 +03:00
Sherif A. Nada
b470489983 🧹delete marketo singer source (#6800) 2021-10-06 22:53:21 -07:00
Vadym
351e998394 🎉 Source Hubspot: Support OAuth (#6465)
* Add required scopes to hubspot.md docs

* Update spec with oauth creds.
Update acceptance test config with oauth tests.
Add oauth test creds configs.

* Fix spec.js `oneOf` usage

* Add CDK authenticators usage.
Update source acceptance tests.
Update spec.json.

* Fix source unit tests

* Update hubspot.md scopes docs

* Add spec field titles.
Update comments.

* Update spec.json oauthFlowOutputParameters.

* Update spec.json to pass SAT.
2021-10-04 17:11:34 +03:00
oneshcheret
11561290be 🎉 Source MSSQL: Implement SSH bastion for acceptance/integration tests (#6608)
* 🎉 Source MSSQL: Implement SSH bastion for acceptance/integration tests

* 🎉 Source MSSQL: remove ci creds for mssql source
2021-10-04 16:08:41 +03:00
VitaliiMaltsev
c2561141fc 🎉 Destination Oracle - Added support for connection via ssh tunnel (#6370)
* Oracle destination with ssh tunneling

* add ssh key integration testing for Oracle destination

* fix checkstyle

* add container approach to Oracle Destination Integration Tests

* add container approach to Oracle Destination Integration Tests

* add container approach to Oracle Source Integration Tests/ delete secrets for Oracle db

* Oracle Destination - bump version to 0.1.8

* remove oracle ssh secrets from ci_credentials.sh

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
2021-09-28 14:25:33 +03:00
Baz
2f835e4e39 🎉 New Source: LinkedIn Ads (#5285)
Resolving: https://github.com/airbytehq/airbyte/issues/2589

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-09-25 01:58:58 +03:00
Eugene
6658ecd5aa 🎉 Source Oracle: Added support for connection via SSH tunnels (#6356)
* [5968] Source Oracle - Added option to connect to Oracle DB via SSH (aka bastion server)
2021-09-24 20:52:47 +03:00
irynakruk
5551698002 Source MongoDb: added support via TLS/SSL (#6364)
* updated ssl configs;
fixed replica connection;
fixed incremental read;

* added MongoDbSourceAtlasAcceptanceTest with ssl enabled

* updated docs, moved TLS option for standalone instance type, enabled it by default for other types

* Update mongodb-v2.md

* updated README.md

* updated spec.json

* Code review changes

* updated ci_credentials.sh
2021-09-24 09:22:54 -04:00
Serhii Lazebnyi
26c9546a1c 🎉 New Source: Marketo (#5863)
* Add Marketo implementation

* Updated PR in docs

* Add bootstrap file

* Update to review

* Add gcc to docker
2021-09-23 18:51:47 +03:00
Yevhenii
63425923d0 Remove deprecated connectors: source-google-adwords-singer, source-google-search-console-singer, source-googleanalytics-singer (#6378) 2021-09-23 15:24:49 +03:00
Yevhenii
73162062db 🎉 New Source: Lever Hiring (#6141)
* Lever Hiring - New Connector
2021-09-22 12:51:00 +03:00
Oleksandr Sheheda
59cd5ca308 🎉 Source MSSQL: Added support for connection via SSH tunnels (#5966) (#6318)
* 🎉 Source MSSQL: Added support for connection via SSH tunnels (#5966)

* 🎉 Source MSSQL: Added support for connection via SSH tunnels (#5966)
updated image version

* 🎉 Source MSSQL: Added support for connection via SSH tunnels (#5966)
correction of remarks

* 🎉 Source MSSQL: Added support for connection via SSH tunnels (#5966)
bumb image version
2021-09-21 17:25:04 +03:00
midavadim
12cb3e3f9f 🎉 Source greenhouse: added identification of accessible streams for API keys with limited permissions (#6238)
* Added identification of accessible streams for API keys with limited permissions

* added credentials for API key with limited permissions (for user stream only)

* update docs and image version

* updated error message

* fixed formatting
2021-09-21 15:02:12 +03:00
Eugene
11f64882a9 🎉Source MySQl - Added support for connection via ssh (aka bastion server) (#6146)
* Source MySQL - Added ssh support
2021-09-21 10:57:31 +03:00
Dmytro
7a6da86baa add coverage report (#6045)
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-09-15 10:50:07 +03:00
LiRen Tu
e837048418 🎉 New Destination: Databricks (#5998)
Implement new destination connector for databricks delta lake.
Resolves #2075.

Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-09-14 16:55:53 -07:00
Anna Lvova
03e1e08047 🎉 New Source: Close.com (#5366)
* added close com source

* added docs

* added activities streams

* added tasks streams

* updated doc

* fixed indent

* added docs

* add PR to doc

* fix doc

* fix doc

* update after review

* update after review

* remove streams from empty_streams

* move path from streams to Mixin classes

* update README

* add streams

* update doc

* fix opportunities cursor_field

* remove Roles stream from tests

* add Roles stream to SAT after fix in 5738 PR

* add bootstrap.md

* add bootstrap.md

* update after review

* update after review

* remove state_checkpoint_interval
2021-09-13 17:44:31 +03:00
Arthur Galuza
c0d4652852 🎉 New Source: Google Search Console (#5350)
* Generate Google Search Console connector

* Add schema

* Upd schema

* Upd authenticator

* Add creds retrieving script

* Remove legacy dep

* Upd dockerfile base image

* Add sample config

* Upd source definitions, add ci_credentials injection

* Upd schema

* Upd GSC creds injection

* Cleanup

* Add tzdata

* Upd tzdata installing

* Change base docker image

* Upd streams

* Fix typo

* Upd supported_sync_modes list

* Add multiple site, service account, search type support

* Fix typo

* Upd streams, pagination, multi site support

* Add service account secrets

* Remove source-google-search-console-singer from source definitions

* Upd creds retrieving base image

* Upd schema

* Upd docs

* Add badge

* Upd schema

* Upd docs

* Move the cursor field to the top

* Upd docs
2021-09-10 11:47:02 +03:00
vovavovavovavova
aa9786df4e 🎉 Google Ads improvement: Support user-specified queries (#5302)
*Add google ads custom queries stream

*Display link to gradle scan on PR comment if test build failed
2021-09-10 09:17:16 +03:00
Eugene
bb043b7f50 🐛 Destination BigQuery Denormalized: Fixed compilation error (#5917)
* Fixed destination bigquery denormalized compilation error (caused by https://github.com/airbytehq/airbyte/pull/5614)
2021-09-09 12:54:46 +03:00
Yevhenii
fdb4c852a8 🎉 Salesforce Native Connector: init version (#5619)
* Native Salesforce Connector

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-09-08 14:00:08 +03:00
Charles
8ad43afb07 SSH for Postgres Destination (#5743)
Co-authored-by: George Claireaux <phlair@users.noreply.github.com>
2021-09-07 17:06:25 -07:00
Maksym Pavlenok
e5c44e64b1 🎉 Source S3: support of Parquet format (#5305)
* add parquet parser

* add integration tests for partquet formats

* add unit tests for parquet

* update docs and secrets

* fix incorrect import for tests

* add lib pandas for unit tests

* revert changes of foreign connectors

* update secret settings

* fix config values

* Update airbyte-integrations/connectors/source-s3/source_s3/source_files_abstract/formats/parquet_spec.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* Update airbyte-integrations/connectors/source-s3/source_s3/source_files_abstract/formats/parquet_spec.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* remove some unused default options

* update tests

* update docs

* bump its version

* fix expected test

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
2021-09-05 02:40:49 +03:00
Charles
7bf531a967 SSH for Postgres Source (#5742) 2021-09-02 11:32:04 -07:00
Arthur Galuza
736126330a 🎉 New Source: Facebook Pages (#5158)
* [WIP] Add basic facebook pages support

* Upd schema

* Hardcode metrics to reduce API call amount

* Refactoring

* Upd test configs

* Add secrets to CI

* Fix typo

* Add a source definition

* fix secrets in CI

* Cleanup with black

* Upd metrics

* Cleanup

* Rm start_date, add test

* Upd schema

* Cleanup

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-09-01 19:11:35 +03:00
Eugene
61842ed7c3 🎉 Destination Azure blob storage: introduced new connector with jsonl and csv formats (#5332)
* [3447] Added destination Azure blob storage connector (CSV and JSONL formats)
2021-08-29 12:23:59 +03:00