1
0
mirror of synced 2026-01-03 15:04:01 -05:00
Commit Graph

323 Commits

Author SHA1 Message Date
Marcos Marx
0e09ac3032 Publish PR 6963: new source freshsales (#7585)
* initial commit

* finish implementing full_refresh

* add other tables

* cleaning up

* add docs and use requests_native_auth

* fix function return different number of values

* change package author

* fix schema

* fix schema bug

* linting

* fix unit test

* clean up

* add null for schemas

* remove fc_widget_collaboration col

* change accpt test

* add creds

* run format

* add config files

* run format

Co-authored-by: Tuan Nguyen <anhtuan.nguyen@me.com>
2021-11-03 01:06:38 -03:00
Harshith Mullapudi
8ddf686e9e added credentials to CI for new source sentry
* add credentials to CI for new source sentry

* add ci cred
2021-11-02 19:55:58 +05:30
Dmytro
85381bde3c Surveymonkey: oAuth backend and connector's spec change. (#7524) 2021-11-02 10:40:39 +02:00
Marcos Marx
c846188772 Publish PR 6966: new source retently (#7552)
* retently

* PR comments

* PR comments

* mask api keys

* README

* keep gradle happy

* add github creds

* run format

Co-authored-by: Subhash <subhash@kloud.io>
2021-11-01 23:58:16 -03:00
Marcos Marx
fe118e676e Publish PR 4551: new source delighted correct format (#7540)
* Add Delighted source with people, unsubscribes and bounces streams

* Add survey responses stream

* Add documentation for Delighted source

* publish delighted source

* correct acceptance test

Co-authored-by: Rodrigo Parra <rodpar07@gmail.com>
2021-11-01 22:09:13 -03:00
Harshith Mullapudi
bdaec1cf3f add credentials in for New Source Paystack: #7214 (#7529)
* added credentials to ci for new source paystack

* format
2021-11-01 23:49:53 +05:30
Yevhenii
269298c854 🎉 Source File: Migrate File source to CDK structure (#7387)
* Migrate File source to CDK structure

* fix .dockerignore file

* remove SAT requirements

* update Dockerfile

* change Dockerfile to base images python:3.7-slim

* add SAT tests

* update tests

* add secret/config.json for source-file

* update changelogs
2021-10-29 13:15:13 +03:00
Marcos Marx
2c8151fef2 Add creds for strava, lemlist, amazon sqs and freshservice (#7388)
* add creds for strava, lemlist, amazon sqs and freshservice

* correct source name in ci_creds
2021-10-26 11:42:36 -03:00
Yevhenii
928aa6280c 🐛 Source Hubspot: Fix issue with getting 414 HTTP error for streams (#6954)
* Source Hubspot: Fix issue with getting 414 HTTP error for streams

* update code and schemas

* bump version
2021-10-26 15:37:07 +03:00
Eugene
9a0b8dac6c 🎉Source Clickhouse: added ssl support and "strict-encrypt" connector (#7127)
* [Ticket 6435] Source-ClickHouse: added ssl support

* [Ticket 6716] Source-ClickHouse: added secure only connector
2021-10-25 13:33:25 +03:00
Marcos Marx
c83cafc5c1 add salesloft credentials github wf and ci script (#7275) 2021-10-22 01:28:46 -03:00
Marcos Marx
4efadb228c add onesignal credentials + gradlew format (#7270) 2021-10-21 23:55:06 -03:00
irynakruk
265986b5bc Create a secure-only MongoDb destination (#6945)
* Added mongodb destination strict encrypt
2021-10-20 09:04:30 -04:00
Maksym Pavlenok
059645fad4 🎉 Source File: separate secure fork (#6768)
* create a fork for the source-file connector

* update docs and tests

* update docs and tests

* add secrets

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* refactoring after reviews

* correction of doc

* update spec file



Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
2021-10-20 12:28:26 +03:00
Harsha Teja Kanna
3e7f95c25a 🎉 Support build on MacOS M1 (Apple Silicon) (#7104)
- See this doc for details: https://github.com/airbytehq/airbyte/blob/master/docs/contributing-to-airbyte/developing-locally.md
- Unit test does not work yet.
2021-10-19 11:20:21 -07:00
Jared Rhizor
0f93224769 kube build fixes (#7139)
* upgrade ec2-github-runner

* remaining build fixes

* fix build
2021-10-18 14:46:17 -07:00
Charles
f194f354c7 move basic normalization to connectors_base build (#7068) 2021-10-15 17:26:08 -07:00
Maksym Pavlenok
592f7fe2b2 🐛 Source ZenDesk Support: fix 404 responses for the ticket_comments stream (#6513)
* fix 404 responses for the ticket_comments stream

* add unit test

* add unit test

* add oauth2 access token

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* Update docs/integrations/sources/zendesk-support.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* add log message for 404 errors

* Update docs/integrations/sources/zendesk-support.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* remove oauth logic from this PR

* remove oauth logic from this PR

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-16 01:20:41 +03:00
Maksym Pavlenok
713fe61efd 🎉 New Source: TikTok Marketing (#5887)
* create a new connector folder

* add base classes

* add schemas

* add ads/ad_groups streams

* update tests and docs

* add a bootstrap file

* update the base domain name for API

* update the base domain name for API

* update Dockerfile libs

* remove unused comments

* Update airbyte-integrations/connectors/source-tiktok-marketing/bootstrap.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* Update airbyte-integrations/connectors/source-tiktok-marketing/bootstrap.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* fix UI error with input parameters

* fix problem with updated state

* Update airbyte-integrations/connectors/source-tiktok-marketing/source_tiktok_marketing/spec.py

Co-authored-by: Davin Chia <davinchia@gmail.com>

* add an unit test

* update README.md

* bump version

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-15 19:24:14 +03:00
Baz
cfae84cbb0 🎉 Source Shopify: support oauth flow (#6951) 2021-10-14 22:37:30 +03:00
Christophe Duong
c4620559d7 🎉 Refactor Normalization docker images and upgrade to use dbt 0.21.0 (#6959)
* Split normalization docker images for some connectors with specifics dependencies

* Regenerate (#7003)
2021-10-14 20:29:16 +02:00
Arthur Galuza
bfe865aebf 🎉 Source Google Sheets: Support connecting via oAuth webflow (#6354)
* Add service account support

* Upd oauth support

* Upd auth creds selector
2021-10-13 23:02:22 +03:00
Arthur Galuza
99b1e91c65 🎉 Source Pipedrive: add oAuth support (#6821)
* Add oauth support

* Upd changelog

* Update airbyte-integrations/connectors/source-pipedrive/source_pipedrive/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-10-12 09:45:46 +03:00
irynakruk
77fd7d9e34 Added Mongodb secure only source (#6859)
* Added mongodb secure source
2021-10-11 12:04:27 -04:00
Arthur Galuza
bf3189cd12 🎉 Source Google Analytics v4: Declare oauth parameters in google sources (#6414)
* Upd auth: oauth support

* Rename authorization/credentials, upd spec, refactor

* Add backward compatibility

* Upd CI

* Bump version

* Upd changelog
2021-10-08 12:38:40 +03:00
Yevhenii
00d56e7711 🎉 Source Slack: Implement OAuth support with OAuth authenticator (#6570)
* Source Slack: Implement OAuth support with OAuth authenticator
2021-10-07 14:01:15 +03:00
Dmytro
ae63696d2b Asana source: Add oauth init flow parameters support (#6832) 2021-10-07 10:15:38 +03:00
Sherif Nada
31af81e03a remove marketo singer credentials 2021-10-06 23:33:43 -07:00
Vadym
351e998394 🎉 Source Hubspot: Support OAuth (#6465)
* Add required scopes to hubspot.md docs

* Update spec with oauth creds.
Update acceptance test config with oauth tests.
Add oauth test creds configs.

* Fix spec.js `oneOf` usage

* Add CDK authenticators usage.
Update source acceptance tests.
Update spec.json.

* Fix source unit tests

* Update hubspot.md scopes docs

* Add spec field titles.
Update comments.

* Update spec.json oauthFlowOutputParameters.

* Update spec.json to pass SAT.
2021-10-04 17:11:34 +03:00
oneshcheret
11561290be 🎉 Source MSSQL: Implement SSH bastion for acceptance/integration tests (#6608)
* 🎉 Source MSSQL: Implement SSH bastion for acceptance/integration tests

* 🎉 Source MSSQL: remove ci creds for mssql source
2021-10-04 16:08:41 +03:00
Yaroslav Dudar
7ec756fe12 🎆 Spec.json linter (#6366)
added python script linter for spec.json validation
2021-09-30 13:01:20 +03:00
Jared Rhizor
f88b8313a8 add the ability to use a secret persistence (#6415)
* test exposing secrets in configrepo

* fix local persistence sql

* working propagation, just without check/discover replacements and without feature flagging

* switch if statement

* set up secret persistence for google secrets manager

* add ttl-based secret persistence for check/discover usage in the future

* set up check/discover to pass around necessary parts

* Revert "set up check/discover to pass around necessary parts"

This reverts commit 489d2d5f5d.

* working updates + check/discover operations

* fix additional configs created on deletion

* clean up docker compose file

* finish up configrepo

* make api path optional

* clean up schedulerapp and local testing persistence

* make optional in the worker app

* add rest of feature flagging

* fmt

* remove completed todo

* fix refactoring typo

* fix another refactoring typo

* fix compilation error in test case

* fix tests

* final cleanups

* fix conditional

* address a couple of things

* add hydrator interface

* add replaceAllConfigs

* specfetcher handling

* fix constructor

* fix test

* fix typo

* fix merge build error

* remove extra config

* fix integration test

* fix final piece
2021-09-29 11:53:29 -07:00
Dmytro
b95f366cee Fix publish script for non-connector images (#6537) 2021-09-29 19:34:38 +03:00
Peter McConnell
3c40e989eb improvement: modifying the cli tool so that it can accept API_URL (#6507)
This solves the problem where you may be running airbyte on an endpoint
that isn't http://localhost:8001

Additionally it improves the error experience when this script can not
reach the API.
2021-09-28 08:31:48 -07:00
VitaliiMaltsev
c2561141fc 🎉 Destination Oracle - Added support for connection via ssh tunnel (#6370)
* Oracle destination with ssh tunneling

* add ssh key integration testing for Oracle destination

* fix checkstyle

* add container approach to Oracle Destination Integration Tests

* add container approach to Oracle Destination Integration Tests

* add container approach to Oracle Source Integration Tests/ delete secrets for Oracle db

* Oracle Destination - bump version to 0.1.8

* remove oracle ssh secrets from ci_credentials.sh

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
2021-09-28 14:25:33 +03:00
andriikorotkov
8fa15713c3 🎉 Destination MySQl - Added support for connection via ssh (aka bastion server) (#6317)
* updated mysql tests

* updated mysql tests

* added mysql ssh tunnel tests by key

* fixed remarks

* fixed remarks

* updated DatabricksStreamCopier

* switch to custom file for ssh config in normalization

* updated MySQL SSH tests

* bump version

* get local port properly

* updated assertSameValue for MySQL ssh tunnel

* updated image version and documentation

* updated code style

* updated CI credentials

* updated normalization documentation

Co-authored-by: George Claireaux <george@claireaux.co.uk>
2021-09-28 13:11:32 +03:00
Baz
2f835e4e39 🎉 New Source: LinkedIn Ads (#5285)
Resolving: https://github.com/airbytehq/airbyte/issues/2589

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-09-25 01:58:58 +03:00
Eugene
6658ecd5aa 🎉 Source Oracle: Added support for connection via SSH tunnels (#6356)
* [5968] Source Oracle - Added option to connect to Oracle DB via SSH (aka bastion server)
2021-09-24 20:52:47 +03:00
VitaliiMaltsev
ec3951ba62 🎉 Integration Testing for SSH using a docker container | Postgres Source and Destination update integration tests using ssh bastion in docker container (#6312)
* ssh-test

* add authentification via ssh tunnel with bastion docker host and postgres testcontainer

* created SshBastion class in base-java module

* implement Postgres source basic ssh tunneling connection for integration tests

* implement Postgres source ssh tunneling connection and refactoring SshBastion

* generate keys inside a bastion container

* remove throwing Exception from startTestContainers method

* fix checkstyle

* add documentation and changelog for Posthres source and destination

* update documentation for ssh readme.md | update version fo Postgres source and destination to 0.3.12

* update version of Postgres source and destination to 0.3.12

* removed static variables, removed version bump, rename class to SshBastionContainer, removed ci credentials for ssh Postgres Source and Destination

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
2021-09-24 20:00:06 +03:00
irynakruk
5551698002 Source MongoDb: added support via TLS/SSL (#6364)
* updated ssl configs;
fixed replica connection;
fixed incremental read;

* added MongoDbSourceAtlasAcceptanceTest with ssl enabled

* updated docs, moved TLS option for standalone instance type, enabled it by default for other types

* Update mongodb-v2.md

* updated README.md

* updated spec.json

* Code review changes

* updated ci_credentials.sh
2021-09-24 09:22:54 -04:00
Serhii Lazebnyi
26c9546a1c 🎉 New Source: Marketo (#5863)
* Add Marketo implementation

* Updated PR in docs

* Add bootstrap file

* Update to review

* Add gcc to docker
2021-09-23 18:51:47 +03:00
Christophe Duong
992bdb1cd5 Skip updating spec cache when publishing normalization docker image (#6399) 2021-09-23 15:14:06 +02:00
Yevhenii
63425923d0 Remove deprecated connectors: source-google-adwords-singer, source-google-search-console-singer, source-googleanalytics-singer (#6378) 2021-09-23 15:24:49 +03:00
Yevhenii
73162062db 🎉 New Source: Lever Hiring (#6141)
* Lever Hiring - New Connector
2021-09-22 12:51:00 +03:00
Cameron
a6c48dcc7b Update README.md (#6350)
Minor typo fixes
2021-09-21 22:33:49 -03:00
Oleksandr Sheheda
59cd5ca308 🎉 Source MSSQL: Added support for connection via SSH tunnels (#5966) (#6318)
* 🎉 Source MSSQL: Added support for connection via SSH tunnels (#5966)

* 🎉 Source MSSQL: Added support for connection via SSH tunnels (#5966)
updated image version

* 🎉 Source MSSQL: Added support for connection via SSH tunnels (#5966)
correction of remarks

* 🎉 Source MSSQL: Added support for connection via SSH tunnels (#5966)
bumb image version
2021-09-21 17:25:04 +03:00
midavadim
12cb3e3f9f 🎉 Source greenhouse: added identification of accessible streams for API keys with limited permissions (#6238)
* Added identification of accessible streams for API keys with limited permissions

* added credentials for API key with limited permissions (for user stream only)

* update docs and image version

* updated error message

* fixed formatting
2021-09-21 15:02:12 +03:00
Eugene
11f64882a9 🎉Source MySQl - Added support for connection via ssh (aka bastion server) (#6146)
* Source MySQL - Added ssh support
2021-09-21 10:57:31 +03:00
Jenny Brown
a9aba9c1c7 Create secrets manager API against Google Secrets store, feature flag (#6113)
* Create secrets manager API against Google Secrets store, plus feature flag on use.
* Run integration test at the right time, with config, for secrets persistence.
* Provide credentials for google secrets store access to tests
* Separate out secrets persistence test from kube for stability
2021-09-17 10:52:33 -05:00
Dmytro
7a6da86baa add coverage report (#6045)
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-09-15 10:50:07 +03:00