1
0
mirror of synced 2026-01-01 00:02:54 -05:00
Commit Graph

171 Commits

Author SHA1 Message Date
LiRen Tu
60f78a9d7c Move the comment step in test and publish commands to the front (#13238) 2022-05-26 13:25:02 -07:00
Brian Leonard
e82e6fc0a2 Show integration test results on the Action Summary (#13072)
* Show summary on action results for failed test

* Write title and steps to log

* Log test

* Remove debugging

* Support python and java output

* Extra line break
2022-05-23 11:23:18 -07:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Marcos Marx
ba72a6514b github workflow fix spell of need to needs (#11592) 2022-03-30 13:45:43 -03:00
Topher Lubaway
90100e013c Increase 2 minute timer to 5 minutes for ec2 (#11519)
same intended commit as before, i didn't tag correctly
we are missing some builds due to the timer and can afford
some wasted actions
2022-03-30 10:35:14 -05:00
Topher Lubaway
a562c2963b More ec2 pipelines using dynamic PATs (#11525)
parker noticed we missed a few
reviewer: please help me make sure the function exists
for `find_valid_pat` in each place it's called
2022-03-29 17:30:21 -05:00
Topher Lubaway
232d9914e6 Revert "Increase runner timeout to 5 minutes from 2 (#11485)" (#11504)
This reverts commit 8cfee1066a.
2022-03-29 10:44:04 -05:00
Topher Lubaway
8cfee1066a Increase runner timeout to 5 minutes from 2 (#11485)
seeing failing builds from 2 and we have higher API
limits now
2022-03-29 10:15:56 -05:00
Topher Lubaway
2693107d60 Adds valid PAT testing to existing ec2 pipes (#11486)
* Adds valid PAT testing to existing ec2 pipes

adds the same workflow seen in gradle.yml to other pipelines

* Reorder PAT selection

rename SELF_RUNNER_GITHUB_ACCESS_TOKEN_1 ->
DAVINCHIA_PAT

remove LABEL_BOT_USERNAME
2022-03-29 10:03:59 -05:00
Topher Lubaway
906887c327 Use shell hacks to find valid PAT (#11439)
* Use shell hacks to find valid PAT

if rate limit exceeded try another key

* Use tim's version of the ec2 runner

this is a weird solution with weird limitations.  should address the short term pain (march 2022)
2022-03-28 15:17:07 -05:00
Topher Lubaway
e3f36eca96 Increase the AWS runner timer via tool update (#11471) 2022-03-28 09:00:56 -05:00
Topher Lubaway
3a8df2fb23 Thottle and retry AWS runner creation in CI (#11435)
* Adds tim's ocotocat rate limiter

* Change all the actions to our homebrew ec2 fork

* Compiles the dist in npm

* Update action to v15.0.1

* Update to v15.0.2

* Update to v15.0.3

* Update to v15.0.4

Co-authored-by: Tim Roes <tim@airbyte.io>
2022-03-25 17:09:26 +01:00
Tim Roes
3df0619b84 Allow slash commands on PRs from forks (#9742)
This enables all slash commands to also work on PRs created from forks. This will not run CI on PRs coming from a fork. There are unfortunately some limitations around injecting secrets in pull_request actions, which I described in detail in this comment.

I've done most of the testing for this in https://github.com/timroes/github-actions-test to test out that the context are set the way I'd expect them.

There's one risk: if any of the actual build scripts called by one of the slash commands would use this repository hard-coded they might fail. I've tried to search through the whole code base and don't believe this is the case.
2022-03-08 18:15:18 +08:00
LiRen Tu
679ce18042 🎉 Destination databricks: move driver downloading to gradle (#9995)
* Download jdbc driver in dockerfile

* Download jdbc driver in dockerfile

* Remove driver downloading from ci

* Update document

* Update public doc

* Download jdbc driver in gradle

* Clean up gradle file

* Clean up dockerfile

* Fix typo
2022-02-02 02:16:53 -08:00
LiRen Tu
2bad2cf3cf Use new github action runner ami with 100 gb disk (#9450)
* Use new ami with 100 gb for action runner

* Revert slash command ami

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
2022-01-15 20:38:56 -08:00
LiRen Tu
dfb7b8cd3f 🐞 Destination databricks: update jdbc driver to patch log4j (#7622)
* Download spark jdbc driver in build cmd

* Download jdbc driver in ci integration test

* Update comments

* Set up cloud sdk

* Add comments

* Download jdbc driver from databricks directly

* Update readme

* Use unzip command

* Install unzip for databricks

* Add databricks build status

* Close database

* Log more error information

* Close database when checking connection

* Update spec
2022-01-06 20:22:56 -08:00
Maksym Pavlenok
d3b0c998b8 🎉 CI secrets: full migration to GSM (#8561)
* add python packages for CI scripts

* add tox config for all subpackages

* draft version

* init venv for scripts

* fix venv

* remove used comments

* fix run test

* change base folder

* update secret format

* update docs

* remove an unused file

* remove github secrets logic fully

* fix base_folder balue

* add functions desc
2021-12-15 15:08:46 +02:00
Jared Rhizor
25674fc306 upgrade to Gradle 7.3.1 / Java 17 (#7964)
* upgrade gradle

* upgrade to Java 17 (and fix a few of the node versioning misses)

* oops

* try to run a different format version

* fix spotless by upgrading / reformatting some files

* fix ci settings

* upgrade mockito to avoid other errors

* undo bad format

* fix "incorrect" sql comments

* fmt

* add debug flag

* remove

* bump

* bump jooq to a version that has a java 17 dist

* fix

* remove logs

* oops

* revert jooq upgrade

* fix

* set up java for connector test

* fix yaml

* generate std source tests

* fail zombie job attempts and add failure reason (#8709)

* fail zombie job attempts and add failure reason

* remove failure reason

* bump gcp dependencies to pick up grpc update (#8713)

* Bump Airbyte version from 0.33.9-alpha to 0.33.10-alpha (#8714)

Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>

* Change CDK "Caching" header to "nested streams & caching"

* Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable (#8524)

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* move S3Config into destination-s3; update dependencies accordingly (#8562)

Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Iryna Grankova <87977540+igrankova@users.noreply.github.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
2021-12-10 16:57:54 -08:00
oleh.zorenko
fdef48c760 Add info about skipped failed tests in /test command message (#8654)
* 7947 Add info about skipped/failed tests to github test message

* 7947 Update docs

* 7947 Add failed test to source stripe

* 7947 Move logic connected to skipped_failed_info info function

* 7947 Remove failed test from source stripe

* 7947 Remove test changes from source-stripe
2021-12-10 17:06:40 +02:00
Maksym Pavlenok
c0cc288f97 🎉 CI secrets: migration to GSM (#7528)
* fix 404 responses for the ticket_comments stream

* add unit test

* add unit test

* add oauth2 access token

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* update spec file

* update CI secrets logic

* update CI secrets logic

* remove debug data

* add a debug message

* fix json convertation

* fix json convertation

* support one secret by several connectors

* Update tools/bin/ci_credentials.sh

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update tools/bin/ci_credentials.sh

Co-authored-by: LiRen Tu <tuliren@gmail.com>

* update function names

* update docs

* reset failed changes

* update json set value

* update secrets json key

* Update tools/bin/ci_credentials.sh


* add JSON validation

* Update docs/connector-development/README.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* update doc

* Update README.md

* rename the function write_standard_creds

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: LiRen Tu <tuliren@gmail.com>
2021-11-10 10:49:44 +02:00
midavadim
c3c3e318c6 🎉 intercom added oauth support (#7060)
* Added oauth support

* fixed description of access_token field in Oauth authentication

* test selenium

* add java implementation

* remove commented dependencies| remove star import

* remove System.out.println

* added support for old format of spec.json files.

* added support for old format of spec.json files.

* added support for old format of spec.json files

* added credential with new spec format

* bumped connector version

* Reverted format of spec.json file

* Override extractRefreshToken method

* java oauth test fixes, minor updates, cloud spec version update

* Fixed docs strings

* Added HttpClient argument

* revert changes in seed folder to follow new dev flow

* Updated image version in seed

* Fixed import of IntercomOAuthFlow

* updated IntercomOAuthFlow to match base class changes

* Change image version to 0.1.8

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
2021-11-09 22:13:46 +02:00
Serhii Lazebnyi
2d2965b879 🎉 New Source: Pinterest (#7493)
* Add Pinterest implementation

* Updated to review

* Added rate limits and PR number to doc

* Fixed unit tests and data slicing bug

* Update airbyte-integrations/connectors/source-pinterest/source_pinterest/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-pinterest/build.gradle

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-pinterest/source_pinterest/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Updated to review

* Updated window in days

* Fixed unit test

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-11-09 12:54:12 +02:00
James Sewa Wilson
46fa44ab80 🎉 New Source: Commerce Tools (#5957)
* adding new source connector Commercetools

* adding new source connector Commercetools

* quick fix

* quick config

* changes

* format files

* add ci credentials run

* correct bin creds

* add publish cmd

* add spec host enum

* correct itnegration tests

* run seed file

Co-authored-by: jsewawilson <jwilson@pebblepost.com>
Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2021-11-08 23:59:29 -03:00
Tuan Nguyen
0b9b8ba472 🎉 New Source: Monday (#7168)
* add items stream

* add boards stream

* add remaining streams

* check point

* lint

* fix unit test & linting

* Delete launch.json

* fix schema

* add retry logic

* Update airbyte-integrations/connectors/source-monday/source_monday/source.py

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* small fix + add creds

* bump version and config file

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2021-11-06 17:32:13 -03:00
vitaliizazmic
1f295f24e6 🎉 Source Google Directory: support oauth
* Source Google Directory #6265 - add oauth support

* Source Google Directory #6265 - update credentials

* Source Google Directory #6265 - fixing according to PR

* Source Google directory #6265 - update docs

* Source Google directory #5190 - update doc

* Source Google Directory #6265 - resolve merge conflict

* Source Google Directory #6265 - SAT for oauth

* Source Google Directory #6265 - bump version and update changelog

* Source Google Directory #6265 - bump version and update changelog (update publish)
2021-11-05 21:40:56 +02:00
Harshith Mullapudi
af1c62f0c6 Publish new source Confluence #7241 (#7666)
* initial commit

* add remaining streams

* fix warning and unit test

* clean up

* revert unrelated files

* remove state files

* lint

* add documentation strings

* added confluence credentials to ci

* fix: removed audit from configured catalog as it cannot be pulled in free account

* fix: format

Co-authored-by: Tuan Nguyen <anhtuan.nguyen@me.com>
2021-11-05 15:17:26 +05:30
Marcos Marx
0e09ac3032 Publish PR 6963: new source freshsales (#7585)
* initial commit

* finish implementing full_refresh

* add other tables

* cleaning up

* add docs and use requests_native_auth

* fix function return different number of values

* change package author

* fix schema

* fix schema bug

* linting

* fix unit test

* clean up

* add null for schemas

* remove fc_widget_collaboration col

* change accpt test

* add creds

* run format

* add config files

* run format

Co-authored-by: Tuan Nguyen <anhtuan.nguyen@me.com>
2021-11-03 01:06:38 -03:00
Harshith Mullapudi
8ddf686e9e added credentials to CI for new source sentry
* add credentials to CI for new source sentry

* add ci cred
2021-11-02 19:55:58 +05:30
Marcos Marx
c846188772 Publish PR 6966: new source retently (#7552)
* retently

* PR comments

* PR comments

* mask api keys

* README

* keep gradle happy

* add github creds

* run format

Co-authored-by: Subhash <subhash@kloud.io>
2021-11-01 23:58:16 -03:00
Marcos Marx
fe118e676e Publish PR 4551: new source delighted correct format (#7540)
* Add Delighted source with people, unsubscribes and bounces streams

* Add survey responses stream

* Add documentation for Delighted source

* publish delighted source

* correct acceptance test

Co-authored-by: Rodrigo Parra <rodpar07@gmail.com>
2021-11-01 22:09:13 -03:00
Harshith Mullapudi
bdaec1cf3f add credentials in for New Source Paystack: #7214 (#7529)
* added credentials to ci for new source paystack

* format
2021-11-01 23:49:53 +05:30
Marcos Marx
2c8151fef2 Add creds for strava, lemlist, amazon sqs and freshservice (#7388)
* add creds for strava, lemlist, amazon sqs and freshservice

* correct source name in ci_creds
2021-10-26 11:42:36 -03:00
Marcos Marx
c83cafc5c1 add salesloft credentials github wf and ci script (#7275) 2021-10-22 01:28:46 -03:00
Marcos Marx
4efadb228c add onesignal credentials + gradlew format (#7270) 2021-10-21 23:55:06 -03:00
Maksym Pavlenok
059645fad4 🎉 Source File: separate secure fork (#6768)
* create a fork for the source-file connector

* update docs and tests

* update docs and tests

* add secrets

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* refactoring after reviews

* correction of doc

* update spec file



Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
2021-10-20 12:28:26 +03:00
George Claireaux
1d3a17a8fb 🎉 Source S3 - memory & performance optimisations + advanced CSV options (#6615)
* memory & performance optimisations

* address comments

* version bump

* added advanced_options for reading csv without header, and more custom pyarrow ReadOptions

* updated to use the latest airbyte-cdk

* updated docs

* bump source-s3 to 0.1.6

* remove unneeded lines

* Use the all dep ami for python builds.

* ec2-instance-id should be ec2-image-id

* ec2-instance-id should be ec2-image-id

Co-authored-by: Jingkun Zhuang <Jingkun.Zhuang@icims.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-19 16:50:51 +01:00
Davin Chia
214b158bde Add labels to EC2 build machines. Refactor actions to reduce duplicate start ec2 runner actions. (#7119)
As the title.

I also took the chance to compose the actions, instead of injecting it everywhere.
2021-10-19 18:02:18 +08:00
Maksym Pavlenok
592f7fe2b2 🐛 Source ZenDesk Support: fix 404 responses for the ticket_comments stream (#6513)
* fix 404 responses for the ticket_comments stream

* add unit test

* add unit test

* add oauth2 access token

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* Update docs/integrations/sources/zendesk-support.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* add log message for 404 errors

* Update docs/integrations/sources/zendesk-support.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* remove oauth logic from this PR

* remove oauth logic from this PR

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-16 01:20:41 +03:00
Maksym Pavlenok
713fe61efd 🎉 New Source: TikTok Marketing (#5887)
* create a new connector folder

* add base classes

* add schemas

* add ads/ad_groups streams

* update tests and docs

* add a bootstrap file

* update the base domain name for API

* update the base domain name for API

* update Dockerfile libs

* remove unused comments

* Update airbyte-integrations/connectors/source-tiktok-marketing/bootstrap.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* Update airbyte-integrations/connectors/source-tiktok-marketing/bootstrap.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* fix UI error with input parameters

* fix problem with updated state

* Update airbyte-integrations/connectors/source-tiktok-marketing/source_tiktok_marketing/spec.py

Co-authored-by: Davin Chia <davinchia@gmail.com>

* add an unit test

* update README.md

* bump version

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-15 19:24:14 +03:00
Baz
cfae84cbb0 🎉 Source Shopify: support oauth flow (#6951) 2021-10-14 22:37:30 +03:00
Arthur Galuza
bfe865aebf 🎉 Source Google Sheets: Support connecting via oAuth webflow (#6354)
* Add service account support

* Upd oauth support

* Upd auth creds selector
2021-10-13 23:02:22 +03:00
Arthur Galuza
99b1e91c65 🎉 Source Pipedrive: add oAuth support (#6821)
* Add oauth support

* Upd changelog

* Update airbyte-integrations/connectors/source-pipedrive/source_pipedrive/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-10-12 09:45:46 +03:00
Arthur Galuza
bf3189cd12 🎉 Source Google Analytics v4: Declare oauth parameters in google sources (#6414)
* Upd auth: oauth support

* Rename authorization/credentials, upd spec, refactor

* Add backward compatibility

* Upd CI

* Bump version

* Upd changelog
2021-10-08 12:38:40 +03:00
Yevhenii
00d56e7711 🎉 Source Slack: Implement OAuth support with OAuth authenticator (#6570)
* Source Slack: Implement OAuth support with OAuth authenticator
2021-10-07 14:01:15 +03:00
Sherif A. Nada
b470489983 🧹delete marketo singer source (#6800) 2021-10-06 22:53:21 -07:00
Vadym
351e998394 🎉 Source Hubspot: Support OAuth (#6465)
* Add required scopes to hubspot.md docs

* Update spec with oauth creds.
Update acceptance test config with oauth tests.
Add oauth test creds configs.

* Fix spec.js `oneOf` usage

* Add CDK authenticators usage.
Update source acceptance tests.
Update spec.json.

* Fix source unit tests

* Update hubspot.md scopes docs

* Add spec field titles.
Update comments.

* Update spec.json oauthFlowOutputParameters.

* Update spec.json to pass SAT.
2021-10-04 17:11:34 +03:00
oneshcheret
11561290be 🎉 Source MSSQL: Implement SSH bastion for acceptance/integration tests (#6608)
* 🎉 Source MSSQL: Implement SSH bastion for acceptance/integration tests

* 🎉 Source MSSQL: remove ci creds for mssql source
2021-10-04 16:08:41 +03:00
VitaliiMaltsev
c2561141fc 🎉 Destination Oracle - Added support for connection via ssh tunnel (#6370)
* Oracle destination with ssh tunneling

* add ssh key integration testing for Oracle destination

* fix checkstyle

* add container approach to Oracle Destination Integration Tests

* add container approach to Oracle Destination Integration Tests

* add container approach to Oracle Source Integration Tests/ delete secrets for Oracle db

* Oracle Destination - bump version to 0.1.8

* remove oracle ssh secrets from ci_credentials.sh

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
2021-09-28 14:25:33 +03:00
Baz
2f835e4e39 🎉 New Source: LinkedIn Ads (#5285)
Resolving: https://github.com/airbytehq/airbyte/issues/2589

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-09-25 01:58:58 +03:00
Eugene
6658ecd5aa 🎉 Source Oracle: Added support for connection via SSH tunnels (#6356)
* [5968] Source Oracle - Added option to connect to Oracle DB via SSH (aka bastion server)
2021-09-24 20:52:47 +03:00