1
0
mirror of synced 2026-01-03 15:04:01 -05:00
Commit Graph

159 Commits

Author SHA1 Message Date
George Claireaux
dd2d5d0129 in case runners fail to spin up, this needs to run on github-hosted (#13996) 2022-06-22 11:45:00 +01:00
George Claireaux
e7dae06168 Enhance /publish to allow for multiple connectors and parallel execution (#13864)
* start

* revert

* azblob

* bq

* bq denorm

* megapublish baaaabyyyy

* fix needs

* matrix connectors

* auto-bump connector version

* dont failfast and max parallel 5

* multi runno

* minor

* testing matrix agents

* name

* testing multi agents

* tmp fix

* new multi agents

* multi test

* tryy

* let's do this

* magico

* fix

* label test

* couple more connector bumps

* temp

* things

* check this

* lets gooo

* more connectors

* Delete TEMP-testing-command.yml

* auto-bump connector version

* added comment describing bash part

* running single thread

* catch sentry cli

* auto-bump connector version

* destinations

* + snowflake

* saved

* auto-bump connector version

* auto-bump connector version

* java source bumps

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* remove twice-defined methods

* label things

* revert action

* using the new test action

* point at action

* wrong tag on action

* update pool label

* update to use new ec2-github-runner fork

* this needs to be more generic than publisher

* change publish to run on pool

* add comment about runner-pool usage

* updated publish command docs for multi & parallel connector runs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* unbump failed publish versions

* missed dockerfiles

* remove failed docs

* mssql fix

* overhauled the git comment output

* bumping a test connector that should work

* slight order switcheroo

* output connectors properly in first message

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-21 18:40:33 +01:00
Sherif A. Nada
4f48748fd1 remove slack notification from publish command (#13710)
This actually has not been working in a long time, and no one seems to need it, so just removing it
2022-06-13 07:15:28 -07:00
Sherif Nada
b763bdf4ab Revert "add github workflows for publishing ERDs"
This reverts commit c7a23c680b.
2022-06-08 03:32:56 -07:00
Sherif Nada
c7a23c680b add github workflows for publishing ERDs 2022-06-07 22:42:13 -07:00
Evan Tahler
d1ff8dd8b4 [Github Actions] Rename docker secrets and parameterize docker user (#13297)
* Rename docker secrets and parameterize docker user

* Apply suggestions from code review

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
2022-05-27 18:48:26 -07:00
Davin Chia
65f79d26b9 Test using default AMI with buildx to publish connectors. (#13282) 2022-05-27 23:30:37 +08:00
LiRen Tu
60f78a9d7c Move the comment step in test and publish commands to the front (#13238) 2022-05-26 13:25:02 -07:00
noahkawasaki-airbyte
7a79216d01 Make publish command work with private repos by adding JWT into curl headers (#13141)
* Make publish command work with private repos by adding in JWT token to headers for curl

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-24 10:55:46 -07:00
Greg Solovyev
60cbf0d0d9 Add timeout to publish-image task (#12679)
publish-image task can take a while, because it is also running
integration tests by default. test-command workflow has a timeout
of 240 minutes. I added the same timeout value to publish-image task.
2022-05-06 11:18:18 -07:00
George Claireaux
7e1d95c783 🤖 publish command: fix for version bumping multiple connectors in same PR (#12397)
* add a git pull from origin master before commit auto changes

* dummy bump 2 connectors

* pull from current branch not master

* auto-bump connector version

* dummy bump 2 connectors

* revert faker bump

* auto-bump connector version

* better descript in changelog

* dummy bump... again

* commit first, then pull merge, then push

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-27 16:18:22 +01:00
Serhii Chvaliuk
5c8bd3a46d remove --ignore-empty option because sentry 2.0.0 (#11926)
check https://github.com/getsentry/sentry-cli/releases/
Breaking Changes:
    ref: Make ignore-empty for releases set-commits a default behavior and remove --ignore-empty flag (remove --ignore-empty usage) (breaking)

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-04-12 18:48:00 +03:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
George Claireaux
7bb7d23e60 Fixing publish command gitref for bot push (#11767)
* use proper gitref

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-06 19:22:22 +01:00
George Claireaux
f7d3a706b2 Publish connector command now auto bumps version (#11712)
* using yq + more success/failure comments

* added gradle process + git push

* dummy bump to test publish

* dummy bump to test publish

* dummy bump to test publish

* bump version

* fix connector variable

* bump version

* only git add necessary files

* remove git config

* bump version

* making octavia user

* version bump

* auto-bump connector version

* added docs and auto-bumo flag

* bump version

* separate IMAGE_NAME and IMAGE_VERSION env vars from sentry prep

* version bump

* auto-bump connector version

* added entry to apify changelog for clarity

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-05 23:21:55 +01:00
Topher Lubaway
90100e013c Increase 2 minute timer to 5 minutes for ec2 (#11519)
same intended commit as before, i didn't tag correctly
we are missing some builds due to the timer and can afford
some wasted actions
2022-03-30 10:35:14 -05:00
Topher Lubaway
232d9914e6 Revert "Increase runner timeout to 5 minutes from 2 (#11485)" (#11504)
This reverts commit 8cfee1066a.
2022-03-29 10:44:04 -05:00
Topher Lubaway
8cfee1066a Increase runner timeout to 5 minutes from 2 (#11485)
seeing failing builds from 2 and we have higher API
limits now
2022-03-29 10:15:56 -05:00
Topher Lubaway
2693107d60 Adds valid PAT testing to existing ec2 pipes (#11486)
* Adds valid PAT testing to existing ec2 pipes

adds the same workflow seen in gradle.yml to other pipelines

* Reorder PAT selection

rename SELF_RUNNER_GITHUB_ACCESS_TOKEN_1 ->
DAVINCHIA_PAT

remove LABEL_BOT_USERNAME
2022-03-29 10:03:59 -05:00
Topher Lubaway
906887c327 Use shell hacks to find valid PAT (#11439)
* Use shell hacks to find valid PAT

if rate limit exceeded try another key

* Use tim's version of the ec2 runner

this is a weird solution with weird limitations.  should address the short term pain (march 2022)
2022-03-28 15:17:07 -05:00
Topher Lubaway
e3f36eca96 Increase the AWS runner timer via tool update (#11471) 2022-03-28 09:00:56 -05:00
Topher Lubaway
3a8df2fb23 Thottle and retry AWS runner creation in CI (#11435)
* Adds tim's ocotocat rate limiter

* Change all the actions to our homebrew ec2 fork

* Compiles the dist in npm

* Update action to v15.0.1

* Update to v15.0.2

* Update to v15.0.3

* Update to v15.0.4

Co-authored-by: Tim Roes <tim@airbyte.io>
2022-03-25 17:09:26 +01:00
Baz
1f98286667 fixed /publish command by changing the reference from @master go @v0 (#11320) 2022-03-22 18:56:58 +02:00
LiRen Tu
b166ba6d4f 🐞 Add ignore-empty flag to sentry release (#11009) 2022-03-09 15:24:11 -08:00
Tim Roes
3df0619b84 Allow slash commands on PRs from forks (#9742)
This enables all slash commands to also work on PRs created from forks. This will not run CI on PRs coming from a fork. There are unfortunately some limitations around injecting secrets in pull_request actions, which I described in detail in this comment.

I've done most of the testing for this in https://github.com/timroes/github-actions-test to test out that the context are set the way I'd expect them.

There's one risk: if any of the actual build scripts called by one of the slash commands would use this repository hard-coded they might fail. I've tried to search through the whole code base and don't believe this is the case.
2022-03-08 18:15:18 +08:00
LiRen Tu
679ce18042 🎉 Destination databricks: move driver downloading to gradle (#9995)
* Download jdbc driver in dockerfile

* Download jdbc driver in dockerfile

* Remove driver downloading from ci

* Update document

* Update public doc

* Download jdbc driver in gradle

* Clean up gradle file

* Clean up dockerfile

* Fix typo
2022-02-02 02:16:53 -08:00
LiRen Tu
aa3e00d37c Add sentry release for connectors (#9940)
* Test sentry release step

* Fix bug

* Fix typo

* Cut off prefix

* Fix env variables

* Add back steps

* Fix syntax error

* Use startsWith to replace IMAGE_TYPE

* Update title

* Bump version in seed
2022-01-31 16:58:39 -08:00
LiRen Tu
2bad2cf3cf Use new github action runner ami with 100 gb disk (#9450)
* Use new ami with 100 gb for action runner

* Revert slash command ami

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
2022-01-15 20:38:56 -08:00
LiRen Tu
dfb7b8cd3f 🐞 Destination databricks: update jdbc driver to patch log4j (#7622)
* Download spark jdbc driver in build cmd

* Download jdbc driver in ci integration test

* Update comments

* Set up cloud sdk

* Add comments

* Download jdbc driver from databricks directly

* Update readme

* Use unzip command

* Install unzip for databricks

* Add databricks build status

* Close database

* Log more error information

* Close database when checking connection

* Update spec
2022-01-06 20:22:56 -08:00
Maksym Pavlenok
d3b0c998b8 🎉 CI secrets: full migration to GSM (#8561)
* add python packages for CI scripts

* add tox config for all subpackages

* draft version

* init venv for scripts

* fix venv

* remove used comments

* fix run test

* change base folder

* update secret format

* update docs

* remove an unused file

* remove github secrets logic fully

* fix base_folder balue

* add functions desc
2021-12-15 15:08:46 +02:00
LiRen Tu
4276b2f3d6 Bump the minor version for mysql source (#8731)
* Bump the minor version for mysql source

* Setup java 17 in the publish command

* Bump version in seed
2021-12-12 16:30:54 -08:00
Maksym Pavlenok
c0cc288f97 🎉 CI secrets: migration to GSM (#7528)
* fix 404 responses for the ticket_comments stream

* add unit test

* add unit test

* add oauth2 access token

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* update spec file

* update CI secrets logic

* update CI secrets logic

* remove debug data

* add a debug message

* fix json convertation

* fix json convertation

* support one secret by several connectors

* Update tools/bin/ci_credentials.sh

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update tools/bin/ci_credentials.sh

Co-authored-by: LiRen Tu <tuliren@gmail.com>

* update function names

* update docs

* reset failed changes

* update json set value

* update secrets json key

* Update tools/bin/ci_credentials.sh


* add JSON validation

* Update docs/connector-development/README.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* update doc

* Update README.md

* rename the function write_standard_creds

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: LiRen Tu <tuliren@gmail.com>
2021-11-10 10:49:44 +02:00
midavadim
c3c3e318c6 🎉 intercom added oauth support (#7060)
* Added oauth support

* fixed description of access_token field in Oauth authentication

* test selenium

* add java implementation

* remove commented dependencies| remove star import

* remove System.out.println

* added support for old format of spec.json files.

* added support for old format of spec.json files.

* added support for old format of spec.json files

* added credential with new spec format

* bumped connector version

* Reverted format of spec.json file

* Override extractRefreshToken method

* java oauth test fixes, minor updates, cloud spec version update

* Fixed docs strings

* Added HttpClient argument

* revert changes in seed folder to follow new dev flow

* Updated image version in seed

* Fixed import of IntercomOAuthFlow

* updated IntercomOAuthFlow to match base class changes

* Change image version to 0.1.8

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
2021-11-09 22:13:46 +02:00
Serhii Lazebnyi
2d2965b879 🎉 New Source: Pinterest (#7493)
* Add Pinterest implementation

* Updated to review

* Added rate limits and PR number to doc

* Fixed unit tests and data slicing bug

* Update airbyte-integrations/connectors/source-pinterest/source_pinterest/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-pinterest/build.gradle

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-pinterest/source_pinterest/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Updated to review

* Updated window in days

* Fixed unit test

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-11-09 12:54:12 +02:00
James Sewa Wilson
46fa44ab80 🎉 New Source: Commerce Tools (#5957)
* adding new source connector Commercetools

* adding new source connector Commercetools

* quick fix

* quick config

* changes

* format files

* add ci credentials run

* correct bin creds

* add publish cmd

* add spec host enum

* correct itnegration tests

* run seed file

Co-authored-by: jsewawilson <jwilson@pebblepost.com>
Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2021-11-08 23:59:29 -03:00
Tuan Nguyen
0b9b8ba472 🎉 New Source: Monday (#7168)
* add items stream

* add boards stream

* add remaining streams

* check point

* lint

* fix unit test & linting

* Delete launch.json

* fix schema

* add retry logic

* Update airbyte-integrations/connectors/source-monday/source_monday/source.py

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* small fix + add creds

* bump version and config file

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2021-11-06 17:32:13 -03:00
vitaliizazmic
1f295f24e6 🎉 Source Google Directory: support oauth
* Source Google Directory #6265 - add oauth support

* Source Google Directory #6265 - update credentials

* Source Google Directory #6265 - fixing according to PR

* Source Google directory #6265 - update docs

* Source Google directory #5190 - update doc

* Source Google Directory #6265 - resolve merge conflict

* Source Google Directory #6265 - SAT for oauth

* Source Google Directory #6265 - bump version and update changelog

* Source Google Directory #6265 - bump version and update changelog (update publish)
2021-11-05 21:40:56 +02:00
Harshith Mullapudi
af1c62f0c6 Publish new source Confluence #7241 (#7666)
* initial commit

* add remaining streams

* fix warning and unit test

* clean up

* revert unrelated files

* remove state files

* lint

* add documentation strings

* added confluence credentials to ci

* fix: removed audit from configured catalog as it cannot be pulled in free account

* fix: format

Co-authored-by: Tuan Nguyen <anhtuan.nguyen@me.com>
2021-11-05 15:17:26 +05:30
Marcos Marx
0e09ac3032 Publish PR 6963: new source freshsales (#7585)
* initial commit

* finish implementing full_refresh

* add other tables

* cleaning up

* add docs and use requests_native_auth

* fix function return different number of values

* change package author

* fix schema

* fix schema bug

* linting

* fix unit test

* clean up

* add null for schemas

* remove fc_widget_collaboration col

* change accpt test

* add creds

* run format

* add config files

* run format

Co-authored-by: Tuan Nguyen <anhtuan.nguyen@me.com>
2021-11-03 01:06:38 -03:00
Harshith Mullapudi
8ddf686e9e added credentials to CI for new source sentry
* add credentials to CI for new source sentry

* add ci cred
2021-11-02 19:55:58 +05:30
Marcos Marx
c846188772 Publish PR 6966: new source retently (#7552)
* retently

* PR comments

* PR comments

* mask api keys

* README

* keep gradle happy

* add github creds

* run format

Co-authored-by: Subhash <subhash@kloud.io>
2021-11-01 23:58:16 -03:00
Marcos Marx
fe118e676e Publish PR 4551: new source delighted correct format (#7540)
* Add Delighted source with people, unsubscribes and bounces streams

* Add survey responses stream

* Add documentation for Delighted source

* publish delighted source

* correct acceptance test

Co-authored-by: Rodrigo Parra <rodpar07@gmail.com>
2021-11-01 22:09:13 -03:00
Harshith Mullapudi
bdaec1cf3f add credentials in for New Source Paystack: #7214 (#7529)
* added credentials to ci for new source paystack

* format
2021-11-01 23:49:53 +05:30
Marcos Marx
2c8151fef2 Add creds for strava, lemlist, amazon sqs and freshservice (#7388)
* add creds for strava, lemlist, amazon sqs and freshservice

* correct source name in ci_creds
2021-10-26 11:42:36 -03:00
Marcos Marx
c83cafc5c1 add salesloft credentials github wf and ci script (#7275) 2021-10-22 01:28:46 -03:00
Marcos Marx
4efadb228c add onesignal credentials + gradlew format (#7270) 2021-10-21 23:55:06 -03:00
Maksym Pavlenok
bc78e1c9b2 add secret name for publish config (#7221) 2021-10-20 20:03:52 +03:00
George Claireaux
1d3a17a8fb 🎉 Source S3 - memory & performance optimisations + advanced CSV options (#6615)
* memory & performance optimisations

* address comments

* version bump

* added advanced_options for reading csv without header, and more custom pyarrow ReadOptions

* updated to use the latest airbyte-cdk

* updated docs

* bump source-s3 to 0.1.6

* remove unneeded lines

* Use the all dep ami for python builds.

* ec2-instance-id should be ec2-image-id

* ec2-instance-id should be ec2-image-id

Co-authored-by: Jingkun Zhuang <Jingkun.Zhuang@icims.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-19 16:50:51 +01:00
Davin Chia
214b158bde Add labels to EC2 build machines. Refactor actions to reduce duplicate start ec2 runner actions. (#7119)
As the title.

I also took the chance to compose the actions, instead of injecting it everywhere.
2021-10-19 18:02:18 +08:00
Maksym Pavlenok
592f7fe2b2 🐛 Source ZenDesk Support: fix 404 responses for the ticket_comments stream (#6513)
* fix 404 responses for the ticket_comments stream

* add unit test

* add unit test

* add oauth2 access token

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* Update docs/integrations/sources/zendesk-support.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* add log message for 404 errors

* Update docs/integrations/sources/zendesk-support.md

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* remove oauth logic from this PR

* remove oauth logic from this PR

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-10-16 01:20:41 +03:00