1
0
mirror of synced 2026-01-02 12:02:47 -05:00
Commit Graph

198 Commits

Author SHA1 Message Date
Aaron ("AJ") Steers
5629bb9e6b CI: fix missing slash command input args (#37564) 2024-04-25 08:58:55 -07:00
Augustin
5b8200181c connectors-ci: deprecate slash test (#27200) 2023-06-14 18:19:13 +02:00
Augustin
4749b779d1 connectors-ci: fix typo in observability report (#24113)
* connectors-ci: fix typo in observability report

* connectors-ci: fix typo in observability report
2023-03-15 14:36:47 -05:00
Augustin
507845cf37 connectors-ci: make /test and dagger pipeline write report in a common way for observability (#24050) 2023-03-15 10:44:56 +01:00
Catherine Noll
d4ec2d952e Update /test to optionally test against the local CDK (#23434) 2023-02-28 11:53:51 -05:00
Augustin
727229cb5f connector testing: remove superfluous tasks (#23208) 2023-02-21 22:24:40 +01:00
Augustin
994ca25a0b test-command: change report status condition (#23244) 2023-02-21 20:18:02 +01:00
Conor
7953c45a09 delete platform code and workflows (#23214)
* delete platform code and workflows

* delete more platform workflows

* sunset kustomize (#23187)

* sunset kustomize

* remove build steps that use kustomize

* remove k8s kustomize gke-kube-acceptance-tests

* delete docker-compose files

* remove platform from check_images_exist.sh

* Working replacement script for docker

* rename start script and add download files to gitignore

* add args and download only option

* integration test downloads docker compose files

* remove helm charts

* remove more platform files

* add flags.yml

* use version from gradle.properties instead of .env

* Working replacement script for docker (#23227)

* Working replacement script for docker

* Working debugging line

* rename start script and add download files to gitignore

* add args and download only option

* Made flags cleaner and more cleanup

i've never used this flag pattern, its much
easier than my usual way

* warn about a sharp edge

* Don't run when we don't know a command

* better help spacing

* add flags.yml

---------

Co-authored-by: cpdeethree <conor@airbyte.io>
Co-authored-by: Conor <cpdeethree@users.noreply.github.com>

* remove more-secrets env

---------

Co-authored-by: Charles <charles@airbyte.io>
Co-authored-by: Topher Lubaway <asimplechris@gmail.com>
2023-02-19 09:42:08 -06:00
Ben Church
35ea6e9cf7 Enable qa-checks in test-command (#22184) 2023-02-07 09:08:48 -08:00
Mykyta Serbynevskiy
d4bfcf3372 introduce new pattern of GH PAT usage for #2641 (#22177) 2023-02-01 20:58:12 +02:00
Ben Church
7d3b0fe255 Fix test-command quotes (#22146) 2023-01-31 07:37:23 -08:00
Ben Church
4f6d7d14b5 Disable qa engine check (#22131) 2023-01-31 03:49:12 -08:00
Augustin
616e2e1af0 QA checks: implement icon and documentation checks (#21845) 2023-01-26 16:03:09 +01:00
Augustin
d77514dcfe CI connector ops: introduce QA checks in /test (#21699) 2023-01-23 18:25:53 +01:00
Augustin
295fcecde0 ci_credentials: disable tooling test run by tox (#21580)
* disable tox

* rename steps

* revert changes on experimental workflow

* do not install tox
2023-01-19 03:42:56 -06:00
Augustin
f57fcdf87c test/publish commands: always run update secrets (#20183) 2022-12-07 17:13:20 +01:00
Augustin
9cb5714cd1 ci-credentials: update GSM secrets with updated configuration values (#20076) 2022-12-07 11:53:22 +01:00
Peter Hu
6770b5e5b6 test new AMI (#19694)
* test new AMI

https://internal-docs.airbyte.io/Things-To-Know/Build-Runner-Images

* update docs link

* remove ami override for test-command
2022-11-29 08:22:22 -08:00
Mykyta Serbynevskiy
db8024aa1a Adding retry attempts in OSS repo workflows (#19610)
* Adding retry attempts in OSS repo workflows
* remove timeout from 'Prepare Sentry' step in gradle.yml
* fix: spelling fix
* fix: identation fix
2022-11-29 16:23:25 +02:00
Conor
acb049602f fix: yaml syntax (#19775) 2022-11-23 16:10:40 -06:00
Peter Hu
0c73b70ac7 find_non_rate_limited_PAT (#19736)
* find_non_rate_limited_PAT

* checkout repo to find pat
2022-11-22 16:33:38 -08:00
Volodymyr Pochtar
7454a42055 ci: bump marketplace actions due to deprecations (#19126) 2022-11-10 16:01:35 +02:00
Serhii Chvaliuk
a0f01ad20b ami-0d648081937c75a73 -> ami-06cf12549e3d9c522 (#18667)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-10-30 21:41:52 +02:00
Conor
21ddc93731 chore: cleanup actions and workflows (#16139)
* chore: cleanup actions

* Update .github/workflows/shared-issues.yml

Co-authored-by: Topher Lubaway <asimplechris@gmail.com>

* Update .github/workflows/shared-issues.yml

Co-authored-by: Topher Lubaway <asimplechris@gmail.com>

* Update .github/workflows/shared-pulls.yml

Co-authored-by: Topher Lubaway <asimplechris@gmail.com>

* Update .github/workflows/test-performance-command.yml

Co-authored-by: Topher Lubaway <asimplechris@gmail.com>

Co-authored-by: Topher Lubaway <asimplechris@gmail.com>
2022-09-01 08:46:50 -05:00
Serhii Chvaliuk
c5a98f3ecf GitHub Actions - workflow Connector Integration Tests - add retry (#14452)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-08-01 17:37:20 +03:00
Edward Gao
6b897c46e5 Fix normalization integration test workflow (#14454) 2022-07-06 09:14:54 -07:00
Edward Gao
bb74373c8b Run some DATs as part of base-normalization tests (#14312) 2022-07-01 14:55:23 -07:00
LiRen Tu
60f78a9d7c Move the comment step in test and publish commands to the front (#13238) 2022-05-26 13:25:02 -07:00
Brian Leonard
e82e6fc0a2 Show integration test results on the Action Summary (#13072)
* Show summary on action results for failed test

* Write title and steps to log

* Log test

* Remove debugging

* Support python and java output

* Extra line break
2022-05-23 11:23:18 -07:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Marcos Marx
ba72a6514b github workflow fix spell of need to needs (#11592) 2022-03-30 13:45:43 -03:00
Topher Lubaway
90100e013c Increase 2 minute timer to 5 minutes for ec2 (#11519)
same intended commit as before, i didn't tag correctly
we are missing some builds due to the timer and can afford
some wasted actions
2022-03-30 10:35:14 -05:00
Topher Lubaway
a562c2963b More ec2 pipelines using dynamic PATs (#11525)
parker noticed we missed a few
reviewer: please help me make sure the function exists
for `find_valid_pat` in each place it's called
2022-03-29 17:30:21 -05:00
Topher Lubaway
232d9914e6 Revert "Increase runner timeout to 5 minutes from 2 (#11485)" (#11504)
This reverts commit 8cfee1066a.
2022-03-29 10:44:04 -05:00
Topher Lubaway
8cfee1066a Increase runner timeout to 5 minutes from 2 (#11485)
seeing failing builds from 2 and we have higher API
limits now
2022-03-29 10:15:56 -05:00
Topher Lubaway
2693107d60 Adds valid PAT testing to existing ec2 pipes (#11486)
* Adds valid PAT testing to existing ec2 pipes

adds the same workflow seen in gradle.yml to other pipelines

* Reorder PAT selection

rename SELF_RUNNER_GITHUB_ACCESS_TOKEN_1 ->
DAVINCHIA_PAT

remove LABEL_BOT_USERNAME
2022-03-29 10:03:59 -05:00
Topher Lubaway
906887c327 Use shell hacks to find valid PAT (#11439)
* Use shell hacks to find valid PAT

if rate limit exceeded try another key

* Use tim's version of the ec2 runner

this is a weird solution with weird limitations.  should address the short term pain (march 2022)
2022-03-28 15:17:07 -05:00
Topher Lubaway
e3f36eca96 Increase the AWS runner timer via tool update (#11471) 2022-03-28 09:00:56 -05:00
Topher Lubaway
3a8df2fb23 Thottle and retry AWS runner creation in CI (#11435)
* Adds tim's ocotocat rate limiter

* Change all the actions to our homebrew ec2 fork

* Compiles the dist in npm

* Update action to v15.0.1

* Update to v15.0.2

* Update to v15.0.3

* Update to v15.0.4

Co-authored-by: Tim Roes <tim@airbyte.io>
2022-03-25 17:09:26 +01:00
Tim Roes
3df0619b84 Allow slash commands on PRs from forks (#9742)
This enables all slash commands to also work on PRs created from forks. This will not run CI on PRs coming from a fork. There are unfortunately some limitations around injecting secrets in pull_request actions, which I described in detail in this comment.

I've done most of the testing for this in https://github.com/timroes/github-actions-test to test out that the context are set the way I'd expect them.

There's one risk: if any of the actual build scripts called by one of the slash commands would use this repository hard-coded they might fail. I've tried to search through the whole code base and don't believe this is the case.
2022-03-08 18:15:18 +08:00
LiRen Tu
679ce18042 🎉 Destination databricks: move driver downloading to gradle (#9995)
* Download jdbc driver in dockerfile

* Download jdbc driver in dockerfile

* Remove driver downloading from ci

* Update document

* Update public doc

* Download jdbc driver in gradle

* Clean up gradle file

* Clean up dockerfile

* Fix typo
2022-02-02 02:16:53 -08:00
LiRen Tu
2bad2cf3cf Use new github action runner ami with 100 gb disk (#9450)
* Use new ami with 100 gb for action runner

* Revert slash command ami

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
2022-01-15 20:38:56 -08:00
LiRen Tu
dfb7b8cd3f 🐞 Destination databricks: update jdbc driver to patch log4j (#7622)
* Download spark jdbc driver in build cmd

* Download jdbc driver in ci integration test

* Update comments

* Set up cloud sdk

* Add comments

* Download jdbc driver from databricks directly

* Update readme

* Use unzip command

* Install unzip for databricks

* Add databricks build status

* Close database

* Log more error information

* Close database when checking connection

* Update spec
2022-01-06 20:22:56 -08:00
Maksym Pavlenok
d3b0c998b8 🎉 CI secrets: full migration to GSM (#8561)
* add python packages for CI scripts

* add tox config for all subpackages

* draft version

* init venv for scripts

* fix venv

* remove used comments

* fix run test

* change base folder

* update secret format

* update docs

* remove an unused file

* remove github secrets logic fully

* fix base_folder balue

* add functions desc
2021-12-15 15:08:46 +02:00
Jared Rhizor
25674fc306 upgrade to Gradle 7.3.1 / Java 17 (#7964)
* upgrade gradle

* upgrade to Java 17 (and fix a few of the node versioning misses)

* oops

* try to run a different format version

* fix spotless by upgrading / reformatting some files

* fix ci settings

* upgrade mockito to avoid other errors

* undo bad format

* fix "incorrect" sql comments

* fmt

* add debug flag

* remove

* bump

* bump jooq to a version that has a java 17 dist

* fix

* remove logs

* oops

* revert jooq upgrade

* fix

* set up java for connector test

* fix yaml

* generate std source tests

* fail zombie job attempts and add failure reason (#8709)

* fail zombie job attempts and add failure reason

* remove failure reason

* bump gcp dependencies to pick up grpc update (#8713)

* Bump Airbyte version from 0.33.9-alpha to 0.33.10-alpha (#8714)

Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>

* Change CDK "Caching" header to "nested streams & caching"

* Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable (#8524)

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* move S3Config into destination-s3; update dependencies accordingly (#8562)

Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Iryna Grankova <87977540+igrankova@users.noreply.github.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
2021-12-10 16:57:54 -08:00
oleh.zorenko
fdef48c760 Add info about skipped failed tests in /test command message (#8654)
* 7947 Add info about skipped/failed tests to github test message

* 7947 Update docs

* 7947 Add failed test to source stripe

* 7947 Move logic connected to skipped_failed_info info function

* 7947 Remove failed test from source stripe

* 7947 Remove test changes from source-stripe
2021-12-10 17:06:40 +02:00
Maksym Pavlenok
c0cc288f97 🎉 CI secrets: migration to GSM (#7528)
* fix 404 responses for the ticket_comments stream

* add unit test

* add unit test

* add oauth2 access token

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* update spec file

* update CI secrets logic

* update CI secrets logic

* remove debug data

* add a debug message

* fix json convertation

* fix json convertation

* support one secret by several connectors

* Update tools/bin/ci_credentials.sh

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update tools/bin/ci_credentials.sh

Co-authored-by: LiRen Tu <tuliren@gmail.com>

* update function names

* update docs

* reset failed changes

* update json set value

* update secrets json key

* Update tools/bin/ci_credentials.sh


* add JSON validation

* Update docs/connector-development/README.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* update doc

* Update README.md

* rename the function write_standard_creds

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: LiRen Tu <tuliren@gmail.com>
2021-11-10 10:49:44 +02:00
midavadim
c3c3e318c6 🎉 intercom added oauth support (#7060)
* Added oauth support

* fixed description of access_token field in Oauth authentication

* test selenium

* add java implementation

* remove commented dependencies| remove star import

* remove System.out.println

* added support for old format of spec.json files.

* added support for old format of spec.json files.

* added support for old format of spec.json files

* added credential with new spec format

* bumped connector version

* Reverted format of spec.json file

* Override extractRefreshToken method

* java oauth test fixes, minor updates, cloud spec version update

* Fixed docs strings

* Added HttpClient argument

* revert changes in seed folder to follow new dev flow

* Updated image version in seed

* Fixed import of IntercomOAuthFlow

* updated IntercomOAuthFlow to match base class changes

* Change image version to 0.1.8

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
2021-11-09 22:13:46 +02:00
Serhii Lazebnyi
2d2965b879 🎉 New Source: Pinterest (#7493)
* Add Pinterest implementation

* Updated to review

* Added rate limits and PR number to doc

* Fixed unit tests and data slicing bug

* Update airbyte-integrations/connectors/source-pinterest/source_pinterest/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-pinterest/build.gradle

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-pinterest/source_pinterest/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Updated to review

* Updated window in days

* Fixed unit test

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-11-09 12:54:12 +02:00
James Sewa Wilson
46fa44ab80 🎉 New Source: Commerce Tools (#5957)
* adding new source connector Commercetools

* adding new source connector Commercetools

* quick fix

* quick config

* changes

* format files

* add ci credentials run

* correct bin creds

* add publish cmd

* add spec host enum

* correct itnegration tests

* run seed file

Co-authored-by: jsewawilson <jwilson@pebblepost.com>
Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2021-11-08 23:59:29 -03:00