1
0
mirror of synced 2026-01-27 07:02:03 -05:00
Commit Graph

401 Commits

Author SHA1 Message Date
Tim Roes
0854adac43 Only run slash commands on PRs (#11017) 2022-03-10 14:50:37 +01:00
LiRen Tu
b166ba6d4f 🐞 Add ignore-empty flag to sentry release (#11009) 2022-03-09 15:24:11 -08:00
Benoit Moriceau
7fb09936a3 Stop docker acceptance instances (#10961)
* Stop docker acceptance instances

* Fix typo
2022-03-08 17:06:30 -08:00
Tim Roes
3df0619b84 Allow slash commands on PRs from forks (#9742)
This enables all slash commands to also work on PRs created from forks. This will not run CI on PRs coming from a fork. There are unfortunately some limitations around injecting secrets in pull_request actions, which I described in detail in this comment.

I've done most of the testing for this in https://github.com/timroes/github-actions-test to test out that the context are set the way I'd expect them.

There's one risk: if any of the actual build scripts called by one of the slash commands would use this repository hard-coded they might fail. I've tried to search through the whole code base and don't believe this is the case.
2022-03-08 18:15:18 +08:00
Parker Mossman
ed19bcc9c2 Parker/acceptance test off benoit branch (#10843)
* Revert "Rm flaky test (#9628)"

This reverts commit 16133cf5e7.

* Restore the acceptance test running with the new scheduler

* Add timeout

* Isolate the new acceptance test

* Update github action name

* Attemptp to fix checkpointing

* Check the state retrieval instead of the existance of the workflow

* fix build

* Add concurrent list for test

* Do not wait for the workflow to be potentially destroy

* Silencely ignore the cancel exception

* Format

* Trigger build

* format

* Remove unrelated changes

* Update acceptance

* Try to fix race condition

* Try to slow down the connection

* Disable test

* Move the sleep

* Rm useless sleep

* Fix missing return

* add repeated

* try using infinite feed source for cancellation test

* set limits on infinite feed source

* misunderstood waitForJob, now correctly waiting for job to be in RUNNING

* clean up PR, DRY create definition methods, clearer method name for waiting on job

* fix acceptance tests action name

* fix imports

* more cleanup

* revert temporalClient do-while change

* fix workflow step names

Co-authored-by: Benoit Moriceau <benoit@airbyte.io>
2022-03-04 08:49:59 -08:00
Tim Roes
d73af697e4 Use cypress dashboard and stabilize e2e tests (#10807)
* Record e2e tests to cypress dashboard

* Make env variable accessible in script

* Improve e2e_test script

* Properly wait for server to be ready

* Isolate test suites better

* More test isolation

* Revert baseUrl for development
2022-03-03 18:29:07 +01:00
Davin Chia
ba3265a76b Add caching to all jobs in the main build. (#10801)
Add build dependency caching to all jobs in the main build.

This speeds things up by 5 mins over the previously uncached time.
2022-03-03 21:55:32 +08:00
Davin Chia
91b8d0e292 Add another token to alleviate API limit pressure. (#10826)
We are running into Github API rate limits.

This PR:
- introduces another token as a temp solution.
- reorganises the workflow file.
2022-03-03 21:17:46 +08:00
Maksym Pavlenok
2bba529803 🎉 Source Salesforce: speed up discovery >20x by leveraging parallel API calls (#10516) 2022-02-27 19:03:39 -08:00
Jared Rhizor
c317b2a8f3 exclude workers test from connectors builds on CI (#10615) 2022-02-24 11:55:04 -08:00
Davin Chia
3190233252 Use AMI with socat installed. (#9719)
* Use AMI with socat installed with disk space increased up to 150 GB.

* Also use the same kube acceptance test set up for the orchestrator.
2022-02-22 23:42:31 +08:00
Maksym Pavlenok
61b0e9e196 🎉 Single py checker (#10246) 2022-02-21 14:01:42 +02:00
Tim Roes
828ebc0e51 Use Octavia account when pinging teams 2022-02-16 15:47:55 +01:00
Tim Roes
f10ebcd6d7 Fix failing sonar-scan without changes (#10278) 2022-02-14 10:16:09 +01:00
Jared Rhizor
6829e86771 full scheduler v2 CI testing (#10199)
* fix normalization output processing in container orchestrator

* add full scheduler v2 acceptance tests

* speed up tests

* fixes

* clean up
2022-02-11 16:47:17 -08:00
Tim Roes
910c5983ba Add CODEOWNERS and notify workflow (#10275)
* Create CODEOWNERS for frontend code

* Add notification on label workflow

* Add empty nl
2022-02-11 17:11:11 +01:00
LiRen Tu
679ce18042 🎉 Destination databricks: move driver downloading to gradle (#9995)
* Download jdbc driver in dockerfile

* Download jdbc driver in dockerfile

* Remove driver downloading from ci

* Update document

* Update public doc

* Download jdbc driver in gradle

* Clean up gradle file

* Clean up dockerfile

* Fix typo
2022-02-02 02:16:53 -08:00
LiRen Tu
aa3e00d37c Add sentry release for connectors (#9940)
* Test sentry release step

* Fix bug

* Fix typo

* Cut off prefix

* Fix env variables

* Add back steps

* Fix syntax error

* Use startsWith to replace IMAGE_TYPE

* Update title

* Bump version in seed
2022-01-31 16:58:39 -08:00
pmossman
1096a5a365 remove unused github workflow 2022-01-31 09:00:50 -08:00
pmossman
c51608a57f add new platform-project-start-sprint workflow skeleton 2022-01-28 10:16:33 -08:00
Parker Mossman
86b506f0e0 Update platform-project-automation.yml 2022-01-26 13:33:16 -08:00
Parker Mossman
d08afa8965 Update platform-project-automation.yml 2022-01-26 13:27:06 -08:00
Parker Mossman
147d0bb478 Update platform-project-automation.yml 2022-01-26 13:25:05 -08:00
Parker Mossman
1f139da4f6 Update platform-project-automation.yml 2022-01-26 13:19:21 -08:00
pmossman
4bed6029cb try adding custom fields 2022-01-26 13:16:41 -08:00
pmossman
ac32b8ef5f use project-beta-automations action 2022-01-26 12:06:11 -08:00
Parker Mossman
5a8fdc6b14 Platform project automation (#9825)
* fix double with:

* rename env MY_GITHUB_TOKEN
2022-01-26 11:43:32 -08:00
Parker Mossman
16a16ec638 fix double with: (#9823) 2022-01-26 11:35:29 -08:00
Parker Mossman
09a202db3d add platform project automation github workflow (#9821) 2022-01-26 11:28:15 -08:00
Harshith Mullapudi
dda6902cec feat: add publish-external command to slash commands to publish external images (#9634)
* feat: add publish-external command to slash commands to publish external connector images

* fix: publish only stpec to cache

* fix: according to new script changes

* fix: removed tox installation

* fix: version is not read from command
2022-01-26 23:44:10 +05:30
Jared Rhizor
ded87beb6d use path for GOOGLE_APPLICATION_CREDENTIALS (#9763) 2022-01-24 12:02:32 -08:00
Maksym Pavlenok
aa31f6abad 🎉 Collapse sonar reports (#9699) 2022-01-21 19:39:49 +02:00
Benoit Moriceau
16133cf5e7 Rm flaky test (#9628) 2022-01-19 18:56:34 -08:00
Benoit Moriceau
e7da9232bb Fix record count and add acceptance test to the new scheduler (#9487)
* Add a job notification

The new scheduler was missing a notification step after the job is done.

This is needed in order to report the number of record of a sync.

* Acceptance test with the new scheduler

Add a new github action task to run the acceptances test with the new scheduler

* Retry if the failure

* PR comments
2022-01-19 18:16:19 -08:00
h-chlor
1832c69c26 Add SonarQube to CI (#8362) 2022-01-19 18:25:43 +02:00
Tim Roes
cb8644e2d5 Remove empty helm workflow (#9592) 2022-01-19 12:11:07 +01:00
LiRen Tu
2bad2cf3cf Use new github action runner ami with 100 gb disk (#9450)
* Use new ami with 100 gb for action runner

* Revert slash command ami

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
2022-01-15 20:38:56 -08:00
Davin Chia
01d64861b1 Limit release workflow to run one at a time. (#9507)
Limit the release workflow to only one with a queue of capacity one. This should help reduce the chance of release clashing.
2022-01-15 01:25:25 +08:00
Parker Mossman
9cff110510 Run the 'Release Airbyte' workflow on EC2 runners (#9361)
* use ec2 runner for release-airbyte-os github workflow

* use setup-python to install pip, which is needed for release_version.sh
2022-01-11 13:48:58 -08:00
Charles
086426339c remove helm install build steps (#9024) 2022-01-07 07:39:55 -08:00
LiRen Tu
dfb7b8cd3f 🐞 Destination databricks: update jdbc driver to patch log4j (#7622)
* Download spark jdbc driver in build cmd

* Download jdbc driver in ci integration test

* Update comments

* Set up cloud sdk

* Add comments

* Download jdbc driver from databricks directly

* Update readme

* Use unzip command

* Install unzip for databricks

* Add databricks build status

* Close database

* Log more error information

* Close database when checking connection

* Update spec
2022-01-06 20:22:56 -08:00
Augustin
34c9366f2b 🐙 Bootstrap octavia-cli project (#9070) 2022-01-05 21:19:40 +01:00
Vadym
ff45a31eac SonarQube QA GitHub jobs (#8668)
* Add detect changed modules and build reports workflow.
Add python scripts for given workflow.
Add unit tests for detect_changed_modules.py.

* Update build_static_checkers_reports.py.
Update task.py to work within any python module.
Fixed report generation.

* Update Detect Changed Modules action.

* Fix action.
Update unit tests.
Code cleanup and fixes.

* Wrap ci_static_check_reports into bash command.

* Detect Changed Modules add install requirements command.

* Update ci_detect_changed_modules package.

* Rename ci_build_checkers_reports -> ci_build_python_checkers_reports.

* Return json as ci_detect_changed_modules output.

* Update Detect Changed Modules set output.

* Update unit tests.
Add changed module filter.
2021-12-30 19:37:56 +02:00
Jared Rhizor
115b0f9875 attach kubernetes logs instead of writing to step output (#8979)
* attach kubernetes logs instead of writing to step output

* it's a pod
2021-12-21 08:54:38 -08:00
Jared Rhizor
20c488948e add container orchestrator in feature flag (#8015)
* it's working

* clean up process factory

* fix and add tests

* clarify airbyte_default

* fix build

* fix kube acceptance test (maybe)

* oops

* fix output prop issue

* fix output propagation regression

* fix kube singleton problem

* sync passing on kube but getting wrong exit code of 7

* misc

* fix port usage

* remove host port that causes conflicts

* eliminate envconfigs static usage

* this took way too long to figure out

* get rid of annoying ==== this is new log messages

* finally successfully completing syncs

* stop using magic strings and clean up logging

* misc minor cleanups

* fmt

* misc

* correct

* misc fixes

* rename + misc

* better logs

* logging fix 1

* logging fix 1 -- fixed

* finally get logging working nicely

* add comment for simplification

* fmt

* misc

* fmt

* break into separate class

* remove comment

* remove flaky multi-node testing

* try to fix connector build

* remove separate node check

* switch to new configs type

* fix regression from logging config changes

* only log path one time

* remove misleading setting terminology

* fix connector build

* fix earlier merge conflict

* fix runtime kubernetes bug

* fix connector build (again)

* greatly simplify logging config by forcing the container-orchestrator to use default (non-job) logging

* add secret insertion for orchestrator on kube

* fix k8s ports

* add four ports

* fix logging test regression

* temporarily disable kube tests to check logging

* improve comments

* make Docker run more secure by limiting env vars transferred

* re-enable kubernetes tests

* fix conflict

* fix docker launching

* revert temporal hacks

* match master

* fix typo

* remove completed todo

* fix conflict

* increase memory requirement to something reasonable

* Update airbyte-container-orchestrator/Dockerfile

Co-authored-by: Davin Chia <davinchia@gmail.com>

* Update airbyte-container-orchestrator/Dockerfile

Co-authored-by: Davin Chia <davinchia@gmail.com>

* see if this stabilitizes tests

* address review comments

* bump new container version

* revert temporary addition

* change port back to 9000

* make re-initialization actually a no-op

* add feature flag

* fix version from merging earlier

* fix dockerfile

* fix connector build

* fix

* bump node version

* fix dockerfile

Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-12-17 16:51:17 -08:00
andriikorotkov
0c99c5d7a5 🎉 Setting CPU and Memory limits for performance tests via github (#8807)
* updated ci performance test sh script

* updated ci performance test sh script

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* run tests

* update ci creds

* update performance test yaml file

* update performance test yaml file

* update performance test yaml file

* update performance test yaml file

* update performance test sh file

* update performance test sh file

* update performance test sh file

* update performance test sh file

* update java to 17 version in yaml file for performance tests

* update java to 17 version in yaml file for performance tests

* update java to 17 version in yaml file for performance tests

* update java to 17 version in yaml file for performance tests

* update java to 17 version in yaml file for performance tests

* clean test logs

* remove test logs

* remove test logs

* remove test logs

* return performance test logs

* add readme by performance tests
2021-12-17 12:47:45 +02:00
Davin Chia
abc50145aa Update more scripts. (#8833) 2021-12-16 16:13:26 +08:00
Jared Rhizor
183a79040a increase reliability of frontend and kubernetes tests (#8827)
* increase reliability of frontend and kubernetes tests

* oops
2021-12-15 19:46:06 -08:00
Jared Rhizor
3a8d262b41 remove unused secret (#8828)
* remove unused secret

* trigger buid

* trigger build

* disable e2e tests to test logging faster

* add back in earlier steps
2021-12-15 19:43:53 -08:00
Maksym Pavlenok
d3b0c998b8 🎉 CI secrets: full migration to GSM (#8561)
* add python packages for CI scripts

* add tox config for all subpackages

* draft version

* init venv for scripts

* fix venv

* remove used comments

* fix run test

* change base folder

* update secret format

* update docs

* remove an unused file

* remove github secrets logic fully

* fix base_folder balue

* add functions desc
2021-12-15 15:08:46 +02:00