1
0
mirror of synced 2025-12-31 15:03:11 -05:00
Commit Graph

490 Commits

Author SHA1 Message Date
George Claireaux
dd2d5d0129 in case runners fail to spin up, this needs to run on github-hosted (#13996) 2022-06-22 11:45:00 +01:00
Peter Hu
bcc20b4db1 publish oss for cloud (#13978)
workflow to publish oss artifacts that cloud needs to build against
use docker buildx to create arm images for local development
2022-06-21 14:30:42 -07:00
Topher Lubaway
7acf0c54df Adds test for new workflow (#13986)
* Adds test for new workflow

* Adds airbyte repo

* remove testing line
2022-06-21 14:58:27 -05:00
George Claireaux
e7dae06168 Enhance /publish to allow for multiple connectors and parallel execution (#13864)
* start

* revert

* azblob

* bq

* bq denorm

* megapublish baaaabyyyy

* fix needs

* matrix connectors

* auto-bump connector version

* dont failfast and max parallel 5

* multi runno

* minor

* testing matrix agents

* name

* testing multi agents

* tmp fix

* new multi agents

* multi test

* tryy

* let's do this

* magico

* fix

* label test

* couple more connector bumps

* temp

* things

* check this

* lets gooo

* more connectors

* Delete TEMP-testing-command.yml

* auto-bump connector version

* added comment describing bash part

* running single thread

* catch sentry cli

* auto-bump connector version

* destinations

* + snowflake

* saved

* auto-bump connector version

* auto-bump connector version

* java source bumps

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* remove twice-defined methods

* label things

* revert action

* using the new test action

* point at action

* wrong tag on action

* update pool label

* update to use new ec2-github-runner fork

* this needs to be more generic than publisher

* change publish to run on pool

* add comment about runner-pool usage

* updated publish command docs for multi & parallel connector runs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* unbump failed publish versions

* missed dockerfiles

* remove failed docs

* mssql fix

* overhauled the git comment output

* bumping a test connector that should work

* slight order switcheroo

* output connectors properly in first message

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-21 18:40:33 +01:00
Topher Lubaway
5e1f968751 Adds zombie removal tool (#13718)
* Adds zombie removal tool

* Corrects endpoint adds comments

* Adds API links

* Changes search logic, fixes escape character

* Corrects help text
2022-06-13 13:28:32 -05:00
Sherif A. Nada
4f48748fd1 remove slack notification from publish command (#13710)
This actually has not been working in a long time, and no one seems to need it, so just removing it
2022-06-13 07:15:28 -07:00
Lake Mossman
30161726c2 escape backticks in create-release action so that they are handled properly (#13672)
* escape backticks in create-release action so that they are handled properly

* reword comment

* fix typo
2022-06-10 08:59:22 -07:00
Tim Roes
fe7e1c2272 Upgrade node to latest LTS (#13683)
* Upgrade node to latest LTS

* Upgrade to newer setup-node action
2022-06-10 17:07:57 +02:00
lmossman
2610ffcefc rename automated release creation action to be less similar to the existing Release OSS Airbyte action 2022-06-08 09:23:00 -07:00
Lake Mossman
527e391c16 add Release Airbyte action and update version bump PR description (#13584) 2022-06-08 09:09:28 -07:00
Tim Roes
4c97893363 Validate links in frontend in CI (#13600)
* Validate doc links during building
* Add scheduled link validation
* Remove accidentally commited file
* Remove unnecessary file
* Review feedback
2022-06-08 15:45:50 +02:00
Sherif Nada
b763bdf4ab Revert "add github workflows for publishing ERDs"
This reverts commit c7a23c680b.
2022-06-08 03:32:56 -07:00
Sherif Nada
c7a23c680b add github workflows for publishing ERDs 2022-06-07 22:42:13 -07:00
Lake Mossman
73034c64da Sweep old scheduler code (#13400)
* sweep all scheduler application code and new-scheduler conditional logic

* remove airbyte-scheduler from deployments and docs

* format

* remove 'v2' from github actions

* add back scheduler in delete deployment command

* remove scheduler parameters from helm chart values

* add back job cleaner + test and add comment

* remove now-unused env vars from code and docs

* format

* remove feature flags from web backend connection handler as it is no longer needed

* remove feature flags from config api as it is now longer needed

* remove feature flags input from config api test

* format + shorter url

* remove scheduler parameters from helm chart readme
2022-06-06 10:49:17 -07:00
Brian Leonard
faab304883 Add more pr actions to tag pull requests (#13437) 2022-06-02 12:35:13 -07:00
Alexandre Girard
d9fa24ffff Low code connectors: implement components for sendgrid (#12853)
* checkout from alex/cac

* checkout from alex/cac

* checkout from alex/cac

* Add missing tests

* Add missing files

* Add missing tests

* add missing file

* missing file

* missing file

* rename

* doc

* doc

* remove broken test

* rename

* jinja dependency

* Add comment

* comment

* comment

* pyjq dependency

* rename file

* delete unused file

* Revert "delete unused file"

This reverts commit 758e939367.

* fix

* rename

* abstract property

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* isort

* update state

* fix imports

* update dependency

* format

* rename file

* decoder

* Use decoder

* Update comment

* dict_state is actually backed by a dict

* Add a comment

* update state takes kwargs

* move state out of offset paginator

* update jq parameter order

* update

* remove incremental mixin

* delete comment

* update comments

* update comments

* remove no_state

* rename package

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* remove print

* move test

* delete duplicates

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* pass parameters through kwargs

* update interface to pass source in interface

* update interface to pass source in interface

* rename to stream_slicer

* Allow passing a string or an enum

* Define StateType enum

* convert state_type if not of type type

* convert state_type if not of type type

* Low code connectors: string interpolation with jinja (#12852)

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* move test to right module

* Add missing test

* Use authbase instead of deprecated class

* leverage generator

* Delete dead code

* rename methods

* rename to declarative

* rename the classes too

* Try to install packages to build jq

* isort

* only automake

* Revert "only automake"

This reverts commit c8fe154ffc.

* remove git

* format

* Add jq dependency

* Use request header provider

* rename

* rename field

* remove get_context method

* rename
2022-06-01 07:02:35 -07:00
Evan Tahler
d1ff8dd8b4 [Github Actions] Rename docker secrets and parameterize docker user (#13297)
* Rename docker secrets and parameterize docker user

* Apply suggestions from code review

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
2022-05-27 18:48:26 -07:00
Davin Chia
65f79d26b9 Test using default AMI with buildx to publish connectors. (#13282) 2022-05-27 23:30:37 +08:00
LiRen Tu
60f78a9d7c Move the comment step in test and publish commands to the front (#13238) 2022-05-26 13:25:02 -07:00
noahkawasaki-airbyte
7a79216d01 Make publish command work with private repos by adding JWT into curl headers (#13141)
* Make publish command work with private repos by adding in JWT token to headers for curl

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-24 10:55:46 -07:00
Tim Roes
5237d26ffd Change frontend label to team/frontend (#13093) 2022-05-24 16:13:04 +02:00
Brian Leonard
e82e6fc0a2 Show integration test results on the Action Summary (#13072)
* Show summary on action results for failed test

* Write title and steps to log

* Log test

* Remove debugging

* Support python and java output

* Extra line break
2022-05-23 11:23:18 -07:00
Lake Mossman
26ed3856e1 Migrate OSS to temporal scheduler (#12757)
* Migrate OSS to temporal scheduler

* add comment about migration being performed in server

* add comments about removing migration logic

* formatting and add tests for migration logic

* rm duplicated test

* remove more duplicated build task

* remove retry

* disable acceptance tests that call temporal directly when on kube

* set NEW_SCHEDULER and CONTAINER_ORCHESTRATOR_ENABLED env vars to true to be consistent

* set default value of container orchestrator enabled to true

* Revert "set default value of container orchestrator enabled to true"

This reverts commit 21b36703a9.

* Revert "set NEW_SCHEDULER and CONTAINER_ORCHESTRATOR_ENABLED env vars to true to be consistent"

This reverts commit 6dd2ec04a2.

* Revert "Revert "set NEW_SCHEDULER and CONTAINER_ORCHESTRATOR_ENABLED env vars to true to be consistent""

This reverts commit 2f40f9da50.

* Revert "Revert "set default value of container orchestrator enabled to true""

This reverts commit 26068d5b31.

* fix sync workflow test

* remove defunct cancellation tests due to internal temporal error

* format - remove unused imports

* revert changes that set container orchestrator enabled to true everywhere

* remove NEW_SCHEDULER feature flag from .env files, and set CONTAINER_ORCHESTRATOR_ENABLED flag to true for kube .env files

Co-authored-by: Benoit Moriceau <benoit@airbyte.io>
2022-05-18 17:05:42 -07:00
Topher Lubaway
2c690f2144 More helpful error message for detected changes (#12956)
* More helpful error message for detected changes

* Wrote the check backwards

* more clear errors

* Newline cinema
2022-05-18 09:01:18 -05:00
Topher Lubaway
6bbd715d59 Removes GH branch sync used by gitbook (#12078)
we aren't using this anymore
2022-05-17 08:51:55 -05:00
Brian Leonard
8a49addc67 Switch to shared code (#12590) 2022-05-13 14:22:59 -07:00
Davin Chia
04eb3b402e Run connector build when airbyte protocol is updated. (#12831) 2022-05-13 09:47:11 +08:00
Sherif A. Nada
fa22d32e14 Filter daily build report to only GA & Beta connectors (#12684) 2022-05-06 16:41:38 -07:00
Greg Solovyev
60cbf0d0d9 Add timeout to publish-image task (#12679)
publish-image task can take a while, because it is also running
integration tests by default. test-command workflow has a timeout
of 240 minutes. I added the same timeout value to publish-image task.
2022-05-06 11:18:18 -07:00
Denys Krytskyi
905807591b Modify OSS Airbyte and Cloud Airbyte to use Catalogs - 809 (#12610)
This PR provides 'version catalog' plugin, Where catalog is defined by using declared dependency list in 'build.gradle'.
In this pr all dependencies were transferred to 'libs' using 'dependencyResolutionManagement -> versionCatalogs -> libs'
2022-05-06 02:23:01 +08:00
Brian Leonard
763dea255d Uses private action for workflow (#12525) 2022-05-02 13:44:13 -07:00
Brian Leonard
4ee1fa6c5a Use shared workflow action on issues (#12382)
* Use shared workflow action on issues

* Rename to token

* Also on unlabeled
2022-05-01 14:38:43 -07:00
noahkawasaki-airbyte
e8406a2cd6 Initial PR /publish workflow into pre-merge /build-connector and post-merge /publish-connector workflows (#12405) 2022-04-29 07:03:27 -07:00
noahkawasaki-airbyte
4a9f663565 Add experimental build-connector and publish-connector slash commands (#12442) 2022-04-28 11:25:31 -07:00
Topher Lubaway
1413aca478 Toph deploy docs ssh repo check (#12323)
* Check for ssh github repo

because assumptions mostly.
more clear fail messages for an unsupported workflow
also removes the pipeline which now is redundant with local workflow

* WIP1

* working test

* More clear git test

* less weird characters for grep

* remove verbose for excho
2022-04-27 10:26:35 -05:00
George Claireaux
7e1d95c783 🤖 publish command: fix for version bumping multiple connectors in same PR (#12397)
* add a git pull from origin master before commit auto changes

* dummy bump 2 connectors

* pull from current branch not master

* auto-bump connector version

* dummy bump 2 connectors

* revert faker bump

* auto-bump connector version

* better descript in changelog

* dummy bump... again

* commit first, then pull merge, then push

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-27 16:18:22 +01:00
Davin Chia
e93bb85dc7 Fix build. (#12242) 2022-04-21 20:46:31 +08:00
Peter Hu
0d3088e456 notify cloud repo on push to master (#12229)
* notify cloud repo on push to master

* split out build-branch action

so we can use it in cloud workflow
2022-04-20 16:53:20 -07:00
Topher Lubaway
f065d361b1 Toph docs pipeline fixes bugs n gitbook (#12114)
* WIP

* WIP2

* WIP3

* Remove comment/better comments

* Remote deployment updates

* Adds GH user for yarn deploy

* WIP 6

* WIP7

* WIP8

* WIP 10

* Working easy from home

not so much in the cloud
2022-04-19 14:49:04 -05:00
Topher Lubaway
e615b2faaa Addreses another docs build error (#12084)
* Addreses another docs build error

* Remove two typos

* Adds a test pattern

* Another stab

* More test

* Removing pipeline until more testing

docusaurus deploy isn't into automation
2022-04-15 15:37:51 -05:00
Topher Lubaway
a379fc07e6 Fixes typo in docs deployment workflows (#12079)
adds newline for proper yml syntax
2022-04-15 13:51:42 -05:00
Topher Lubaway
6dd1daf6a0 Toph docs deployment workflow (#11991)
* Adds deploy docusaurus command

(won't work yet. commit for testing)

not force pushing

* Adds GH token workflow for GH actions

* Small comment changes

* Adds branch name to push

* adds gh switch

* Commit for clear dir for testing

* Addresses comments by adding comments
2022-04-15 13:32:55 -05:00
Davin Chia
efb969ca1d Add fix build publish step to master build. (#12065)
Also publish to slack if and only if the master build was fixed. This will make alerts more actionable.
2022-04-15 22:03:56 +08:00
Pedro S. Lopez
fbaf83eeeb Only build CDK with supported python versions (#12019)
* Only build CDK with supported python version

* also build 3.10

* use quotes

* back to only 3.9
2022-04-14 08:54:01 -07:00
Davin Chia
acfb606ce2 Actually remember to remove the force failure. (#12011) 2022-04-14 19:40:31 +08:00
Davin Chia
e3583d3a37 Remove force failure for Slack notification testing. (#12009)
Remove forced failure for Slack notification testing.

Also remove old build notifications that are being routed to the old Airbyte slack instance.
2022-04-14 19:34:44 +08:00
Davin Chia
ae8c4f2b3a Notify Slack Build Failure. (#12007)
To build awareness with the Airbyte dev team, we want to start notifying folks on build failures. This PR allows for that. This also forces the build to fail so we can test this.

Some modifications here to make it easy to test on Master.
2022-04-14 19:25:34 +08:00
Serhii Chvaliuk
5c8bd3a46d remove --ignore-empty option because sentry 2.0.0 (#11926)
check https://github.com/getsentry/sentry-cli/releases/
Breaking Changes:
    ref: Make ignore-empty for releases set-commits a default behavior and remove --ignore-empty flag (remove --ignore-empty usage) (breaking)

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-04-12 18:48:00 +03:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Tim Roes
82fd13ab40 Merge project assignment workflows (#11874)
* Merge project assignment workflows

* File naming suggestion
2022-04-11 19:13:28 +02:00