1
0
mirror of synced 2026-01-01 00:02:54 -05:00
Commit Graph

564 Commits

Author SHA1 Message Date
George Claireaux
dd2d5d0129 in case runners fail to spin up, this needs to run on github-hosted (#13996) 2022-06-22 11:45:00 +01:00
Peter Hu
bcc20b4db1 publish oss for cloud (#13978)
workflow to publish oss artifacts that cloud needs to build against
use docker buildx to create arm images for local development
2022-06-21 14:30:42 -07:00
Topher Lubaway
7acf0c54df Adds test for new workflow (#13986)
* Adds test for new workflow

* Adds airbyte repo

* remove testing line
2022-06-21 14:58:27 -05:00
George Claireaux
e7dae06168 Enhance /publish to allow for multiple connectors and parallel execution (#13864)
* start

* revert

* azblob

* bq

* bq denorm

* megapublish baaaabyyyy

* fix needs

* matrix connectors

* auto-bump connector version

* dont failfast and max parallel 5

* multi runno

* minor

* testing matrix agents

* name

* testing multi agents

* tmp fix

* new multi agents

* multi test

* tryy

* let's do this

* magico

* fix

* label test

* couple more connector bumps

* temp

* things

* check this

* lets gooo

* more connectors

* Delete TEMP-testing-command.yml

* auto-bump connector version

* added comment describing bash part

* running single thread

* catch sentry cli

* auto-bump connector version

* destinations

* + snowflake

* saved

* auto-bump connector version

* auto-bump connector version

* java source bumps

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* remove twice-defined methods

* label things

* revert action

* using the new test action

* point at action

* wrong tag on action

* update pool label

* update to use new ec2-github-runner fork

* this needs to be more generic than publisher

* change publish to run on pool

* add comment about runner-pool usage

* updated publish command docs for multi & parallel connector runs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* unbump failed publish versions

* missed dockerfiles

* remove failed docs

* mssql fix

* overhauled the git comment output

* bumping a test connector that should work

* slight order switcheroo

* output connectors properly in first message

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-21 18:40:33 +01:00
George Claireaux
b9f79ccdf0 remove test action (#13944) 2022-06-20 17:45:10 +01:00
George Claireaux
546d633337 newaction (#13942) 2022-06-20 17:07:20 +01:00
Jonathan Pearlin
abe3024511 Update airbyte-protocol.md (#13892)
* Update airbyte-protocol.md

* Fix typo

* Fix prose

* Add protocol reviewers for protocol documentation

* Remove duplicate
2022-06-17 14:06:53 -04:00
Charles
5d1b902f6d add evan, jonathan, charles as protocol reviews in codeowners (#13776) 2022-06-16 07:56:50 -07:00
Topher Lubaway
5e1f968751 Adds zombie removal tool (#13718)
* Adds zombie removal tool

* Corrects endpoint adds comments

* Adds API links

* Changes search logic, fixes escape character

* Corrects help text
2022-06-13 13:28:32 -05:00
Sherif A. Nada
4f48748fd1 remove slack notification from publish command (#13710)
This actually has not been working in a long time, and no one seems to need it, so just removing it
2022-06-13 07:15:28 -07:00
Lake Mossman
30161726c2 escape backticks in create-release action so that they are handled properly (#13672)
* escape backticks in create-release action so that they are handled properly

* reword comment

* fix typo
2022-06-10 08:59:22 -07:00
Tim Roes
fe7e1c2272 Upgrade node to latest LTS (#13683)
* Upgrade node to latest LTS

* Upgrade to newer setup-node action
2022-06-10 17:07:57 +02:00
lmossman
2610ffcefc rename automated release creation action to be less similar to the existing Release OSS Airbyte action 2022-06-08 09:23:00 -07:00
Lake Mossman
527e391c16 add Release Airbyte action and update version bump PR description (#13584) 2022-06-08 09:09:28 -07:00
Tim Roes
4c97893363 Validate links in frontend in CI (#13600)
* Validate doc links during building
* Add scheduled link validation
* Remove accidentally commited file
* Remove unnecessary file
* Review feedback
2022-06-08 15:45:50 +02:00
Sherif Nada
b763bdf4ab Revert "add github workflows for publishing ERDs"
This reverts commit c7a23c680b.
2022-06-08 03:32:56 -07:00
Sherif Nada
c7a23c680b add github workflows for publishing ERDs 2022-06-07 22:42:13 -07:00
Sherif A. Nada
0ce18fb56d Auto-add connector docs pages to catalog sidebar and update checklist (#13467) 2022-06-06 13:30:24 -07:00
Lake Mossman
73034c64da Sweep old scheduler code (#13400)
* sweep all scheduler application code and new-scheduler conditional logic

* remove airbyte-scheduler from deployments and docs

* format

* remove 'v2' from github actions

* add back scheduler in delete deployment command

* remove scheduler parameters from helm chart values

* add back job cleaner + test and add comment

* remove now-unused env vars from code and docs

* format

* remove feature flags from web backend connection handler as it is no longer needed

* remove feature flags from config api as it is now longer needed

* remove feature flags input from config api test

* format + shorter url

* remove scheduler parameters from helm chart readme
2022-06-06 10:49:17 -07:00
Brian Leonard
faab304883 Add more pr actions to tag pull requests (#13437) 2022-06-02 12:35:13 -07:00
Alexandre Girard
d9fa24ffff Low code connectors: implement components for sendgrid (#12853)
* checkout from alex/cac

* checkout from alex/cac

* checkout from alex/cac

* Add missing tests

* Add missing files

* Add missing tests

* add missing file

* missing file

* missing file

* rename

* doc

* doc

* remove broken test

* rename

* jinja dependency

* Add comment

* comment

* comment

* pyjq dependency

* rename file

* delete unused file

* Revert "delete unused file"

This reverts commit 758e939367.

* fix

* rename

* abstract property

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* isort

* update state

* fix imports

* update dependency

* format

* rename file

* decoder

* Use decoder

* Update comment

* dict_state is actually backed by a dict

* Add a comment

* update state takes kwargs

* move state out of offset paginator

* update jq parameter order

* update

* remove incremental mixin

* delete comment

* update comments

* update comments

* remove no_state

* rename package

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* remove print

* move test

* delete duplicates

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* pass parameters through kwargs

* update interface to pass source in interface

* update interface to pass source in interface

* rename to stream_slicer

* Allow passing a string or an enum

* Define StateType enum

* convert state_type if not of type type

* convert state_type if not of type type

* Low code connectors: string interpolation with jinja (#12852)

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* move test to right module

* Add missing test

* Use authbase instead of deprecated class

* leverage generator

* Delete dead code

* rename methods

* rename to declarative

* rename the classes too

* Try to install packages to build jq

* isort

* only automake

* Revert "only automake"

This reverts commit c8fe154ffc.

* remove git

* format

* Add jq dependency

* Use request header provider

* rename

* rename field

* remove get_context method

* rename
2022-06-01 07:02:35 -07:00
Marcos Marx
53eeb09da8 Github Template: Update bug-report.md (#13316) 2022-05-31 19:32:14 -03:00
Parker Mossman
aaf34cfc20 update OSS action to take in docker username input (#13352) 2022-05-31 13:41:40 -07:00
Evan Tahler
d1ff8dd8b4 [Github Actions] Rename docker secrets and parameterize docker user (#13297)
* Rename docker secrets and parameterize docker user

* Apply suggestions from code review

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
2022-05-27 18:48:26 -07:00
Davin Chia
65f79d26b9 Test using default AMI with buildx to publish connectors. (#13282) 2022-05-27 23:30:37 +08:00
LiRen Tu
60f78a9d7c Move the comment step in test and publish commands to the front (#13238) 2022-05-26 13:25:02 -07:00
noahkawasaki-airbyte
7a79216d01 Make publish command work with private repos by adding JWT into curl headers (#13141)
* Make publish command work with private repos by adding in JWT token to headers for curl

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-24 10:55:46 -07:00
Tim Roes
5237d26ffd Change frontend label to team/frontend (#13093) 2022-05-24 16:13:04 +02:00
Brian Leonard
e82e6fc0a2 Show integration test results on the Action Summary (#13072)
* Show summary on action results for failed test

* Write title and steps to log

* Log test

* Remove debugging

* Support python and java output

* Extra line break
2022-05-23 11:23:18 -07:00
Lake Mossman
26ed3856e1 Migrate OSS to temporal scheduler (#12757)
* Migrate OSS to temporal scheduler

* add comment about migration being performed in server

* add comments about removing migration logic

* formatting and add tests for migration logic

* rm duplicated test

* remove more duplicated build task

* remove retry

* disable acceptance tests that call temporal directly when on kube

* set NEW_SCHEDULER and CONTAINER_ORCHESTRATOR_ENABLED env vars to true to be consistent

* set default value of container orchestrator enabled to true

* Revert "set default value of container orchestrator enabled to true"

This reverts commit 21b36703a9.

* Revert "set NEW_SCHEDULER and CONTAINER_ORCHESTRATOR_ENABLED env vars to true to be consistent"

This reverts commit 6dd2ec04a2.

* Revert "Revert "set NEW_SCHEDULER and CONTAINER_ORCHESTRATOR_ENABLED env vars to true to be consistent""

This reverts commit 2f40f9da50.

* Revert "Revert "set default value of container orchestrator enabled to true""

This reverts commit 26068d5b31.

* fix sync workflow test

* remove defunct cancellation tests due to internal temporal error

* format - remove unused imports

* revert changes that set container orchestrator enabled to true everywhere

* remove NEW_SCHEDULER feature flag from .env files, and set CONTAINER_ORCHESTRATOR_ENABLED flag to true for kube .env files

Co-authored-by: Benoit Moriceau <benoit@airbyte.io>
2022-05-18 17:05:42 -07:00
Marcos Marx
0a311e6c8d Update Issue Bug Template: remove log summary and update log message (#12982)
* remove log summary and update log message

* apply suggestions
2022-05-18 15:38:38 -04:00
Topher Lubaway
2c690f2144 More helpful error message for detected changes (#12956)
* More helpful error message for detected changes

* Wrote the check backwards

* more clear errors

* Newline cinema
2022-05-18 09:01:18 -05:00
Topher Lubaway
6bbd715d59 Removes GH branch sync used by gitbook (#12078)
we aren't using this anymore
2022-05-17 08:51:55 -05:00
Brian Leonard
8a49addc67 Switch to shared code (#12590) 2022-05-13 14:22:59 -07:00
Davin Chia
04eb3b402e Run connector build when airbyte protocol is updated. (#12831) 2022-05-13 09:47:11 +08:00
a-honcharenko
570770c10b A honcharenko/multiarch build 801 (#12570)
* added multiarch image publishing/modified dockerfiles

* added new ami

* changed version to test

* rollback version

* check version test

* env vars temp fix

* apt-utils error fix

* disabled failed test

* remove excluded tests

* Excluded :airbyte-db:lib:test

* Excluded :airbyte-db:lib:test

* static jdk version for test

* ok test

* ok test

* ok test

* test vars

* quemu issue fix

* Returned version vars

* Scripts update

* Version vars change

* Comment fix

* Added comments, minor changes and comments in dockerfiles

* Uncomment line to push images
2022-05-11 20:07:13 +03:00
Sherif A. Nada
fa22d32e14 Filter daily build report to only GA & Beta connectors (#12684) 2022-05-06 16:41:38 -07:00
Greg Solovyev
60cbf0d0d9 Add timeout to publish-image task (#12679)
publish-image task can take a while, because it is also running
integration tests by default. test-command workflow has a timeout
of 240 minutes. I added the same timeout value to publish-image task.
2022-05-06 11:18:18 -07:00
Denys Krytskyi
905807591b Modify OSS Airbyte and Cloud Airbyte to use Catalogs - 809 (#12610)
This PR provides 'version catalog' plugin, Where catalog is defined by using declared dependency list in 'build.gradle'.
In this pr all dependencies were transferred to 'libs' using 'dependencyResolutionManagement -> versionCatalogs -> libs'
2022-05-06 02:23:01 +08:00
Brian Leonard
763dea255d Uses private action for workflow (#12525) 2022-05-02 13:44:13 -07:00
Brian Leonard
4ee1fa6c5a Use shared workflow action on issues (#12382)
* Use shared workflow action on issues

* Rename to token

* Also on unlabeled
2022-05-01 14:38:43 -07:00
noahkawasaki-airbyte
e8406a2cd6 Initial PR /publish workflow into pre-merge /build-connector and post-merge /publish-connector workflows (#12405) 2022-04-29 07:03:27 -07:00
noahkawasaki-airbyte
4a9f663565 Add experimental build-connector and publish-connector slash commands (#12442) 2022-04-28 11:25:31 -07:00
Topher Lubaway
1413aca478 Toph deploy docs ssh repo check (#12323)
* Check for ssh github repo

because assumptions mostly.
more clear fail messages for an unsupported workflow
also removes the pipeline which now is redundant with local workflow

* WIP1

* working test

* More clear git test

* less weird characters for grep

* remove verbose for excho
2022-04-27 10:26:35 -05:00
George Claireaux
7e1d95c783 🤖 publish command: fix for version bumping multiple connectors in same PR (#12397)
* add a git pull from origin master before commit auto changes

* dummy bump 2 connectors

* pull from current branch not master

* auto-bump connector version

* dummy bump 2 connectors

* revert faker bump

* auto-bump connector version

* better descript in changelog

* dummy bump... again

* commit first, then pull merge, then push

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-27 16:18:22 +01:00
Davin Chia
e93bb85dc7 Fix build. (#12242) 2022-04-21 20:46:31 +08:00
Peter Hu
0d3088e456 notify cloud repo on push to master (#12229)
* notify cloud repo on push to master

* split out build-branch action

so we can use it in cloud workflow
2022-04-20 16:53:20 -07:00
Topher Lubaway
f065d361b1 Toph docs pipeline fixes bugs n gitbook (#12114)
* WIP

* WIP2

* WIP3

* Remove comment/better comments

* Remote deployment updates

* Adds GH user for yarn deploy

* WIP 6

* WIP7

* WIP8

* WIP 10

* Working easy from home

not so much in the cloud
2022-04-19 14:49:04 -05:00
Topher Lubaway
e615b2faaa Addreses another docs build error (#12084)
* Addreses another docs build error

* Remove two typos

* Adds a test pattern

* Another stab

* More test

* Removing pipeline until more testing

docusaurus deploy isn't into automation
2022-04-15 15:37:51 -05:00
Topher Lubaway
a379fc07e6 Fixes typo in docs deployment workflows (#12079)
adds newline for proper yml syntax
2022-04-15 13:51:42 -05:00