1
0
mirror of synced 2026-01-02 03:02:26 -05:00
Commit Graph

379 Commits

Author SHA1 Message Date
Topher Lubaway
5e1f968751 Adds zombie removal tool (#13718)
* Adds zombie removal tool

* Corrects endpoint adds comments

* Adds API links

* Changes search logic, fixes escape character

* Corrects help text
2022-06-13 13:28:32 -05:00
Denis Davydov
20bd923c63 fix source-acceptance-test build (#13485) 2022-06-10 10:11:49 +03:00
lmossman
2610ffcefc rename automated release creation action to be less similar to the existing Release OSS Airbyte action 2022-06-08 09:23:00 -07:00
Lake Mossman
527e391c16 add Release Airbyte action and update version bump PR description (#13584) 2022-06-08 09:09:28 -07:00
Davin Chia
40cb78e5c7 Fix the acceptance tests. (#13501)
We've seen errors like:
- https://github.com/airbytehq/airbyte/runs/6758654948?check_suite_focus=true#step:11:52833 - from trying to list bootloader logs. This is no longer possible since we remove the bootloader pod.
- https://github.com/airbytehq/airbyte/runs/6746572522?check_suite_focus=true#step:11:52164 - errors while tearing down the test.

I will do a follow up PR to refactor tests to hopefully speed things up:
- split Kube from normal tests.
- explore not recreating the db on each time.

In the mean time, this should stabilise the tests and get us back to green build.
2022-06-07 02:38:19 +08:00
Lake Mossman
da22f0320a remove scheduler/app from publish_docker.sh (#13507) 2022-06-06 11:34:08 -07:00
Lake Mossman
73034c64da Sweep old scheduler code (#13400)
* sweep all scheduler application code and new-scheduler conditional logic

* remove airbyte-scheduler from deployments and docs

* format

* remove 'v2' from github actions

* add back scheduler in delete deployment command

* remove scheduler parameters from helm chart values

* add back job cleaner + test and add comment

* remove now-unused env vars from code and docs

* format

* remove feature flags from web backend connection handler as it is no longer needed

* remove feature flags from config api as it is now longer needed

* remove feature flags input from config api test

* format + shorter url

* remove scheduler parameters from helm chart readme
2022-06-06 10:49:17 -07:00
Davin Chia
adc590e385 Update publishing scripts after renaming dirs. (#13486)
* Update dockerfile to match.

* Update publishing script to match.
2022-06-06 18:11:02 +08:00
Davin Chia
eb99f47746 Fat Jar: Rename Dir Part 2 (#13478)
## What
Part 2 of https://github.com/airbytehq/airbyte/pull/13122.

Follow up to #13476 .

Explanation for what is happening:

Identically named subprojects have the following issues:

* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.

* Rename airbyte-config:models to airbyte-config:config-models.
* Rename airbyte-config:persistence to airbyte-config:config-persistence.
2022-06-06 02:21:54 +08:00
Evan Tahler
d1ff8dd8b4 [Github Actions] Rename docker secrets and parameterize docker user (#13297)
* Rename docker secrets and parameterize docker user

* Apply suggestions from code review

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
2022-05-27 18:48:26 -07:00
Sherif A. Nada
981dc655ee include SAT and normalization in build reports (#13216)
* include SAT and normalization in build reports

* uncomment
2022-05-25 19:47:29 -07:00
Evan Tahler
be8d8574f2 Remove M1-specific docs - Airbyte works on M1 Macs now (#11856) 2022-05-24 16:46:26 -07:00
Brian Leonard
4d3e0418a4 Java integration test update (#13113)
* Get more java error messages

* Bolder result

* Debug file contents

* Just give an output message
2022-05-23 15:36:53 -07:00
Brian Leonard
e82e6fc0a2 Show integration test results on the Action Summary (#13072)
* Show summary on action results for failed test

* Write title and steps to log

* Log test

* Remove debugging

* Support python and java output

* Extra line break
2022-05-23 11:23:18 -07:00
Davin Chia
bdbd68f9bb Remove old test scripts. (#13014)
No longer needed since we removed the old scheduler.
2022-05-19 22:51:57 +08:00
Topher Lubaway
2c690f2144 More helpful error message for detected changes (#12956)
* More helpful error message for detected changes

* Wrote the check backwards

* more clear errors

* Newline cinema
2022-05-18 09:01:18 -05:00
Lake Mossman
a82f41edcf Acceptance test scripts - kill containers using volumes on shutdown (#12887)
* kill containers using volumes on shutdown in acceptance test scripts

* apply change to octavia integration tests as well

* escape inner command to make it be executed at the right time
2022-05-16 17:09:46 -07:00
Sherif A. Nada
7ec7ba33e0 continue sending connector build reports (#12879) 2022-05-16 10:41:37 -07:00
a-honcharenko
852528533d Fix naming in script (#12786)
* Artifact naming fix

* Indent properly
2022-05-11 22:20:27 +03:00
a-honcharenko
7b43e39219 Added init exception to script (#12783) 2022-05-12 02:17:54 +08:00
a-honcharenko
570770c10b A honcharenko/multiarch build 801 (#12570)
* added multiarch image publishing/modified dockerfiles

* added new ami

* changed version to test

* rollback version

* check version test

* env vars temp fix

* apt-utils error fix

* disabled failed test

* remove excluded tests

* Excluded :airbyte-db:lib:test

* Excluded :airbyte-db:lib:test

* static jdk version for test

* ok test

* ok test

* ok test

* test vars

* quemu issue fix

* Returned version vars

* Scripts update

* Version vars change

* Comment fix

* Added comments, minor changes and comments in dockerfiles

* Uncomment line to push images
2022-05-11 20:07:13 +03:00
Sherif A. Nada
fa22d32e14 Filter daily build report to only GA & Beta connectors (#12684) 2022-05-06 16:41:38 -07:00
Topher Lubaway
2119f4d3a2 docs deploy command works in more cases (#12622)
if the user has more than one page of branches
they will be prompted in less
this fixes that
2022-05-05 15:18:39 -05:00
Topher Lubaway
fced083be5 Comments and aids to help future devs build OSS (#12571) 2022-05-05 07:26:31 -05:00
Topher Lubaway
cc24c5dda3 Adds error message for missing homebrew (#12547)
tested by Brian locally
2022-05-04 09:05:28 -05:00
Topher Lubaway
1413aca478 Toph deploy docs ssh repo check (#12323)
* Check for ssh github repo

because assumptions mostly.
more clear fail messages for an unsupported workflow
also removes the pipeline which now is redundant with local workflow

* WIP1

* working test

* More clear git test

* less weird characters for grep

* remove verbose for excho
2022-04-27 10:26:35 -05:00
Topher Lubaway
f065d361b1 Toph docs pipeline fixes bugs n gitbook (#12114)
* WIP

* WIP2

* WIP3

* Remove comment/better comments

* Remote deployment updates

* Adds GH user for yarn deploy

* WIP 6

* WIP7

* WIP8

* WIP 10

* Working easy from home

not so much in the cloud
2022-04-19 14:49:04 -05:00
Topher Lubaway
e615b2faaa Addreses another docs build error (#12084)
* Addreses another docs build error

* Remove two typos

* Adds a test pattern

* Another stab

* More test

* Removing pipeline until more testing

docusaurus deploy isn't into automation
2022-04-15 15:37:51 -05:00
Topher Lubaway
d6b6e4b065 Addressing docusaurus deploy (#12083)
error is unclear and only happens in cloud
2022-04-15 14:43:13 -05:00
Topher Lubaway
ff7b2eecf5 Specifies a git user to address build error (#12082)
error:
[INFO] Deploy command invoked...
Error:  Error: Please set the GIT_USER environment variable,
or explicitly specify USE_SSH instead!
2022-04-15 14:16:32 -05:00
Topher Lubaway
8f6acc72be Cloud's version of which doesn't support -s (#12080)
so we removed -s and sent the output to the dark place
2022-04-15 14:02:16 -05:00
Topher Lubaway
6dd1daf6a0 Toph docs deployment workflow (#11991)
* Adds deploy docusaurus command

(won't work yet. commit for testing)

not force pushing

* Adds GH token workflow for GH actions

* Small comment changes

* Adds branch name to push

* adds gh switch

* Commit for clear dir for testing

* Addresses comments by adding comments
2022-04-15 13:32:55 -05:00
Parker Mossman
197806102e make sure VERSION is up to date after bumpversion (#11945) 2022-04-12 15:36:43 -07:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Alexandre Girard
266f05cecd Add script to setup venv for a connector (#11699)
* Add script to setup venv for a connector

* new line

* echo

* Add missing requirements file

* Add script to add venvs to intellij

* script to install venv

* Update

* This should be easier to use

* Better default

* Add unit tests

* add install venv option flag

* remove references to my own own home directory

* Call tools/bin/setup_connector_venv.sh

* Skip module if it already exists in the jdk table

* update

* Set right test name

* better interface

* update

* rename flag

* move to a function

* fix variable name

* derp

* Use real path

* Update setup instructions

* update instructions

* Instructions to both install the venv and update intellij

* update link

* Add command to install venv for all connectors

* <>

* newline
2022-04-08 17:18:14 -07:00
Augustin
582baf3b68 gradle: create a separate sub build for all connectors (#11833) 2022-04-08 10:35:37 -07:00
Topher Lubaway
c8337ea3ba Add docs sidebar (#11817)
* Brings pairity for doc checks from cloud

* WIP Sidebar config

It's sketchy JSON let's commit before we break i

* MVP sidebars

summary.md is no longer functional
it's higherarchy was translated into sidebar.js
as a JSON format that docusaurus likes

* Removes another unused readme
2022-04-07 14:16:35 -07:00
Jared Rhizor
a809514ea7 improve naming of cloud integration tests (#11713) 2022-04-04 16:01:22 -07:00
Topher Lubaway
d82c2a0505 Same logic as before with more clear user facing (#11700)
Old code was hard to understand errors from.

this should communicate problems more clearly
2022-04-04 13:16:16 -07:00
Greg Solovyev
afc1976a80 Skip empty line output from gradle (#11667)
Currently, every time this script attempts to launch a github action for an empty connector name
  see log output here: https://github.com/airbytehq/airbyte/runs/5786705935?check_suite_focus=true
  This change fixes this minor bug.
2022-04-04 09:47:47 -07:00
Jared Rhizor
493f0ea9f6 use Kubernetes watch api for retrieving exit codes (#11083)
* use kubernetes api for retrieving exit codes

* undelete test

* clean up more status check interval

* fmt

* wip

* clean up

* smarter filtering

* reordering

* exception handling

* better logging for test + speed up acceptance tests temp

* re-enable running on branch

* fix race condition in test

* add log

* trigger build

* trigger build

* re-run tests with everything enabled

* run tests

* run tests

* clean up

* respond to comments

* fix formatting

* fix whitespace

* remove comment

* 10 -> 5

* log exit code error message
2022-03-31 04:01:55 -07:00
Augustin
2d8f66f9e1 🐛 octavia-cli: specific release job for octavia (#11517) 2022-03-30 20:47:56 +02:00
Augustin
b42639a0e7 rollback build and publish of octavia (#11518) 2022-03-29 15:45:04 -07:00
Augustin
9d46ef83ba 🐙 octavia-cli: bump and publish with same version as Airbyte core (#11464) 2022-03-29 19:22:04 +02:00
Augustin
55ae3f856a 🐛 octavia-cli: propagate open api spec update (#11441) 2022-03-29 09:31:00 +02:00
Topher Lubaway
906887c327 Use shell hacks to find valid PAT (#11439)
* Use shell hacks to find valid PAT

if rate limit exceeded try another key

* Use tim's version of the ec2 runner

this is a weird solution with weird limitations.  should address the short term pain (march 2022)
2022-03-28 15:17:07 -05:00
Jared Rhizor
0191af9bf3 build acceptance test connector images for m1 (#11374)
* build acceptance test connector images for m1

* executable

* rename file

* add comment
2022-03-24 06:53:27 -07:00
girarda
fd13ab1970 Fix CONNECTORS_BASE build for M1 (#10925)
* Pass in arm64

* default value

* fix
2022-03-07 17:30:41 -08:00
Tim Roes
d73af697e4 Use cypress dashboard and stabilize e2e tests (#10807)
* Record e2e tests to cypress dashboard

* Make env variable accessible in script

* Improve e2e_test script

* Properly wait for server to be ready

* Isolate test suites better

* More test isolation

* Revert baseUrl for development
2022-03-03 18:29:07 +01:00
Octavia Squidington III
74a05e4604 Bump Airbyte version from 0.35.43-alpha to 0.35.44-alpha (#10789)
* Bump Airbyte version from 0.35.43-alpha to 0.35.44-alpha

* Commit.

* Add exception block.

* Why would having try catch work?

* Add logging to figure out.

* Undo all debugging changes.

* Better comments.

Co-authored-by: davinchia <davinchia@users.noreply.github.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
2022-03-02 22:47:26 +08:00