1
0
mirror of synced 2026-01-02 21:02:43 -05:00
Commit Graph

435 Commits

Author SHA1 Message Date
Alexandre Girard
64b50adfdd Connector-builder server stub (#18410)
* init

* bad copy/paste

* move to top level

* Revert "move to top level"

This reverts commit aca3534d38.

* attempt to wire up connector builder frontend to server

* copy from octaviacli

* fix connection to builder server

* update

* delete

* Update

* delete python-version

* Revert "delete python-version"

This reverts commit f9258a7755.

* setup python

* install python

* rename

* kube stuff

* Install python

* missing kube file

* rename

* Update files

* Update bumpversion

* install python

* try with different entrypoint

* rename container

* point to docker-compose.yaml file

* derp

* copy acceptance_test.sh

* copy from acceptance tests

* delete cruft

* update

* remove application env

* reset

* reset to master

* update

* skip comprehensive incremental tests

* Revert "skip comprehensive incremental tests"

This reverts commit 9cee657596.

* reset to master

* remove cruft

* delete superfluous steps

* update port to 8003

* reset to master

* Update publish docker

* move openapi spec to airbyte-connector-builder

* point to openapi spec

* dont expose the connector builder to localhost

* reset FE components to master

* Don't deploy the connector-builder

* Revert "Don't deploy the connector-builder"

This reverts commit 3d157494cf.

* Revert "Revert "Don't deploy the connector-builder""

This reverts commit beac3d48f0.

* comment out more things related to connector builder server

* more attempts at removing the connector builder

* comment out more things

* Apply suggestions from code review

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update airbyte-webapp/src/config/configProviders.ts

Co-authored-by: Lake Mossman <lake@airbyte.io>

* update

* rename

* indent

* Revert "move openapi spec to airbyte-connector-builder"

This reverts commit 57dda04723.

* Revert "rename"

This reverts commit b2d802b8fa.

* Revert "Revert "rename""

This reverts commit 91db24fd4a.

* point to wrong file in case it fixes the build

* point to right openapi file

* Revert "Revert "move openapi spec to airbyte-connector-builder""

This reverts commit e46a837454.

* point to moved file

* fix path

* Update from master

* newline

* Add failing test

* Revert "Add failing test"

This reverts commit ed9fea09b5.

* comment

* update commented requires

* Add a comment

* 2022

* rename to connector-builder-server

* typo

Co-authored-by: lmossman <lake@airbyte.io>
2022-11-09 13:07:26 -08:00
Augustin
3feda06a14 ci_integration_test.sh: cut GITHUB_STEP_SUMMARY (#18895) 2022-11-03 14:54:32 +01:00
perangel
c5936b0841 [CI] Speed up check_images_exist (#18873)
* [CI] Speed up check_images_exist

* Refactor to use helper function and cleanup output

* Remove test for `compose_output` file
2022-11-02 22:02:09 -04:00
Liren Tu
1dc6874c59 Show version and changelog status for affected connectors (#18845)
* Add placeholder change

* List connectors and versions in a table

* Add changelog status

* Add changelog annotation

* Add publish version status

* Add version bump for demo

* Update status

* Add strict-encrypt verion check

* Add overall status summary

* Fix version status format

* Fix format

* Update styling

* Add notes

* Add a special case for mongodb

* Update details open / close logic

* Add link to dockerfile

* Revert demo changes

* Remove version link

* Delete report when it is deprecated

* Revert "Delete report when it is deprecated"

This reverts commit f0008e4976.
2022-11-02 11:56:38 -07:00
Volodymyr Pochtar
b77d039e6b ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables (#18809)
* ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables

* ci: replace github set-ouput with new syntax in ./tools/bin/
2022-11-02 17:57:48 +02:00
Conor
1167d35ce7 ci: add job and run id to test reports (#18832) 2022-11-01 18:49:16 -07:00
Conor
dc15f56e31 ci: use custom test-reporter action to upload job results (#18004)
* ci: use custom action to upload job results
2022-11-01 13:59:19 -05:00
Ella Rohm-Ensing
385ab7e6d2 improve connector dependency github action (#18480)
* Use more specific variable name

* Write affected sources and destinations to files

* Use Markdown template to format results

* Update workflow to edit existing comment instead of making a new one

* Check whether changed_files.txt is empty before proceeding

* move template to .github/

* Use different quotes for true

* Move logic about continuing back to after dependency report because of ignores

* Use correct filename in conditional

* Use if to avoid early exit

* Create folder for comment templates and use constant for file path

* Add handling for non-source non-destination info, pull out markdown listing
2022-11-01 11:37:01 -04:00
Davin Chia
f0954597c3 Remove unused test script. (#18715) 2022-10-31 09:58:41 -07:00
Taras Korenko
a9a2bd4da0 Simplify the OSS documentation deploy system (#2670) (#18436)
+ Updates tools/bin/deploy_docusaurus to work on Github Runners
+ Adds 'Deploy docs.airbyte.com' Github workflow
2022-10-27 18:56:06 +03:00
Jonathan Pearlin
5d326a3cae Make script both Unix/Linux and OSX compatible (#18493) 2022-10-26 12:53:42 -04:00
terencecho
a714175997 Change port for acceptance test destination db (#18391) 2022-10-24 15:38:55 -07:00
Parker Mossman
0abc5f6373 only rm file if it exists (#18310) 2022-10-21 11:34:47 -07:00
pmossman
ec52a637b9 remove file that should be gitignored 2022-10-20 15:09:33 -07:00
Parker Mossman
c5336ce1db use basic auth username and password from .env (#18273) 2022-10-20 15:05:59 -07:00
Evan Tahler
e232ffa9fc Skip basic auth for Octavia integration tests (#18270) 2022-10-20 14:10:24 -07:00
Evan Tahler
ec304fd376 Disable basic auth for frontend acceptance tests (#18255)
* Disable auth for frontend acceptance tests

* Update auth flags
2022-10-20 10:41:48 -07:00
Evan Tahler
8fba3ac95b Remove airbyte-cli from build scrips (#18251) 2022-10-20 09:08:50 -07:00
Parker Mossman
7cedfa48de Load Testing Script (#18020)
* script skeleton

* add API call to source_definitions to fetch E2E Test Source definition ID

* createSource implementation

* add destination creation logic implementation

* get definition IDs, catalogId, and implement connection creation

* add cleanup script and write created ids to a file that can be cleaned up

* make cloud header a command-line argument, other cleanup

* script comments fix

* remove kube references and fix indentation

* temp commit - don't push

* remove discover catalog function

* more cleanups

* more cleanups

* cleanup help text

* exit codes and show how many connections left

* add README

Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
2022-10-19 17:56:12 -07:00
Xiaohan Song
6f88bdba39 query to find out jobs running unusally long (#17978)
* query to find out jobs running unusally long

* comments fix

* add more indents for better readability

* Update airbyte-metrics/reporter/src/main/java/io/airbyte/metrics/reporter/MetricRepository.java

Co-authored-by: Davin Chia <davinchia@gmail.com>

* formatting

Co-authored-by: Davin Chia <davinchia@gmail.com>
2022-10-19 16:21:05 -07:00
Evan Tahler
3d053e32ee Use Nginx + Basic Auth to secure OSS Airbyte (#17694)
* Use Nginx + Basic Auth to secure OSS Airbyte

* use local passwords

* Use gradle builds

* K8s setup and source values from ENV

* note about disabling

* add back defaults

* custom 401 page

* update http message

* update docs

* remove kube files

* additional doc updates

* Add a test suite

* fix failure exit codes

* doc updates

* Add docs

* bump to re-test

* add more sleep in tests for CI

* better sleep in test

* Update docs/operator-guides/security.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* PR updates

* test comment

* change test host on CI

* update tests and nginx to boot without backend

* proxy updates for docker DNS

* simpler test for uptime

* acceptance test skips PWs

* remove resolver madness

* fixup tests

* more proxy_pass revert

* update acceptance test exit codes

* relax test expectations

* add temporal mount back for testing

* Update docs/operator-guides/security.md

Co-authored-by: swyx <shawnthe1@gmail.com>

* Update airbyte-proxy/401.html

Co-authored-by: swyx <shawnthe1@gmail.com>

* more doc updates

* Octavia CLI uses Basic Auth  (#17982)

* [WIP] Octavia CLI uses Basic Auth

* readme

* augustin: add basic auth headers to clien

* augustin: add basic auth headers to client

* tests passing

* lint

* docs

* Move monkey patch to test

* coerce headers into strings

* monkey patch get_basic_auth_token

Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>

* fix launch permissions

* Keep worker port internal

* more readme

Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: swyx <shawnthe1@gmail.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
2022-10-19 15:52:01 -07:00
Benoit Moriceau
260ab6bdea Move the port forward outside of the main docker-compose (#17864) 2022-10-17 12:59:28 -07:00
Evan Tahler
792fa7cead Remove unused airbyte-cli (#18009) 2022-10-17 10:28:53 -07:00
Topher Lubaway
804bf47484 Major overhaul of check images (#17979)
* Major overhaul of check images

actually fail on missed images
detect rate limit for dockerhub
sleep when hit rate limit and registry
more clear user messaging
Reduce output from 5000 lines to ~ 1000
More clear help messages for failures

* Addresses GH feedback

minor bugfixes
2022-10-14 12:58:27 -05:00
Edward Gao
808af16023 Fix Docker image existence check (normalization) (#17908)
* fix path to NormalizationRunnerFactory

* check for file existence also
2022-10-12 14:08:46 -07:00
Topher Lubaway
9c46e5a560 Fixes relative links above docs to be URIs (#17850)
also adds a missing exit 1 to deploy docs
when the deploy SHOULD fail
2022-10-11 11:36:33 -05:00
Volodymyr Pochtar
a142141c98 Revert "Revert "feat: replace openjdk with amazoncorretto:17.0.4 on connectors for seсurity compliance (#17511)" (#17820)" (#17833)
This reverts commit e47c94ecff.
2022-10-11 14:22:54 +03:00
Volodymyr Pochtar
e47c94ecff Revert "feat: replace openjdk with amazoncorretto:17.0.4 on connectors for seсurity compliance (#17511)" (#17820)
This reverts commit ab71f5bc29.
2022-10-11 11:58:01 +03:00
Volodymyr Pochtar
ab71f5bc29 feat: replace openjdk with amazoncorretto:17.0.4 on connectors for seсurity compliance (#17511) 2022-10-10 12:35:39 +03:00
Michael Siega
84f0bacaa2 support large schema discovery (#17394)
* support large schema discovery

* update generic source tests to handle new approach to schema discovery

* readability improvements related to schema discovery and large schema support

* update internal ScheduleHandler method name

* update source tests per new schema discovery interface
2022-10-05 20:33:43 +02:00
Jimmy Ma
d2a717594c Add some services start validation to acceptance_tests.sh (#17425) 2022-09-30 10:35:27 -07:00
Volodymyr Pochtar
a72cb7d1f8 feat: replace openjdk with amazoncorretto:17.0.4 on platform for security compliance (#16919) 2022-09-20 19:01:48 +03:00
Jimmy Ma
fa79bc4dfe Add missing references to cron in the acceptance tests (#16820)
* Add missing image
* More missed references to cron in the test scripts
2022-09-16 14:39:43 -07:00
Topher Lubaway
b5de9803ac Adds exit (#16702) 2022-09-14 09:07:02 -05:00
Topher Lubaway
7de9e25798 Fixes case where there is nothing to kill (#16661) 2022-09-13 14:01:23 -05:00
Evan Tahler
6249b485cb clean-local.sh to remove local builds and venvs (#16597) 2022-09-12 14:06:20 -07:00
Serhii Chvaliuk
ba1154c677 Tools: script for scanning all non-secret spec fields (#16390)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-09-09 17:05:32 +03:00
Kyryl Skobylko
3b807c8e35 fix: Update oss charts. Add acceptance tests for helm (#16131)
* fix: revert extraEnv. Move updated extraEnv to env_vars

* fix: Add node pool selector feature for gke deployments

* fix: hotfix. Fix webapp values and imagepullsecrets

* fix: remove nodeSelector duplicate

* fix: revert imagePullSecrets changes

* fix nit

* fix: fix bootloader imagepullsecrets

* fix: fix image pull secrets again...

* feat: add acceptance tests for helm

* add step to install unzip

* fix incorrect command spelling(helm)

* update slack notification sending for Airbyte CI workflow

* Rename Helm E2E test report name

* fix: Update helm charts

* fix: fix temporal DB Port bug

* fix nits

* do a small testing for sake of science

* Update test chart.yaml with note block

* Revert changes for AC tests

* fix: Include #16166

* fix: add metrics to the ci workflow

* fix: revert changes for temporal service. move them to server service;

* fix: revert fullstory deletion

* fix: change way of defining image tag and make it to be set in global values. Add values.yaml.test for integration testing and local deployment

* fix: udpate docs, update values.yaml

* fix: fix helpers.tpl in all subcharts

* Update chart docs

Co-authored-by: Davin Chia <davinchia@gmail.com>
2022-09-09 15:58:58 +03:00
Tim Roes
e8bc3d7e03 Improve body of release PR (#16330) 2022-09-06 23:21:00 +02:00
Tim Chan
a4621f8c96 use refactored find a pat script that writes token to env (not base64 encoded) (#16321) 2022-09-06 09:30:32 -07:00
Conor
ce2bcd4207 fix: run.delete() not available in pygithub (#16127) 2022-08-30 09:55:06 -05:00
Conor
6074ad5d83 feat: tools to identify and clean up unused workflows (#16019)
* feat: tools to identify and clean up unused workflows

* fix: shorten title

Co-authored-by: Topher Lubaway <asimplechris@gmail.com>

* fix: make slack token optional

* fix: review comments

* ci: remove todo's in workflow run for merge

Co-authored-by: Topher Lubaway <asimplechris@gmail.com>
2022-08-29 13:45:38 -05:00
Yevhen Sukhomud
c402264664 11815 New github action to test projects that depend on the project being changed (#15783)
* 11815 New github action to test projects that depend on the project being changed
2022-08-25 10:00:19 +07:00
Evan Tahler
9a531bc524 Add airbyte/cron docker image to publish script (#15860)
* Add `airbyte/cron` docker image to publish script

* add publish comment
2022-08-22 15:10:17 -07:00
Evan Tahler
4774949ab0 Remove -alpha tags from Airbyte release versions (#15766)
* Remove alpha tags in autobump files

* remove more alpha tags
2022-08-22 11:06:49 -07:00
Davin Chia
fd0b7699a4 Add JUnit Test Reports to build outputs. (#15271)
It's difficult today to parse all the logs from tests. Engineers waste time scrolling through the log outputs and looking for the relevant stack trace.

This PR adds an action to generate a JUnit report so devs can understand test results at a glance. This generates 3 reports for each of the main build jobs when the build completes. We leave the frontend build out since this is aggregated by cypress.

See https://github.com/airbytehq/airbyte/pull/15271/checks?check_run_id=7683783016 for an example of how this works.

Use the https://github.com/dorny/test-reporter action and configure this to look at the Jacoco test report output for top level and second level builds. Note that most of the parameters into the action don't work.

Comment out the trap commands to output logs. Though this can be useful for debugging, there is little practical use in the day-to-day, and results in extremely noisy logs.
2022-08-08 17:04:36 -07:00
Davin Chia
f9026b0dfc Remove unnecessary test logging. (#15384)
This logging makes it tough to figure out what is happening in the tests.
2022-08-05 17:20:36 -07:00
Serhii Chvaliuk
c5a98f3ecf GitHub Actions - workflow Connector Integration Tests - add retry (#14452)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-08-01 17:37:20 +03:00
Davin Chia
cd5d8c2cbd Make sure Airbyte release process uses JDK 19. (#14993)
Follow up to #14971 . Make sure to update this for the OSS publishing process as well.

Also update all the dockerfiles.

Connectors are not touched.
2022-07-25 08:53:41 -07:00
Peter Hu
b4d2348fcf sleep 1 second every connector lookup (#14808)
* sleep 1 second every connector lookup

to prevent running into docker hub API rate limits

* comment on why we sleep

Co-authored-by: Davin Chia <davinchia@gmail.com>
2022-07-18 12:12:20 -07:00