1
0
mirror of synced 2025-12-20 18:39:31 -05:00
Commit Graph

851 Commits

Author SHA1 Message Date
Ella Rohm-Ensing
c472340ad3 add base-normalization to dependency checker action (#19313)
* Add normalization to list of modules to check dependencies for

* Remove now unneeded empty lists

* Refactor how gradle files are retrieved

* Let method set the dict key for module

* Rename CONNECTOR_PATH, ensure that path for get_gradle_file_for_path is always a folder

* Add typing on potentially confusing method

* Make comment template clearer

* Move modules into details

* Auto-close sources and destinations, add emoji and count for other modules

* Rename function to get dependent modules
2022-11-18 08:55:28 -05:00
perangel
9d03921908 Retry docker pulls in CI (#19384)
* Retry docker pulls in CI

* Notify caller of retry

* Invoke pulls with `-i`

* Cleanup retrie default value
2022-11-16 13:44:21 -05:00
Augustin
1c961f41bf ci-connector-ops: Check test strictness level on GA source connectors (#19383) 2022-11-15 00:23:33 +01:00
Alexandre Girard
64b50adfdd Connector-builder server stub (#18410)
* init

* bad copy/paste

* move to top level

* Revert "move to top level"

This reverts commit aca3534d38.

* attempt to wire up connector builder frontend to server

* copy from octaviacli

* fix connection to builder server

* update

* delete

* Update

* delete python-version

* Revert "delete python-version"

This reverts commit f9258a7755.

* setup python

* install python

* rename

* kube stuff

* Install python

* missing kube file

* rename

* Update files

* Update bumpversion

* install python

* try with different entrypoint

* rename container

* point to docker-compose.yaml file

* derp

* copy acceptance_test.sh

* copy from acceptance tests

* delete cruft

* update

* remove application env

* reset

* reset to master

* update

* skip comprehensive incremental tests

* Revert "skip comprehensive incremental tests"

This reverts commit 9cee657596.

* reset to master

* remove cruft

* delete superfluous steps

* update port to 8003

* reset to master

* Update publish docker

* move openapi spec to airbyte-connector-builder

* point to openapi spec

* dont expose the connector builder to localhost

* reset FE components to master

* Don't deploy the connector-builder

* Revert "Don't deploy the connector-builder"

This reverts commit 3d157494cf.

* Revert "Revert "Don't deploy the connector-builder""

This reverts commit beac3d48f0.

* comment out more things related to connector builder server

* more attempts at removing the connector builder

* comment out more things

* Apply suggestions from code review

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update airbyte-webapp/src/config/configProviders.ts

Co-authored-by: Lake Mossman <lake@airbyte.io>

* update

* rename

* indent

* Revert "move openapi spec to airbyte-connector-builder"

This reverts commit 57dda04723.

* Revert "rename"

This reverts commit b2d802b8fa.

* Revert "Revert "rename""

This reverts commit 91db24fd4a.

* point to wrong file in case it fixes the build

* point to right openapi file

* Revert "Revert "move openapi spec to airbyte-connector-builder""

This reverts commit e46a837454.

* point to moved file

* fix path

* Update from master

* newline

* Add failing test

* Revert "Add failing test"

This reverts commit ed9fea09b5.

* comment

* update commented requires

* Add a comment

* 2022

* rename to connector-builder-server

* typo

Co-authored-by: lmossman <lake@airbyte.io>
2022-11-09 13:07:26 -08:00
Augustin
3feda06a14 ci_integration_test.sh: cut GITHUB_STEP_SUMMARY (#18895) 2022-11-03 14:54:32 +01:00
perangel
c5936b0841 [CI] Speed up check_images_exist (#18873)
* [CI] Speed up check_images_exist

* Refactor to use helper function and cleanup output

* Remove test for `compose_output` file
2022-11-02 22:02:09 -04:00
Anne
d26e5bc248 DiscoverSchema endpoints calculates diff and breaking change (#18571)
* update discover schema endpoint to calculate diff
2022-11-02 14:10:07 -07:00
Liren Tu
1dc6874c59 Show version and changelog status for affected connectors (#18845)
* Add placeholder change

* List connectors and versions in a table

* Add changelog status

* Add changelog annotation

* Add publish version status

* Add version bump for demo

* Update status

* Add strict-encrypt verion check

* Add overall status summary

* Fix version status format

* Fix format

* Update styling

* Add notes

* Add a special case for mongodb

* Update details open / close logic

* Add link to dockerfile

* Revert demo changes

* Remove version link

* Delete report when it is deprecated

* Revert "Delete report when it is deprecated"

This reverts commit f0008e4976.
2022-11-02 11:56:38 -07:00
Volodymyr Pochtar
b77d039e6b ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables (#18809)
* ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables

* ci: replace github set-ouput with new syntax in ./tools/bin/
2022-11-02 17:57:48 +02:00
Conor
1167d35ce7 ci: add job and run id to test reports (#18832) 2022-11-01 18:49:16 -07:00
Conor
dc15f56e31 ci: use custom test-reporter action to upload job results (#18004)
* ci: use custom action to upload job results
2022-11-01 13:59:19 -05:00
Ella Rohm-Ensing
385ab7e6d2 improve connector dependency github action (#18480)
* Use more specific variable name

* Write affected sources and destinations to files

* Use Markdown template to format results

* Update workflow to edit existing comment instead of making a new one

* Check whether changed_files.txt is empty before proceeding

* move template to .github/

* Use different quotes for true

* Move logic about continuing back to after dependency report because of ignores

* Use correct filename in conditional

* Use if to avoid early exit

* Create folder for comment templates and use constant for file path

* Add handling for non-source non-destination info, pull out markdown listing
2022-11-01 11:37:01 -04:00
Anne
0c183334bf add nonBreakingChangePreference and notifySchemaChanges (#18636)
* add nonBreakingChangePreference and notifySchemaChanges
2022-10-31 12:58:39 -07:00
Davin Chia
f0954597c3 Remove unused test script. (#18715) 2022-10-31 09:58:41 -07:00
Taras Korenko
a9a2bd4da0 Simplify the OSS documentation deploy system (#2670) (#18436)
+ Updates tools/bin/deploy_docusaurus to work on Github Runners
+ Adds 'Deploy docs.airbyte.com' Github workflow
2022-10-27 18:56:06 +03:00
Jonathan Pearlin
5d326a3cae Make script both Unix/Linux and OSX compatible (#18493) 2022-10-26 12:53:42 -04:00
terencecho
a714175997 Change port for acceptance test destination db (#18391) 2022-10-24 15:38:55 -07:00
Anne
5082ebfe2a Add SchemaChange to WebBackendConnectionRead object (#17969)
* add schemaChange field to WebBackendConnectionRead and breakingChange to ConnectionRead
2022-10-21 13:08:03 -07:00
Parker Mossman
0abc5f6373 only rm file if it exists (#18310) 2022-10-21 11:34:47 -07:00
pmossman
ec52a637b9 remove file that should be gitignored 2022-10-20 15:09:33 -07:00
Parker Mossman
c5336ce1db use basic auth username and password from .env (#18273) 2022-10-20 15:05:59 -07:00
Evan Tahler
e232ffa9fc Skip basic auth for Octavia integration tests (#18270) 2022-10-20 14:10:24 -07:00
Evan Tahler
ec304fd376 Disable basic auth for frontend acceptance tests (#18255)
* Disable auth for frontend acceptance tests

* Update auth flags
2022-10-20 10:41:48 -07:00
Evan Tahler
8fba3ac95b Remove airbyte-cli from build scrips (#18251) 2022-10-20 09:08:50 -07:00
Parker Mossman
7cedfa48de Load Testing Script (#18020)
* script skeleton

* add API call to source_definitions to fetch E2E Test Source definition ID

* createSource implementation

* add destination creation logic implementation

* get definition IDs, catalogId, and implement connection creation

* add cleanup script and write created ids to a file that can be cleaned up

* make cloud header a command-line argument, other cleanup

* script comments fix

* remove kube references and fix indentation

* temp commit - don't push

* remove discover catalog function

* more cleanups

* more cleanups

* cleanup help text

* exit codes and show how many connections left

* add README

Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
2022-10-19 17:56:12 -07:00
Xiaohan Song
6f88bdba39 query to find out jobs running unusally long (#17978)
* query to find out jobs running unusally long

* comments fix

* add more indents for better readability

* Update airbyte-metrics/reporter/src/main/java/io/airbyte/metrics/reporter/MetricRepository.java

Co-authored-by: Davin Chia <davinchia@gmail.com>

* formatting

Co-authored-by: Davin Chia <davinchia@gmail.com>
2022-10-19 16:21:05 -07:00
Evan Tahler
3d053e32ee Use Nginx + Basic Auth to secure OSS Airbyte (#17694)
* Use Nginx + Basic Auth to secure OSS Airbyte

* use local passwords

* Use gradle builds

* K8s setup and source values from ENV

* note about disabling

* add back defaults

* custom 401 page

* update http message

* update docs

* remove kube files

* additional doc updates

* Add a test suite

* fix failure exit codes

* doc updates

* Add docs

* bump to re-test

* add more sleep in tests for CI

* better sleep in test

* Update docs/operator-guides/security.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* PR updates

* test comment

* change test host on CI

* update tests and nginx to boot without backend

* proxy updates for docker DNS

* simpler test for uptime

* acceptance test skips PWs

* remove resolver madness

* fixup tests

* more proxy_pass revert

* update acceptance test exit codes

* relax test expectations

* add temporal mount back for testing

* Update docs/operator-guides/security.md

Co-authored-by: swyx <shawnthe1@gmail.com>

* Update airbyte-proxy/401.html

Co-authored-by: swyx <shawnthe1@gmail.com>

* more doc updates

* Octavia CLI uses Basic Auth  (#17982)

* [WIP] Octavia CLI uses Basic Auth

* readme

* augustin: add basic auth headers to clien

* augustin: add basic auth headers to client

* tests passing

* lint

* docs

* Move monkey patch to test

* coerce headers into strings

* monkey patch get_basic_auth_token

Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>

* fix launch permissions

* Keep worker port internal

* more readme

Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: swyx <shawnthe1@gmail.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
2022-10-19 15:52:01 -07:00
Pedro S. Lopez
db0227e051 publish: increase delay to delete temp arch tags (#18137) 2022-10-19 17:18:30 -04:00
Benoit Moriceau
260ab6bdea Move the port forward outside of the main docker-compose (#17864) 2022-10-17 12:59:28 -07:00
Evan Tahler
792fa7cead Remove unused airbyte-cli (#18009) 2022-10-17 10:28:53 -07:00
Topher Lubaway
804bf47484 Major overhaul of check images (#17979)
* Major overhaul of check images

actually fail on missed images
detect rate limit for dockerhub
sleep when hit rate limit and registry
more clear user messaging
Reduce output from 5000 lines to ~ 1000
More clear help messages for failures

* Addresses GH feedback

minor bugfixes
2022-10-14 12:58:27 -05:00
Edward Gao
808af16023 Fix Docker image existence check (normalization) (#17908)
* fix path to NormalizationRunnerFactory

* check for file existence also
2022-10-12 14:08:46 -07:00
Serhii Chvaliuk
6a8b053fb5 Use locally-installed qemu rather than docker-installed version (2) (#17889)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-10-12 17:46:24 +03:00
Evan Tahler
2d049f18cf Use locally-installed qemu rather than docker-installed version (#17848)
* Use locally-installed qemu rather than docker-installed version

* Bump faker for test

* Update docs/integrations/sources/faker.md

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>

* auto-bump connector version [ci skip]

Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-11 10:32:57 -07:00
Topher Lubaway
9c46e5a560 Fixes relative links above docs to be URIs (#17850)
also adds a missing exit 1 to deploy docs
when the deploy SHOULD fail
2022-10-11 11:36:33 -05:00
Volodymyr Pochtar
a142141c98 Revert "Revert "feat: replace openjdk with amazoncorretto:17.0.4 on connectors for seсurity compliance (#17511)" (#17820)" (#17833)
This reverts commit e47c94ecff.
2022-10-11 14:22:54 +03:00
Volodymyr Pochtar
e47c94ecff Revert "feat: replace openjdk with amazoncorretto:17.0.4 on connectors for seсurity compliance (#17511)" (#17820)
This reverts commit ab71f5bc29.
2022-10-11 11:58:01 +03:00
Volodymyr Pochtar
ab71f5bc29 feat: replace openjdk with amazoncorretto:17.0.4 on connectors for seсurity compliance (#17511) 2022-10-10 12:35:39 +03:00
Michael Siega
84f0bacaa2 support large schema discovery (#17394)
* support large schema discovery

* update generic source tests to handle new approach to schema discovery

* readability improvements related to schema discovery and large schema support

* update internal ScheduleHandler method name

* update source tests per new schema discovery interface
2022-10-05 20:33:43 +02:00
Jimmy Ma
d2a717594c Add some services start validation to acceptance_tests.sh (#17425) 2022-09-30 10:35:27 -07:00
Volodymyr Pochtar
a72cb7d1f8 feat: replace openjdk with amazoncorretto:17.0.4 on platform for security compliance (#16919) 2022-09-20 19:01:48 +03:00
Jimmy Ma
fa79bc4dfe Add missing references to cron in the acceptance tests (#16820)
* Add missing image
* More missed references to cron in the test scripts
2022-09-16 14:39:43 -07:00
Topher Lubaway
b5de9803ac Adds exit (#16702) 2022-09-14 09:07:02 -05:00
Topher Lubaway
7de9e25798 Fixes case where there is nothing to kill (#16661) 2022-09-13 14:01:23 -05:00
Lake Mossman
a15288ad0f Improve job list API with more fetching capabilities (#16415)
* start implementation of new persistence method

* add includingJobId and totalJobCount to job list request

* format

* update local openapi as well

* refactor queries into JOOQ and return empty list if target job cannot be found

* fix descriptions and undo changes from other branch

* switch including job to starting job

* fix job history handler tests

* rewrite jobs subqueries in jooq

* fix multiple config type querying

* remove unnecessary casts

* switch back to 'including' and return multiple of page size necessary to include job

* undo webapp changes

* fix test description

* format
2022-09-12 16:32:04 -07:00
Evan Tahler
6249b485cb clean-local.sh to remove local builds and venvs (#16597) 2022-09-12 14:06:20 -07:00
Serhii Chvaliuk
ba1154c677 Tools: script for scanning all non-secret spec fields (#16390)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-09-09 17:05:32 +03:00
Kyryl Skobylko
3b807c8e35 fix: Update oss charts. Add acceptance tests for helm (#16131)
* fix: revert extraEnv. Move updated extraEnv to env_vars

* fix: Add node pool selector feature for gke deployments

* fix: hotfix. Fix webapp values and imagepullsecrets

* fix: remove nodeSelector duplicate

* fix: revert imagePullSecrets changes

* fix nit

* fix: fix bootloader imagepullsecrets

* fix: fix image pull secrets again...

* feat: add acceptance tests for helm

* add step to install unzip

* fix incorrect command spelling(helm)

* update slack notification sending for Airbyte CI workflow

* Rename Helm E2E test report name

* fix: Update helm charts

* fix: fix temporal DB Port bug

* fix nits

* do a small testing for sake of science

* Update test chart.yaml with note block

* Revert changes for AC tests

* fix: Include #16166

* fix: add metrics to the ci workflow

* fix: revert changes for temporal service. move them to server service;

* fix: revert fullstory deletion

* fix: change way of defining image tag and make it to be set in global values. Add values.yaml.test for integration testing and local deployment

* fix: udpate docs, update values.yaml

* fix: fix helpers.tpl in all subcharts

* Update chart docs

Co-authored-by: Davin Chia <davinchia@gmail.com>
2022-09-09 15:58:58 +03:00
Octavia Squidington III
4b3e0f52bc fix: tools/code-generator/Dockerfile to reduce vulnerabilities (#16502)
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-ALPINE315-LIBTIRPC-2975614
- https://snyk.io/vuln/SNYK-ALPINE315-NCURSES-2952568
- https://snyk.io/vuln/SNYK-ALPINE315-NCURSES-2952568
- https://snyk.io/vuln/SNYK-ALPINE315-ZLIB-2434420
- https://snyk.io/vuln/SNYK-ALPINE315-ZLIB-2976173

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2022-09-08 15:40:24 -07:00
Anne
d856731029 Implement more Error Prone PMD rules (#15491)
* AvoidFieldNameMatchingTypeName rule
* AvoidInstanceofChecksInCatchClause
* compareObjectsWithEquals
* DoNotTerminateVM and ConstructorCallsOverridableMethod
* EmptyIfStmt and EmptyStatementNotInLoop
* ImplicitSwitchFallThrough, InvalidLogMessageFormat, and MoreThanOneLogger
2022-09-07 13:20:32 -07:00