1
0
mirror of synced 2026-01-14 12:07:57 -05:00
Commit Graph

461 Commits

Author SHA1 Message Date
Mykyta Serbynevskiy
9e1504521c improve logging of find-valid-pat script (#21789) 2023-01-24 16:45:05 +02:00
Augustin
d77514dcfe CI connector ops: introduce QA checks in /test (#21699) 2023-01-23 18:25:53 +01:00
Topher Lubaway
e67bb115ba Changes AC test check to check exited ps (#21672)
some docker compose changes no longer show exited
processes.  this broke out test

this change should fix master

tested in a runner that failed
2023-01-20 13:11:09 -06:00
Jimmy Ma
c6b8279386 Add dependencies to docker-compose (#19321)
* Revert "Revert "Add dependencies to docker-compose (#19257)" (#19306)"

This reverts commit b16f28fd4d.

* Bump docker-compose version

* test

* Revert "test"

This reverts commit c3b28c9ec3.

* Bump docker compose version to 3.8

* test

* test

* Use a newer docker for platform build

* test

* Use docker compose v2

* Update docker for frontend e2e

* Use docker compose v2 for e2e tests

* Update tools to use docker compose v2

* Update more docker compose references to v2

* Update docker compose usage in docs

* Update deploying airbyte docs

* Fix permission

* Case insensistive check

* Add extra mention for docker compose in upgrading airbyte

* Improve upgrade docker note
2023-01-12 10:41:00 -08:00
Volodymyr Pochtar
702027e10a feat: mention user on failed master branch build (#21201) 2023-01-12 12:30:26 +02:00
Joe Reuter
f61a790a07 Connector builder: E2e tests (#21122)
* wip

* wip

* e2e tests for connector builder server

* rename function

* clean up

* clean up a bit more

* fix path

* fix and add documentation

* more documentation

* stabilze

* review comments
2023-01-10 12:44:09 +00:00
Greg Solovyev
80504d998e Switch order of namespaces and make Destination Default is the defaul… (#21047)
* Switch order of namespace options to make Destination Default appear as the first option
* Destination Default is the default option
* Update expected HTML
2023-01-05 10:20:26 -08:00
Kyryl Skobylko
2485576a97 fix: Disable fluentbit deployment for local ac tests (#21072) 2023-01-05 19:59:27 +02:00
Volodymyr Pochtar
45f5d76e85 Revert "build: added script that match github user to slack user/added action/changed workflow with user mentioning (#20815)" (#21018)
This reverts commit 31f96cef76.
2023-01-04 15:26:13 +00:00
Andrii Honcharenko
31f96cef76 build: added script that match github user to slack user/added action/changed workflow with user mentioning (#20815) 2023-01-03 22:41:18 +02:00
Kyryl Skobylko
9da0de1079 fix: Improve Airbyte CI logging (#20517)
* add fluent-bit logging to k3s setup

* Fix --set for helm

* add --generate-name

* fix slash in multiline bash

* Update --set for helm

* Use generate-name instead

* Add debug

* Remove deubg for helm

* Fix env again

* Disable build

* Remove everything

* Disable everything again

* Snap back to reallity

* Update fluent_values.yaml

* Use role instead of AWS creds

* Use new bucket

* Revert changes

* Update bucket name

* Play around with fluent-bit config

* Escape string in helm --set arg

* Update fluent-bit config + update --set argument

* Update fluentbit config so it"ll create s3 path correctly

* Add logging to kustomize

* Revert debug mode
2022-12-27 19:53:24 +02:00
Evan Tahler
a55eb7df2d Fix CI Dependency Check Failures (#20666)
* pardot

* plaid

* quickbooks

* appfollow

* appstore

* cloudtrail

* clickup

* clockify

* coda

* Coinmarketcap

* cooper

* dixa

* dv-360

* exchange-rates

* file

* gridly

* Hellobaton

* kustomer

* mailersend

* microsoft dataverse

* n8n

* PersistIq

* Survey Sparrow

* Twilio Taskrouter

* YouTube Analytics Business

* Younium

* Yahoo Finance Price

* Yandex Metrica

* Xero

* WooCommerce

* XKCD

* Webflow

* US Census API

* Qonto

* Pivotal Tracker

* KVDB

* Firestore

* Ignore even more connectors

* test run

* SFTP JSON

* cleanup

* move pardot changelog

* update links

* remove testing HACK

* Update docs/integrations/sources/dixa.md

Co-authored-by: Augustin <augustin@airbyte.io>

* Update docs/integrations/sources/kustomer-singer.md

Co-authored-by: Augustin <augustin@airbyte.io>

* Update docs/integrations/sources/pardot.md

Co-authored-by: Augustin <augustin@airbyte.io>

* Update docs/integrations/sources/kustomer-singer.md

Co-authored-by: Augustin <augustin@airbyte.io>

* Update docs/integrations/sources/pardot.md

Co-authored-by: Augustin <augustin@airbyte.io>

Co-authored-by: Augustin <augustin@airbyte.io>
2022-12-20 16:57:25 -06:00
Taras Korenko
79bfe56a8f Fix OSS integration tests for docs (#3663) (#20578)
+ Routes Slack notifications to '#docs'
+ Sets default value of SKIP_DEPLOY variable in deploy_docusaurus
  script
2022-12-16 20:47:16 +02:00
Taras Korenko
228a524d63 Fix OSS integration tests for docs (#3663) (#20590)
+ In "Docs PR check mode" stop before "yarn deploy"
2022-12-16 19:13:00 +02:00
Topher Lubaway
4625e8bd3d removes skip_deploy from docs push (#20587) 2022-12-16 10:51:46 -06:00
Taras Korenko
2720326599 Fix OSS integration tests for docs (#3663) (#20563)
+ Changes 'deploy-docs-site' workflow and 'deploy_docusaurus'
  script so these two things are used for doc PR verification
  as well.
2022-12-16 17:34:33 +02:00
andriikorotkov
eecfafd10e Remove NormalizationRunnerFactory (#20245)
* updated IntegrationLauncherConfig.yaml and added to this class suportDBT and normalizationImage fields. Added to the GenerateInputActivityImpl and TemporalClient classes code parts for read destination_definition.yaml and get suportDBT and normalizationImage fields. Added logging and comparing normalization images from NormalizationRunnerFactory and destination_definition.yaml

* updated minor remarks

* updated minor remarks

* fixed minor remarks

* added normalization data to the tests

* fixed minor remarks

* removed NormalizationRunnerFactory

* fixed remarks

* fixed remarks

* fixed remarks

* updated acceptance tests

* updated acceptance tests

* updated check_images_exist.sh script

* updated method for get normalization image name for destination acceptance test

* fixed code style

* fixed code style and removed tests data

* updated JobErrorReporterTest.java

* updated JobErrorReporterTest.java

* fixed remarks

* added integration type field to the dectination_definition file and actor_definition table

* fixed tests

* fixed tests

* fixed minor changes after pulling master changes

* fixed minor changes after pulling master changes

* renamed integrationType to normalizationIntegrationType/ fixed minor remarks

* renamed extra dependencies

* updated docs

* updated docs

* fixed minor remarks

* added NormalizationDestinationDefinitionConfig.yaml for StandardDestinationDefinition.yaml and updated configuration

* updated normalization tag

* updated DestinationAcceptanceTest.java

* updated DestinationAcceptanceTest.java

* updated imports and descriptions
2022-12-15 12:15:34 +02:00
Michael Siega
0fac8c82e6 implement column filtering in the replication workflow (#20369)
* implement column filtering in the replication workflow

* fixes to column selection in replication workflow

* add a basic acceptance test for column selection

* make CI acceptance tests run with new field selection flag enabled

* fix format

* readability improvements around columns selection tests and other small fixes
2022-12-13 01:02:09 +01:00
Evan Tahler
c9483316a8 More destination fixes for ci_check_dependency (#20274) 2022-12-09 11:51:14 -08:00
Evan Tahler
92ad0fdc07 Destination doc and warning updates (#20110)
* Doc updates

* Bigquery Denormalized

* bump faker for change

* ignore missing strict-encrypt connectors from ci check

* Apply suggestions from code review

Co-authored-by: Augustin <augustin@airbyte.io>

* Fix MD titles

Co-authored-by: Augustin <augustin@airbyte.io>
2022-12-06 12:10:13 -08:00
Ella Rohm-Ensing
4bef54d1b9 Add README info about using your local CDK locally and in docker (#20082)
* Add README info about installing the local CDK locally and in Docker

* Add '|| true' to validate_dockerignore()

* Update airbyte-cdk/python/README.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-12-05 17:52:35 -05:00
Evan Tahler
ea58882fa3 Fix affected connectors (#19865)
* source-appsflyer

* source-cart

* source-chargebee

* source-lemlist

* source-linnworks

* source-mailgun

* source-pardot

* source-plaid

* source-prestashop

* source-quickbooks-singer

* source-recharge

* source-tplcentral

* remove icons that aren't real

* Remove TODOs

* add missing specs

* Update docs/integrations/sources/primetric.md

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* bump readmes

* revert ci_check_deps logic

* New name

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
2022-11-30 09:51:21 -08:00
Sofiia Zaitseva
d16f673c87 🪟 🚥 Add tests for verify sync mode options (#16549)
add tests for checking sync mode options
2022-11-30 12:03:23 +02:00
Kyryl Skobylko
d663f5a82f fix: Fix Helm Acceptance Tests (#19483) 2022-11-18 19:11:32 +02:00
Ella Rohm-Ensing
c472340ad3 add base-normalization to dependency checker action (#19313)
* Add normalization to list of modules to check dependencies for

* Remove now unneeded empty lists

* Refactor how gradle files are retrieved

* Let method set the dict key for module

* Rename CONNECTOR_PATH, ensure that path for get_gradle_file_for_path is always a folder

* Add typing on potentially confusing method

* Make comment template clearer

* Move modules into details

* Auto-close sources and destinations, add emoji and count for other modules

* Rename function to get dependent modules
2022-11-18 08:55:28 -05:00
perangel
9d03921908 Retry docker pulls in CI (#19384)
* Retry docker pulls in CI

* Notify caller of retry

* Invoke pulls with `-i`

* Cleanup retrie default value
2022-11-16 13:44:21 -05:00
Alexandre Girard
64b50adfdd Connector-builder server stub (#18410)
* init

* bad copy/paste

* move to top level

* Revert "move to top level"

This reverts commit aca3534d38.

* attempt to wire up connector builder frontend to server

* copy from octaviacli

* fix connection to builder server

* update

* delete

* Update

* delete python-version

* Revert "delete python-version"

This reverts commit f9258a7755.

* setup python

* install python

* rename

* kube stuff

* Install python

* missing kube file

* rename

* Update files

* Update bumpversion

* install python

* try with different entrypoint

* rename container

* point to docker-compose.yaml file

* derp

* copy acceptance_test.sh

* copy from acceptance tests

* delete cruft

* update

* remove application env

* reset

* reset to master

* update

* skip comprehensive incremental tests

* Revert "skip comprehensive incremental tests"

This reverts commit 9cee657596.

* reset to master

* remove cruft

* delete superfluous steps

* update port to 8003

* reset to master

* Update publish docker

* move openapi spec to airbyte-connector-builder

* point to openapi spec

* dont expose the connector builder to localhost

* reset FE components to master

* Don't deploy the connector-builder

* Revert "Don't deploy the connector-builder"

This reverts commit 3d157494cf.

* Revert "Revert "Don't deploy the connector-builder""

This reverts commit beac3d48f0.

* comment out more things related to connector builder server

* more attempts at removing the connector builder

* comment out more things

* Apply suggestions from code review

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update airbyte-webapp/src/config/configProviders.ts

Co-authored-by: Lake Mossman <lake@airbyte.io>

* update

* rename

* indent

* Revert "move openapi spec to airbyte-connector-builder"

This reverts commit 57dda04723.

* Revert "rename"

This reverts commit b2d802b8fa.

* Revert "Revert "rename""

This reverts commit 91db24fd4a.

* point to wrong file in case it fixes the build

* point to right openapi file

* Revert "Revert "move openapi spec to airbyte-connector-builder""

This reverts commit e46a837454.

* point to moved file

* fix path

* Update from master

* newline

* Add failing test

* Revert "Add failing test"

This reverts commit ed9fea09b5.

* comment

* update commented requires

* Add a comment

* 2022

* rename to connector-builder-server

* typo

Co-authored-by: lmossman <lake@airbyte.io>
2022-11-09 13:07:26 -08:00
Augustin
3feda06a14 ci_integration_test.sh: cut GITHUB_STEP_SUMMARY (#18895) 2022-11-03 14:54:32 +01:00
perangel
c5936b0841 [CI] Speed up check_images_exist (#18873)
* [CI] Speed up check_images_exist

* Refactor to use helper function and cleanup output

* Remove test for `compose_output` file
2022-11-02 22:02:09 -04:00
Liren Tu
1dc6874c59 Show version and changelog status for affected connectors (#18845)
* Add placeholder change

* List connectors and versions in a table

* Add changelog status

* Add changelog annotation

* Add publish version status

* Add version bump for demo

* Update status

* Add strict-encrypt verion check

* Add overall status summary

* Fix version status format

* Fix format

* Update styling

* Add notes

* Add a special case for mongodb

* Update details open / close logic

* Add link to dockerfile

* Revert demo changes

* Remove version link

* Delete report when it is deprecated

* Revert "Delete report when it is deprecated"

This reverts commit f0008e4976.
2022-11-02 11:56:38 -07:00
Volodymyr Pochtar
b77d039e6b ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables (#18809)
* ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables

* ci: replace github set-ouput with new syntax in ./tools/bin/
2022-11-02 17:57:48 +02:00
Conor
1167d35ce7 ci: add job and run id to test reports (#18832) 2022-11-01 18:49:16 -07:00
Conor
dc15f56e31 ci: use custom test-reporter action to upload job results (#18004)
* ci: use custom action to upload job results
2022-11-01 13:59:19 -05:00
Ella Rohm-Ensing
385ab7e6d2 improve connector dependency github action (#18480)
* Use more specific variable name

* Write affected sources and destinations to files

* Use Markdown template to format results

* Update workflow to edit existing comment instead of making a new one

* Check whether changed_files.txt is empty before proceeding

* move template to .github/

* Use different quotes for true

* Move logic about continuing back to after dependency report because of ignores

* Use correct filename in conditional

* Use if to avoid early exit

* Create folder for comment templates and use constant for file path

* Add handling for non-source non-destination info, pull out markdown listing
2022-11-01 11:37:01 -04:00
Davin Chia
f0954597c3 Remove unused test script. (#18715) 2022-10-31 09:58:41 -07:00
Taras Korenko
a9a2bd4da0 Simplify the OSS documentation deploy system (#2670) (#18436)
+ Updates tools/bin/deploy_docusaurus to work on Github Runners
+ Adds 'Deploy docs.airbyte.com' Github workflow
2022-10-27 18:56:06 +03:00
Jonathan Pearlin
5d326a3cae Make script both Unix/Linux and OSX compatible (#18493) 2022-10-26 12:53:42 -04:00
terencecho
a714175997 Change port for acceptance test destination db (#18391) 2022-10-24 15:38:55 -07:00
Parker Mossman
0abc5f6373 only rm file if it exists (#18310) 2022-10-21 11:34:47 -07:00
pmossman
ec52a637b9 remove file that should be gitignored 2022-10-20 15:09:33 -07:00
Parker Mossman
c5336ce1db use basic auth username and password from .env (#18273) 2022-10-20 15:05:59 -07:00
Evan Tahler
e232ffa9fc Skip basic auth for Octavia integration tests (#18270) 2022-10-20 14:10:24 -07:00
Evan Tahler
ec304fd376 Disable basic auth for frontend acceptance tests (#18255)
* Disable auth for frontend acceptance tests

* Update auth flags
2022-10-20 10:41:48 -07:00
Evan Tahler
8fba3ac95b Remove airbyte-cli from build scrips (#18251) 2022-10-20 09:08:50 -07:00
Parker Mossman
7cedfa48de Load Testing Script (#18020)
* script skeleton

* add API call to source_definitions to fetch E2E Test Source definition ID

* createSource implementation

* add destination creation logic implementation

* get definition IDs, catalogId, and implement connection creation

* add cleanup script and write created ids to a file that can be cleaned up

* make cloud header a command-line argument, other cleanup

* script comments fix

* remove kube references and fix indentation

* temp commit - don't push

* remove discover catalog function

* more cleanups

* more cleanups

* cleanup help text

* exit codes and show how many connections left

* add README

Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
2022-10-19 17:56:12 -07:00
Xiaohan Song
6f88bdba39 query to find out jobs running unusally long (#17978)
* query to find out jobs running unusally long

* comments fix

* add more indents for better readability

* Update airbyte-metrics/reporter/src/main/java/io/airbyte/metrics/reporter/MetricRepository.java

Co-authored-by: Davin Chia <davinchia@gmail.com>

* formatting

Co-authored-by: Davin Chia <davinchia@gmail.com>
2022-10-19 16:21:05 -07:00
Evan Tahler
3d053e32ee Use Nginx + Basic Auth to secure OSS Airbyte (#17694)
* Use Nginx + Basic Auth to secure OSS Airbyte

* use local passwords

* Use gradle builds

* K8s setup and source values from ENV

* note about disabling

* add back defaults

* custom 401 page

* update http message

* update docs

* remove kube files

* additional doc updates

* Add a test suite

* fix failure exit codes

* doc updates

* Add docs

* bump to re-test

* add more sleep in tests for CI

* better sleep in test

* Update docs/operator-guides/security.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* PR updates

* test comment

* change test host on CI

* update tests and nginx to boot without backend

* proxy updates for docker DNS

* simpler test for uptime

* acceptance test skips PWs

* remove resolver madness

* fixup tests

* more proxy_pass revert

* update acceptance test exit codes

* relax test expectations

* add temporal mount back for testing

* Update docs/operator-guides/security.md

Co-authored-by: swyx <shawnthe1@gmail.com>

* Update airbyte-proxy/401.html

Co-authored-by: swyx <shawnthe1@gmail.com>

* more doc updates

* Octavia CLI uses Basic Auth  (#17982)

* [WIP] Octavia CLI uses Basic Auth

* readme

* augustin: add basic auth headers to clien

* augustin: add basic auth headers to client

* tests passing

* lint

* docs

* Move monkey patch to test

* coerce headers into strings

* monkey patch get_basic_auth_token

Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>

* fix launch permissions

* Keep worker port internal

* more readme

Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: swyx <shawnthe1@gmail.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
2022-10-19 15:52:01 -07:00
Benoit Moriceau
260ab6bdea Move the port forward outside of the main docker-compose (#17864) 2022-10-17 12:59:28 -07:00
Evan Tahler
792fa7cead Remove unused airbyte-cli (#18009) 2022-10-17 10:28:53 -07:00
Topher Lubaway
804bf47484 Major overhaul of check images (#17979)
* Major overhaul of check images

actually fail on missed images
detect rate limit for dockerhub
sleep when hit rate limit and registry
more clear user messaging
Reduce output from 5000 lines to ~ 1000
More clear help messages for failures

* Addresses GH feedback

minor bugfixes
2022-10-14 12:58:27 -05:00