1
0
mirror of synced 2026-01-13 18:02:36 -05:00
Commit Graph

588 Commits

Author SHA1 Message Date
Augustin
3f154590ae qa-engine: implement fetch_adoption_metrics_per_connector_version (#21840) 2023-01-26 08:44:20 +00:00
Augustin
734d5484c2 qa-engine: implement early enrichments and validations on QA report (#21776) 2023-01-25 17:38:52 +01:00
Mykyta Serbynevskiy
9e1504521c improve logging of find-valid-pat script (#21789) 2023-01-24 16:45:05 +02:00
Augustin
f7ce2041b7 qa-engine: fix typo (#21773)
* fix typo

* fix typo

* fix typo
2023-01-24 02:45:05 -06:00
Augustin
4770a8fa43 connector-ops-ci: bootstrap qa engine (#21709) 2023-01-24 09:35:06 +01:00
Augustin
d77514dcfe CI connector ops: introduce QA checks in /test (#21699) 2023-01-23 18:25:53 +01:00
Topher Lubaway
e67bb115ba Changes AC test check to check exited ps (#21672)
some docker compose changes no longer show exited
processes.  this broke out test

this change should fix master

tested in a runner that failed
2023-01-20 13:11:09 -06:00
Jimmy Ma
c6b8279386 Add dependencies to docker-compose (#19321)
* Revert "Revert "Add dependencies to docker-compose (#19257)" (#19306)"

This reverts commit b16f28fd4d.

* Bump docker-compose version

* test

* Revert "test"

This reverts commit c3b28c9ec3.

* Bump docker compose version to 3.8

* test

* test

* Use a newer docker for platform build

* test

* Use docker compose v2

* Update docker for frontend e2e

* Use docker compose v2 for e2e tests

* Update tools to use docker compose v2

* Update more docker compose references to v2

* Update docker compose usage in docs

* Update deploying airbyte docs

* Fix permission

* Case insensistive check

* Add extra mention for docker compose in upgrading airbyte

* Improve upgrade docker note
2023-01-12 10:41:00 -08:00
Volodymyr Pochtar
702027e10a feat: mention user on failed master branch build (#21201) 2023-01-12 12:30:26 +02:00
Augustin
524f0717f2 ci-connector-ops: change required reviewers logic (#21158) 2023-01-10 14:50:33 +00:00
Joe Reuter
f61a790a07 Connector builder: E2e tests (#21122)
* wip

* wip

* e2e tests for connector builder server

* rename function

* clean up

* clean up a bit more

* fix path

* fix and add documentation

* more documentation

* stabilze

* review comments
2023-01-10 12:44:09 +00:00
Greg Solovyev
80504d998e Switch order of namespaces and make Destination Default is the defaul… (#21047)
* Switch order of namespace options to make Destination Default appear as the first option
* Destination Default is the default option
* Update expected HTML
2023-01-05 10:20:26 -08:00
Kyryl Skobylko
2485576a97 fix: Disable fluentbit deployment for local ac tests (#21072) 2023-01-05 19:59:27 +02:00
Volodymyr Pochtar
45f5d76e85 Revert "build: added script that match github user to slack user/added action/changed workflow with user mentioning (#20815)" (#21018)
This reverts commit 31f96cef76.
2023-01-04 15:26:13 +00:00
Andrii Honcharenko
31f96cef76 build: added script that match github user to slack user/added action/changed workflow with user mentioning (#20815) 2023-01-03 22:41:18 +02:00
Davin Chia
2deac0f244 Prep for Java 19 Upgrade (#20911)
Upgrade all the necessary tooling dependencies to get us to Java 19.
2022-12-29 12:32:42 -08:00
Kyryl Skobylko
9da0de1079 fix: Improve Airbyte CI logging (#20517)
* add fluent-bit logging to k3s setup

* Fix --set for helm

* add --generate-name

* fix slash in multiline bash

* Update --set for helm

* Use generate-name instead

* Add debug

* Remove deubg for helm

* Fix env again

* Disable build

* Remove everything

* Disable everything again

* Snap back to reallity

* Update fluent_values.yaml

* Use role instead of AWS creds

* Use new bucket

* Revert changes

* Update bucket name

* Play around with fluent-bit config

* Escape string in helm --set arg

* Update fluent-bit config + update --set argument

* Update fluentbit config so it"ll create s3 path correctly

* Add logging to kustomize

* Revert debug mode
2022-12-27 19:53:24 +02:00
Evan Tahler
a55eb7df2d Fix CI Dependency Check Failures (#20666)
* pardot

* plaid

* quickbooks

* appfollow

* appstore

* cloudtrail

* clickup

* clockify

* coda

* Coinmarketcap

* cooper

* dixa

* dv-360

* exchange-rates

* file

* gridly

* Hellobaton

* kustomer

* mailersend

* microsoft dataverse

* n8n

* PersistIq

* Survey Sparrow

* Twilio Taskrouter

* YouTube Analytics Business

* Younium

* Yahoo Finance Price

* Yandex Metrica

* Xero

* WooCommerce

* XKCD

* Webflow

* US Census API

* Qonto

* Pivotal Tracker

* KVDB

* Firestore

* Ignore even more connectors

* test run

* SFTP JSON

* cleanup

* move pardot changelog

* update links

* remove testing HACK

* Update docs/integrations/sources/dixa.md

Co-authored-by: Augustin <augustin@airbyte.io>

* Update docs/integrations/sources/kustomer-singer.md

Co-authored-by: Augustin <augustin@airbyte.io>

* Update docs/integrations/sources/pardot.md

Co-authored-by: Augustin <augustin@airbyte.io>

* Update docs/integrations/sources/kustomer-singer.md

Co-authored-by: Augustin <augustin@airbyte.io>

* Update docs/integrations/sources/pardot.md

Co-authored-by: Augustin <augustin@airbyte.io>

Co-authored-by: Augustin <augustin@airbyte.io>
2022-12-20 16:57:25 -06:00
Taras Korenko
79bfe56a8f Fix OSS integration tests for docs (#3663) (#20578)
+ Routes Slack notifications to '#docs'
+ Sets default value of SKIP_DEPLOY variable in deploy_docusaurus
  script
2022-12-16 20:47:16 +02:00
Taras Korenko
228a524d63 Fix OSS integration tests for docs (#3663) (#20590)
+ In "Docs PR check mode" stop before "yarn deploy"
2022-12-16 19:13:00 +02:00
Topher Lubaway
4625e8bd3d removes skip_deploy from docs push (#20587) 2022-12-16 10:51:46 -06:00
Taras Korenko
2720326599 Fix OSS integration tests for docs (#3663) (#20563)
+ Changes 'deploy-docs-site' workflow and 'deploy_docusaurus'
  script so these two things are used for doc PR verification
  as well.
2022-12-16 17:34:33 +02:00
andriikorotkov
eecfafd10e Remove NormalizationRunnerFactory (#20245)
* updated IntegrationLauncherConfig.yaml and added to this class suportDBT and normalizationImage fields. Added to the GenerateInputActivityImpl and TemporalClient classes code parts for read destination_definition.yaml and get suportDBT and normalizationImage fields. Added logging and comparing normalization images from NormalizationRunnerFactory and destination_definition.yaml

* updated minor remarks

* updated minor remarks

* fixed minor remarks

* added normalization data to the tests

* fixed minor remarks

* removed NormalizationRunnerFactory

* fixed remarks

* fixed remarks

* fixed remarks

* updated acceptance tests

* updated acceptance tests

* updated check_images_exist.sh script

* updated method for get normalization image name for destination acceptance test

* fixed code style

* fixed code style and removed tests data

* updated JobErrorReporterTest.java

* updated JobErrorReporterTest.java

* fixed remarks

* added integration type field to the dectination_definition file and actor_definition table

* fixed tests

* fixed tests

* fixed minor changes after pulling master changes

* fixed minor changes after pulling master changes

* renamed integrationType to normalizationIntegrationType/ fixed minor remarks

* renamed extra dependencies

* updated docs

* updated docs

* fixed minor remarks

* added NormalizationDestinationDefinitionConfig.yaml for StandardDestinationDefinition.yaml and updated configuration

* updated normalization tag

* updated DestinationAcceptanceTest.java

* updated DestinationAcceptanceTest.java

* updated imports and descriptions
2022-12-15 12:15:34 +02:00
Augustin
c2895c1537 ci-connector-ops: declare review requirements on source connectors changes (#20303) 2022-12-13 23:55:50 +01:00
Michael Siega
0fac8c82e6 implement column filtering in the replication workflow (#20369)
* implement column filtering in the replication workflow

* fixes to column selection in replication workflow

* add a basic acceptance test for column selection

* make CI acceptance tests run with new field selection flag enabled

* fix format

* readability improvements around columns selection tests and other small fixes
2022-12-13 01:02:09 +01:00
Augustin
9eca59e6b4 ci-credentials: fix typo when no secret found (#20353) 2022-12-12 10:04:43 +01:00
Evan Tahler
c9483316a8 More destination fixes for ci_check_dependency (#20274) 2022-12-09 11:51:14 -08:00
Augustin
9cb5714cd1 ci-credentials: update GSM secrets with updated configuration values (#20076) 2022-12-07 11:53:22 +01:00
Evan Tahler
92ad0fdc07 Destination doc and warning updates (#20110)
* Doc updates

* Bigquery Denormalized

* bump faker for change

* ignore missing strict-encrypt connectors from ci check

* Apply suggestions from code review

Co-authored-by: Augustin <augustin@airbyte.io>

* Fix MD titles

Co-authored-by: Augustin <augustin@airbyte.io>
2022-12-06 12:10:13 -08:00
Ella Rohm-Ensing
4bef54d1b9 Add README info about using your local CDK locally and in docker (#20082)
* Add README info about installing the local CDK locally and in Docker

* Add '|| true' to validate_dockerignore()

* Update airbyte-cdk/python/README.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-12-05 17:52:35 -05:00
Jonathan Pearlin
201df24f4d Enable PMD rule to prevent == string comparisons (#20080) 2022-12-05 12:54:36 -05:00
Evan Tahler
ea58882fa3 Fix affected connectors (#19865)
* source-appsflyer

* source-cart

* source-chargebee

* source-lemlist

* source-linnworks

* source-mailgun

* source-pardot

* source-plaid

* source-prestashop

* source-quickbooks-singer

* source-recharge

* source-tplcentral

* remove icons that aren't real

* Remove TODOs

* add missing specs

* Update docs/integrations/sources/primetric.md

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* bump readmes

* revert ci_check_deps logic

* New name

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
2022-11-30 09:51:21 -08:00
Sofiia Zaitseva
d16f673c87 🪟 🚥 Add tests for verify sync mode options (#16549)
add tests for checking sync mode options
2022-11-30 12:03:23 +02:00
Anne
f07cef808d Disable broken connections when feature flag is on (#19730)
When auto-detect schema changes feature flag is on, disable connections that have breaking schema changes and connections that have any schema changes where the user has set their preference to disable.
2022-11-29 12:20:26 -08:00
Augustin
8d4886ad12 connector-ops ci check: handle missing definitions (#19873) 2022-11-29 14:51:51 +01:00
Anne
8287362247 Add schema change attrs to connection create (#19771)
* Add schema change attrs to connection create
2022-11-28 16:53:19 -08:00
Kyryl Skobylko
d663f5a82f fix: Fix Helm Acceptance Tests (#19483) 2022-11-18 19:11:32 +02:00
Ella Rohm-Ensing
c472340ad3 add base-normalization to dependency checker action (#19313)
* Add normalization to list of modules to check dependencies for

* Remove now unneeded empty lists

* Refactor how gradle files are retrieved

* Let method set the dict key for module

* Rename CONNECTOR_PATH, ensure that path for get_gradle_file_for_path is always a folder

* Add typing on potentially confusing method

* Make comment template clearer

* Move modules into details

* Auto-close sources and destinations, add emoji and count for other modules

* Rename function to get dependent modules
2022-11-18 08:55:28 -05:00
perangel
9d03921908 Retry docker pulls in CI (#19384)
* Retry docker pulls in CI

* Notify caller of retry

* Invoke pulls with `-i`

* Cleanup retrie default value
2022-11-16 13:44:21 -05:00
Augustin
1c961f41bf ci-connector-ops: Check test strictness level on GA source connectors (#19383) 2022-11-15 00:23:33 +01:00
Alexandre Girard
64b50adfdd Connector-builder server stub (#18410)
* init

* bad copy/paste

* move to top level

* Revert "move to top level"

This reverts commit aca3534d38.

* attempt to wire up connector builder frontend to server

* copy from octaviacli

* fix connection to builder server

* update

* delete

* Update

* delete python-version

* Revert "delete python-version"

This reverts commit f9258a7755.

* setup python

* install python

* rename

* kube stuff

* Install python

* missing kube file

* rename

* Update files

* Update bumpversion

* install python

* try with different entrypoint

* rename container

* point to docker-compose.yaml file

* derp

* copy acceptance_test.sh

* copy from acceptance tests

* delete cruft

* update

* remove application env

* reset

* reset to master

* update

* skip comprehensive incremental tests

* Revert "skip comprehensive incremental tests"

This reverts commit 9cee657596.

* reset to master

* remove cruft

* delete superfluous steps

* update port to 8003

* reset to master

* Update publish docker

* move openapi spec to airbyte-connector-builder

* point to openapi spec

* dont expose the connector builder to localhost

* reset FE components to master

* Don't deploy the connector-builder

* Revert "Don't deploy the connector-builder"

This reverts commit 3d157494cf.

* Revert "Revert "Don't deploy the connector-builder""

This reverts commit beac3d48f0.

* comment out more things related to connector builder server

* more attempts at removing the connector builder

* comment out more things

* Apply suggestions from code review

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update airbyte-webapp/src/config/configProviders.ts

Co-authored-by: Lake Mossman <lake@airbyte.io>

* update

* rename

* indent

* Revert "move openapi spec to airbyte-connector-builder"

This reverts commit 57dda04723.

* Revert "rename"

This reverts commit b2d802b8fa.

* Revert "Revert "rename""

This reverts commit 91db24fd4a.

* point to wrong file in case it fixes the build

* point to right openapi file

* Revert "Revert "move openapi spec to airbyte-connector-builder""

This reverts commit e46a837454.

* point to moved file

* fix path

* Update from master

* newline

* Add failing test

* Revert "Add failing test"

This reverts commit ed9fea09b5.

* comment

* update commented requires

* Add a comment

* 2022

* rename to connector-builder-server

* typo

Co-authored-by: lmossman <lake@airbyte.io>
2022-11-09 13:07:26 -08:00
Augustin
3feda06a14 ci_integration_test.sh: cut GITHUB_STEP_SUMMARY (#18895) 2022-11-03 14:54:32 +01:00
perangel
c5936b0841 [CI] Speed up check_images_exist (#18873)
* [CI] Speed up check_images_exist

* Refactor to use helper function and cleanup output

* Remove test for `compose_output` file
2022-11-02 22:02:09 -04:00
Anne
d26e5bc248 DiscoverSchema endpoints calculates diff and breaking change (#18571)
* update discover schema endpoint to calculate diff
2022-11-02 14:10:07 -07:00
Liren Tu
1dc6874c59 Show version and changelog status for affected connectors (#18845)
* Add placeholder change

* List connectors and versions in a table

* Add changelog status

* Add changelog annotation

* Add publish version status

* Add version bump for demo

* Update status

* Add strict-encrypt verion check

* Add overall status summary

* Fix version status format

* Fix format

* Update styling

* Add notes

* Add a special case for mongodb

* Update details open / close logic

* Add link to dockerfile

* Revert demo changes

* Remove version link

* Delete report when it is deprecated

* Revert "Delete report when it is deprecated"

This reverts commit f0008e4976.
2022-11-02 11:56:38 -07:00
Volodymyr Pochtar
b77d039e6b ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables (#18809)
* ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables

* ci: replace github set-ouput with new syntax in ./tools/bin/
2022-11-02 17:57:48 +02:00
Conor
1167d35ce7 ci: add job and run id to test reports (#18832) 2022-11-01 18:49:16 -07:00
Conor
dc15f56e31 ci: use custom test-reporter action to upload job results (#18004)
* ci: use custom action to upload job results
2022-11-01 13:59:19 -05:00
Ella Rohm-Ensing
385ab7e6d2 improve connector dependency github action (#18480)
* Use more specific variable name

* Write affected sources and destinations to files

* Use Markdown template to format results

* Update workflow to edit existing comment instead of making a new one

* Check whether changed_files.txt is empty before proceeding

* move template to .github/

* Use different quotes for true

* Move logic about continuing back to after dependency report because of ignores

* Use correct filename in conditional

* Use if to avoid early exit

* Create folder for comment templates and use constant for file path

* Add handling for non-source non-destination info, pull out markdown listing
2022-11-01 11:37:01 -04:00
Anne
0c183334bf add nonBreakingChangePreference and notifySchemaChanges (#18636)
* add nonBreakingChangePreference and notifySchemaChanges
2022-10-31 12:58:39 -07:00