1
0
mirror of synced 2025-12-30 03:02:21 -05:00
Commit Graph

985 Commits

Author SHA1 Message Date
Ben Church
fb7258e2bd Move tools/ci_* projects to airbyte-ci, update to use Poetry, bump to python 3.10 (#27957)
* Move ci_connector_ops

* Move ci_credentials

* Move tools/ci_common_utils

* Rename tools to airbyte-ci

* Move to ci

* Convert ci_credentials

* Convert ci_common_utls

* Convert ci_connector_ops

* Get pipelines running

* Move pipelines to own poetry project

* Update readme

* Delete

* Add ci_code_validator

* Use pipx to install gha deps

* Fix'

* Ensure every thing is running

* Automated Commit - Formatting Changes

* Gitignore miss

* Add pipx installer

* Get local pipx dependencies

* Fix paths

* Install pipx

* ceremonial source-faker change

* Add installation step for ci_code_validator

* Add comment

* remove ci_code_validator

* Address code review comments

* add pipx install to acceptance-test-docker.sh

* Run formater

* Revert "ceremonial source-faker change"

This reverts commit 26884cd0db.

* gitignore lecacy pipeline report path

* update poetry.lock

* skip upload if logs do not exist

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
2023-07-26 15:49:59 +00:00
Marcos Marx
bdaddd4c20 Docs: update PR template and remove python destination as accepted contributions (#28665)
* update PR template and remove python destination as accepted contributions

* fix warning callout

* improve speeling
2023-07-26 10:09:53 -03:00
Sherif A. Nada
7f4a90a5a1 specify extensibility as Python CDK CODEOWNER not all CDK 2023-07-25 19:06:39 -07:00
Alexandre Girard
221cbdc093 Update the cloud dockerfile too (#28640) 2023-07-25 15:25:09 -07:00
Conor
71211e8591 add a stub to test a java cdk workflow (#28703) 2023-07-25 17:18:27 -05:00
Augustin
9915bcd17a connectors-ci: extend nightly build timeout (#28669) 2023-07-25 09:35:25 -05:00
Maxime Carbonneau-Leclerc
8d206e3e05 Update automatic PR following atelier removal (#28522) 2023-07-20 12:35:18 -04:00
Ben Church
baf510b619 Remove python and lowcode specifier (#28376) 2023-07-19 18:57:53 -05:00
Augustin
65227f614c connectors-ci: upgrade to Dagger 0.6.4 (#28008)
* connectors-ci: upgrade to Dagger 0.6.3

* redeclare _EXPERIMENTAL_DAGGER_RUNNER_HOST

* skip-ci: bump dagger version to 0.6.4

* remove unused dagger-cli-commit input
2023-07-19 18:24:36 -04:00
Augustin
a9bfe9b452 connectors-ci: pin Dagger CLI to new version (#28370) 2023-07-17 23:25:07 +02:00
Pedro S. Lopez
7befa93003 Stop publishing java connectors on merge to master (#28344)
* process language after modified flag

* modify some connectors

* empty commit

* another bump

* modified connectors are already filtered

* change some connectors

* change some connectors again

* revert to master version

* nit

* filter for python/lowcode connectors
2023-07-14 21:53:02 -04:00
Alexandre Girard
97a353d5c5 Run mypy on airbyte-cdk as part of the build pipeline and fix typing issues in the file-based module (#27790)
* Try running only on modified files

* make a change

* return something with the wrong type

* Revert "return something with the wrong type"

This reverts commit 23b828371e.

* fix typing in file-based

* format

* Mypy

* fix

* leave as Mapping

* Revert "leave as Mapping"

This reverts commit 908f063f70.

* Use Dict

* update

* move dict()

* Revert "move dict()"

This reverts commit fa347a8236.

* Revert "Revert "move dict()""

This reverts commit c9237df2e4.

* Revert "Revert "Revert "move dict()"""

This reverts commit 5ac1616414.

* use Mapping

* point to config file

* comment

* strict = False

* remove --

* Revert "comment"

This reverts commit 6000814a82.

* install types

* install types in same command as mypy runs

* non-interactive

* freeze version

* pydantic plugin

* plugins

* update

* ignore missing import

* Revert "ignore missing import"

This reverts commit 1da7930fb7.

* Install pydantic instead

* fix

* this passes locally

* strict = true

* format

* explicitly import models

* Update

* remove old mypy.ini config

* temporarily disable mypy

* format

* any

* format

* fix tests

* format

* Automated Commit - Formatting Changes

* Revert "temporarily disable mypy"

This reverts commit eb8470fa3f.

* implicit reexport

* update test

* fix mypy

* Automated Commit - Formatting Changes

* fix some errors in tests

* more type fixes

* more fixes

* more

* .

* done with tests

* fix last files

* format

* Update gradle

* change source-stripe

* only run mypy on cdk

* remove strict

* Add more rules

* update

* ignore missing imports

* cast to string

* Allow untyped decorator

* reset to master

* move to the cdk

* derp

* move explicit imports around

* Automated Commit - Formatting Changes

* Revert "move explicit imports around"

This reverts commit 56e306b72f.

* move explicit imports around

* Upgrade mypy version

* point to config file

* Update readme

* Ignore errors in the models module

* Automated Commit - Formatting Changes

* move check to gradle build

* Any

* try checking out master too

* Revert "try checking out master too"

This reverts commit 8a8f3e373c.

* fetch master

* install mypy

* try without origin

* fetch from the script

* checkout master

* ls the branches

* remotes/origin/master

* remove some cruft

* comment

* remove pydantic types

* unpin mypy

* fetch from the script

* Update connectors base too

* modify a non-cdk file to confirm it doesn't get checked by mypy

* run mypy after generateComponentManifestClassFiles

* run from the venv

* pass files as arguments

* update

* fix when running without args

* with subdir

* path

* try without /

* ./

* remove filter

* try resetting

* Revert "try resetting"

This reverts commit 3a54c424de.

* exclude autogen file

* do not use the github action

* works locally

* remove extra fetch

* run on connectors base

* try bad  typing

* Revert "try bad  typing"

This reverts commit 33b512a3e4.

* reset stripe

* Revert "reset stripe"

This reverts commit 28f23fc6dd.

* Revert "Revert "reset stripe""

This reverts commit 5bf5dee371.

* missing return type

* do not ignore the autogen file

* remove extra installs

* run from venv

* Only check files modified on current branch

* Revert "Only check files modified on current branch"

This reverts commit b4b728e654.

* use merge-base

* Revert "use merge-base"

This reverts commit 3136670cbf.

* try with updated mypy

* bump

* run other steps after mypy

* reset task ordering

* run mypy though

* looser config

* tests pass

* fix mypy issues

* type: ignore

* optional

* this is always a bool

* ignore

* fix typing issues

* remove ignore

* remove mapping

* Automated Commit - Formatting Changes

* Revert "remove ignore"

This reverts commit 9ffeeb6cb1.

* update config

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
2023-07-13 16:55:48 -07:00
Augustin
086ddb750b connectors-ci: revert to Dagger 0.5.4 (#27976) 2023-07-05 20:40:22 +02:00
Augustin
efe21d1659 connectors-ci: better capture stderr (#27833) 2023-07-04 11:13:12 +02:00
Sherif A. Nada
c33ea9926c make publish_option required in publish connectors workflow (#27879) 2023-06-30 13:06:02 -07:00
Conor
f2d5b7449a Revert "connectors-ci: switch back to large-runner" (#27767) 2023-06-28 17:29:45 +02:00
Augustin
58dd54f97f change default bucket (#27749) 2023-06-27 03:27:52 -05:00
Augustin
8a03adcd98 switch back to large-runner (#27748) 2023-06-27 02:47:12 -05:00
Ben Church
543897c2ad Remove Affected connector report (#27622)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-26 09:54:58 -07:00
Augustin
2f02d7fb96 weekly build: fix typo on workflow name (#27707) 2023-06-26 07:56:50 -05:00
Augustin
d3c33a6321 connector-ci: better and lighter logging (#27606) 2023-06-24 09:17:46 +00:00
Ella Rohm-Ensing
12c7da7f11 fix docker auth for metadata tests and orchestrator deploy (#27655)
* test change in metadata library to trigger tests

* add credentials when calling dagger from metadata_service_tests_dagger.yml

* Fix env for metadata_service_deploy_orchestrator_dagger.yml

* revert adding metadata api token input

* reorder inputs by how they are listed in workflow

* add context for orchestrator deploy
2023-06-23 14:57:55 +00:00
Augustin
4638209e0d connectors-ci: update runners to new node pools (#27615) 2023-06-22 19:54:42 +02:00
Ben Church
3bd340a971 Update to use rev-parse (#27598)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-21 20:42:56 -05:00
Ryan Fu
4d76ac5809 Turn off java formatter (#27579) 2023-06-21 21:13:20 +00:00
Augustin
3050c24ab4 fix typo in env var name (#27553) 2023-06-21 10:54:20 -05:00
Augustin
22480ac642 ref: broaden the use of the run-dagger-pipeline action (#27506) 2023-06-21 06:47:33 -05:00
Ben Church
943150a19b Add any glob to filter to ensure all steps are run in CI (#27525)
* Ensure all steps are run

* Update comment

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-20 15:14:28 -07:00
Augustin
e9d7c54adc connectors ci: auto format connectors code (#27353) 2023-06-19 17:44:43 +00:00
Augustin
12c1602883 fix: retry action latest version is not stable (#27465) 2023-06-19 05:37:11 -05:00
Augustin
55423c2495 connectors-ci: do not run tests on fork PR (#27402) 2023-06-15 18:14:47 +00:00
Augustin
d5dc309560 gradle.yml: remove Cache Build Artifacts (#27409) 2023-06-15 19:55:00 +02:00
Augustin
ecc310e69d fix trigger on format workflow (#27391) 2023-06-15 03:56:27 -05:00
Augustin
02a592dd4a create new workflow to format connectors (#27389) 2023-06-15 03:41:22 -05:00
Ben Church
68c0d6dbbc Connector CI: update badges test summary (#27215)
* Rename to connector test report

* Get all test outputs

* Get last 10 output for all connectors

* Get runs by connector

* Crudely write badge and summary

* Add render functions

* Update connector report badge and link

* Fix url link

* Fix qa report tests

* update qa-engine tests

* Move regex to resource

* Add cron

* Run format

* New line

* Make badge green if last is successful

* Fix test

* Remove report.sh and build_report.py

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-14 11:12:47 -07:00
Augustin
5b8200181c connectors-ci: deprecate slash test (#27200) 2023-06-14 18:19:13 +02:00
Augustin
ad1a992097 connectors-ci: move connectors_weekly_build.yml to workflows dir (#27330) 2023-06-13 14:41:37 -05:00
Augustin
6879cdf5d6 weekly build: add missing env var (#27320) 2023-06-13 11:22:45 -05:00
Augustin
09ec1efcc4 connectors-ci: weekly build for alpha connectors (#27300) 2023-06-13 18:17:14 +02:00
Evan Tahler
521a798fb5 Experiments to fix CI (#27262)
* don't pin virtualenv version

* comments

* relax python version

* keep virtualenv relax

* remove cache

* add note

* Update .github/workflows/gradle.yml

Co-authored-by: Ben Church <ben@airbyte.io>

---------

Co-authored-by: Ben Church <ben@airbyte.io>
2023-06-12 17:59:57 -04:00
Tim Roes
cdfa75e781 Remove old growth related labeling code (#27248) 2023-06-12 20:18:53 +02:00
Augustin
09ae1bafb0 connectors-ci: disable concurrency for Java connectors testing (#27012) 2023-06-09 08:39:23 +00:00
Ben Church
2df72cc63c Connector CI: Update nightlies to write to GCS (#26929)
* Generate report output prefix

* DNC

* Upload to gcs bucket

* Revert dev comments

* Format

* Reenable for cron

* remove reports

* Connector CI reports

* Reremove AWS

* Missed some env vars

* Move bucket to CI

* Bad merge

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-06 16:37:19 +00:00
Augustin
1ab95effbb connectors-ci: re-enable concurrency on java connectors (#27052) 2023-06-06 10:06:24 +02:00
Ben Church
723093b095 Metadata: Validate that metadata file exists on new connectors (#26933)
* Ensure we test to see IF there is a metadata file

* Run format

* Add modifed files as path

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-05 17:52:07 -07:00
Edward Gao
ae43522659 Destination TiDB: install normalization (#27025)
* install normalization

* changelog

* add to codeowners

* update changelog messages
2023-06-06 00:05:50 +00:00
Augustin
80584edea1 fix typo on workflow (#27005) 2023-06-05 01:54:31 -05:00
Augustin
4e88af04bb connectors-ci: no concurrency on java connectors for nightly builds (#27003) 2023-06-05 01:48:47 -05:00
Augustin
af890515ee from 6 to 10hours timeout (#26999) 2023-06-04 15:30:19 -05:00
Conor
6324b28ba3 Use repository tags to clean up zombine instances instead of size (#26919) 2023-06-02 10:01:53 -05:00