jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-05 21:02:13 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,570 Commits 5,162 Branches 518 Tags
516a8a59c4eccbfdbfde27cb9bca8b6647ea5036
Commit Graph

767 Commits

Author SHA1 Message Date
Augustin
e742f8d5b5 connectors-ci: better connector test debugging experience (1/2) (#27550) 2023-06-21 17:28:04 +00:00
Augustin
22480ac642 ref: broaden the use of the run-dagger-pipeline action (#27506) 2023-06-21 06:47:33 -05:00
Augustin
2e3ee2deed qa-check: add pydash deps (#27544) 2023-06-21 02:28:50 -05:00
Augustin
473112548e connectors-ci: fix CAT not run for java connectors (#27474) 2023-06-20 23:54:40 +02:00
Ben Church
38a189e829 Breaking Changes: QA migration check (#27294) 
* Add migration guide check

* Add test todo

* migration_guide to -migrations

* Update tests

* Valid tests

* Add tests

* Update tools/ci_connector_ops/ci_connector_ops/utils.py

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* Ensure that the migration path is checked

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
 2023-06-20 09:30:23 -07:00
Augustin
e9d7c54adc connectors ci: auto format connectors code (#27353) 2023-06-19 17:44:43 +00:00
Ben Church
48d09730f6 Remove old qa-engine logic (#27439) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-16 12:05:18 -05:00
Ben Church
59dbc609e0 Handle the case of no metadata on master for new connectors (#27377) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-14 15:01:22 -05:00
Ben Church
68c0d6dbbc Connector CI: update badges test summary (#27215) 
* Rename to connector test report

* Get all test outputs

* Get last 10 output for all connectors

* Get runs by connector

* Crudely write badge and summary

* Add render functions

* Update connector report badge and link

* Fix url link

* Fix qa report tests

* update qa-engine tests

* Move regex to resource

* Add cron

* Run format

* New line

* Make badge green if last is successful

* Fix test

* Remove report.sh and build_report.py

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-14 11:12:47 -07:00
Augustin
5b8200181c connectors-ci: deprecate slash test (#27200) 2023-06-14 18:19:13 +02:00
Augustin
ef860d3d2a connectors-ci: minor cleaning (#27348) 
* remove debug instructions

* do not consider markdown file in dependency resolution
 2023-06-14 03:26:47 -05:00
Augustin
6e9be02172 connectors-ci: improve run duration computation (#27304) 2023-06-13 16:21:50 -07:00
Augustin
25ef3a2686 connectors-ci: disable format check on python connectors (#27301) 
* disable format check on python connectors

* Automated Commit - Formatting Changes

* disable format check on python connectors

---------

Co-authored-by: alafanechere <alafanechere@users.noreply.github.com>
 2023-06-13 06:17:05 -05:00
Augustin
9a040af212 connectors-ci: run dockerd in a background task (#27267) 
* connectors-ci: long running but ephemeral dockerd service

* Automated Commit - Formatting Changes

* to revert

* Revert "to revert"

This reverts commit 168ed30695.

---------

Co-authored-by: alafanechere <alafanechere@users.noreply.github.com>
 2023-06-13 06:14:38 -05:00
Ben Church
fb1b58d004 Add CI_JOB_KEY to override log paths (#27210) 
* Add CI_JOB_KEY

* Update to none

* ternary operator override

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
 2023-06-12 07:39:01 -05:00
Augustin
50c18d8825 connector_ci: more granular mgmt of concurrency (#27204) 2023-06-09 08:33:44 -05:00
Augustin
09ae1bafb0 connectors-ci: disable concurrency for Java connectors testing (#27012) 2023-06-09 08:39:23 +00:00
Ben Church
1315f5a768 Nightly Reports: Generate markdown slack report from new nightlies (#26990) 
* Add github asset

* reduce columns in asset

* Add sensor

* remove sensor calls

* Remove bucket manager

* Get nightly resources

* Add nightly report data

* TODO

* Generate usable data frame

* Add template

* Add all data to report

* Generalize gcs blob sensor

* Remove old metadata sensor

* Send report to slack

* Complete report

* Remove overly verbose comments

* Add generics

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-08 10:45:13 -07:00
Augustin
c2b7043f3f connectors-ci: make source-file testable in airbyte-ci (#27107) 2023-06-08 15:03:59 +00:00
Conor
4c9acc55a2 allow specifying override version in bump_version.sh (#27127) 
* allow specifying override version in bump_version.sh

* Automated Commit - Format and Process Resources Changes
 2023-06-07 16:04:34 -05:00
Augustin
85ca7ff8b0 airbyte-ci: missing missing module errors (#27104) 2023-06-07 02:11:59 -05:00
Ben Church
8bc8414212 Connector CI: Upload complete.json file on pipeline complete (#27051) 
* Write complete.json file at end of run

* Move state tranform

* Move file_path_key logic into report

* Use property decorator

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-06 22:26:17 +00:00
Augustin
ee1be355d4 connectors-ci: mask secrets in GHA logs with ::add-mask:: (#27087) 2023-06-06 21:19:46 +00:00
Ben Church
2df72cc63c Connector CI: Update nightlies to write to GCS (#26929) 
* Generate report output prefix

* DNC

* Upload to gcs bucket

* Revert dev comments

* Format

* Reenable for cron

* remove reports

* Connector CI reports

* Reremove AWS

* Missed some env vars

* Move bucket to CI

* Bad merge

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-06 16:37:19 +00:00
Augustin
e24e18837d connectors-ci: dockerd as a singleton service (#27021) 2023-06-06 07:55:57 +00:00
Augustin
43098ded79 connectors-ci: write pytest logs to host (#27042) 
* connectors-ci: write pytest logs to host

* do no pass log as output artifact as they are already available as stdout or stderr

* only write acceptance test logs when local
 2023-06-06 02:38:19 -05:00
Ben Church
723093b095 Metadata: Validate that metadata file exists on new connectors (#26933) 
* Ensure we test to see IF there is a metadata file

* Run format

* Add modifed files as path

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-05 17:52:07 -07:00
Edward Gao
ae43522659 Destination TiDB: install normalization (#27025) 
* install normalization

* changelog

* add to codeowners

* update changelog messages
 2023-06-06 00:05:50 +00:00
Edward Gao
bd22b25ccf Destination Mssql: install normalization (#27034) 
* install in mssql

* changelog+metadata

* update changelog message
 2023-06-06 00:00:39 +00:00
cpdeethree
d47fded347 bump to 0.44.11 to catch up to airbyte-platform and fix bumpversion script 2023-06-04 17:05:20 -05:00
Augustin
5aeb956d47 connectors-ci: add retry logic at step level (#26888) 2023-06-02 23:06:07 +00:00
Conor
6f2cdf4c26 Fix grep versionbump (#26979) 
* bump to 0.44.9 to catch up with airbyte-platform

* fix bug in grep that looks for new version
 2023-06-02 14:00:30 -05:00
Augustin
4aa76a48ad connectors-ci: do not run VersionCheck on nightlies (#26954) 2023-06-02 09:10:35 -05:00
Augustin
1166d41d18 connectors-ci: get modified files from PR API (#26889) 2023-06-02 00:46:37 +02:00
Edward Gao
7a75c0c0e6 Destination Snowflake: Enable in-connector normalization (#25782) 
* add support for snowflake

* do useful copy things

* corrections

* add build dependency

* deeeerp

* version bumps + changelog

* try installing gcc-c++

* move stuff around

* copy snowflake

* bump version

---------

Co-authored-by: Cynthia Yin <cynthia@airbyte.io>
 2023-06-01 14:14:28 -07:00
Augustin
a63041603e connectors-ci: dynamic dagger pipeline name (#26886) 
* connectors-ci: dynamic dagger pipeline name

* tmp

* add retry logic

* tmp

* Revert "tmp"

This reverts commit 976feeb584.

* Revert "tmp"

This reverts commit 5682a9a064.

* Revert "add retry logic"

This reverts commit 71e074991d.

* fix typo on GHA input
 2023-06-01 03:18:15 -05:00
Ben Church
4befe546c5 Connector CI: Ensure Crane has appropriate auth (#26812) 
* Add crane login

* Save env state

* Add comments

* Run format

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-05-31 09:31:46 -07:00
Augustin
801dea2129 connectors-ci: dynamic dependency resolution and test trigger (#26740) 2023-05-31 14:50:22 +00:00
Augustin
c094df4613 connectors-ci: publish strict-encrypt first (#26644) 2023-05-31 08:06:57 +00:00
Ben Church
5e0e317eac Move qa engine to aibyte-ci (#26377) 
* Add change notes

* Add qa-engine poetry

* Port qa_engine code

* Add missing fixture

* Update import

* Update git code

* Delete old qa-engine files

* convert tests

* Add path

* Fix mocks

* Fix test_enable_in_cloud

* Fix metadata path test

* Run formater

* Automated Commit - Formatting Changes

* Update readme

* Remove duplicates

* Remove redundant if

* Add ruamel

* Use ruamel

* Remove safe load

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
 2023-05-31 05:26:28 +00:00
Conor
b32357c684 update bump_version to use shell script (#26801) 2023-05-30 16:45:15 -05:00
Augustin
c5a58feeff fix: modified files detection (#26744) 2023-05-30 04:36:50 -05:00
Augustin
312a3615f3 connectors-ci: pass PR id to airbyte-ci (#26504) 2023-05-30 08:15:53 +00:00
Augustin
f2c5f1df25 connectors-ci: run metadata validation on test (#26646) 2023-05-30 09:44:05 +02:00
Augustin
2c9192e7a0 connectors-ci: improve global status reporting to GitHub for tests (#26575) 2023-05-29 19:53:59 +02:00
Ben Church
5a975c31ef Connectors CI: spec upload on pre-release publish (#26691) 
* upload spec on prepublish

* remove test comments

* Fix up unit test

* Fix test_publish

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-05-29 10:35:57 -04:00
Ben Church
1dabc6208e Metadata: add tags field (#26320) 
* Add optional tags field

* Remove duplicate icons

* Add programming tags to all

* Update docs

* supportUrl -> documentationUrl

* Ensure one language tag is applied

* Add keyvalue check

* rebase and fix tests

* Format

* Add cache buster

* Improve test

* Automated Commit - Formatting Changes

* Update error

* Fix missing tags

* Fix scaffold

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
 2023-05-26 16:13:09 -07:00
Augustin
75d4c09cd7 retrieve CDK version, set it in context and report + set image labels for CDK version and dagger version (#26572) 2023-05-26 08:12:43 +02:00
Edward Gao
bbdb3be868 Destination Redshift: Install normalization (#26165) 
* do things in-connector

* Automated Change

* version bumps + changelog

* remove unnecessary yum package

* derp

* also build.gradle

* Automated Change

---------

Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-05-25 23:40:48 +00:00
Augustin
42f3d4559f connectors-ci: re-upload spec if image exists + fix pre-release (#26604) 
* connectors-ci: fix pre-release

* fix edge case

* Automated Change

---------

Co-authored-by: alafanechere <alafanechere@users.noreply.github.com>
 2023-05-25 21:53:51 +02:00