jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 02:51:29 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

851 Commits

Author SHA1 Message Date
Augustin
09ae1bafb0 connectors-ci: disable concurrency for Java connectors testing (#27012) 2023-06-09 08:39:23 +00:00
Ben Church
1315f5a768 Nightly Reports: Generate markdown slack report from new nightlies (#26990) 
* Add github asset

* reduce columns in asset

* Add sensor

* remove sensor calls

* Remove bucket manager

* Get nightly resources

* Add nightly report data

* TODO

* Generate usable data frame

* Add template

* Add all data to report

* Generalize gcs blob sensor

* Remove old metadata sensor

* Send report to slack

* Complete report

* Remove overly verbose comments

* Add generics

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-08 10:45:13 -07:00
Augustin
c2b7043f3f connectors-ci: make source-file testable in airbyte-ci (#27107) 2023-06-08 15:03:59 +00:00
Conor
4c9acc55a2 allow specifying override version in bump_version.sh (#27127) 
* allow specifying override version in bump_version.sh

* Automated Commit - Format and Process Resources Changes
 2023-06-07 16:04:34 -05:00
Augustin
85ca7ff8b0 airbyte-ci: missing missing module errors (#27104) 2023-06-07 02:11:59 -05:00
Ben Church
8bc8414212 Connector CI: Upload complete.json file on pipeline complete (#27051) 
* Write complete.json file at end of run

* Move state tranform

* Move file_path_key logic into report

* Use property decorator

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-06 22:26:17 +00:00
Augustin
ee1be355d4 connectors-ci: mask secrets in GHA logs with ::add-mask:: (#27087) 2023-06-06 21:19:46 +00:00
Ben Church
2df72cc63c Connector CI: Update nightlies to write to GCS (#26929) 
* Generate report output prefix

* DNC

* Upload to gcs bucket

* Revert dev comments

* Format

* Reenable for cron

* remove reports

* Connector CI reports

* Reremove AWS

* Missed some env vars

* Move bucket to CI

* Bad merge

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-06 16:37:19 +00:00
Augustin
e24e18837d connectors-ci: dockerd as a singleton service (#27021) 2023-06-06 07:55:57 +00:00
Augustin
43098ded79 connectors-ci: write pytest logs to host (#27042) 
* connectors-ci: write pytest logs to host

* do no pass log as output artifact as they are already available as stdout or stderr

* only write acceptance test logs when local
 2023-06-06 02:38:19 -05:00
Ben Church
723093b095 Metadata: Validate that metadata file exists on new connectors (#26933) 
* Ensure we test to see IF there is a metadata file

* Run format

* Add modifed files as path

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-06-05 17:52:07 -07:00
Edward Gao
ae43522659 Destination TiDB: install normalization (#27025) 
* install normalization

* changelog

* add to codeowners

* update changelog messages
 2023-06-06 00:05:50 +00:00
Edward Gao
bd22b25ccf Destination Mssql: install normalization (#27034) 
* install in mssql

* changelog+metadata

* update changelog message
 2023-06-06 00:00:39 +00:00
cpdeethree
d47fded347 bump to 0.44.11 to catch up to airbyte-platform and fix bumpversion script 2023-06-04 17:05:20 -05:00
Augustin
5aeb956d47 connectors-ci: add retry logic at step level (#26888) 2023-06-02 23:06:07 +00:00
Conor
6f2cdf4c26 Fix grep versionbump (#26979) 
* bump to 0.44.9 to catch up with airbyte-platform

* fix bug in grep that looks for new version
 2023-06-02 14:00:30 -05:00
Augustin
4aa76a48ad connectors-ci: do not run VersionCheck on nightlies (#26954) 2023-06-02 09:10:35 -05:00
Augustin
1166d41d18 connectors-ci: get modified files from PR API (#26889) 2023-06-02 00:46:37 +02:00
Edward Gao
7a75c0c0e6 Destination Snowflake: Enable in-connector normalization (#25782) 
* add support for snowflake

* do useful copy things

* corrections

* add build dependency

* deeeerp

* version bumps + changelog

* try installing gcc-c++

* move stuff around

* copy snowflake

* bump version

---------

Co-authored-by: Cynthia Yin <cynthia@airbyte.io>
 2023-06-01 14:14:28 -07:00
Augustin
a63041603e connectors-ci: dynamic dagger pipeline name (#26886) 
* connectors-ci: dynamic dagger pipeline name

* tmp

* add retry logic

* tmp

* Revert "tmp"

This reverts commit 976feeb584.

* Revert "tmp"

This reverts commit 5682a9a064.

* Revert "add retry logic"

This reverts commit 71e074991d.

* fix typo on GHA input
 2023-06-01 03:18:15 -05:00
Ben Church
4befe546c5 Connector CI: Ensure Crane has appropriate auth (#26812) 
* Add crane login

* Save env state

* Add comments

* Run format

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-05-31 09:31:46 -07:00
Augustin
801dea2129 connectors-ci: dynamic dependency resolution and test trigger (#26740) 2023-05-31 14:50:22 +00:00
Augustin
c094df4613 connectors-ci: publish strict-encrypt first (#26644) 2023-05-31 08:06:57 +00:00
Ben Church
5e0e317eac Move qa engine to aibyte-ci (#26377) 
* Add change notes

* Add qa-engine poetry

* Port qa_engine code

* Add missing fixture

* Update import

* Update git code

* Delete old qa-engine files

* convert tests

* Add path

* Fix mocks

* Fix test_enable_in_cloud

* Fix metadata path test

* Run formater

* Automated Commit - Formatting Changes

* Update readme

* Remove duplicates

* Remove redundant if

* Add ruamel

* Use ruamel

* Remove safe load

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
 2023-05-31 05:26:28 +00:00
Conor
b32357c684 update bump_version to use shell script (#26801) 2023-05-30 16:45:15 -05:00
Augustin
c5a58feeff fix: modified files detection (#26744) 2023-05-30 04:36:50 -05:00
Augustin
312a3615f3 connectors-ci: pass PR id to airbyte-ci (#26504) 2023-05-30 08:15:53 +00:00
Augustin
f2c5f1df25 connectors-ci: run metadata validation on test (#26646) 2023-05-30 09:44:05 +02:00
Augustin
2c9192e7a0 connectors-ci: improve global status reporting to GitHub for tests (#26575) 2023-05-29 19:53:59 +02:00
Ben Church
5a975c31ef Connectors CI: spec upload on pre-release publish (#26691) 
* upload spec on prepublish

* remove test comments

* Fix up unit test

* Fix test_publish

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
 2023-05-29 10:35:57 -04:00
Ben Church
1dabc6208e Metadata: add tags field (#26320) 
* Add optional tags field

* Remove duplicate icons

* Add programming tags to all

* Update docs

* supportUrl -> documentationUrl

* Ensure one language tag is applied

* Add keyvalue check

* rebase and fix tests

* Format

* Add cache buster

* Improve test

* Automated Commit - Formatting Changes

* Update error

* Fix missing tags

* Fix scaffold

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
 2023-05-26 16:13:09 -07:00
Augustin
75d4c09cd7 retrieve CDK version, set it in context and report + set image labels for CDK version and dagger version (#26572) 2023-05-26 08:12:43 +02:00
Edward Gao
bbdb3be868 Destination Redshift: Install normalization (#26165) 
* do things in-connector

* Automated Change

* version bumps + changelog

* remove unnecessary yum package

* derp

* also build.gradle

* Automated Change

---------

Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-05-25 23:40:48 +00:00
Augustin
42f3d4559f connectors-ci: re-upload spec if image exists + fix pre-release (#26604) 
* connectors-ci: fix pre-release

* fix edge case

* Automated Change

---------

Co-authored-by: alafanechere <alafanechere@users.noreply.github.com>
 2023-05-25 21:53:51 +02:00
Augustin
31077bb3a1 ignore SVG from https check (#26567) 2023-05-25 15:03:48 +02:00
Augustin
fd3655707e connectors-ci: add finalize_build logic to handle custom Dockerfiles (#26489) 2023-05-25 09:32:19 +02:00
Ben Church
248bbf94c1 HACKDAY: Icon CDN (#26158) 
* Move icons to connector folder

* Delete old icons

* Update upload logic

* Add icon url to definitions

* Update registry model

* Populate cdn url

* DNC butcher the pipeline

* Low hanging fruit fixes

* Fix bucket name

* Merge old and new approaches

* Fix metadata upload step

* Format

* Fix test
 2023-05-24 17:25:41 -07:00
Augustin
75b051d055 connectors-ci: force compression to GZIP on export for tests (#26500) 2023-05-24 18:16:56 +02:00
Augustin
7d7996fd18 airbyte-ci: add connectors list command (#26496) 2023-05-24 08:56:46 -07:00
Augustin
fef9c2a28e connectors-ci: improve Java connector build logic and Gradle caching strategy (#26438) 2023-05-24 08:54:33 +00:00
Ryan Fu
8849669420 Destination Performance Harness: Adds support for incremental syncs with GLOBAL state message emission (#26443) 
* Adds support for incremental syncs with GLOBAL state message emission

* Adds args to Source to not break, future work is to add same features to Source perf-harness and ./gradlew :spotlessJavaApply
 2023-05-23 15:13:31 -07:00
Augustin
4aca67bb68 connectors-ci: update publish doc (#26344) 2023-05-22 22:26:28 +02:00
Ryan Fu
3689ff3abb Performance Harness: Support for pseudo-parallel streams (#26219) 
* Adds support for pseudo-parallel datasets

* Ran ./gradlew :spotlessJavaApply

* Automated Change

* Fixes issue with parallel datasets credentials

* Fixes filter for parallel credentials

* Adds a new configurable property  to build a pseudo-parallel catalog

* Fixes Github Actions variable to be processed properly with the K8s harness yaml

* Adds unit test for random streams and generating streams within the same configured catalog

* Ran ./gradlew :spotlessJavaApply

* Added additional description for GitHub Actions

* Update connector-performance-command.yml

Moved text up to connect with other argument discussion

* Fixes spotBugs issue

* Automated Commit - Formatting Changes

* Update GitHub Action description

---------

Co-authored-by: ryankfu <ryankfu@users.noreply.github.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
 2023-05-22 14:13:38 -05:00
Augustin
80032f73f9 connectors-ci: deprecate slash publish (#25865) 2023-05-22 10:10:56 +02:00
Augustin
8fff0dbb14 connectors-ci: force image's layers compression to GZIP (#26291) 2023-05-19 12:02:03 +02:00
Augustin
68a2157578 connectors-ci: include destination-bigquery to build denormalized, include destination-e2e-test to build destination-dev-null (#26293) 2023-05-19 03:54:16 -05:00
Augustin
4527542f9a connectors-ci: check compression type on image layers (#26138) 2023-05-19 09:07:37 +02:00
Ben Church
1d856ab943 Feat: Metadata Service remove old catalog system (#26013) 
* Remove Definition File References from Python files (#25590)

* Remove check_images_exist.sh

* Update definitions.py

* Update build_report.py

* Update tools/bin/ci_integration_workflow_launcher.py

* Update tools/bin/ci_check_dependency.py

* tools/bin/scan_all_spec_airbyte_secret.py

* Remove qa engine references

* Revert "Remove check_images_exist.sh"

This reverts commit 7675162789.

* Improve get url function

* Add test

* remove scan_all_spec_airbyte_secret.py

* add additional test

* Remove check_images_exist.sh (#25593)

* Remove Definition File References from Java files (LocalDefinitionProvider), shell scripts and docs (#25592)

* Remove CombinedConnectorCatalogGenerator.java

* Update local definition provider

* Update local def test

* Add spec mask downloader

* Make downloader work

* Delete generators and add tests

* REMOVE THE YAML FILES

* Roughly update docs

* Update shell scripts

* Remove unused

* Add connector metadata file doc

* Apply suggestions from code review

Co-authored-by: Augustin <augustin@airbyte.io>

* Additional PR comments

* Run format tasks

---------

Co-authored-by: Augustin <augustin@airbyte.io>

* Remove unused import

* bundle registry

* Ignore future updates

* Update registry

* new registry

* Readd maskeddatainterceptor

* Automated Change

* Remove icon validation

* Automated Change

* Automated Change

* Source Amazon Ads: get rid of `fail_on_extra_columns: false` in SAT (#25913)

* Source Amazon Ads: small schema fixes

* Source Amazon Ads: update changelog

* Source Amazon Ads: update unittest

* Source Amazon Ads: unittest additional property is boolean

* Source Amazon Ads: bump version

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

* connectors-ci: make spec-cache / metadata bucket and creds not required for pre-release (#26119)

* Automated Change

---------

Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
Co-authored-by: Roman Yermilov [GL] <86300758+roman-yermilov-gl@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-16 11:40:11 -07:00
Augustin
0d115c96b9 connectors-ci: make spec-cache / metadata bucket and creds not required for pre-release (#26119) 2023-05-16 19:48:33 +02:00
Augustin
c4888012bd use crane to check if image exists (#26118) 2023-05-16 19:36:18 +02:00