Ben Church
3bd340a971
Update to use rev-parse ( #27598 )
...
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
2023-06-21 20:42:56 -05:00
Ryan Fu
4d76ac5809
Turn off java formatter ( #27579 )
2023-06-21 21:13:20 +00:00
Augustin
22480ac642
ref: broaden the use of the run-dagger-pipeline action ( #27506 )
2023-06-21 06:47:33 -05:00
Ben Church
943150a19b
Add any glob to filter to ensure all steps are run in CI ( #27525 )
...
* Ensure all steps are run
* Update comment
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
2023-06-20 15:14:28 -07:00
Augustin
e9d7c54adc
connectors ci: auto format connectors code ( #27353 )
2023-06-19 17:44:43 +00:00
Augustin
12c1602883
fix: retry action latest version is not stable ( #27465 )
2023-06-19 05:37:11 -05:00
Augustin
55423c2495
connectors-ci: do not run tests on fork PR ( #27402 )
2023-06-15 18:14:47 +00:00
Augustin
d5dc309560
gradle.yml: remove Cache Build Artifacts ( #27409 )
2023-06-15 19:55:00 +02:00
Augustin
ecc310e69d
fix trigger on format workflow ( #27391 )
2023-06-15 03:56:27 -05:00
Augustin
02a592dd4a
create new workflow to format connectors ( #27389 )
2023-06-15 03:41:22 -05:00
Ben Church
68c0d6dbbc
Connector CI: update badges test summary ( #27215 )
...
* Rename to connector test report
* Get all test outputs
* Get last 10 output for all connectors
* Get runs by connector
* Crudely write badge and summary
* Add render functions
* Update connector report badge and link
* Fix url link
* Fix qa report tests
* update qa-engine tests
* Move regex to resource
* Add cron
* Run format
* New line
* Make badge green if last is successful
* Fix test
* Remove report.sh and build_report.py
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
2023-06-14 11:12:47 -07:00
Augustin
5b8200181c
connectors-ci: deprecate slash test ( #27200 )
2023-06-14 18:19:13 +02:00
Augustin
ad1a992097
connectors-ci: move connectors_weekly_build.yml to workflows dir ( #27330 )
2023-06-13 14:41:37 -05:00
Evan Tahler
521a798fb5
Experiments to fix CI ( #27262 )
...
* don't pin virtualenv version
* comments
* relax python version
* keep virtualenv relax
* remove cache
* add note
* Update .github/workflows/gradle.yml
Co-authored-by: Ben Church <ben@airbyte.io >
---------
Co-authored-by: Ben Church <ben@airbyte.io >
2023-06-12 17:59:57 -04:00
Tim Roes
cdfa75e781
Remove old growth related labeling code ( #27248 )
2023-06-12 20:18:53 +02:00
Augustin
09ae1bafb0
connectors-ci: disable concurrency for Java connectors testing ( #27012 )
2023-06-09 08:39:23 +00:00
Ben Church
2df72cc63c
Connector CI: Update nightlies to write to GCS ( #26929 )
...
* Generate report output prefix
* DNC
* Upload to gcs bucket
* Revert dev comments
* Format
* Reenable for cron
* remove reports
* Connector CI reports
* Reremove AWS
* Missed some env vars
* Move bucket to CI
* Bad merge
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
2023-06-06 16:37:19 +00:00
Augustin
1ab95effbb
connectors-ci: re-enable concurrency on java connectors ( #27052 )
2023-06-06 10:06:24 +02:00
Ben Church
723093b095
Metadata: Validate that metadata file exists on new connectors ( #26933 )
...
* Ensure we test to see IF there is a metadata file
* Run format
* Add modifed files as path
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
2023-06-05 17:52:07 -07:00
Augustin
80584edea1
fix typo on workflow ( #27005 )
2023-06-05 01:54:31 -05:00
Augustin
4e88af04bb
connectors-ci: no concurrency on java connectors for nightly builds ( #27003 )
2023-06-05 01:48:47 -05:00
Augustin
af890515ee
from 6 to 10hours timeout ( #26999 )
2023-06-04 15:30:19 -05:00
Conor
6324b28ba3
Use repository tags to clean up zombine instances instead of size ( #26919 )
2023-06-02 10:01:53 -05:00
Augustin
ecddb3634d
connectors-ci: re-enable scheduling of nightly builds ( #26956 )
...
* connectors-ci: re-enable scheduling of nightly builds
* fix timing
2023-06-02 09:38:47 -05:00
Ben Church
68e4d503ef
Update gradle command to remove gitignored files ( #26924 )
...
* Update gradle command to remove gitignored files
* fix typo
* Automated Commit - Format and Process Resources Changes
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com >
2023-06-02 02:24:59 +00:00
Augustin
d5650eeabd
connectors-ci: use new dagger cli on tests ( #26927 )
...
* connectors-ci: use new dagger cli on tests
* Automated Change
---------
Co-authored-by: alafanechere <alafanechere@users.noreply.github.com >
2023-06-01 17:46:45 -05:00
Augustin
a63041603e
connectors-ci: dynamic dagger pipeline name ( #26886 )
...
* connectors-ci: dynamic dagger pipeline name
* tmp
* add retry logic
* tmp
* Revert "tmp"
This reverts commit 976feeb5845682a9a06471e074991d
2023-06-01 03:18:15 -05:00
Duy Nguyen
88438bc6f5
Add paths to only trigger on source-postgres changes ( #26866 )
...
* Add paths to only trigger on source-postgres changes
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com >
2023-05-31 15:40:15 -07:00
Duy Nguyen
7089869927
Jacoco report for source-postgres ( #26792 )
...
* Add jacoco coverage workflow for source-postgres
* Update min coverage to 64 pct for source-postgres
2023-05-31 13:19:16 -07:00
Ben Church
5e0e317eac
Move qa engine to aibyte-ci ( #26377 )
...
* Add change notes
* Add qa-engine poetry
* Port qa_engine code
* Add missing fixture
* Update import
* Update git code
* Delete old qa-engine files
* convert tests
* Add path
* Fix mocks
* Fix test_enable_in_cloud
* Fix metadata path test
* Run formater
* Automated Commit - Formatting Changes
* Update readme
* Remove duplicates
* Remove redundant if
* Add ruamel
* Use ruamel
* Remove safe load
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com >
2023-05-31 05:26:28 +00:00
Augustin
9e96effc89
connectors-ci: connectors base build not required for majority of connectors ( #26739 )
2023-05-30 22:55:36 +02:00
Augustin
fc0c5fbc34
connectors-ci: renable automated tests on PRs ( #26742 )
2023-05-30 04:14:23 -05:00
Augustin
312a3615f3
connectors-ci: pass PR id to airbyte-ci ( #26504 )
2023-05-30 08:15:53 +00:00
Augustin
2c9192e7a0
connectors-ci: improve global status reporting to GitHub for tests ( #26575 )
2023-05-29 19:53:59 +02:00
Augustin
f23a6c93f8
connectors-ci: avoid rate limit on Python install ( #26659 )
...
* connectors-ci: avoid rate limit on Python install
* connectors-ci: avoid rate limit on Python install
2023-05-26 07:50:37 -05:00
Duy Nguyen
84da242f5b
specified java version to zulu 17 ( #26621 )
2023-05-25 14:37:27 -05:00
Duy Nguyen
108fb2c818
Removed test comment and update java version to v3 ( #26617 )
2023-05-25 14:22:34 -05:00
Duy Nguyen
e7a685f434
first commit Jacoco report ( #26611 )
2023-05-25 13:59:50 -05:00
Ryan Fu
c87ffb3149
Adds the bottleneck datasources and respective catalogs and updated description for the harness ( #26503 )
...
* Adds the bottleneck datasources and respective catalogs and updated description for the harness
* Automated Commit - Formatting Changes
---------
Co-authored-by: ryankfu <ryankfu@users.noreply.github.com >
2023-05-24 12:17:34 -05:00
Augustin
fef9c2a28e
connectors-ci: improve Java connector build logic and Gradle caching strategy ( #26438 )
2023-05-24 08:54:33 +00:00
Ryan Fu
8849669420
Destination Performance Harness: Adds support for incremental syncs with GLOBAL state message emission ( #26443 )
...
* Adds support for incremental syncs with GLOBAL state message emission
* Adds args to Source to not break, future work is to add same features to Source perf-harness and ./gradlew :spotlessJavaApply
2023-05-23 15:13:31 -07:00
Ryan Fu
3689ff3abb
Performance Harness: Support for pseudo-parallel streams ( #26219 )
...
* Adds support for pseudo-parallel datasets
* Ran ./gradlew :spotlessJavaApply
* Automated Change
* Fixes issue with parallel datasets credentials
* Fixes filter for parallel credentials
* Adds a new configurable property to build a pseudo-parallel catalog
* Fixes Github Actions variable to be processed properly with the K8s harness yaml
* Adds unit test for random streams and generating streams within the same configured catalog
* Ran ./gradlew :spotlessJavaApply
* Added additional description for GitHub Actions
* Update connector-performance-command.yml
Moved text up to connect with other argument discussion
* Fixes spotBugs issue
* Automated Commit - Formatting Changes
* Update GitHub Action description
---------
Co-authored-by: ryankfu <ryankfu@users.noreply.github.com >
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com >
2023-05-22 14:13:38 -05:00
Augustin
80032f73f9
connectors-ci: deprecate slash publish ( #25865 )
2023-05-22 10:10:56 +02:00
Ben Church
7fc6a9efd1
Fix dagster deploy ( #26325 )
2023-05-19 19:03:32 -05:00
Augustin
a02d0044d1
connectors-ci: disable automated tests ( #26294 )
2023-05-19 12:06:08 +02:00
Augustin
8fff0dbb14
connectors-ci: force image's layers compression to GZIP ( #26291 )
2023-05-19 12:02:03 +02:00
Edward Gao
d5e6bc0bfd
tiny fix in /publish output ( #26267 )
2023-05-18 22:09:17 +00:00
Ben Church
8557eb96d9
Remove autobump from /publish ( #26166 )
2023-05-16 18:44:37 -05:00
Ben Church
1d856ab943
Feat: Metadata Service remove old catalog system ( #26013 )
...
* Remove Definition File References from Python files (#25590 )
* Remove check_images_exist.sh
* Update definitions.py
* Update build_report.py
* Update tools/bin/ci_integration_workflow_launcher.py
* Update tools/bin/ci_check_dependency.py
* tools/bin/scan_all_spec_airbyte_secret.py
* Remove qa engine references
* Revert "Remove check_images_exist.sh"
This reverts commit 7675162789#25593 )
* Remove Definition File References from Java files (LocalDefinitionProvider), shell scripts and docs (#25592 )
* Remove CombinedConnectorCatalogGenerator.java
* Update local definition provider
* Update local def test
* Add spec mask downloader
* Make downloader work
* Delete generators and add tests
* REMOVE THE YAML FILES
* Roughly update docs
* Update shell scripts
* Remove unused
* Add connector metadata file doc
* Apply suggestions from code review
Co-authored-by: Augustin <augustin@airbyte.io >
* Additional PR comments
* Run format tasks
---------
Co-authored-by: Augustin <augustin@airbyte.io >
* Remove unused import
* bundle registry
* Ignore future updates
* Update registry
* new registry
* Readd maskeddatainterceptor
* Automated Change
* Remove icon validation
* Automated Change
* Automated Change
* Source Amazon Ads: get rid of `fail_on_extra_columns: false` in SAT (#25913 )
* Source Amazon Ads: small schema fixes
* Source Amazon Ads: update changelog
* Source Amazon Ads: update unittest
* Source Amazon Ads: unittest additional property is boolean
* Source Amazon Ads: bump version
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com >
* connectors-ci: make spec-cache / metadata bucket and creds not required for pre-release (#26119 )
* Automated Change
---------
Co-authored-by: Augustin <augustin@airbyte.io >
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com >
Co-authored-by: Roman Yermilov [GL] <86300758+roman-yermilov-gl@users.noreply.github.com >
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com >
2023-05-16 11:40:11 -07:00
Rodi Reich Zilberman
6374f4780f
destination harness ( #25030 )
...
* destination harness
* Automated Commit - Formatting Changes
* refactor to support multiple connectors
* better byte counting
* Automated Change
* fix main cp
* harness type
* harness type
* harness type
* harness type
* harness type
* harness type
* harness type
* speed up harness build
* harness type
* harness type
* harness type
* harness type
* 1m catalog
* 1m catalog
* Automated Commit - Formatting Changes
* 1m catalog
* Automated Commit - Formatting Changes
* 1m catalog
* Automated Commit - Formatting Changes
* 1m catalog
* 1m catalog
* Automated Commit - Formatting Changes
* 1m catalog
* Automated Commit - Formatting Changes
* Automated Change
* add some additional logging for debugging
* test
* test
* debugging code
* Automated Commit - Formatting Changes
* debugging code
* debugging code
* debugging code
* Automated Commit - Formatting Changes
* Checkpoint: This is working!.
* Automated Commit - Formatting Changes
* Updates plumbing for Snowflake with static records - Working
* Automated Commit - Formatting Changes
* working 10m
* Automated Commit - Formatting Changes
* add destination datasource
* Automated Commit - Formatting Changes
* sanity
* sanity
* Delete 1m_catalog.json
* sanity
* Update airbyte-integrations/connectors-performance/destination-harness/README.md
Co-authored-by: Davin Chia <davinchia@gmail.com >
* sanity
* Update airbyte-integrations/connectors/destination-e2e-test/src/main/java/io/airbyte/integrations/destination/e2e_test/TestingDestinations.java
Co-authored-by: Davin Chia <davinchia@gmail.com >
* Splits performance harness into separated methods
* Automated Commit - Formatting Changes
* Updates test description to force Github builds
---------
Co-authored-by: rodireich <rodireich@users.noreply.github.com >
Co-authored-by: Jeff Cowan (Airbyte) <4992320+jcowanpdx@users.noreply.github.com >
Co-authored-by: Davin Chia <davinchia@gmail.com >
Co-authored-by: davinchia <davinchia@users.noreply.github.com >
Co-authored-by: ryankfu <ryan.fu@airbyte.io >
Co-authored-by: ryankfu <ryankfu@users.noreply.github.com >
2023-05-16 06:37:00 +00:00
