1
0
mirror of synced 2026-01-07 09:05:45 -05:00
Commit Graph

9 Commits

Author SHA1 Message Date
Charles
480a43b9ab grab bag of non-controversial clean up tasks (#26702) 2023-05-31 08:53:27 -07:00
Ryan Fu
c87ffb3149 Adds the bottleneck datasources and respective catalogs and updated description for the harness (#26503)
* Adds the bottleneck datasources and respective catalogs and updated description for the harness

* Automated Commit - Formatting Changes

---------

Co-authored-by: ryankfu <ryankfu@users.noreply.github.com>
2023-05-24 12:17:34 -05:00
Ryan Fu
8849669420 Destination Performance Harness: Adds support for incremental syncs with GLOBAL state message emission (#26443)
* Adds support for incremental syncs with GLOBAL state message emission

* Adds args to Source to not break, future work is to add same features to Source perf-harness and ./gradlew :spotlessJavaApply
2023-05-23 15:13:31 -07:00
Ryan Fu
3689ff3abb Performance Harness: Support for pseudo-parallel streams (#26219)
* Adds support for pseudo-parallel datasets

* Ran ./gradlew :spotlessJavaApply

* Automated Change

* Fixes issue with parallel datasets credentials

* Fixes filter for parallel credentials

* Adds a new configurable property  to build a pseudo-parallel catalog

* Fixes Github Actions variable to be processed properly with the K8s harness yaml

* Adds unit test for random streams and generating streams within the same configured catalog

* Ran ./gradlew :spotlessJavaApply

* Added additional description for GitHub Actions

* Update connector-performance-command.yml

Moved text up to connect with other argument discussion

* Fixes spotBugs issue

* Automated Commit - Formatting Changes

* Update GitHub Action description

---------

Co-authored-by: ryankfu <ryankfu@users.noreply.github.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
2023-05-22 14:13:38 -05:00
Rodi Reich Zilberman
6374f4780f destination harness (#25030)
* destination harness

* Automated Commit - Formatting Changes

* refactor to support multiple connectors

* better byte counting

* Automated Change

* fix main cp

* harness type

* harness type

* harness type

* harness type

* harness type

* harness type

* harness type

* speed up harness build

* harness type

* harness type

* harness type

* harness type

* 1m catalog

* 1m catalog

* Automated Commit - Formatting Changes

* 1m catalog

* Automated Commit - Formatting Changes

* 1m catalog

* Automated Commit - Formatting Changes

* 1m catalog

* 1m catalog

* Automated Commit - Formatting Changes

* 1m catalog

* Automated Commit - Formatting Changes

* Automated Change

* add some additional logging for debugging

* test

* test

* debugging code

* Automated Commit - Formatting Changes

* debugging code

* debugging code

* debugging code

* Automated Commit - Formatting Changes

* Checkpoint: This is working!.

* Automated Commit - Formatting Changes

* Updates plumbing for Snowflake with static records - Working

* Automated Commit - Formatting Changes

* working 10m

* Automated Commit - Formatting Changes

* add destination datasource

* Automated Commit - Formatting Changes

* sanity

* sanity

* Delete 1m_catalog.json

* sanity

* Update airbyte-integrations/connectors-performance/destination-harness/README.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* sanity

* Update airbyte-integrations/connectors/destination-e2e-test/src/main/java/io/airbyte/integrations/destination/e2e_test/TestingDestinations.java

Co-authored-by: Davin Chia <davinchia@gmail.com>

* Splits performance harness into separated methods

* Automated Commit - Formatting Changes

* Updates test description to force Github builds

---------

Co-authored-by: rodireich <rodireich@users.noreply.github.com>
Co-authored-by: Jeff Cowan (Airbyte) <4992320+jcowanpdx@users.noreply.github.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: davinchia <davinchia@users.noreply.github.com>
Co-authored-by: ryankfu <ryan.fu@airbyte.io>
Co-authored-by: ryankfu <ryankfu@users.noreply.github.com>
2023-05-16 06:37:00 +00:00
Rodi Reich Zilberman
5ff0cfd588 refactor source harness and GitHub action to add source-mysql (#25036)
* refactor to add source-mysql

* refactor to add source-mysql

* refactor to add source-mysql

* test

* Automated Change

* fix if condition

* try multiline

* new db

* add mysql catalogs

---------

Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-04-13 13:47:20 -07:00
Rodi Reich Zilberman
32b9760cff Update seeds in new location under config-oss (#24966)
* update seeds in new location under config-oss

* Automated Change

* force change

---------

Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-04-06 13:27:04 -05:00
Rodi Reich Zilberman
0bab1756b8 Rename airbyte-config module (#24885)
* rename airbyte-config module

* Automated Commit - Formatting Changes

* sanity

* update import

* update import

* update script

* update script

* update script

* update script

* Automated Change

* Automated Change

* Automated Change

* Automated Change

* update awsdatalake icon

* point slash commands to new path

* sanity

* Automated Commit - Formatting Changes

* sanity

* Automated Change

* Automated Change

* sanity

---------

Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-04-06 10:47:30 -07:00
Rodi Reich Zilberman
0a38d4e1ca Source connector performance harness (#23786)
* remove arguments

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* Source Performance harness (#23922)

* test

* initial commit

* Working dev

* src-platform-dst

* Integrate secrets

* Update .dockerignore

* Update build.gradle

* Update base.sh

* Build harness module and use it to measure

* Build harness module and use it to measure

* Build harness module and use it to measure

* Automated Change

* Integrate secrets

* Integrate secrets

* Integrate secrets

* Integrate secrets

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* Automated Change

* test

* Automated Change

* better error handling

* temp

* Automated Change

* use connector image name

* use connector image name

* use connector image name

* use connector image name

* use connector image name

* use connector image name

* use connector image name

* use connector image name

* add test name argument

* add test name argument

* add test name argument

* add test name argument

* add test name argument

* add test name argument

* Automated Change

* add bottleneck_stream1 dataset

* Automated Change

* fix dependency

* fix dependency

* test downgrade of commons to 41.1

* Automated Change

* test downgrade of commons to 41.0

* test

* test

* test

* test

* test

* exit condition

* exit condition

* exit condition

* test

* test

* test

* test

* test

* test

* sanity

* readme

* change command name

* update gradle settings

* sanity

* architecture aware docker build

* sanity

* put run harness step in a script

* sanity

* sanity

---------

Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-03-30 10:26:31 -07:00