* Add begining of command
* Add invalid metadata file
* Add begining of pipx
* DNC
* Get pipx working
* Add env var fix
* Fix the stupid path issue
* Include all connectors
* add new command structure
* properly handle directory vs filepath
* Back to my roots of using a step
* install module in base pipeline
* Multi step
* Move concurrent run to util
* Add actions
* Run black
* Remove metadata files
* Remove logs
* Fix typo
* Add metadata test files
* Address pr comments
* Address pr comments
* Run black
* Use path read contents
* Remove metadata files
* Run black and flake
* Refactor poetry for multiple poetry packages
* add with packages
* Change poetry base to debian
* Add github action
* Add pipelien command decorator
* Fix action
* Add pipeline click command class
* Use directory as param type
* Bump version
* Debug
* Bring the editable flag back
* Update install script
* Add skelton
* Run metadata unit tests
* Move poetry set up to environment
* Set up poetry caching
* Add top level airbyte-ci command
* Move connectors_ci under airbyte_pipeline command
* Update command usage
* Add pipeline context
* Properly handle exit codes
* Check for changed metadata lib paths
* Run black and flake
* Update environment to use context
* Add workflow
* Get rid of log name
* Add github action
* Add workflow dispatch
* Apply suggestions from code review
Co-authored-by: Augustin <augustin@airbyte.io>
* rename airbyte_ci_pipeline to airbyte_ci
* Update lib
* Add cicontext enum, safe logging, move call to super.__init__
* Run black
* Refactor to use test
* Move test logic into pipeline
* Add reporting back
* Rename commands
* Move metadata run to pipeline
* Move connectors run to pipeline
* Run black
* Increase connector ops version
* Clear pip
* Different uninstall command
* omit git
* Silly shot in the dark -e
---------
Co-authored-by: Augustin <augustin@airbyte.io>
* remove arguments
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* Source Performance harness (#23922)
* test
* initial commit
* Working dev
* src-platform-dst
* Integrate secrets
* Update .dockerignore
* Update build.gradle
* Update base.sh
* Build harness module and use it to measure
* Build harness module and use it to measure
* Build harness module and use it to measure
* Automated Change
* Integrate secrets
* Integrate secrets
* Integrate secrets
* Integrate secrets
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* test
* Automated Change
* test
* Automated Change
* better error handling
* temp
* Automated Change
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* use connector image name
* add test name argument
* add test name argument
* add test name argument
* add test name argument
* add test name argument
* add test name argument
* Automated Change
* add bottleneck_stream1 dataset
* Automated Change
* fix dependency
* fix dependency
* test downgrade of commons to 41.1
* Automated Change
* test downgrade of commons to 41.0
* test
* test
* test
* test
* test
* exit condition
* exit condition
* exit condition
* test
* test
* test
* test
* test
* test
* sanity
* readme
* change command name
* update gradle settings
* sanity
* architecture aware docker build
* sanity
* put run harness step in a script
* sanity
* sanity
---------
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
* Add compare version tool
* Delete empty line
* Updated script to sync with state and add doc to README.md
* Add STATE to filter and move pull during run
* Pass argument along, add test that should pass and test that should fail
* Add tests with additionalProperties
* Set additionalproperties=false when not set|
* Parametrize test cases
* Make the behavior 'optional'
* Fix parametrization for all combinations
* Improve CI credentials README and rename param
* Update naming to be clearer about columns only
* record_has_unexpected_field > record_has_unexpected_column
* Automated Change
* Add bypass for beta + ga connectors that failed
* Update docs and TODOs
* Update changelog and dockerfile
* Update TODO
* Update a few neglected connectors
* Remove uploaded file
* Update dockerfile after merge conflict
---------
Co-authored-by: marcosmarxm <marcosmarxm@users.noreply.github.com>
* connectors-ci: only send commit status on PR
* increase timeout
* try semaphore
* disable qa check code format for testing semaphore
* fix typo and change semaphore
* disable acceptance tests
* use semaphore
* restore ga builds
* use CacheSharingMode.SHARED, renable acceptance tests, semaphore of 5
* expose concurrency as a CLI option, set it to 10 on the workflow
* clean
* use xlarge runner
* // 15
* declare inputs
* better job naming
* better job naming
* use run-name
* use run-name
* copy tests from other branch
* switch to >
* [wip] wire up tests
* make tests work
* fixes
* nicer test structure
* maybe add feature flag?
* pattern matching
* also add version check
* formatting
* refactor test also
* extract test + fix method call
* minor tweaks
* add context to log message
* put workspace id in normalization input
* use non-semver tag
* add flag for version of normalization
* also flag old version
* add test
* missed part of the commit
* format
* add test for null workspace ID
* Revert "also flag old version"
This reverts commit 3be601d16c.
* Revert "missed part of the commit"
This reverts commit 47a67b4631.
* always apply flag, even if we're behind a version
* derp
* Add more logging to the normalization activity
* Update charts and kustomize for the feature flag
* fix clickhouse integration test
* remove replace_identifiers
* Revert "remove replace_identifiers"
This reverts commit 0e7ded5a7b.
* fix replace_identifiers
* garbage debug logs
* stop trying to setup duckdb test
* wake up and choose violence
* fix mssql
* exclude duckdb from tests
* make snowflake happy
* uncomment tests
* derp
* derpderp
* format
* format
* also fix redshift???
* maybe now everything works???
* remove debug logs
* use special docker tag
* bump to new tag
* use random test schema in publish also
* properly cleanup
* remove feature flag stuff
* version bump + changelog
* Automated Commit - Formatting Changes
* bump definitions
---------
Co-authored-by: Jimmy Ma <gosusnp@users.noreply.github.com>
Co-authored-by: Jimmy Ma <jimmy@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-bot@airbyte.io>
Co-authored-by: edgao <edgao@users.noreply.github.com>
* Add instructions to install reqs and run tests locally
* add passing tests with old behavior
* Make it like test strictness level
* Remove expected teams from non-tracked change test since they are not asserted against
* Make changed GA connector list object same object format as regex checker to be able to compare them. Fix tests
* Refactor and align naming
* Update all find methods to return Connector objects
* Bump setup.py
* Updates docusaurus
* Small docs fixes
deploy_docusaurus info messages are changed to
reflect reality
updates a secret to exist
* Go back to the right branch after deploy
dev runs of this pr can be found here: https://github.com/airbytehq/airbyte/actions/workflows/release-airbyte-os.yml
I've tested the `create_release` step for both `airbyte` and `airbyte-platform`.
What I haven't tested are:
- publishing docker images step for airbyte-platform
- the auto commit steps for both.
What also should be noted: the artifacts from the `run-ab-platform.sh` script need to be commited into the repo for them to be included in the release.
* delete platform code and workflows
* delete more platform workflows
* sunset kustomize (#23187)
* sunset kustomize
* remove build steps that use kustomize
* remove k8s kustomize gke-kube-acceptance-tests
* delete docker-compose files
* remove platform from check_images_exist.sh
* Working replacement script for docker
* rename start script and add download files to gitignore
* add args and download only option
* integration test downloads docker compose files
* remove helm charts
* remove more platform files
* add flags.yml
* use version from gradle.properties instead of .env
* Working replacement script for docker (#23227)
* Working replacement script for docker
* Working debugging line
* rename start script and add download files to gitignore
* add args and download only option
* Made flags cleaner and more cleanup
i've never used this flag pattern, its much
easier than my usual way
* warn about a sharp edge
* Don't run when we don't know a command
* better help spacing
* add flags.yml
---------
Co-authored-by: cpdeethree <conor@airbyte.io>
Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
* remove more-secrets env
---------
Co-authored-by: Charles <charles@airbyte.io>
Co-authored-by: Topher Lubaway <asimplechris@gmail.com>
* cloud-availability-updater: enable it
* cloud-availability-updater: enable it
* cloud-availability-updater: enable it
* cloud-availability-updater: enable it
* cloud-availability-updater: enable it
* cloud-availability-updater: enable it
* cloud-availability-updater: enable it
* cloud-availability-updater: enable it
* cloud-availability-updater: enable it
* cloud-availability-updater: enable it
* cloud-availability-updater: enable it
* tools changes only
* normalize line to lower
* improve url prefix ignore
* more robust tests
* ignore acceptance_tests_logs
* qa-checks: validate the connector code does not contain http:// urls
* qa-checks: validate the connector code does not contain http:// urls
