* init
* bad copy/paste
* move to top level
* Revert "move to top level"
This reverts commit aca3534d38.
* attempt to wire up connector builder frontend to server
* copy from octaviacli
* fix connection to builder server
* update
* delete
* Update
* delete python-version
* Revert "delete python-version"
This reverts commit f9258a7755.
* setup python
* install python
* rename
* kube stuff
* Install python
* missing kube file
* rename
* Update files
* Update bumpversion
* install python
* try with different entrypoint
* rename container
* point to docker-compose.yaml file
* derp
* copy acceptance_test.sh
* copy from acceptance tests
* delete cruft
* update
* remove application env
* reset
* reset to master
* update
* skip comprehensive incremental tests
* Revert "skip comprehensive incremental tests"
This reverts commit 9cee657596.
* reset to master
* remove cruft
* readme
* readme
* save
* buncha stuff
* it works
* improvements
* comment
* codesnippet
* delete superfluous steps
* update port to 8003
* reset to master
* Update publish docker
* move openapi spec to airbyte-connector-builder
* point to openapi spec
* dont expose the connector builder to localhost
* reset FE components to master
* Don't deploy the connector-builder
* Revert "Don't deploy the connector-builder"
This reverts commit 3d157494cf.
* Revert "Revert "Don't deploy the connector-builder""
This reverts commit beac3d48f0.
* comment out more things related to connector builder server
* more attempts at removing the connector builder
* comment out more things
* Update airbyte-connector-builder/README.md
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
* fixes
* rename gradle task
* save
* copyright header
* Apply suggestions from code review
Co-authored-by: Lake Mossman <lake@airbyte.io>
* Update airbyte-webapp/src/config/configProviders.ts
Co-authored-by: Lake Mossman <lake@airbyte.io>
* update
* rename
* indent
* Revert "move openapi spec to airbyte-connector-builder"
This reverts commit 57dda04723.
* Revert "rename"
This reverts commit b2d802b8fa.
* Revert "Revert "rename""
This reverts commit 91db24fd4a.
* point to wrong file in case it fixes the build
* point to right openapi file
* Revert "Revert "move openapi spec to airbyte-connector-builder""
This reverts commit e46a837454.
* point to moved file
* fix path
* Update from master
* newline
* Add failing test
* Revert "Add failing test"
This reverts commit ed9fea09b5.
* comment
* update commented requires
* Add a comment
* comments
* save
* formatting is for losers anyways
* pyproject
* toml
* toml
* saved
* nice
* anothaone
* fmt
* fixmypy
* fix autogen notice header location
* improvecomment
* save working branch
* fmt
* delete license and disable caching
* removeyaml
* fix merge and uncomment flakeCheck
* undo css change
* fix typo
* format
* enable CORS
* move template to backend server
* try adding connector builder server to 'generate-docker' task to see if it fixes CI
* try adding buildDockerImage task
* try adding gradle.properties
* Revert "try adding buildDockerImage task"
This reverts commit 2102009ddb.
* Revert "try adding connector builder server to 'generate-docker' task to see if it fixes CI"
This reverts commit 40b5ae0fcd.
* process detail in apiOverride
* fix gradle
* fix scrollbars
* Run connector builder server tests and static analysis from a docker container (#19495)
* Revert "try adding buildDockerImage task"
This reverts commit 2102009ddb.
* Revert "try adding connector builder server to 'generate-docker' task to see if it fixes CI"
This reverts commit 40b5ae0fcd.
* missing file
* build airbyte-connector-builder-server image
* Add .
* comment
* build connector builder server as part of octavia_cli
* build connector builder too
* patch
* missing file
* comment
* comment
* run unit tests in a container
* Run integration tests
* run
* move to platform build
* format
* rootdir
* get the path from the project
* Update
* pyproject
* delete a bunch of code
* parameterize
* run tests
* update
* rename
* comment
* reset
* uncomment
* dont format
* run everything except isort
* delete dead code
* Revert "delete dead code"
This reverts commit b11df716a0.
* comment
* Update module
* reset to master
* mount in home
* commit formatted files
* try running format
* Better comment
* reset
* Revert "reset"
This reverts commit 20808f8262.
* Update buildSrc/src/main/groovy/airbyte-python-docker.gradle
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* comment
* Add everything except build and hidden directories
* symlink
* Update airbyte-connector-builder-server/run_format.sh
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Update airbyte-connector-builder-server/run_tests.sh
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: sherifnada <snadalive@gmail.com>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
* init
* bad copy/paste
* move to top level
* Revert "move to top level"
This reverts commit aca3534d38.
* attempt to wire up connector builder frontend to server
* copy from octaviacli
* fix connection to builder server
* update
* delete
* Update
* delete python-version
* Revert "delete python-version"
This reverts commit f9258a7755.
* setup python
* install python
* rename
* kube stuff
* Install python
* missing kube file
* rename
* Update files
* Update bumpversion
* install python
* try with different entrypoint
* rename container
* point to docker-compose.yaml file
* derp
* copy acceptance_test.sh
* copy from acceptance tests
* delete cruft
* update
* remove application env
* reset
* reset to master
* update
* skip comprehensive incremental tests
* Revert "skip comprehensive incremental tests"
This reverts commit 9cee657596.
* reset to master
* remove cruft
* delete superfluous steps
* update port to 8003
* reset to master
* Update publish docker
* move openapi spec to airbyte-connector-builder
* point to openapi spec
* dont expose the connector builder to localhost
* reset FE components to master
* Don't deploy the connector-builder
* Revert "Don't deploy the connector-builder"
This reverts commit 3d157494cf.
* Revert "Revert "Don't deploy the connector-builder""
This reverts commit beac3d48f0.
* comment out more things related to connector builder server
* more attempts at removing the connector builder
* comment out more things
* Apply suggestions from code review
Co-authored-by: Lake Mossman <lake@airbyte.io>
* Update airbyte-webapp/src/config/configProviders.ts
Co-authored-by: Lake Mossman <lake@airbyte.io>
* update
* rename
* indent
* Revert "move openapi spec to airbyte-connector-builder"
This reverts commit 57dda04723.
* Revert "rename"
This reverts commit b2d802b8fa.
* Revert "Revert "rename""
This reverts commit 91db24fd4a.
* point to wrong file in case it fixes the build
* point to right openapi file
* Revert "Revert "move openapi spec to airbyte-connector-builder""
This reverts commit e46a837454.
* point to moved file
* fix path
* Update from master
* newline
* Add failing test
* Revert "Add failing test"
This reverts commit ed9fea09b5.
* comment
* update commented requires
* Add a comment
* 2022
* rename to connector-builder-server
* typo
Co-authored-by: lmossman <lake@airbyte.io>
* upgrade debezium version to 1.9.6
* test override not required
* bump version
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Revert "Revert "Bmoric/temporal cleaning cron (#16414)" (#17192)"
This reverts commit 751667949e.
* Generate an artifact
* Change missing property from value to property
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Add the injected temporal client to airbyte-cron
* Update cron to make it work
* Remove useless Bean factory
* PR comments
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Improve conditional bean check
* Match existing Optional functionality
* Add notEquals check
* Add missing env var to Helm chart
* Fix typo
* Mark LogConfigs as Singleton
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Improve conditional bean check
* Match existing Optional functionality
* Add notEquals check
* Add missing env var to Helm chart
* Fix typo
* Mark LogConfigs as Singleton
* Env vars for log/state storage type
* Remove use of Optional in bean declarations
* Fix typo in config property name
* Support Temporal Cloud namespace
* Change to @Value
* Use correct value for conditional check
* Upgrade Micronaut
* Fix merge conflict
* Formatting
* Add missing env var
* Use sync task queue environment variable
* Handle sync task queue as set
* format and force http
* Handle case where sync task queue is empty
* Add correct path to config property
* Remove unused import
* Remove conflict
* Remove unused parameter
* Formatting
* Use pattern for condition process factory beans
* Cleanup
* PR feedback
* Revert hack for testing
* Fix temporal restart by status (#16447)
* Update application.yml
* Re add worker dep
* Add missing env var
* PR comments
* Bmoric/move temporal client (#16778)
* tmp
* tmp
* View diff
* Move part of the temporal client
* tmp
* copy Temporal Utils test
* Uniq Temporal Utils
* Uniq Temporal Workflow Utils
* Move CancellationHandler
* Move commons-temporal to being shared between platform, connector and CLI
* Rm worker dependency from cron
* Fix build
* Update with right value for cron.
* Fix dep conflict
Co-authored-by: jdpgrailsdev <jpearlin1@gmail.com>
Co-authored-by: pmossman <parker@airbyte.io>
Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io>
* added destination-r2 spec
added base-java-s3 module
updated common s3 lib
local
changed dependency of s3 in all realted modules
minor style fixes
added common module for s3 integration tests
* fix configuration for r2 integration tests
* fix configuration for r2 integration tests
* added docs for destination-r2
* minor import fixes
* mark test as disabled
* fixed imports in destination-snowflake
* restored styling task
* added tests for S3DEstinationConfig
* added upload threads count configuration for r2(due to limitations)
* deleted parquet format type
* Fix import in gcs
* Fix import in redshift
* Fix import in snowflake
* Fix one more import in gcs
* Fix one more import in redshift
* Fix import in databricks
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
* wip for moving scheduler-persistence to airbyte-persistence
* move main/resources
* move settings include to match existing includes
* fix incorrect import paths
* fix import order
* sweep all scheduler application code and new-scheduler conditional logic
* remove airbyte-scheduler from deployments and docs
* format
* remove 'v2' from github actions
* add back scheduler in delete deployment command
* remove scheduler parameters from helm chart values
* add back job cleaner + test and add comment
* remove now-unused env vars from code and docs
* format
* remove feature flags from web backend connection handler as it is no longer needed
* remove feature flags from config api as it is now longer needed
* remove feature flags input from config api test
* format + shorter url
* remove scheduler parameters from helm chart readme
## What
Part 3 of https://github.com/airbytehq/airbyte/pull/13122.
Follow up to #13478 .
Explanation for what is happening:
Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
## How
Rename airbyte-scheduler:models to airbyte-scheduler:scheduler-models.
Rename airbyte-scheduler:persistence to airbyte-scheduler:scheduler-persistence.
## What
Part 2 of https://github.com/airbytehq/airbyte/pull/13122.
Follow up to #13476 .
Explanation for what is happening:
Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
* Rename airbyte-config:models to airbyte-config:config-models.
* Rename airbyte-config:persistence to airbyte-config:config-persistence.
Part 1 of #13122.
Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.
Explanation for what is happening:
Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
This PR provides 'version catalog' plugin, Where catalog is defined by using declared dependency list in 'build.gradle'.
In this pr all dependencies were transferred to 'libs' using 'dependencyResolutionManagement -> versionCatalogs -> libs'
* Switch json-avro-converter to jitpack
* Switch jsongenerator to jitpack
* Fix default config
* Fix one more default config use case
* Fix jitpack dependency
* Move jitpack repo to root build.gradle
Part 1 of https://docs.google.com/document/d/11pEUsHyKUhh4CtV3aReau3SUG-ncEvy6ROJRVln6YB4/edit?usp=sharing.
This is the initial set up of the reporter. Will add actual metrics in the follow up PR.
Since the reporter is primarily for Cloud use and limited to DD to begin with, we are keeping it outside the regular Airbyte docker and Kube deploys for now.
Add ReadMes to better define various modules.
Part 1 of #8303.
Essentially, create a separate application that will run all set up operations for Airbyte, including database migrations and metadata setup.
Part 2 will be removing these operations from the server, modifying start up conditions to look for migration versions (#8302). I'll then release a minor version.
This sequencing felt the most natural to me, since it allows for the new code to exist in master without any breaking changes. It also preserves the ServerApp class for comparison.
