* Revert "Revert "Add dependencies to docker-compose (#19257)" (#19306)"
This reverts commit b16f28fd4d.
* Bump docker-compose version
* test
* Revert "test"
This reverts commit c3b28c9ec3.
* Bump docker compose version to 3.8
* test
* test
* Use a newer docker for platform build
* test
* Use docker compose v2
* Update docker for frontend e2e
* Use docker compose v2 for e2e tests
* Update tools to use docker compose v2
* Update more docker compose references to v2
* Update docker compose usage in docs
* Update deploying airbyte docs
* Fix permission
* Case insensistive check
* Add extra mention for docker compose in upgrading airbyte
* Improve upgrade docker note
* Migrate airbyte-bootloader to Micronaut
* PR feedback
* More PR feedback
* Rename variable for clarity
* Add properties to support cloud
* Formatting
* Use default values for env vars
* Re-organization to support cloud overrides
* Disable conditional logging
* More singleton cleanup
* test: try CI without fluentbit
* Revert "test: try CI without fluentbit"
This reverts commit 8fa0f74106.
* test: enable SSH on EC2 runner
* Revert "test: enable SSH on EC2 runner"
This reverts commit e4867aae09.
* Avoid early database connection on startup
* Fix compile issues from refactor
* Formatting
Co-authored-by: perangel <perangel@gmail.com>
* pass workspace id to sync workflow and use it to selectively enable field selection
* fix tests around workspace id in job creation
* make sure field selection environment variables get passed through properly
* clean up handling around field selection flags
* debug logging for field selection
* properly handle empty field selection feature flag
* fix pmd
* actually fix pmd
* init
* bad copy/paste
* move to top level
* Revert "move to top level"
This reverts commit aca3534d38.
* attempt to wire up connector builder frontend to server
* copy from octaviacli
* fix connection to builder server
* update
* delete
* Update
* delete python-version
* Revert "delete python-version"
This reverts commit f9258a7755.
* setup python
* install python
* rename
* kube stuff
* Install python
* missing kube file
* rename
* Update files
* Update bumpversion
* install python
* try with different entrypoint
* rename container
* point to docker-compose.yaml file
* derp
* copy acceptance_test.sh
* copy from acceptance tests
* delete cruft
* update
* remove application env
* reset
* reset to master
* update
* skip comprehensive incremental tests
* Revert "skip comprehensive incremental tests"
This reverts commit 9cee657596.
* reset to master
* remove cruft
* readme
* readme
* save
* buncha stuff
* it works
* improvements
* comment
* codesnippet
* delete superfluous steps
* update port to 8003
* reset to master
* Update publish docker
* move openapi spec to airbyte-connector-builder
* point to openapi spec
* dont expose the connector builder to localhost
* reset FE components to master
* Don't deploy the connector-builder
* Revert "Don't deploy the connector-builder"
This reverts commit 3d157494cf.
* Revert "Revert "Don't deploy the connector-builder""
This reverts commit beac3d48f0.
* comment out more things related to connector builder server
* more attempts at removing the connector builder
* comment out more things
* Update airbyte-connector-builder/README.md
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
* fixes
* rename gradle task
* save
* copyright header
* Apply suggestions from code review
Co-authored-by: Lake Mossman <lake@airbyte.io>
* Update airbyte-webapp/src/config/configProviders.ts
Co-authored-by: Lake Mossman <lake@airbyte.io>
* update
* rename
* indent
* Revert "move openapi spec to airbyte-connector-builder"
This reverts commit 57dda04723.
* Revert "rename"
This reverts commit b2d802b8fa.
* Revert "Revert "rename""
This reverts commit 91db24fd4a.
* point to wrong file in case it fixes the build
* point to right openapi file
* Revert "Revert "move openapi spec to airbyte-connector-builder""
This reverts commit e46a837454.
* point to moved file
* fix path
* Update from master
* newline
* Add failing test
* Revert "Add failing test"
This reverts commit ed9fea09b5.
* comment
* update commented requires
* Add a comment
* comments
* save
* formatting is for losers anyways
* pyproject
* toml
* toml
* saved
* nice
* anothaone
* fmt
* fixmypy
* fix autogen notice header location
* improvecomment
* save working branch
* fmt
* delete license and disable caching
* removeyaml
* fix merge and uncomment flakeCheck
* undo css change
* fix typo
* format
* enable CORS
* move template to backend server
* try adding connector builder server to 'generate-docker' task to see if it fixes CI
* try adding buildDockerImage task
* try adding gradle.properties
* Revert "try adding buildDockerImage task"
This reverts commit 2102009ddb.
* Revert "try adding connector builder server to 'generate-docker' task to see if it fixes CI"
This reverts commit 40b5ae0fcd.
* process detail in apiOverride
* fix gradle
* fix scrollbars
* Run connector builder server tests and static analysis from a docker container (#19495)
* Revert "try adding buildDockerImage task"
This reverts commit 2102009ddb.
* Revert "try adding connector builder server to 'generate-docker' task to see if it fixes CI"
This reverts commit 40b5ae0fcd.
* missing file
* build airbyte-connector-builder-server image
* Add .
* comment
* build connector builder server as part of octavia_cli
* build connector builder too
* patch
* missing file
* comment
* comment
* run unit tests in a container
* Run integration tests
* run
* move to platform build
* format
* rootdir
* get the path from the project
* Update
* pyproject
* delete a bunch of code
* parameterize
* run tests
* update
* rename
* comment
* reset
* uncomment
* dont format
* run everything except isort
* delete dead code
* Revert "delete dead code"
This reverts commit b11df716a0.
* comment
* Update module
* reset to master
* mount in home
* commit formatted files
* try running format
* Better comment
* reset
* Revert "reset"
This reverts commit 20808f8262.
* Update buildSrc/src/main/groovy/airbyte-python-docker.gradle
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* comment
* Add everything except build and hidden directories
* symlink
* Update airbyte-connector-builder-server/run_format.sh
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Update airbyte-connector-builder-server/run_tests.sh
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: sherifnada <snadalive@gmail.com>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
* init
* bad copy/paste
* move to top level
* Revert "move to top level"
This reverts commit aca3534d38.
* attempt to wire up connector builder frontend to server
* copy from octaviacli
* fix connection to builder server
* update
* delete
* Update
* delete python-version
* Revert "delete python-version"
This reverts commit f9258a7755.
* setup python
* install python
* rename
* kube stuff
* Install python
* missing kube file
* rename
* Update files
* Update bumpversion
* install python
* try with different entrypoint
* rename container
* point to docker-compose.yaml file
* derp
* copy acceptance_test.sh
* copy from acceptance tests
* delete cruft
* update
* remove application env
* reset
* reset to master
* update
* skip comprehensive incremental tests
* Revert "skip comprehensive incremental tests"
This reverts commit 9cee657596.
* reset to master
* remove cruft
* delete superfluous steps
* update port to 8003
* reset to master
* Update publish docker
* move openapi spec to airbyte-connector-builder
* point to openapi spec
* dont expose the connector builder to localhost
* reset FE components to master
* Don't deploy the connector-builder
* Revert "Don't deploy the connector-builder"
This reverts commit 3d157494cf.
* Revert "Revert "Don't deploy the connector-builder""
This reverts commit beac3d48f0.
* comment out more things related to connector builder server
* more attempts at removing the connector builder
* comment out more things
* Apply suggestions from code review
Co-authored-by: Lake Mossman <lake@airbyte.io>
* Update airbyte-webapp/src/config/configProviders.ts
Co-authored-by: Lake Mossman <lake@airbyte.io>
* update
* rename
* indent
* Revert "move openapi spec to airbyte-connector-builder"
This reverts commit 57dda04723.
* Revert "rename"
This reverts commit b2d802b8fa.
* Revert "Revert "rename""
This reverts commit 91db24fd4a.
* point to wrong file in case it fixes the build
* point to right openapi file
* Revert "Revert "move openapi spec to airbyte-connector-builder""
This reverts commit e46a837454.
* point to moved file
* fix path
* Update from master
* newline
* Add failing test
* Revert "Add failing test"
This reverts commit ed9fea09b5.
* comment
* update commented requires
* Add a comment
* 2022
* rename to connector-builder-server
* typo
Co-authored-by: lmossman <lake@airbyte.io>
* Use Micronaut auto-detected environment for docker vs kubernetes choices
* Handle set, but blank env var
* Formatting
* Revert change
* Add default value
* Explicitly set WORKER_ENVIRONMENT for container orchestrator
* Make public
* Add sleep to allow Temporal cache to populate
Co-authored-by: Jimmy Ma <gosusnp@users.noreply.github.com>
* Cleanup application.yml
* Clean up cron application.yml and kube overlay
* Avoid flooding the logs with waiting for temporal namespace message
* Rm common app initializer
* Try to re-introduce an initializer
* Revert "Revert "Bmoric/temporal cleaning cron (#16414)" (#17192)"
This reverts commit 751667949e.
* Generate an artifact
* Change missing property from value to property
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Add the injected temporal client to airbyte-cron
* Update cron to make it work
* Remove useless Bean factory
* PR comments
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Improve conditional bean check
* Match existing Optional functionality
* Add notEquals check
* Add missing env var to Helm chart
* Fix typo
* Mark LogConfigs as Singleton
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Improve conditional bean check
* Match existing Optional functionality
* Add notEquals check
* Add missing env var to Helm chart
* Fix typo
* Mark LogConfigs as Singleton
* Env vars for log/state storage type
* Remove use of Optional in bean declarations
* Fix typo in config property name
* Support Temporal Cloud namespace
* Change to @Value
* Use correct value for conditional check
* Upgrade Micronaut
* Fix merge conflict
* Formatting
* Add missing env var
* Use sync task queue environment variable
* Handle sync task queue as set
* format and force http
* Handle case where sync task queue is empty
* Add correct path to config property
* Remove unused import
* Remove conflict
* Remove unused parameter
* Formatting
* Use pattern for condition process factory beans
* Cleanup
* PR feedback
* Revert hack for testing
* Fix temporal restart by status (#16447)
* Update application.yml
* Re add worker dep
* Add missing env var
* PR comments
* Bmoric/move temporal client (#16778)
* tmp
* tmp
* View diff
* Move part of the temporal client
* tmp
* copy Temporal Utils test
* Uniq Temporal Utils
* Uniq Temporal Workflow Utils
* Move CancellationHandler
* Move commons-temporal to being shared between platform, connector and CLI
* Rm worker dependency from cron
* Fix build
* Update with right value for cron.
* Fix dep conflict
Co-authored-by: jdpgrailsdev <jpearlin1@gmail.com>
Co-authored-by: pmossman <parker@airbyte.io>
Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io>
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Improve conditional bean check
* Match existing Optional functionality
* Add notEquals check
* Add missing env var to Helm chart
* Fix typo
* Mark LogConfigs as Singleton
* Env vars for log/state storage type
* Remove use of Optional in bean declarations
* Fix typo in config property name
* Support Temporal Cloud namespace
* Change to @Value
* Use correct value for conditional check
* Upgrade Micronaut
* Fix merge conflict
* Formatting
* Add missing env var
* Use sync task queue environment variable
* Handle sync task queue as set
* format and force http
* Handle case where sync task queue is empty
* Add correct path to config property
* Remove unused import
* Remove unused parameter
* Formatting
* Use pattern for condition process factory beans
* Cleanup
* PR feedback
* Revert hack for testing
Co-authored-by: pmossman <parker@airbyte.io>
* Add airbyte-cron to bumpversion
* Update airbyte-cron version to current
* Add workspace clean up job
* Add missing env var to docker-compose
* Update file deletion logging
* add new Configs for multi-cloud
* add api endpoints for setting workflow attempt id and createOrUpdate state
* update activities to call APIs instead of persistence
* workerApp refactor to separately initialize control or data plane dependencies
* modify syncWorkflow to call new activity that decides which task queue for data plane tasks
* misc to get build working
* move StateConverter to worker, so that server and worker can both access without needing to introduce any new dependencies
* update configs - remove extranneous helpers, clarify naming and comments, removed COMBINED value
* forgot to actually remove COMBINED enum value, this removes it
* add WorkerApp todo for breaking API Client into a scoped client
* rename decideTaskQueueActivity var to routeToTaskQueueActivity
* pr comments
* naming fix
* refactor secretHydrator instantiation
* WorkerApp PR feedback: move API client logic to separate class, use updated configs, etc
* add a RouterService class that is injected into RouteToTaskQueueActivityImpl
* AttemptApi cleanup and added unit test coverage
* fix confusion between AttemptId and AttemptNumber in new AttemptApi
* remove unused getDataPlaneSecretsHydrator
* remove unused import
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
* `LOG_CONNECTOR_MESSAGES` to enable logging of all messages seen by the worker in a sync
* also log initial state
* linting
* getState 2x for to match input file
* use `Jsons.serialize` utility
* add note to `developing-locally`
* fix bad merge
Today we are running into OOM exceptions with normalization. Normalization itself also inherits the destination's resource requirements. After work to bring destination memory usage down, this is no longer ideal, since most destinations use less memory than normalization needs.
This PR makes it possible to specify the general resource the normalization pod is provided via env vars.
Notes:
- Add env vars. Default to the various job main container resources if these are not set.
- Instead of using the destination's memory, use the normalization specify env vars.
* report failures for synchronous check/discover, refactor common logic
* allow null workspace, send spec errors
* add failure origin, format
* rm connector_type, fix failing tests
* add tests for other job types
* log instead of throw
* move swallow to common spot
* connector jobs use context instead of passing full config
* sync jobs use context instead of passing raw config
* fix failing test
* fix failing scheduler client test
* skeleton for reporting connector errors to sentry
* report on job failures instead of attempt failures
* report sync job failures with relevant metadata using JobErrorReporter
* send stack traces from python connectors to sentry
* test JobCreationAndStatusUpdate and JobErrorReporter
* logs
* refactor into helper, initial tests
* using sentry
* run format
* load reporting client from env
* load sentry dsn from env
* send java stack traces to sentry
* test sentryclient, refactor to use Hub instance
* ErrorReportingClient.report -> .reportJobFailureReason
* inject exception helper, test stack trace parse error tagging
* rm logs
* more stack trace tests
* remove logs
* fix failing tests
* rename ErrorReportingClient to JobErrorReportingClient
* rename vars in docker-compose
* Return an Optional instead of null when parsing stack traces
* dont remove airbyte prefix when setting release name
* from_trace_message static
* remove failureSummary from jobfailure input, get from Job
* send stacktrace string if we weren't able to parse
* set deployment mode tag
* update .env
* just log if something goes wrong
* use SHORT_ACTIVITY_OPTIONS on check connection activity so that it has retries
* retry workflow after delay instead of quarantining
* allow activity env vars to be configured in docker and kube
* add env var for workflow restart delay and refactor slightly
* update tests to handle new restart behavior
* update test name
* add empty env var values to .env files
* fail attempt before job in cleanJobState to prevent state machine failure
* change default value of max activity attempt retries from 10 to 5
* Add deployment mode to env shared with jobs
* Add adaptive runners
* Migrate postgres source to use adaptive runner
* Add an array of specs in docker image spec definition
* Add copyright
* Parse docker image spec with specs list
* Update spec yaml files
* Pass in DEPLOYMENT_MODE to docker compose file
* Revert "Parse docker image spec with specs list"
This reverts commit 8fe41dd3b7.
* Revert changes in docker image spec
* Read cloud specific spec files based on deployment mode
* Revert "Update spec yaml files"
This reverts commit 059f326432.
* Publish cloud spec file if necessary
* Fix upload script
* Move test files
* Update docker compose file
* Format code
* Add comment about spec filename
* Add unit tests
* Remove redundant jdbc acceptance test
When running `PostgresStrictEncryptJdbcSourceAcceptanceTest`, the `discover` method tests always fail because there are unexpected columns in the catalog:
- `wakeup_at`
- `last_visited_at`
- `last_comment_at`
These columns only exist in `PostgresJdbcSourceAcceptanceTest`. And this failure cannot be reproduced locally.
The hypothesis is that when the JDBC unit tests are run on CI, they are run in parallel, and the same testcontainer is used for both tests. That's why the strict encrypt test can discover columns from the oridinary unit test.
Given that the JDBC strict encrypt test is basically redundant, it is removed.
* Create interface, factory for metric client
* remove unused func
* change count val to use long
* PR fix
* otel metric client implementation
* merge conflicts resolve
* build fix
* add a test, moved version into deps catalog
* fix test
* sweep all scheduler application code and new-scheduler conditional logic
* remove airbyte-scheduler from deployments and docs
* format
* remove 'v2' from github actions
* add back scheduler in delete deployment command
* remove scheduler parameters from helm chart values
* add back job cleaner + test and add comment
* remove now-unused env vars from code and docs
* format
* remove feature flags from web backend connection handler as it is no longer needed
* remove feature flags from config api as it is now longer needed
* remove feature flags input from config api test
* format + shorter url
* remove scheduler parameters from helm chart readme
* Add Disable Failing Connections feature
* Rename and cleanup
* list jobs based off connection id
* Move variables to env config and update unit tests
* Fix env flag name
* Fix missing name changes
* Add comments to unit test
* Address PR comments
* Support multiple config types
* Update unit tests
* Remove the attemptId notion in the connectionManagerWorkflow (#10780)
This is removing the attemptId from the create attempt activity to replace it with the attemptNumber. This will be modified in the workflow in a later commit.
* Revert "Remove the attemptId notion in the connectionManagerWorkflow (#10780)" (#11057)
This reverts commit 99338c852a.
* Revert "Revert "Remove the attemptId notion in the connectionManagerWorkflow (#10780)" (#11057)" (#11073)
This reverts commit 892dc7ec66.
* Revert "Revert "Revert "Remove the attemptId notion in the connectionManagerWorkflow (#10780)" (#11057)" (#11073)" (#11081)
This reverts commit e27bb74050.
* Add Disable Failing Connections feature
* Rename and cleanup
* Fix rebase
* only disable if first job is older than max days
* Return boolean for activity
* Return boolean for activity
* Add unit tests for ConnectionManagerWorkflow
* Utilize object output for activity and ignore non success or failed runs
* Utilize object output for activity and ignore non success or failed runs
Co-authored-by: Benoit Moriceau <benoit@airbyte.io>
* Add a job notification
The new scheduler was missing a notification step after the job is done.
This is needed in order to report the number of record of a sync.
* Acceptance test with the new scheduler
Add a new github action task to run the acceptances test with the new scheduler
* Retry if the failure
* PR comments
- sort docker env vars.
- remove all non-docker related env vars.
- add what is missing.
For the .env file:
- sort the file to match the Configs.java lay out for better reading.
- get rid of env vars that are not used in docker
- get rid of env vars that have defaults, with the exception of var that are for scaling e.g. submitter_num_threads, worker related vars to prevent the env file from getting too large
- add a header to clarify when/where to add env vars to the file
For the docker compose file:
- sort the env vars alphabetically
- get rid of env vars that aren't used in that application
- add missing env vars into the worker application
This getting rid of scheduling with the scheduler app and the job submitter. It is replaced by a temporal workflow which will be responsible to schedule the syncs on time.
