* init
* bad copy/paste
* move to top level
* Revert "move to top level"
This reverts commit aca3534d38.
* attempt to wire up connector builder frontend to server
* copy from octaviacli
* fix connection to builder server
* update
* delete
* Update
* delete python-version
* Revert "delete python-version"
This reverts commit f9258a7755.
* setup python
* install python
* rename
* kube stuff
* Install python
* missing kube file
* rename
* Update files
* Update bumpversion
* install python
* try with different entrypoint
* rename container
* point to docker-compose.yaml file
* derp
* copy acceptance_test.sh
* copy from acceptance tests
* delete cruft
* update
* remove application env
* reset
* reset to master
* update
* skip comprehensive incremental tests
* Revert "skip comprehensive incremental tests"
This reverts commit 9cee657596.
* reset to master
* remove cruft
* delete superfluous steps
* update port to 8003
* reset to master
* Update publish docker
* move openapi spec to airbyte-connector-builder
* point to openapi spec
* dont expose the connector builder to localhost
* reset FE components to master
* Don't deploy the connector-builder
* Revert "Don't deploy the connector-builder"
This reverts commit 3d157494cf.
* Revert "Revert "Don't deploy the connector-builder""
This reverts commit beac3d48f0.
* comment out more things related to connector builder server
* more attempts at removing the connector builder
* comment out more things
* Apply suggestions from code review
Co-authored-by: Lake Mossman <lake@airbyte.io>
* Update airbyte-webapp/src/config/configProviders.ts
Co-authored-by: Lake Mossman <lake@airbyte.io>
* update
* rename
* indent
* Revert "move openapi spec to airbyte-connector-builder"
This reverts commit 57dda04723.
* Revert "rename"
This reverts commit b2d802b8fa.
* Revert "Revert "rename""
This reverts commit 91db24fd4a.
* point to wrong file in case it fixes the build
* point to right openapi file
* Revert "Revert "move openapi spec to airbyte-connector-builder""
This reverts commit e46a837454.
* point to moved file
* fix path
* Update from master
* newline
* Add failing test
* Revert "Add failing test"
This reverts commit ed9fea09b5.
* comment
* update commented requires
* Add a comment
* 2022
* rename to connector-builder-server
* typo
Co-authored-by: lmossman <lake@airbyte.io>
* Use Micronaut auto-detected environment for docker vs kubernetes choices
* Handle set, but blank env var
* Formatting
* Revert change
* Add default value
* Explicitly set WORKER_ENVIRONMENT for container orchestrator
* Make public
* Add sleep to allow Temporal cache to populate
Co-authored-by: Jimmy Ma <gosusnp@users.noreply.github.com>
* Cleanup application.yml
* Clean up cron application.yml and kube overlay
* Avoid flooding the logs with waiting for temporal namespace message
* Rm common app initializer
* Try to re-introduce an initializer
* Revert "Revert "Bmoric/temporal cleaning cron (#16414)" (#17192)"
This reverts commit 751667949e.
* Generate an artifact
* Change missing property from value to property
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Add the injected temporal client to airbyte-cron
* Update cron to make it work
* Remove useless Bean factory
* PR comments
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Improve conditional bean check
* Match existing Optional functionality
* Add notEquals check
* Add missing env var to Helm chart
* Fix typo
* Mark LogConfigs as Singleton
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Improve conditional bean check
* Match existing Optional functionality
* Add notEquals check
* Add missing env var to Helm chart
* Fix typo
* Mark LogConfigs as Singleton
* Env vars for log/state storage type
* Remove use of Optional in bean declarations
* Fix typo in config property name
* Support Temporal Cloud namespace
* Change to @Value
* Use correct value for conditional check
* Upgrade Micronaut
* Fix merge conflict
* Formatting
* Add missing env var
* Use sync task queue environment variable
* Handle sync task queue as set
* format and force http
* Handle case where sync task queue is empty
* Add correct path to config property
* Remove unused import
* Remove conflict
* Remove unused parameter
* Formatting
* Use pattern for condition process factory beans
* Cleanup
* PR feedback
* Revert hack for testing
* Fix temporal restart by status (#16447)
* Update application.yml
* Re add worker dep
* Add missing env var
* PR comments
* Bmoric/move temporal client (#16778)
* tmp
* tmp
* View diff
* Move part of the temporal client
* tmp
* copy Temporal Utils test
* Uniq Temporal Utils
* Uniq Temporal Workflow Utils
* Move CancellationHandler
* Move commons-temporal to being shared between platform, connector and CLI
* Rm worker dependency from cron
* Fix build
* Update with right value for cron.
* Fix dep conflict
Co-authored-by: jdpgrailsdev <jpearlin1@gmail.com>
Co-authored-by: pmossman <parker@airbyte.io>
Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io>
* WIP Convert airbyte-workers to Micronaut framework
* Rebase cleanup
* Fix broken tests
* Simplify code
* Support control vs data plane configuration
* make WORFKLOW_PROXY_CACHE non-static to avoid cacheing mocks in unit tests
* Formatting
* Pairing on Worker Micronaut (#16364)
* add RouteToSyncTaskQueue activity
* use new route activity in connection manager workflow
* format
* call router service for task queue
* Revert temporal proxy changes
* Formatting
* Fix default value
* register new route activity in test
* fix SyncWorkflowTest now that it isn't doing any routing
* Update dependencies
* More dependency updates
* Update dependencies
* Improve conditional bean check
* Match existing Optional functionality
* Add notEquals check
* Add missing env var to Helm chart
* Fix typo
* Mark LogConfigs as Singleton
* Env vars for log/state storage type
* Remove use of Optional in bean declarations
* Fix typo in config property name
* Support Temporal Cloud namespace
* Change to @Value
* Use correct value for conditional check
* Upgrade Micronaut
* Fix merge conflict
* Formatting
* Add missing env var
* Use sync task queue environment variable
* Handle sync task queue as set
* format and force http
* Handle case where sync task queue is empty
* Add correct path to config property
* Remove unused import
* Remove unused parameter
* Formatting
* Use pattern for condition process factory beans
* Cleanup
* PR feedback
* Revert hack for testing
Co-authored-by: pmossman <parker@airbyte.io>
* Add airbyte-cron to bumpversion
* Update airbyte-cron version to current
* Add workspace clean up job
* Add missing env var to docker-compose
* Update file deletion logging
* add new Configs for multi-cloud
* add api endpoints for setting workflow attempt id and createOrUpdate state
* update activities to call APIs instead of persistence
* workerApp refactor to separately initialize control or data plane dependencies
* modify syncWorkflow to call new activity that decides which task queue for data plane tasks
* misc to get build working
* move StateConverter to worker, so that server and worker can both access without needing to introduce any new dependencies
* update configs - remove extranneous helpers, clarify naming and comments, removed COMBINED value
* forgot to actually remove COMBINED enum value, this removes it
* add WorkerApp todo for breaking API Client into a scoped client
* rename decideTaskQueueActivity var to routeToTaskQueueActivity
* pr comments
* naming fix
* refactor secretHydrator instantiation
* WorkerApp PR feedback: move API client logic to separate class, use updated configs, etc
* add a RouterService class that is injected into RouteToTaskQueueActivityImpl
* AttemptApi cleanup and added unit test coverage
* fix confusion between AttemptId and AttemptNumber in new AttemptApi
* remove unused getDataPlaneSecretsHydrator
* remove unused import
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
* `LOG_CONNECTOR_MESSAGES` to enable logging of all messages seen by the worker in a sync
* also log initial state
* linting
* getState 2x for to match input file
* use `Jsons.serialize` utility
* add note to `developing-locally`
* fix bad merge
Today we are running into OOM exceptions with normalization. Normalization itself also inherits the destination's resource requirements. After work to bring destination memory usage down, this is no longer ideal, since most destinations use less memory than normalization needs.
This PR makes it possible to specify the general resource the normalization pod is provided via env vars.
Notes:
- Add env vars. Default to the various job main container resources if these are not set.
- Instead of using the destination's memory, use the normalization specify env vars.
* report failures for synchronous check/discover, refactor common logic
* allow null workspace, send spec errors
* add failure origin, format
* rm connector_type, fix failing tests
* add tests for other job types
* log instead of throw
* move swallow to common spot
* connector jobs use context instead of passing full config
* sync jobs use context instead of passing raw config
* fix failing test
* fix failing scheduler client test
* skeleton for reporting connector errors to sentry
* report on job failures instead of attempt failures
* report sync job failures with relevant metadata using JobErrorReporter
* send stack traces from python connectors to sentry
* test JobCreationAndStatusUpdate and JobErrorReporter
* logs
* refactor into helper, initial tests
* using sentry
* run format
* load reporting client from env
* load sentry dsn from env
* send java stack traces to sentry
* test sentryclient, refactor to use Hub instance
* ErrorReportingClient.report -> .reportJobFailureReason
* inject exception helper, test stack trace parse error tagging
* rm logs
* more stack trace tests
* remove logs
* fix failing tests
* rename ErrorReportingClient to JobErrorReportingClient
* rename vars in docker-compose
* Return an Optional instead of null when parsing stack traces
* dont remove airbyte prefix when setting release name
* from_trace_message static
* remove failureSummary from jobfailure input, get from Job
* send stacktrace string if we weren't able to parse
* set deployment mode tag
* update .env
* just log if something goes wrong
* use SHORT_ACTIVITY_OPTIONS on check connection activity so that it has retries
* retry workflow after delay instead of quarantining
* allow activity env vars to be configured in docker and kube
* add env var for workflow restart delay and refactor slightly
* update tests to handle new restart behavior
* update test name
* add empty env var values to .env files
* fail attempt before job in cleanJobState to prevent state machine failure
* change default value of max activity attempt retries from 10 to 5
* Add deployment mode to env shared with jobs
* Add adaptive runners
* Migrate postgres source to use adaptive runner
* Add an array of specs in docker image spec definition
* Add copyright
* Parse docker image spec with specs list
* Update spec yaml files
* Pass in DEPLOYMENT_MODE to docker compose file
* Revert "Parse docker image spec with specs list"
This reverts commit 8fe41dd3b7.
* Revert changes in docker image spec
* Read cloud specific spec files based on deployment mode
* Revert "Update spec yaml files"
This reverts commit 059f326432.
* Publish cloud spec file if necessary
* Fix upload script
* Move test files
* Update docker compose file
* Format code
* Add comment about spec filename
* Add unit tests
* Remove redundant jdbc acceptance test
When running `PostgresStrictEncryptJdbcSourceAcceptanceTest`, the `discover` method tests always fail because there are unexpected columns in the catalog:
- `wakeup_at`
- `last_visited_at`
- `last_comment_at`
These columns only exist in `PostgresJdbcSourceAcceptanceTest`. And this failure cannot be reproduced locally.
The hypothesis is that when the JDBC unit tests are run on CI, they are run in parallel, and the same testcontainer is used for both tests. That's why the strict encrypt test can discover columns from the oridinary unit test.
Given that the JDBC strict encrypt test is basically redundant, it is removed.
* Create interface, factory for metric client
* remove unused func
* change count val to use long
* PR fix
* otel metric client implementation
* merge conflicts resolve
* build fix
* add a test, moved version into deps catalog
* fix test
* sweep all scheduler application code and new-scheduler conditional logic
* remove airbyte-scheduler from deployments and docs
* format
* remove 'v2' from github actions
* add back scheduler in delete deployment command
* remove scheduler parameters from helm chart values
* add back job cleaner + test and add comment
* remove now-unused env vars from code and docs
* format
* remove feature flags from web backend connection handler as it is no longer needed
* remove feature flags from config api as it is now longer needed
* remove feature flags input from config api test
* format + shorter url
* remove scheduler parameters from helm chart readme
* Add Disable Failing Connections feature
* Rename and cleanup
* list jobs based off connection id
* Move variables to env config and update unit tests
* Fix env flag name
* Fix missing name changes
* Add comments to unit test
* Address PR comments
* Support multiple config types
* Update unit tests
* Remove the attemptId notion in the connectionManagerWorkflow (#10780)
This is removing the attemptId from the create attempt activity to replace it with the attemptNumber. This will be modified in the workflow in a later commit.
* Revert "Remove the attemptId notion in the connectionManagerWorkflow (#10780)" (#11057)
This reverts commit 99338c852a.
* Revert "Revert "Remove the attemptId notion in the connectionManagerWorkflow (#10780)" (#11057)" (#11073)
This reverts commit 892dc7ec66.
* Revert "Revert "Revert "Remove the attemptId notion in the connectionManagerWorkflow (#10780)" (#11057)" (#11073)" (#11081)
This reverts commit e27bb74050.
* Add Disable Failing Connections feature
* Rename and cleanup
* Fix rebase
* only disable if first job is older than max days
* Return boolean for activity
* Return boolean for activity
* Add unit tests for ConnectionManagerWorkflow
* Utilize object output for activity and ignore non success or failed runs
* Utilize object output for activity and ignore non success or failed runs
Co-authored-by: Benoit Moriceau <benoit@airbyte.io>
* Add a job notification
The new scheduler was missing a notification step after the job is done.
This is needed in order to report the number of record of a sync.
* Acceptance test with the new scheduler
Add a new github action task to run the acceptances test with the new scheduler
* Retry if the failure
* PR comments
- sort docker env vars.
- remove all non-docker related env vars.
- add what is missing.
For the .env file:
- sort the file to match the Configs.java lay out for better reading.
- get rid of env vars that are not used in docker
- get rid of env vars that have defaults, with the exception of var that are for scaling e.g. submitter_num_threads, worker related vars to prevent the env file from getting too large
- add a header to clarify when/where to add env vars to the file
For the docker compose file:
- sort the env vars alphabetically
- get rid of env vars that aren't used in that application
- add missing env vars into the worker application
This getting rid of scheduling with the scheduler app and the job submitter. It is replaced by a temporal workflow which will be responsible to schedule the syncs on time.
- Add the CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION and JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION. These are env vars that will determine if the database is ready for an application to start.
- Add the CONFIGS_DATABASE_INITIALIZATION_TIMEOUT_MS and the JOBS_DATABASE_INITIALIZATION_TIMEOUT_MS env vars to determine how long an application should wait for the DB before giving up.
- Create the MinimumFlywayMigrationVersionCheck class. This class contains all the assertions to check if 1) a database is initialised. 2) a database meets the minimum migration version.
- Remove all set up operations from the ServerApp. Use MinimumFlywayMigrationVersionCheck operations instead.
- I also had to modify the Databases and BaseDatabaseInstance classes to support connecting to a database with timeouts. We would previously try forever.
- Add Bootloader to the relevant docker files and Kube files.
- Clean up the migration acceptance tests so it's clear what is happening.
* Update platform containers to use non-root users
* Update kube template for the webapp container to use port 8080
After having updated the webbapp nginx image to expose port 8080 instead of 80
* missing 80 -> 8080 changes
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
* Rename GcsStorageBucket to GcsLogBucket.
* Update all references to GCP_STORAGE_BUCKET to GCS_LOG_BUCKET.
* Undo this for configuration files for older Airbyte versions.
* Clean up Job env vars. (#8462)
* Rename MAX_SYNC_JOB_ATTEMPTS to SYNC_JOB_MAX_ATTEMPTS.
* Rename MAX_SYNC_TIMEOUT_DAYS to SYNC_JOB_MAX_TIMEOUT_DAYS.
* Rename WORKER_POD_TOLERATIONS to JOB_POD_TOLERATIONS.
* Rename WORKER_POD_NODE_SELECTORS to JOB_POD_NODE_SELECTORS.
* Rename JOB_IMAGE_PULL_POLICY to JOB_POD_MAIN_CONTAINER_IMAGE_PULL_POLICY.
* Rename JOBS_IMAGE_PULL_SECRET to JOB_POD_MAIN_CONTAINER_IMAGE_PULL_SECRET.
* Rename JOB_SOCAT_IMAGE to JOB_POD_SOCAT_IMAGE.
* Rename JOB_BUSYBOX_IMAGE to JOB_POD_BUSYBOX_IMAGE.
* Rename JOB_CURL_IMAGE to JOB_POD_CURL_IMAGE.
* Rename KUBE_NAMESPACE to JOB_POD_KUBE_NAMESPACE.
* Rename RESOURCE_CPU_REQUEST to JOB_POD_MAIN_CONTAINER_CPU_REQUEST.
* Rename RESOURCE_CPU_LIMIT to JOB_POD_MAIN_CONTAINER_CPU_LIMIT.
* Rename RESOURCE_MEMORY_REQUEST to JOB_POD_MAIN_CONTAINER_MEMORY_REQUEST.
* Rename RESOURCE_MEMORY_LIMIT to JOB_POD_MAIN_CONTAINER_MEMORY_LIMIT.
* Remove worker suffix from created pods to reduce confusion with actual worker pods.
* Use sync instead of worker to name job pods.
* make spec field required
* remove spec backfill logic
* remove usages of specFetcher.getSpec()
* remove unused code and the caching scheduler clients
* fix tests to work with fetching specs from definitions
* fetch spec from definition in config repository and fix RunMigrationTest
* remove unused SpecFetcher methods/tests
* run gw format
* run gw format
* undo change to main method
* add back newlines
* set additional properties to true on destination definition
* remove now-unused VERSION_0_32_0_FORCE_UPGRADE env var
This is a custom auto-setup script for the temporal environment. Unfortunately there is no other way properly update the DB without copy pasting parts of the temporal auto-setup script. Ideally temporal would provide a dedicated container for it DB but it is not the case right now.
* Add migration to create latest state table
* Log migration name
* Expose db variables to airbyte-db
* Implement migration
* Fix migration test
* temp
* Rebase on master
* Save state in temporal (#7253)
* Copy state to airbyte_configs table
* Add standard sync state
* Move state methods to config repository
* Add unit tests
* Fix unit tests
* Register standard sync state in migration
* Add comment
* Use config model instead of json node
* Add comments
* Remove unnecessary method
* Fix migration query
* Remove unused config database
* Move persist statement and log the call
* Update dev doc
* Add unit tests for sync workflow
Co-authored-by: Charles <giardina.charles@gmail.com>
