* Add migration to create latest state table
* Log migration name
* Expose db variables to airbyte-db
* Implement migration
* Fix migration test
* temp
* Rebase on master
* Save state in temporal (#7253)
* Copy state to airbyte_configs table
* Add standard sync state
* Move state methods to config repository
* Add unit tests
* Fix unit tests
* Register standard sync state in migration
* Add comment
* Use config model instead of json node
* Add comments
* Remove unnecessary method
* Fix migration query
* Remove unused config database
* Move persist statement and log the call
* Update dev doc
* Add unit tests for sync workflow
Co-authored-by: Charles <giardina.charles@gmail.com>
* test exposing secrets in configrepo
* fix local persistence sql
* working propagation, just without check/discover replacements and without feature flagging
* switch if statement
* set up secret persistence for google secrets manager
* add ttl-based secret persistence for check/discover usage in the future
* set up check/discover to pass around necessary parts
* Revert "set up check/discover to pass around necessary parts"
This reverts commit 489d2d5f5d.
* working updates + check/discover operations
* fix additional configs created on deletion
* clean up docker compose file
* finish up configrepo
* make api path optional
* clean up schedulerapp and local testing persistence
* make optional in the worker app
* add rest of feature flagging
* fmt
* remove completed todo
* fix refactoring typo
* fix another refactoring typo
* fix compilation error in test case
* fix tests
* final cleanups
* fix conditional
* address a couple of things
* add hydrator interface
* add replaceAllConfigs
* specfetcher handling
* fix constructor
* fix test
* fix typo
* fix merge build error
* remove extra config
* fix integration test
* fix final piece
* docker-compose split of scheduler and worker
* fix heartbeat location bug + add support for kubernetes
* use two workers in integration tests
* capture logs in AirbyteTestContainer
* add waiting
* rename to make it easier to review
* rename module
* fix remaining conflicts
* allow configuring max workers of each type and document usage
* fix build
* remove comment
* add worker resource requiremetns
* try to fix for connector build
* fix regression in biuld
* add env comments for SUBMITTER_NUM_THREADS
* Update airbyte-workers/src/main/java/io/airbyte/workers/WorkerApp.java
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update airbyte-workers/src/main/java/io/airbyte/workers/temporal/TemporalPool.java
Co-authored-by: Davin Chia <davinchia@gmail.com>
* merge temporalpool into workerapp
* output docker system info
* move check to before
* remove unnecessary parts of the patch
* could this be the problem? i thought i added this
* show disk usage
* add print statements
* add pruning
* fix prune option
* use force
Co-authored-by: Davin Chia <davinchia@gmail.com>
* allow scheduler and server to run on separate nodes
* re-add workspace mount for docker compose only
* remove stacktrace printing
* add affinity testing components
* reorder mounts
* just try a two node cluster
* add waiting log line
* seed containers are now axed
* remove unused var
* add comment
* rename to integration-test
* Implement database migrator
* Add unit tests
* Add RUN_FLYWAY_MIGRATION variable
* Run flyway migration in server
* Add db migration info api
* Add db migration migrate api
* Add unit test
* Remove base airbyte migration
* Implement migration dev helper
* Dry and format code
* Fix url
* Use camel case
* Add db migration page
* Add button to run migration
* Update migration table
* Fix resource warning
* Update readme
* Revert package-lock changes
* Update readme
* Address simple frontend review comments
* Add java migration template (not completed yet)
* Add method to generate migration file
* Set up jooq code generation
* Check in generated code
* Move generated code to build directory
* Exclude db dev center methods in gradle
* Update airbyte-db/README.md
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Mark getMigrator as private
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Address review comments
* Format code
* Fix format output column name
* Remove config persistence builder
* Remove dumpSchemaToFile method
* Run baseline in server
* Rename info to list
Co-authored-by: Charles <giardina.charles@gmail.com>
* Rename executeDbMigrationInfo to listMigrations
Co-authored-by: Charles <giardina.charles@gmail.com>
* Rename RUN_FLYWAY_MIGRATION
* Clean up migration apis
* Remove redundant version comparison
* Refactor db migrator
* Add migration file location parameter back
This is necessary because other databases may exist in a different module and follow different patterns.
* Fix build
* Generate jooq code in gradle
* Remove frontend changes
* Remove testing migration
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Charles <giardina.charles@gmail.com>
See https://airbytehq.slack.com/archives/C019WEENQRM/p1629383779144300 for more details.
TLDR:
Retrying within an attempt is confusing for a UX perspective (we already have attempts), does not provides more value from a retries perspective (we already have attempts) and inefficient from a resource perspective (hogs up api limits and compute/memory resources).
Follow up ticket: #5571
## What
- This is the first PR for #4890.
- This PR does not remove the config volume.
- This PR does not mount the directories for the local connectors.
- Resolves#5373.
## How
- Previously the seed container copies the configs to the storage root, it may take some time for the operation to complete and for the `CONFIG_DIR` to show up. So we cannot infer anything based on the existence of this directory. Now this seed generation step has been removed. So we can tell immediately whether `CONFIG_DIR` exists or not.
- If `CONFIG_DIR` exists, it means the user has just migrated Airbyte from an old version that uses this file system config persistence.
- Otherwise, we can seed the config persistence from the YAML files.
* Implement db config persistence
* Fix database readiness check
* Reduce logging noise
* Setup config database in config persistence factory
* Update documentation
* Load seed from yaml files
* Refactor config persistence factory
* Add one more test to mimic migration
* Remove unnecessary changes
* Run code formatter
* Update placeholder env values
* Set default config database parameters in docker compose
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
* Default setupDatabase to false
* Rename variable
* Set default config db parameters for server
* Remove config db parameters from the env file
* Remove unnecessary environment statements
* Hide config persistence factory (#4772)
* Remove CONFIG_DATABASE_HOST
* Use builder in the test
* Simplify config persistence builder
* Clarify config db connection readiness
* Format code
* Add logging
* Fix typo
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
* Add a config_id only index
* Reuse record insertion code
* Add id field name to config schema
* Support data loading from legacy config schemas
* Log missing logs in migration test
* Move airbyte configs table to separate directory
* Update exception message
* Dump specific tables from the job database
* Remove postgres specific uuid extension
* Comment out future branch
* Default configs db variables to empty
When defaulting them to the jobs db variables, it somehow does not work.
* Log inserted config records
* Log all db write operations
* Add back config db variables in env file to mute warnings
* Log connection exception to debug flaky e2e test
* Leave config db variables empty
`.env` file does not support variable expansion.
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Co-authored-by: Charles <giardina.charles@gmail.com>
* introduce automatic migration at the startup of server
* handle versions with non-zero patch
* it works!!!
* add dummy data
* cleanup orphan configs
* add more assertions
* format + add comments
* move migration acceptance test to acceptance test directory
* add automatic migration test to the build
* address review comments
* missed out on these
* format
* add more assertions
* format
* fix test
* format
* use default port for temporal
* move seed to server + introduce atomice replacement for config
* make tests better
* remove unwanted changes
* move atomic replacement logic behind persistence + pass path to latest seeds
* format
* update seeds
* review comments
* update seeds
* merge latest seeds with configs
* fix bug around latest seed
* update seed
* update seed
* seeds should be populated by separate container
* address review comment + change latest definition url
* update seeds
* format
* update seed references
* update seed
* update seed
* update seed
* update seed references
* update seed references + add Migration Acceptance Test
* update seed container in kube + disable automatic migration for kube + update docs
* update docs
* address review comments from Michel
* update doc
* temporary commmit to see if build becomes green
* delete seeds from airbyte config + undo temp commit
Use Log4j2 appender to support routing logs to S3.
Create LogClient to support reading from S3.
Some clean up of the Log4j2 xml variables.
Several dependency changes to be more explicit when configuring jackson.
* initial attempt at generating local kube setup from docker compose
* update current state
* mounts not working
* working mounts, failing cors
* working UI
* add remaining todos
* update todos
* A
* use kustomize to select image versions
* kube process builder factory
* fix misalignment
* don't allow any retries for requested jobs
* fix log waiting and path handling
* update todos
* local volume handling
* propagate return code correctly
* update todos
* update docs
* fmt
* add to docs
* fix conflicting config file bug
* fmt
* delete unused file
* remove comment
* add job id and attempt as inputs
* rename to WorkerEnvironment
* fix example custom overlay
* less trigger-happy docs
* rename mounts
* show local csv as not working in kube in the docs
* use config maps for everything
* fix paths
* fix build
* fix stripe integration test usage
* fix papercups on kube
* Give names to docker volumes used in airbyte
* Add tutorial on config persistence (#993 and #994)
* Include output results in markdown file
* Update outputs of notebook
* Update docs
* Update workspace numbers
* Tweaks from review
