* init
* bad copy/paste
* move to top level
* Revert "move to top level"
This reverts commit aca3534d38.
* attempt to wire up connector builder frontend to server
* copy from octaviacli
* fix connection to builder server
* update
* delete
* Update
* delete python-version
* Revert "delete python-version"
This reverts commit f9258a7755.
* setup python
* install python
* rename
* kube stuff
* Install python
* missing kube file
* rename
* Update files
* Update bumpversion
* install python
* try with different entrypoint
* rename container
* point to docker-compose.yaml file
* derp
* copy acceptance_test.sh
* copy from acceptance tests
* delete cruft
* update
* remove application env
* reset
* reset to master
* update
* skip comprehensive incremental tests
* Revert "skip comprehensive incremental tests"
This reverts commit 9cee657596.
* reset to master
* remove cruft
* delete superfluous steps
* update port to 8003
* reset to master
* Update publish docker
* move openapi spec to airbyte-connector-builder
* point to openapi spec
* dont expose the connector builder to localhost
* reset FE components to master
* Don't deploy the connector-builder
* Revert "Don't deploy the connector-builder"
This reverts commit 3d157494cf.
* Revert "Revert "Don't deploy the connector-builder""
This reverts commit beac3d48f0.
* comment out more things related to connector builder server
* more attempts at removing the connector builder
* comment out more things
* Apply suggestions from code review
Co-authored-by: Lake Mossman <lake@airbyte.io>
* Update airbyte-webapp/src/config/configProviders.ts
Co-authored-by: Lake Mossman <lake@airbyte.io>
* update
* rename
* indent
* Revert "move openapi spec to airbyte-connector-builder"
This reverts commit 57dda04723.
* Revert "rename"
This reverts commit b2d802b8fa.
* Revert "Revert "rename""
This reverts commit 91db24fd4a.
* point to wrong file in case it fixes the build
* point to right openapi file
* Revert "Revert "move openapi spec to airbyte-connector-builder""
This reverts commit e46a837454.
* point to moved file
* fix path
* Update from master
* newline
* Add failing test
* Revert "Add failing test"
This reverts commit ed9fea09b5.
* comment
* update commented requires
* Add a comment
* 2022
* rename to connector-builder-server
* typo
Co-authored-by: lmossman <lake@airbyte.io>
* sweep all scheduler application code and new-scheduler conditional logic
* remove airbyte-scheduler from deployments and docs
* format
* remove 'v2' from github actions
* add back scheduler in delete deployment command
* remove scheduler parameters from helm chart values
* add back job cleaner + test and add comment
* remove now-unused env vars from code and docs
* format
* remove feature flags from web backend connection handler as it is no longer needed
* remove feature flags from config api as it is now longer needed
* remove feature flags input from config api test
* format + shorter url
* remove scheduler parameters from helm chart readme
- Add the CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION and JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION. These are env vars that will determine if the database is ready for an application to start.
- Add the CONFIGS_DATABASE_INITIALIZATION_TIMEOUT_MS and the JOBS_DATABASE_INITIALIZATION_TIMEOUT_MS env vars to determine how long an application should wait for the DB before giving up.
- Create the MinimumFlywayMigrationVersionCheck class. This class contains all the assertions to check if 1) a database is initialised. 2) a database meets the minimum migration version.
- Remove all set up operations from the ServerApp. Use MinimumFlywayMigrationVersionCheck operations instead.
- I also had to modify the Databases and BaseDatabaseInstance classes to support connecting to a database with timeouts. We would previously try forever.
- Add Bootloader to the relevant docker files and Kube files.
- Clean up the migration acceptance tests so it's clear what is happening.
* docker-compose split of scheduler and worker
* fix heartbeat location bug + add support for kubernetes
* use two workers in integration tests
* capture logs in AirbyteTestContainer
* add waiting
* rename to make it easier to review
* rename module
* fix remaining conflicts
* allow configuring max workers of each type and document usage
* fix build
* remove comment
* add worker resource requiremetns
* try to fix for connector build
* fix regression in biuld
* add env comments for SUBMITTER_NUM_THREADS
* Update airbyte-workers/src/main/java/io/airbyte/workers/WorkerApp.java
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update airbyte-workers/src/main/java/io/airbyte/workers/temporal/TemporalPool.java
Co-authored-by: Davin Chia <davinchia@gmail.com>
* merge temporalpool into workerapp
* output docker system info
* move check to before
* remove unnecessary parts of the patch
* could this be the problem? i thought i added this
* show disk usage
* add print statements
* add pruning
* fix prune option
* use force
Co-authored-by: Davin Chia <davinchia@gmail.com>
# Summary
- A follow-up PR for #5543.
- This PR separates the `airbyte-db` project to two modules:
- `lib` is the original `airbyte-db`.
- `jooq` is for jOOQ code generation.
- This is necessary because the jOOQ generator requires a custom database implementation that can run Flyway migration. So the code generator logic needs to depend on the compilation of the original `airbyte-db` project.
# Commits
* Separate db to lib and jooq modules
* Update dependencies
* Add jobs db migrator test
* Fix compose build
* Add migration dev center
* Add schema dump task
* Update airbyte-db/lib/README.md
* Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update readme
* Remove bom dependency
* Update readme
* Use jooq code in db config persistence
* Remove AirbyteConfigsTable
Co-authored-by: Davin Chia <davinchia@gmail.com>
## What
- This is the first PR for #4890.
- This PR does not remove the config volume.
- This PR does not mount the directories for the local connectors.
- Resolves#5373.
## How
- Previously the seed container copies the configs to the storage root, it may take some time for the operation to complete and for the `CONFIG_DIR` to show up. So we cannot infer anything based on the existence of this directory. Now this seed generation step has been removed. So we can tell immediately whether `CONFIG_DIR` exists or not.
- If `CONFIG_DIR` exists, it means the user has just migrated Airbyte from an old version that uses this file system config persistence.
- Otherwise, we can seed the config persistence from the YAML files.
* introduce automatic migration at the startup of server
* handle versions with non-zero patch
* it works!!!
* add dummy data
* cleanup orphan configs
* add more assertions
* format + add comments
* move migration acceptance test to acceptance test directory
* add automatic migration test to the build
* address review comments
* missed out on these
* format
* add more assertions
* format
* fix test
* format
* use default port for temporal
* move seed to server + introduce atomice replacement for config
* make tests better
* remove unwanted changes
* move atomic replacement logic behind persistence + pass path to latest seeds
* format
* update seeds
* review comments
* update seeds
* merge latest seeds with configs
* fix bug around latest seed
* update seed
* update seed
* seeds should be populated by separate container
* address review comment + change latest definition url
* update seeds
* format
* update seed references
* update seed
* update seed
* update seed
* update seed references
* update seed references + add Migration Acceptance Test
* update seed container in kube + disable automatic migration for kube + update docs
* update docs
* address review comments from Michel
* update doc
* temporary commmit to see if build becomes green
* delete seeds from airbyte config + undo temp commit
Splits scheduler into 4 submodules:
* airbyte-scheduler:app - this is the "scheduler". the thing that runs and schedules work. (no other module should depend on this one.)
*airbyte-scheduler:client - this is the module that anything that needs to submit work to the schedule should depend on. (any module is allowed to depend on this one.)
* airbyte-scheduler:models - pojos / structs that are used as part of interfaces across modules (both internal and external to the scheduler). (any module is allowed to depend on this one.)
* airbyte-scheduler:persistence - code for interacting with the underlying scheduler persistence (database). since the client and main interact via writing to the database both the client and main depend on this. (right now anything that invokes the scheduler client also depends on this unfortunately because the client requires the database to be instantiated. ideally this would only be depended upon by the app (and a server if we ever had one))
* fix mounting
* svc accounts and other fixes
* remove redundant sync closure call
* switch back to using dev tags and format
* do not overwrite in initContainers
* rename to scheduler-service-account
* add documentation
* fix lack of cp -n on busybox
* use kubectl run, still needs affinity
* add affinity
* downgrade kubectl on scheduler to match docker-desktop latest
* fix overrides
* THE MISISNG PIECE
* add attach permissions
* fmt
* update kube documentation
* put docs on the docs site instead
* Adds a seed container that we copy the default configurations onto. Then we mount a volume (data) onto the path where the seed data has been copied on this image. When the volume is created, this mount configuration will copy the contents from the seed container into the volume. On subsequent start ups it will be a no-op, which is exactly what we want.
