* add specs module with logic to fetch specs on build
* format + build and add gradle dependency for new script
* check seed file for existing specs + refactor
* add tests + a bit more refactoring
* run gw format
* update yaml config persistence to merge specs into definitions
* add comment
* delete secrets migration to be consistent with master
* add dep
* add tests for GcsBucketSpecFetcher
* get rid of static block + format
* DRY up parse call
* add GCS details to comment
* formatting + fix test
* update comment
* do not format seed specs files
* change signature of run to allow cloud to reuse this script
* run gw format
* revert commits that change signature of run
* fix comment typo
Co-authored-by: Davin Chia <davinchia@gmail.com>
* rename enum to be distinct from the enum in cloud
* add missing dependencies between modules
* add readme for seed connector spec generator
* reword
* reference readme in comment
* ignore 'spec' field in newFields logic
Co-authored-by: Davin Chia <davinchia@gmail.com>
* store spec in db
* update tests
* run gw format
* add TODOs
* add lmossman to TODOs
* run gw format
* remove redundant DockerImageValidator
* run gw format
* Add migration to create latest state table
* Log migration name
* Expose db variables to airbyte-db
* Implement migration
* Fix migration test
* temp
* Rebase on master
* Save state in temporal (#7253)
* Copy state to airbyte_configs table
* Add standard sync state
* Move state methods to config repository
* Add unit tests
* Fix unit tests
* Register standard sync state in migration
* Add comment
* Use config model instead of json node
* Add comments
* Remove unnecessary method
* Fix migration query
* Remove unused config database
* Move persist statement and log the call
* Update dev doc
* Add unit tests for sync workflow
Co-authored-by: Charles <giardina.charles@gmail.com>
* try fetching specs from definitions first
* refactor specFetcher and update tests
* run gradle format
* format again
* fix comment formatting
* fix test
* merge comment lines into single line
* move duplicate job metadata mocking logic to shared static method
* add todo
* formatting
* use local var and clone
* run gw format
* add todo
* skip spec fetcher in docker image validator and update todos
* run gw format
* adding google sheets oauth flow to server
* fix oauth type in protocol yaml
* bump sheets version in definitions
* added GDrive scope
* update sheets to master changes
* update protocol incl. cdk
* protocol typing for oauth rootobject
* format
* Add API endpoints for connections, sources, and destinations
* Updating tests
* Tests for source and destination handlers
* Reformat code
* Update airbyte-server/src/main/java/io/airbyte/server/handlers/helpers/ConnectionMatcher.java
Co-authored-by: LiRen Tu <tuliren@gmail.com>
* Adding tests
* Format code
Co-authored-by: LiRen Tu <tuliren@gmail.com>
* test exposing secrets in configrepo
* fix local persistence sql
* working propagation, just without check/discover replacements and without feature flagging
* switch if statement
* set up secret persistence for google secrets manager
* add ttl-based secret persistence for check/discover usage in the future
* set up check/discover to pass around necessary parts
* Revert "set up check/discover to pass around necessary parts"
This reverts commit 489d2d5f5d.
* working updates + check/discover operations
* fix additional configs created on deletion
* clean up docker compose file
* finish up configrepo
* make api path optional
* clean up schedulerapp and local testing persistence
* make optional in the worker app
* add rest of feature flagging
* fmt
* remove completed todo
* fix refactoring typo
* fix another refactoring typo
* fix compilation error in test case
* fix tests
* final cleanups
* fix conditional
* address a couple of things
* add hydrator interface
* add replaceAllConfigs
* specfetcher handling
* fix constructor
* fix test
* fix typo
* fix merge build error
* remove extra config
* fix integration test
* fix final piece
* add some initial tests and todos
* add rest of files
* format
* make json secrets processor availabe as helper functions, not injected classes
* get tests working for splitting
* save state
* complete update
* combine working
* format
* add separate test cases
* add oneof test case and fix behavior
* combine support for arrays
* working string arrays
* test object arrays
* add unsupported test case
* clean up
* add airbyte_ prefix
* add ability to test individual version bumping
* add read only persistence exposure
* version handling partially
* version handling fully working yay
* remove comment typo
* add test
* throw an exception when the secret can't be found and test for it
* test for mutations
* remove magic strings
* extract coordinate generation into own functions
* misc cleanup
* massive simplification
* mild cleanup
* remove json path
* fix test run and clean up more
* format
* misc minor cleanups
* remove giant if with filter
* significantly clean up split function
* significant cleanups and add array of oneof test case
* support nestedOneOf test case
* add schema validation test and fix failing cases
* add javadocs
* clarify coordinate conversion
* make sense
* fix build
* revert json secrets processing static-ification to fix build
* add in method
* Remove json schema valication in file-based migration
* Revert attempts schema
* Remove extra log line
* Format code
* Remove invalid input record test
* Revert #45ad88a40
* Run flyway migration only if the file-based migration succeeds
* Load latest configs after flyway migration
* Add back json schema validation in file-based migration
* Format code
* switch most usages of writing sources to using specs
* fix other usages
* fix test
* only wait on the server in the scheduler, not the worker
* fix
* rephrase sanity check and remove stdout
* New abstraction for NoSql database sources
* New MongoDbSource: partial impl
* Added MongoDataType
* Improved MongoDatabase and fixed read method
* code review changes;
* merge clean up;
* Renamed NoSqlDatabase to AbstractDatabase
* formatter changes;
* code review changes: changed mongodb-new to mongodb-v2; left only new connector info in all docs
* code review changes: changed mongodb-new to mongodb-v2; left only new connector info in all docs
* updated spec.json and toDatabaseConfig() method
* updated doc accordingly to spec.json changes
Co-authored-by: Iryna Kruk <iryna.o.kruk@globallogic.com>
A few days ago we removed the workflow volume from the Kubernetes deployment in order to simplify the set up. This also lets us do away with the workspace PVC.
In theory this isn't needed since the volume is mainly used for logs and our Kube deployment logs out to the Cloud storage.
In the process of doing so, we realised the volume is used to store the temporal workflow id that is later used to cancel the workflow. Thus cancellations stopped working.
This PR:
Adds a migration to add the temporalWorkflowId column to the Attempts table. Exposes various persistent methods for this.
Modify temporal to store the workflow id in this column. Modify cancellation to retrieve the workflow id from the table.
Things to call out:
This approach means the worker now requires access to the jobs DB. I think this is reasonable.
Some tests are disabled since we haven't really stabilised the Flyway + older file-base migrations yet. Follow up ticket has been created (Update config and job persistence unit tests to run migrations #5857) and Liren is working on this.
