* pass workspace id to sync workflow and use it to selectively enable field selection
* fix tests around workspace id in job creation
* make sure field selection environment variables get passed through properly
* clean up handling around field selection flags
* debug logging for field selection
* properly handle empty field selection feature flag
* fix pmd
* actually fix pmd
Implement the persistence layer changes following #19191.
This PR handles writing and reading stats to the new stream stat_table and columns in the existing sync_stats table.
At the same time we introduce upserts of stats records - i.e. merge updates into a single record - in preparation for real time stats updates vs the current approach where a new stat record is always written.
There will be two remaining PRs after this:
- First PR will be to fully wire up and test the API.
- Second PR will be to actually save stats while jobs are running.
* Adds javadoc comments to files that touch writing data from destination connector into the destination
* Updates comments for copyTableQuery
* Fixed linter issues
* Fixed remaining formatting issues with javadoc paragraphs
* Make sure decimal numbers are serialized to json as a plain string rather than scientific notation, as it breaks deserialization of integers
* bump version and release note
* Correct PR number on release note
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* implement column filtering in the replication workflow
* fixes to column selection in replication workflow
* add a basic acceptance test for column selection
* make CI acceptance tests run with new field selection flag enabled
* fix format
* readability improvements around columns selection tests and other small fixes
* wip; add micronaut
* add additional json deserializer methods
* wip; converting to micronaut
* misc cleanup
* wip; broken
* wip; still broken
* wip
* formatting
* minor code cleanup; no actual changes
* wip; still broken
* removed commented out code; no longer broken
* wip; clean-up micronaut code
* cleanup; format
* fix pmd issues
* remove unused file
* init ApplicationTest
* edited link (#19444)
* move 'Example values' into intl (#19446)
* Revert "Update action.yml (#19416)" (#19450)
This reverts commit 78fb528a9a.
* Notifications Workflow (#18735)
* notification workflow
* Bmoric/remove unused code (#19188)
* Tmp
* Move when the deletion is performed
* Re-enable disable test
* PR comments
* Use cancel
* rename
* Fix test and version check position
* remove unused temporal deletion code
* Remove false todo
* Rm repeated test
* Rm unused import
* Make sure that long running activity are not retried (#19452)
* Parse list of dicts in json_schema_helper.find_nodes() (#19386)
* Get test on nested list/dict passing - use index to query next object for list
* Fix flakecheck
* Test that get_node provides correct value
* Improve test and test cases
* Rewrite method for better comprehension
* Add test for base-level key. Rewrite method for comprehension and handling this case
* adding tests
* fix test
* formatting
* remove unused dependencies
* add missing test resource
* format
* add missing test resource (real)
* format
* add back protocol-models dep
* format
* pr feedback; log stacktrace
Co-authored-by: Sophia Wiley <106352739+sophia-wiley@users.noreply.github.com>
Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: Topher Lubaway <asimplechris@gmail.com>
Co-authored-by: Anne <102554163+alovew@users.noreply.github.com>
Co-authored-by: Benoit Moriceau <benoit@airbyte.io>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
* Add helper class to check protocol version range
* Check ProtocolVersion when modifying a destination definition
* Check ProtocolVersion when modifying a source definition
* Format
* Add UnsupportedProtocolVersion exception
* Rewrite AirbyteProtocolVersionRange as a record
* Format
* Rename exception
* Changes method from returning null to throwing Exception to match method structure
* Added javadoc explaining that KeyPair should not be returning null
* Moved ConnectionErrorException to commons package
* Adds error handling for SshWrappedDestinations and markdown to include changes
* Bumps version numbers for Postgres, MySQL, MSSQL
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* wip
* handle webhook configs in workspaces endpoint and split/hydrate secrets
* style improvements to documentation around webhook configs
* Clarify documentation around webhook auth tokens
* More documentation clarification around webhook configs
* Format.
* unit test coverage for webhook config handling
* use common json parsing libraries around webhook configs
* clean up around testing webhook operation configs
Co-authored-by: Davin Chia <davinchia@gmail.com>
* More AirbyteVersion references fix
* Propagate protocol version from sourceDef to SchedulerClient
* Propagate protocol version to LauncherConfig
* Add VersionedMigratorFactory
* Update VersionedAirbyteStreamFactory
* Fix Version Json serialization/deserialization
* Plug message migration in CheckConnection for Sources
* Add Airbyte Protocol Range configs
* Refactor metadata read/write
* Add ProtocolVersion Min/Max get/set to JobsPersistence
* Store the supported protocol version range in airbyte_metadata
* Use defaults in EnvConfigs instead of .env
* start implementation of new persistence method
* add includingJobId and totalJobCount to job list request
* format
* update local openapi as well
* refactor queries into JOOQ and return empty list if target job cannot be found
* fix descriptions and undo changes from other branch
* switch including job to starting job
* fix job history handler tests
* rewrite jobs subqueries in jooq
* fix multiple config type querying
* remove unnecessary casts
* switch back to 'including' and return multiple of page size necessary to include job
* undo webapp changes
* fix test description
* format
* Update protocol version from actor defs API operations
* Implement default airbyte protocol version support
* Add version parsing
* Add acceptance tests
* Fix Acceptance Tests
* format
* Make test package private
* `LOG_CONNECTOR_MESSAGES` to enable logging of all messages seen by the worker in a sync
* also log initial state
* linting
* getState 2x for to match input file
* use `Jsons.serialize` utility
* add note to `developing-locally`
* fix bad merge
* sweep all scheduler application code and new-scheduler conditional logic
* remove airbyte-scheduler from deployments and docs
* format
* remove 'v2' from github actions
* add back scheduler in delete deployment command
* remove scheduler parameters from helm chart values
* add back job cleaner + test and add comment
* remove now-unused env vars from code and docs
* format
* remove feature flags from web backend connection handler as it is no longer needed
* remove feature flags from config api as it is now longer needed
* remove feature flags input from config api test
* format + shorter url
* remove scheduler parameters from helm chart readme
