* Update StateDB to support per Stream states.
* Add `StateType` type
* Add `steam_name`, `namespace` and `type` to `state` table.
* Set the default StateType to LEGACY
* Postgres Source: fixed truncated precision if the value of the millisecond or second is 0
* check CI with 1.15.3 testcontainer
* check CI with 1.15.3 testcontainer
* returned latest version of testcontainer
* fixed checkstyle
* fixed checkstyle
* returned latest testcontainer version
* updated CHANGELOG
* bump version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Switch to measure max row byte size
* Reduce fetch size change logs
* Update unit tests
* Determine jdbc buffer size based on max memory
* Bump postgres version
* Bump postgres version
* Bump mysql version
* Bump mssql version
* Format java code
* Increase hikari connection timeout
* Update data source default parameters
* auto-bump connector version
* Mark postgres 0.4.21 as not published
* Revert "Bump mysql version"
This reverts commit ad9135258c.
* Fix unit test
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
## What
Finale of https://github.com/airbytehq/airbyte/pull/13122.
We've renamed all directories in previous PRs. Here we remove the fat jar configuration and add publishing to all subprojects.
Explanation for what is happening:
Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
## How
* Remove fat jar configuration.
* Add publishing to all subprojects.
## What
Part 2 of https://github.com/airbytehq/airbyte/pull/13122.
Follow up to #13476 .
Explanation for what is happening:
Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
* Rename airbyte-config:models to airbyte-config:config-models.
* Rename airbyte-config:persistence to airbyte-config:config-persistence.
Part 1 of #13122.
Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.
Explanation for what is happening:
Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
* Adds a new string cloumn to configs for cron
closes#11418
i'm new to this task in Java please be brutal
* Adds airbyte header
* WIP
* Rebase a week of commits
* WIP for davin
* deps update
* Reorganize code for better readability. Also add a schema.
* Update tests.
* Correct bad test.
* Adds note for testing version change
* formatting change
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Migrate to public databricks jdbc driver
* Update documentation
* Bump version
* Format code
* Check in databricks in seed file
* Check in databricks 0.2.0
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Better database connection handling for connectors
* Log connection error
* Properly close connection
* Remove unused method
* Close data source
* Use utility to close data source
* Use utility to close data source
* PR feedback
* Add Databricks driver
* Use driver class enum
* Use correct config
* Ensure config created before use
* Fix failing integration test
* Create DSLContext before use
* Address integration test failures
* Ensure DSLContext is closed
* Fix compile error
* Use correct datasource
* Use correct connection properties
* Close DSLContext
* Close DSLContext
* Fix integration test failures
* Properly close datasource
* Fix compilation issues
* Use existing database object
* Wrap close in try/finally
* Update test
* Wrap close in try/finally
* Ensure DSLContext is created
* Revert change to test
* Use correct data source
* Remove unused import
* More cleanup
* Add missing annotation
* Only initialize data source once
* Remove unused import
* Force testcontainers version
* Fix testcontainer issue
* Fix failing test
* Properly close all data sources
* Clear data sources after closing
* Fix compile error
* Fix compilation error
* Add missing method
* Default scaffold to use adaptive streaming config
* Switch more connectors to use adaptive streaming config
* Bump version for cockroach db
* Bump version for db2
* Bump mssql version
* Bump mysql version
* Bump oracle version
* Bump postgres version
* Bump redshift version
* Bump snowflake version
* Bump tidb version
* auto-bump connector version
* Fix db2 findbug issue
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
* Fix more findbug issues
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
* Fix findbug issue for mysql-strict-encrypt
* Fix findbugs issue for oracle source
* auto-bump connector version
* Remove suppress warnings annotation
* Fix oracle encrypt tests
* Fix oracle encrypt acceptance test
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Merge all streaming configs to one
* Implement new streaming query config
* Format code
* Fix comparison
* Use double for mean byte size
* Update fetch size only when changed
* Calculate mean size by sampling n rows
* Add javadoc
* Change min fetch size to 1
* Add comment by buffer size
* Update java connector template
* Perform division first
* Add unit test for fetching large rows
* Format code
* Fix connector compilation error
* implement migration to create workspace_service_account table
* make all columns non nullable
* introduce persistence code for service account table (#11944)
* implement persistence code for workspace_service_account table
* update yaml
* implement secret handling for workspace_service_account table (#11946)
* implement secret handling for workspace_service_account table
* add new line to the mock json
* get rid of file
* address review comments
* update method name and add comment
* Revert "Revert "Build OSS branch for deploying to Cloud env (#11474)""
This reverts commit 55e3c1e051.
* add action to get dev branch tag to OSS project instead of doing it in cloud
* remove dev branch version action, going to do this in cloud afterall
* add VERSION buildArg to Dockerfiles, default to current airbyte version but overwritable
* use VERSION env var consistently as Dockerfile buildArg, jar version, and tag
pass version and image_tag into docker build task function
* add github action for building and pushing an OSS branch for Cloud to consume
* allow AirbyteVersion to validate versions containing 'oss-branch' prefix
* change oss-branch prefix to dev for branch-based versions
* better action name
* add docker-compose-cloud.build.yaml to define minimum set of cloud images that are pushed by oss branch action
* update local dev docs to describe optional usage of VERSION env var
* make branch_version_tag input optional, if not provided, generates dev-<commit_hash>
* fix typo
* fix missed merge conflict
* update docker docs
* update integrationRunner isDev check
* Move buildDestinationOAuthParameter to DbConverter
* Migrate getDestinationOAuthParamByDefinitionIdOptional to use direct SQL
* Move buildSourceOAuthParameter to DbConverter
* Change getSourceOAuthParamByDefinitionIdOptional to use direct SQL
* Add tests
* Add new indices for oauth param table
* Fix index creation statement
* migrations for supporting actor grants
* seed yaml definitions are always public, never custom
* add public and custom to actor definition models
* move methods to DbConverter
* add includeTombstones helper
to avoid having conditionals in every method with the tombstone boolean
* remove migration TODO comments
* assert custom is false when loading seeds
* format
* Move Workspace conversion function to DbConverter
* Migrate getWorkspaceBySlug to use direct SQL statement
* Move extraction of notification to converter function
* Add additional indices on workspace table
* Add tests for getWorkspaceBySlug
* Feat: first cut to allow naming for connections
* fix
* fix: migration
* fix: migration
* fix: formatting
* fix: formatting
* fix: tests
* fix: -> is bit outside of what we do generally
* fix: tests are failing
* fix: tests are failing
* fix: tests are failing
* fix: tests are failing
* fix: tests are failing
* Feat: first cut to allow naming for connections
* fix
* fix: migration
* fix: migration
* fix: formatting
* fix: formatting
* fix: tests
* fix: -> is bit outside of what we do generally
* Remove deprecated/unused enum values from json schema, migration to update records to use corrected values
* make migration-specific classes handle any string, and remove extranneous comments/annotations for readability. also test that an unrecognized enum value is left alone and doesn't cause deserialization errors
* gradle format
* fix test
* fix jobTrackerTest with new enum values
