* add WellKnownTypes.yaml
* rename to snakecase + put in airbyte-protocol
* add examples
* more descriptoins
* descriptions, more restrictions, better regex
* update documentation
* explicitly call out BC support
* Add cursor_record_count to db stream state
* Add cursor record count to cursor info
* Emit max cursor record count
* Add original cursor record count
* Unify logging format
* Add backward compatible methods
* Update unit tests for state decorating iterator
* Update test (not done yet)
* Fix one more unit test
* Change where clause operator according to record count
* Add branch for null cursor
* Skip saving record count when it is 0
* Fix log wording
* Set mock record count in test
* Check cursor value instead of cursor info
* Fix source jdbc test
* Read record count from state
* Fix tests
* Add an acceptance test case
* Fix npe
* Change record count from int to long to avoid type conversion
* Fix references
* Fix oracle container
* Use uppercase for snowflake
* Use uppercase for db2
* Fix and use uppercase
* Update test case to include the edge case
* Format code
* Remove extra assertion in clickhouse
* Merge ms sql incremental query method
* Log query for debugging
* Clean up name_and_timestamp table
* Fix db2 tests
* Fix mssql tests
* Fix oracle tests
* Fix oracle tests
* Fix cockroachdb tests
* Fix snowflake tests
* Add changelog
* Fix mssql tests
* Fix db2-strict-encrypt tests
* Fix oracle-strict-encrypt tests
* Bump postgres version
* Fix oracle-strict-encrypt tests
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Add test on the catalog diff logic
* Simplify test case
* Fix the bug
* Rm useless val
* Format
* Update test input
* revert map change
* Add test
* Fix test
* Format
* Use sugar
* Fix PMD errors
Co-authored-by: Malik Diarra <malik@airbyte.io>
* Add Version to AirbyteMessage
* Move protocol version to ConnectorSpecification
* Add cdk generated protocol model
* Add protocol_version to the sample ConnectorSpec in the docs
* Update airbyte-protocol/protocol-models/src/main/resources/airbyte_protocol/airbyte_protocol.yaml
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* update doc
* Update CDK changelog
* Update CDK protocol model
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* generate airbyte_type:integer
* normalization accepts `airbyte_type: integer`
* handles ints+longs
* update avro for consistency
* delete long type for now, treat all ints as longs
* update avro type mappings
{type:number, airbyte_type:integer} -> long
{type:number, airbyte_type:big_integer} -> string (i.e. "unbounded integer")
* fix test
* remove long handling
* Revert "remove long handling"
This reverts commit 33ade8d2831e675c3545ac6019d200ec312e54d9.
* Revert "update avro type mappings"
This reverts commit 5b0349badad7545efe8e1191291a628445fe1c84.
* Revert "delete long type for now, treat all ints as longs"
This reverts commit 018efd4a5d0c59f392fd8e3b0d0967c666b72947.
* Revert "update avro for consistency"
This reverts commit bcf47c6799b5906deb4f219d7f6e64ea73b41b74.
* newline@eof
* update test
* slightly better local tests
* fix test
* missed a few cases
* postgres tests use correct hostnames
* fix normalization
* fix int macro
* add test case
* normalization test output
* handle int/long correctly
* fix types for other DBs
* uint32 -> bigint; tests
* add type value assertions
* more test updates
* regenerate output
* reconcile big_integer to match docs
* update comment
* fix type
* fix mysql constructor call
* bigint only has 38 digits
* fix s3 ints, fix DAT test case
* big_integer should be string
* reduce to 28 digit big_ints
* fix test setup, mysql
* kill big_integer tests
* regenerate output
* version bumps
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Validate only on incremental
* Add test
* Format and pmd
* Update test
* PR comments
* Update airbyte-protocol/protocol-models/src/test/java/io/airbyte/protocol/models/CatalogHelpersTest.java
Co-authored-by: Lake Mossman <lake@airbyte.io>
* Fix test
Co-authored-by: Lake Mossman <lake@airbyte.io>
* BigQuery Denormalized "airbyte_type": "big_integer" to INT64
* updated changelog
* added unit test
* removed star import
* fixed checkstyle
* bump version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Update schema
* generate python
* Stream as an object
* PR comments
* generate python
* rm unused required
* Description the state with no type
* Fix connector build
* Format
* format
Co-authored-by: cgardens <charles@airbyte.io>
## What
Finale of https://github.com/airbytehq/airbyte/pull/13122.
We've renamed all directories in previous PRs. Here we remove the fat jar configuration and add publishing to all subprojects.
Explanation for what is happening:
Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
## How
* Remove fat jar configuration.
* Add publishing to all subprojects.
Part 1 of #13122.
Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.
Explanation for what is happening:
Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
What
Update the airbyte state message to support the per stream state.
The state message still contains the old way of storing the state in the data fields. It introduce 2 new fields to represent the global and the per stream state.
* generate AirbyeTraceMessage `type` enum with descriptive class name
* add comment on `title` usage
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* apply changes to bases/airbyte-protocol
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Source-MySql: transform binary data base64 format, add integration tests
* Source-MySql: fix code style
* Source-MySql: bump versions
* Source-MySql: bump versions in source_specs.yaml
* Source-MySql: added test for stream with binary data for DestinationAbstractTest
* Source-MySql: added format
* Change OAuth API
* Change protocol for new OAuth Spec (#7827)
* Add examples
* Add protocol object to api too
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* add specs module with logic to fetch specs on build
* format + build and add gradle dependency for new script
* check seed file for existing specs + refactor
* add tests + a bit more refactoring
* run gw format
* update yaml config persistence to merge specs into definitions
* add comment
* delete secrets migration to be consistent with master
* add dep
* add tests for GcsBucketSpecFetcher
* get rid of static block + format
* DRY up parse call
* add GCS details to comment
* formatting + fix test
* update comment
* do not format seed specs files
* change signature of run to allow cloud to reuse this script
* run gw format
* revert commits that change signature of run
* fix comment typo
Co-authored-by: Davin Chia <davinchia@gmail.com>
* rename enum to be distinct from the enum in cloud
* add missing dependencies between modules
* add readme for seed connector spec generator
* reword
* reference readme in comment
* ignore 'spec' field in newFields logic
Co-authored-by: Davin Chia <davinchia@gmail.com>
