* Bump version for redshift, bigquery, and snowflake
* auto-bump connector version
* auto-bump connector version
* Log failed refresh token response
* Revert snowflake version bump
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Refactor attempt 1
Increasing readability:
- Performed audit of methods that are marked as public/protected but should be private
- Performed audit of methods that are marked as public but should be protected
- Removing @Override from methods that don't need them (are the very first)
* Remove .sqlite files
* Remove AbstractRelationDbSource class
- Removing AbstractRelationalDbSource abstract class and moving contents of it to a Util class
* Rename utility class
* Add overrides back
* Add cursor_record_count to db stream state
* Add cursor record count to cursor info
* Emit max cursor record count
* Add original cursor record count
* Unify logging format
* Add backward compatible methods
* Update unit tests for state decorating iterator
* Update test (not done yet)
* Fix one more unit test
* Change where clause operator according to record count
* Add branch for null cursor
* Skip saving record count when it is 0
* Fix log wording
* Set mock record count in test
* Check cursor value instead of cursor info
* Fix source jdbc test
* Read record count from state
* Fix tests
* Add an acceptance test case
* Fix npe
* Change record count from int to long to avoid type conversion
* Fix references
* Fix oracle container
* Use uppercase for snowflake
* Use uppercase for db2
* Fix and use uppercase
* Update test case to include the edge case
* Format code
* Remove extra assertion in clickhouse
* Merge ms sql incremental query method
* Log query for debugging
* Clean up name_and_timestamp table
* Fix db2 tests
* Fix mssql tests
* Fix oracle tests
* Fix oracle tests
* Fix cockroachdb tests
* Fix snowflake tests
* Add changelog
* Fix mssql tests
* Fix db2-strict-encrypt tests
* Fix oracle-strict-encrypt tests
* Bump postgres version
* Fix oracle-strict-encrypt tests
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Use LOG4J2 to wrap connectors logs to JSON format
* log connector's stack trace directly as a message
* add stack_trace field to json template
* bump versions
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* revert versions for destinations: postgres, mssql, mysql
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Add order by clause in incremental query
* Support emitting intermediate states
* Add comment
* Log state warning only for final state emission
* Format code
* Add unit tests
* Define message iterator in each test case
* Fix compilation error
* generate airbyte_type:integer
* normalization accepts `airbyte_type: integer`
* handles ints+longs
* update avro for consistency
* delete long type for now, treat all ints as longs
* update avro type mappings
{type:number, airbyte_type:integer} -> long
{type:number, airbyte_type:big_integer} -> string (i.e. "unbounded integer")
* fix test
* remove long handling
* Revert "remove long handling"
This reverts commit 33ade8d2831e675c3545ac6019d200ec312e54d9.
* Revert "update avro type mappings"
This reverts commit 5b0349badad7545efe8e1191291a628445fe1c84.
* Revert "delete long type for now, treat all ints as longs"
This reverts commit 018efd4a5d0c59f392fd8e3b0d0967c666b72947.
* Revert "update avro for consistency"
This reverts commit bcf47c6799b5906deb4f219d7f6e64ea73b41b74.
* newline@eof
* update test
* slightly better local tests
* fix test
* missed a few cases
* postgres tests use correct hostnames
* fix normalization
* fix int macro
* add test case
* normalization test output
* handle int/long correctly
* fix types for other DBs
* uint32 -> bigint; tests
* add type value assertions
* more test updates
* regenerate output
* reconcile big_integer to match docs
* update comment
* fix type
* fix mysql constructor call
* bigint only has 38 digits
* fix s3 ints, fix DAT test case
* big_integer should be string
* reduce to 28 digit big_ints
* fix test setup, mysql
* kill big_integer tests
* regenerate output
* version bumps
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Part 1 of #13122.
Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.
Explanation for what is happening:
Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
* Better database connection handling for connectors
* Log connection error
* Properly close connection
* Remove unused method
* Close data source
* Use utility to close data source
* Use utility to close data source
* PR feedback
* Add Databricks driver
* Use driver class enum
* Use correct config
* Ensure config created before use
* Fix failing integration test
* Create DSLContext before use
* Address integration test failures
* Ensure DSLContext is closed
* Fix compile error
* Use correct datasource
* Use correct connection properties
* Close DSLContext
* Close DSLContext
* Fix integration test failures
* Properly close datasource
* Fix compilation issues
* Use existing database object
* Wrap close in try/finally
* Update test
* Wrap close in try/finally
* Ensure DSLContext is created
* Revert change to test
* Use correct data source
* Remove unused import
* More cleanup
* Add missing annotation
* Only initialize data source once
* Remove unused import
* Force testcontainers version
* Fix testcontainer issue
* Fix failing test
* Properly close all data sources
* Clear data sources after closing
* Fix compile error
* Fix compilation error
* Add missing method
* Add jdbc compatible layer
* Support routine mysql types
* Format code
* Fix build
* Refactor abstract jdbc source and operation classes
* Update mysql source operations
* Test discover command for mysql
* Remove abstract jdbc compatible source layer
* Format code
* Update template
* Fix more types
* Bump version
* Log original field type
* Update comments
* Bump version in seed
* Source BigQuery: Fix for NPE when `dataset_id` isn't provided
* changes for an empty `dataset_id` in the configuration
* added changelog
* bumped version
* Source Bigquery: Version should be 0.1.2 🤦♂️
* Source BigQuery: Fixed formatting
* Fixes from `./gradlew format` run
* Source Bigquery: Bump version to 0.1.3
* New abstraction for NoSql database sources
* New MongoDbSource: partial impl
* Added MongoDataType
* Improved MongoDatabase and fixed read method
* code review changes;
* merge clean up;
* Renamed NoSqlDatabase to AbstractDatabase
* formatter changes;
* code review changes: changed mongodb-new to mongodb-v2; left only new connector info in all docs
* code review changes: changed mongodb-new to mongodb-v2; left only new connector info in all docs
* updated spec.json and toDatabaseConfig() method
* updated doc accordingly to spec.json changes
Co-authored-by: Iryna Kruk <iryna.o.kruk@globallogic.com>
* move logic from static JdbcUtils to JdbcSourceOperations
* format
* Add methods for all types in order to have possibility rewrite them and use common type mapping.
* Make BigQuery in line impl with JDBC changes
* fix binary type
* add database creations methods with sourceOperations input
* Update airbyte-db/lib/src/main/java/io/airbyte/db/SourceOperations.java
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
# Summary
- A follow-up PR for #5543.
- This PR separates the `airbyte-db` project to two modules:
- `lib` is the original `airbyte-db`.
- `jooq` is for jOOQ code generation.
- This is necessary because the jOOQ generator requires a custom database implementation that can run Flyway migration. So the code generator logic needs to depend on the compilation of the original `airbyte-db` project.
# Commits
* Separate db to lib and jooq modules
* Update dependencies
* Add jobs db migrator test
* Fix compose build
* Add migration dev center
* Add schema dump task
* Update airbyte-db/lib/README.md
* Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update readme
* Remove bom dependency
* Update readme
* Use jooq code in db config persistence
* Remove AirbyteConfigsTable
Co-authored-by: Davin Chia <davinchia@gmail.com>
* unfinished jdbcsource separation
* creation AbstactRelation
* Migrate StateManager to new abstract level (JdbcSource -> RelationalSource)
* fix imports
* move configs to Database level + fix MySql source
* make in line jdbc source with a new impl
* Fix ScaffoldJavaJdbcSource template
* rename `AbstractField` to `CommonField`. Now it
s not an abstract class.
+ add default implementation for `AbstractRelationalDbSource.getFullyQualifiedTableName`
* format
* rename generated files in line with their location
* bonus renaming
* move utility methods specific for jdbc source to a proper module
* internal review update
* BigQueryDatabase impl without row transformation
* add Static method for BigQueryDatabase instancing
* remove data type parameter limitation + rename class parameters
* Move DataTypeUtils from jdbs to common + impl basic types BigQueryUtils
* make DB2 in line with new relational abstract classes
* add missing import
* cover all biqquery classes + add type transformation method from StandardSQLTypeName to JsonSchemaPrimitive
* close unused connections
* add table list extract method
* bigquery source connector
* return all tables for a whole project instead of a dataset
* impl incremental fetch
* bigquery source connector
* bigquery source connector
* remove unnecessary databaseid
* add primitive type filtering
* add temporary workaround for test database.
* add dataset location
* fix table info retrieving
* handle dataset config
* Add working comprehensive test without data cases
* minor changes in the source processing
* acceptance tests; discover method fix
* discover method fix
* first comprehensinve test
* Comprehensive tests for the BigQuery source + database timeout config
* bigquery acceptance tests fix; formatting
* fix incremental sync using date, datetime, time and timestamp types
* Implement source checks: basic and dataset
* format
* revert: airbyte_protocol.by
* internal review update
* Add possibility to get list of comprehensive tests in a Markdown table format.
* Update airbyte-integrations/connectors/source-bigquery/src/main/resources/spec.json
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* review update
* Implement processing for arrays and structures
* format
* added bigquery secrets
* added bigquery secrets
* spec fix
* test configs fix
* extend mapping for Arrays and Structs
* Process nested arrays
* handle arrays of records properly.
* format
* BigQuery source docs
* docs readme update
* hide evidences
* fix changlog order
* Add bigquery to source_defintions yaml
Co-authored-by: heade <danildubinin2@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
