1
0
mirror of synced 2025-12-22 03:21:25 -05:00
Commit Graph

1 Commits

Author SHA1 Message Date
Andrii Leonets
9151d83bcf 🐛 BigQuery source: Fix nested arrays (#4981)
* unfinished jdbcsource separation

* creation AbstactRelation

* Migrate StateManager to new abstract level (JdbcSource -> RelationalSource)

* fix imports

* move configs to Database level + fix MySql source

* make in line jdbc source with a new impl

* Fix ScaffoldJavaJdbcSource template

* rename `AbstractField` to `CommonField`. Now it
s not an abstract class.
+ add default implementation for `AbstractRelationalDbSource.getFullyQualifiedTableName`

* format

* rename generated files in line with their location

* bonus renaming

* move utility methods specific for jdbc source to a proper module

* internal review update

* BigQueryDatabase impl without row transformation

* add Static method for BigQueryDatabase instancing

* remove data type parameter limitation + rename class parameters

* Move DataTypeUtils from jdbs to common + impl basic types BigQueryUtils

* make DB2 in line with new relational abstract classes

* add missing import

* cover all biqquery classes + add type transformation method from StandardSQLTypeName to JsonSchemaPrimitive

* close unused connections

* add table list extract method

* bigquery source connector

* return all tables for a whole project instead of a dataset

* impl incremental fetch

* bigquery source connector

* bigquery source connector

* remove unnecessary databaseid

* add primitive type filtering

* add temporary workaround for test database.

* add dataset location

* fix table info retrieving

* handle dataset config

* Add working comprehensive test without data cases

* minor changes in the source processing

* acceptance tests; discover method fix

* discover method fix

* first comprehensinve test

* Comprehensive tests for the BigQuery source + database timeout config

* bigquery acceptance tests fix; formatting

* fix incremental sync using date, datetime, time and timestamp types

* Implement source checks: basic and dataset

* format

* revert: airbyte_protocol.by

* internal review update

* Add possibility to get list of comprehensive tests in a Markdown table format.

* Update airbyte-integrations/connectors/source-bigquery/src/main/resources/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* review update

* Implement processing for arrays and structures

* format

* added bigquery secrets

* added bigquery secrets

* spec fix

* test configs fix

* extend mapping for Arrays and Structs

* Process nested arrays

* handle arrays of records properly.

* format

* BigQuery source docs

* docs readme update

* hide evidences

* fix changlog order

* Add bigquery to source_defintions yaml

Co-authored-by: heade <danildubinin2@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-07-27 20:27:12 +03:00