9151d83bcf
* unfinished jdbcsource separation * creation AbstactRelation * Migrate StateManager to new abstract level (JdbcSource -> RelationalSource) * fix imports * move configs to Database level + fix MySql source * make in line jdbc source with a new impl * Fix ScaffoldJavaJdbcSource template * rename `AbstractField` to `CommonField`. Now it s not an abstract class. + add default implementation for `AbstractRelationalDbSource.getFullyQualifiedTableName` * format * rename generated files in line with their location * bonus renaming * move utility methods specific for jdbc source to a proper module * internal review update * BigQueryDatabase impl without row transformation * add Static method for BigQueryDatabase instancing * remove data type parameter limitation + rename class parameters * Move DataTypeUtils from jdbs to common + impl basic types BigQueryUtils * make DB2 in line with new relational abstract classes * add missing import * cover all biqquery classes + add type transformation method from StandardSQLTypeName to JsonSchemaPrimitive * close unused connections * add table list extract method * bigquery source connector * return all tables for a whole project instead of a dataset * impl incremental fetch * bigquery source connector * bigquery source connector * remove unnecessary databaseid * add primitive type filtering * add temporary workaround for test database. * add dataset location * fix table info retrieving * handle dataset config * Add working comprehensive test without data cases * minor changes in the source processing * acceptance tests; discover method fix * discover method fix * first comprehensinve test * Comprehensive tests for the BigQuery source + database timeout config * bigquery acceptance tests fix; formatting * fix incremental sync using date, datetime, time and timestamp types * Implement source checks: basic and dataset * format * revert: airbyte_protocol.by * internal review update * Add possibility to get list of comprehensive tests in a Markdown table format. * Update airbyte-integrations/connectors/source-bigquery/src/main/resources/spec.json Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * review update * Implement processing for arrays and structures * format * added bigquery secrets * added bigquery secrets * spec fix * test configs fix * extend mapping for Arrays and Structs * Process nested arrays * handle arrays of records properly. * format * BigQuery source docs * docs readme update * hide evidences * fix changlog order * Add bigquery to source_defintions yaml Co-authored-by: heade <danildubinin2@gmail.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
878 B
878 B
BigQuery Test Configuration
In order to test the BigQuery source, you need a service account key file.
Community Contributor
As a community contributor, you will need access to a GCP project and BigQuery to run tests.
- Go to the
Service Accountspage on the GCP console - Click on `+ Create Service Account" button
- Fill out a descriptive name/id/description
- Click the edit icon next to the service account you created on the
IAMpage - Add the
BigQuery Data EditorandBigQuery Userrole - Go back to the
Service Accountspage and use the actions modal toCreate Key - Download this key as a JSON file
- Move and rename this file to
secrets/credentials.json
Airbyte Employee
- Access the
BigQuery Integration Test Usersecret on Rippling under theEngineeringfolder - Create a file with the contents at
secrets/credentials.json