200b035ec5
* initial commit * fix test error * Update get_gcs_blobs logic * add docs * Update source_definitions.yaml * Update airbyte-integrations/connectors/source-gcs/source_gcs/source.py Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com> * Update airbyte-config/init/src/main/resources/seed/source_definitions.yaml Co-authored-by: Denys Davydov <davydov.den18@gmail.com> * Update airbyte-integrations/connectors/source-gcs/source_gcs/helpers.py Co-authored-by: Denys Davydov <davydov.den18@gmail.com> * Update airbyte-integrations/connectors/source-gcs/source_gcs/helpers.py Co-authored-by: Denys Davydov <davydov.den18@gmail.com> * update docker file for pandas package * reimplement read_csv file * add logic to filter selected streams * close file_obj after reading * fix format and tests * add another stream * auto-bump connector version --------- Co-authored-by: Sunny <6833405+sh4sh@users.noreply.github.com> Co-authored-by: Denys Davydov <davydov.den18@gmail.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
1.6 KiB
1.6 KiB
GCS
This page guides you through the process of setting up the GCS source connector. This connector supports loading multiple CSV files (non compressed) from a GCS directory. The conntector will check for all files ending in .csv, even nested files.
Prerequisites
- JSON credentials for the service account that has access to GCS. For more details check instructions
- GCS bucket
- Path to file(s)
Set up Source
Create a Service Account
First, you need to select existing or create a new project in the Google Cloud Console:
- Sign in to the Google Account.
- Go to the Service Accounts page.
- Click
Create service account. - Create a JSON key file for the service user. The contents of this file will be provided as the
service_accountin the UI.
Grant permisison to GCS
Use the service account ID from above, grant read access to your target bucket. Click here for more details.
Set up the source in Airbyte UI
- Paste the service account JSON key to
service_account - Enter your GCS bucket name to
gcs_bucket - Enter path to your file(s) to
gcs_path
Changelog
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.1.0 | 2023-02-16 | 23186 | New Source: GCS |