jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

Add how to copy files from Kubernetes pods. (#7238)

Browse Source 
* Add how to copy files from Kubernetes pods.

* Add instructions on how to browse.

* Format.

* This is needed.

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
This commit is contained in:
Davin Chia
2021-11-18 19:16:36 +08:00
committed by GitHub
parent 8e0e0b4f79
commit 7b54480677
1 changed files with 22 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
docs/operator-guides/browsing-output-logs.md
View File

@@ -93,6 +93,28 @@ docker cp airbyte-server:/tmp/workspace/9/2/catalog.json .
cat catalog.json
```
### Browsing on Kubernetes
If you are running on Kubernetes, use the following commands instead to browsing and copy the files to your local.
To browse, identify the pod you are interested in and exec into it. You will be presented with a terminal that will accept normal linux commands e.g ls.
```bash
kubectl exec -it <pod name> -n <namespace pod is in> -c main bash
e.g.
kubectl exec -it destination-bigquery-worker-3607-0-chlle -n jobs -c main bash
root@destination-bigquery-worker-3607-0-chlle:/config# ls
FINISHED_UPLOADING destination_catalog.json destination_config.json
```
To copy the file on to your local in order to preserve it's contents:
```bash
kubectl cp <namespace pods are in>/<normalisation-pod-name>:/config/destination_catalog.json ./catalog.json
e.g.
kubectl cp jobs/normalization-worker-3605-0-sxtox:/config/destination_catalog.json ./catalog.json
cat ./catalog.json
```
## CSV or JSON local Destinations: Check local data folder
If you setup a pipeline using one of the local File based destinations \(CSV or JSON\), Airbyte is writing the resulting files containing the data in the special `/local/` directory in the container. By default, this volume is mounted from `/tmp/airbyte_local` on the host machine. So you need to navigate to this [local folder](file:///tmp/airbyte_local/) on the filesystem of the machine running the Airbyte deployment to retrieve the local data files.