airbyte/docs/deploying-airbyte/on-gcp-compute-engine.md

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';

On GCP (Compute Engine)

This page guides you through deploying Airbyte Open Source on a Google Cloud Platform (GCP) Compute Engine instance by setting up the deployment environment, installing and starting Airbyte, and connecting it to the GCP instance.

:::info

The instructions have been tested on a Debian GNU/Linux 10 VM instance.

:::

Requirements

• To test Airbyte, we recommend an e2.medium instance and provision at least 30GBs of disk per node.
• To deploy Airbyte in a production environment, we recommend a n1-standard-2 instance.

Set up the environment

1. Create a new GCP instance.
2. Set variables in your local terminal:

PROJECT_ID=PROJECT_ID_WHERE_YOU_CREATED_YOUR_INSTANCE
INSTANCE_NAME=airbyte # or any other name that you've used

3. Install Google Cloud SDK and initialize the gcloud command-line tool using the following commands in your local terminal:

brew install --cask google-cloud-sdk
gcloud init

4. List all instances in your project and verify that you can see the Airbyte instance you created in step 1 in your local terminal:

# Verify you can see your instance
gcloud --project $PROJECT_ID compute instances list
[...] # You should see the airbyte instance you just created

5. Connect to your instance in your local terminal:

gcloud --project=$PROJECT_ID beta compute SSH $INSTANCE_NAME

6. Install Docker on your VM instance by following the below commands in your VM terminal:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add --
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian buster stable"
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
sudo usermod -a -G docker $USER

7. Install docker-compose on your VM instance by following the below commands in your VM terminal:

sudo apt-get -y install wget
sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version

8. Close the SSH connection on your VM instance to ensure the group modification is taken into account by following the below command in your VM terminal:

logout

Install and launch Airbyte

To install and launch Airbyte:

1. In your local terminal, connect to your Google Cloud instance:

gcloud --project=$PROJECT_ID beta compute SSH $INSTANCE_NAME

2. In your VM terminal, install Airbyte:

mkdir airbyte && cd airbyte
wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml}
docker-compose up -d

Connect to Airbyte

:::caution Warning: For security reasons, we strongly recommended not exposing Airbyte publicly. :::

1. In your local terminal, create an SSH tunnel to connect the GCP instance to Airbyte:

gcloud --project=$PROJECT_ID beta compute SSH $INSTANCE_NAME -- -L 8000:localhost:8000 -N -f

2. Verify the connection by visiting http://localhost:8000 in your browser.

Troubleshooting

If you encounter any issues, reach out to our community on Slack.