Deploy on AWS (Amazon EC2)

This page guides you through deploying Airbyte Open Source on an Amazon EC2 instance by setting up the deployment environment, installing and starting Airbyte, and connecting it to the Amazon EC2 instance.

INFO

The instructions have been tested on Amazon Linux 2 AMI (HVM)

Requirements

To test Airbyte, we recommend a t2.medium instance
To deploy Airbyte in a production environment, we recommend a t2.large instance
Make sure your Docker Desktop app is running to ensure all services run smoothly

Create and download an SSH key to connect to the instance

Set up the environment

Create an Amazon EC2 instance.

To connect to your instance, run the following command on your local terminal:

SSH_KEY=~/Downloads/dataline-key-airbyte.pem # the file path you downloaded the key
INSTANCE_IP=REPLACE_WITH_YOUR_INSTANCE_IP # you can find your IP address in the EC2 console under the Instances tab
chmod 400 $SSH_KEY # or ssh will complain that the key has the wrong permissions
ssh -i $SSH_KEY ec2-user@$INSTANCE_IP # connect to the aws ec2 instance AMI and the your private IP address

To install Docker, run the following command in your SSH session on the instance terminal:

sudo yum update -y
sudo yum install -y docker
sudo service docker start
sudo usermod -a -G docker $USER

To install docker-compose, run the following command in your ssh session on the instance terminal:

sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version

To close the SSH connection, run the following command in your SSH session on the instance terminal:

logout

Install and start Airbyte

In your local terminal, run the following commands:

Connect to your instance

ssh -i $SSH_KEY ec2-user@$INSTANCE_IP

Install Airbyte

mkdir airbyte && cd airbyte
wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml}
docker-compose up -d # run the Docker container

Connect to Airbyte

DANGER

We strongly recommend not exposing Airbyte on Internet available ports for security reasons.

Create an SSH tunnel for port 8000

INFO

If you want to use different ports, modify API_URL in your .env file and restart Airbyte. Run the following commands in your workstation terminal from the downloaded key folder:

# In your workstation terminal
SSH_KEY=~/Downloads/dataline-key-airbyte.pem
ssh -i $SSH_KEY -L 8000:localhost:8000 -N -f ec2-user@$INSTANCE_IP

Visit http://localhost:8000 to verify the deployment.

Get Airbyte logs in CloudWatch

Follow this guide to get your logs from your Airbyte Docker containers in CloudWatch.

Troubleshooting

If you encounter any issues, reach out to our community on Slack.

3.1 KiB Raw Blame History