2e3dfccc68
* setup GCP terraform structure for connector dev infra * fxes * Created aws terraform bootstrap infrastructure * Corrected path for S3 state file * Creating ssh tunnel testing infrastructure * Creating a bastion host * Docs * Created bastion host with airbyte unprivileged account for tunneling * Added yum updates * Create a private subnet and a postgres database within it * Identifier for db * Create postgres and bastion within a module. * Set up postgres infrastructure * Don't need this terraform wrapper when we run with tfenv * Dropped incomplete WIP from GCP setup for this PR. * Touchups to how to use terraform * Updated to resolve merge conflict * More separated top level structure to avoid monorepo problems with terraform destroy. * Clarifying directory structure * Migrated directory structure for testing infrastructure Co-authored-by: Jenny Brown <jenny@airbyte.io>
2.0 KiB
SSH Tunnel Testing
This directory creates infrastructure for testing ssh tunneling to databases for airbyte connectors. It sets up:
- a public subnet (for a bastion host and one postgres AZ)
- a private subnet (for postgres secondary AZ that aws insists on)
- two security groups (for the bastion host, for the postgres server)
- a bastion host reachable from the internet, with ssh tunnel support
- a user account on the bastion host
- a postgres database on a private address
All infrastructure for this is kept separate from other airbyte infrastructure, as it's meant to simulate a client's corporate environment and private databases.
This configuration uses the 'tfenv' wrapper on terraform for versioning. Each directory contains a .terraform-version file specifying the compatibility for that terraform instance.
brew install tfenv # install
terraform plan # should use the tfenv wrapper's version of terraform
Public Keys
The bastion host requires an ec2-user (always) and preferably also a non-root capable user named airbyte. The airbyte user is used for ssh tunnel from the connectors, and should not be a priviledged user. These are in the integration test secrets store under the 'infra' prefix.
To create a fresh ssh keypair and set its comment (where the email usually shows), use a command like this:
ssh-keygen -t rsa -f dbtunnel-bastion-ec2-user_rsa -C ec2-user
ssh-keygen -t rsa -f ~/dbtunnel-bastion-airbyte_rsa -C airbyte
The public key from that is used for ec2 instance creation, but the private key should be kept secret.
TODO: The airbyte user will also need password auth allowed on the ssh connection, once we're ready for that.
Database Setup
We don't have yet automation for running the database configuration scripts from infrastructure as code. The sql scripts included should be run once by hand when setting up from scratch. Note that the sql script creating a user has a place to manually change the password.