jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-23 21:03:15 -05:00
Code Issues Packages Projects Releases Wiki Activity

docs: convert deploying oss to a helm installation guide (#41974)

Browse Source
This commit is contained in:
Charles
2024-07-16 16:32:56 -07:00
committed by GitHub
parent 8a04aa5237
commit 3d420150b5
7 changed files with 160 additions and 161 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
docs/deploying-airbyte/creating-secrets.md Normal file
View File

@@ -0,0 +1,41 @@
import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";
# Preconfiguring Kubernetes Secrets
Deploying Airbyte requires specifying a number of sensitive values. These can be API keys, usernames and passwords, etc.
In order to protect these sensitive values, the Helm Chart assumes that these values are pre-configured and stored in a Kubernetes Secret *before* the Helm installation begins. Each [integration](#integrations) will provide the Secret values that are required for the specific integration.
While you can set the name of the secret to whatever you prefer, you will need to set that name in various places in your values.yaml file. For this reason we suggest that you keep the name of `airbyte-config-secrets` unless you have a reason to change it.
<Tabs>
<TabItem value="yaml" label="Creating Secrets with YAML" default>
You can apply your yaml to the cluster with `kubectl apply -f secrets.yaml -n airbyte` to create the secrets.
```yaml
apiVersion: v1
kind: Secret
metadata:
name: airbyte-config-secrets
type: Opaque
stringData:
# Examples
key-1: "value-1"
key-2: "value-2"
```
</TabItem>
<TabItem value="cli" label="Creating secrets with kubectl">
You can also use `kubectl` to create the secret directly from the CLI:
```sh
kubectl create secret generic airbyte-config-secrets \
--from-literal=key-1='value-1' \
--from-literal=key2='value-2' \
--namespace airbyte
```
</TabItem>
</Tabs>

121
docs/deploying-airbyte/deploying-airbyte.md
View File

@@ -27,7 +27,13 @@ Helm is a Kubernetes package manager for automating deployment and management of
The [Infrastructure](infrastructure/aws) section describes the Airbyte's recommended cloud infrastructure to set up for each supported platform. Keep in mind that these guides are meant to assist you, but you are not required to follow them. Airbyte is designed to be as flexible as possible in order to fit into your existing infrastructure.
## Adding the Helm Repository
## Installation Guide
This installation guide walks through how to deploy Airbyte into _any_ kubernetes cluster. It will run through how to deploy a default version of Airbyte. It will, as an optional step, describe you how you can customize that deployment for your cloud provider and integrations (e.g. ingresses, external databases, external loggers, etc).
This guide assumes that you already have a running kubernetes cluster. If you're trying out Airbyte on your local machine, we recommend using [Docker Desktop](https://www.docker.com/products/docker-desktop/) and enabling the kubernetes extension. We've also tested it with kind, k3s, and colima. If you are installing onto a single vm in a cloud provider (e.g. EC2 or GCE), make sure you've installed kubernetes on that machine. This guide also works for multi-node setups (e.g. EKS and GKE).
### 1. Add the Helm Repository
The deployment will use a Helm chart which is a package for Kubernetes applications, acting like a blueprint or template that defines the resources needed to deploy an application on a Kubernetes cluster. Charts are stored in `helm-repo`.
@@ -63,7 +69,7 @@ airbyte/workload-launcher 0.290.0 0.63.6 Helm chart to depl
```
## Creating a Namespace for Airbyte
### 2. Create a Namespace for Airbyte
While it is not strictly necessary to isolate the Airbyte installation into its own namespace, it is good practice and recommended as a part of the installation.
This documentation assumes that you chose the name `airbyte` for the namespace, but you may choose a different name if required.
@@ -74,47 +80,7 @@ To create a namespace run the following:
kubectl create namespace airbyte
```
## Preconfiguring Kubernetes Secrets
Deploying Airbyte requires specifying a number of sensitive values. These can be API keys, usernames and passwords, etc.
In order to protect these sensitive values, the Helm Chart assumes that these values are pre-configured and stored in a Kubernetes Secret *before* the Helm installation begins. Each [integration](#integrations) will provide the Secret values that are required for the specific integration.
While you can set the name of the secret to whatever you prefer, you will need to set that name in various places in your values.yaml file. For this reason we suggest that you keep the name of `airbyte-config-secrets` unless you have a reason to change it.
<Tabs>
<TabItem value="yaml" label="Creating Secrets with YAML" default>
You can apply your yaml to the cluster with `kubectl apply -f secrets.yaml -n airbyte` to create the secrets.
```yaml
apiVersion: v1
kind: Secret
metadata:
name: airbyte-config-secrets
type: Opaque
stringData:
# Examples
key-1: "value-1"
key-2: "value-2"
```
</TabItem>
<TabItem value="cli" label="Creating secrets with kubectl">
You can also use `kubectl` to create the secret directly from the CLI:
```sh
kubectl create secret generic airbyte-config-secrets \
--from-literal=key-1='value-1' \
--from-literal=key2='value-2' \
--namespace airbyte
```
</TabItem>
</Tabs>
## Creating a values.yaml override file
### 3. Create a values.yaml override file
To configure your installation of Airbyte, you will need to override specific parts of the Helm Chart. To do this you should create a new file called `values.yaml` somewhere that is accessible during the installation process.
The documentation has been created to "build up" a values.yaml, so there is no need to copy the whole of the Chart values.yaml. You only need to provide the specific overrides.
@@ -124,37 +90,29 @@ Each [Integration](#integrations) will provide a section of the specific values
```yaml
global:
airbyteUrl: https://airbyte.company.example
storage:
type: "S3"
bucket: ## S3 bucket names that you've created. We recommend storing the following all in one bucket.
log: airbyte-bucket
state: airbyte-bucket
workloadOutput: airbyte-bucket
s3:
region: "us-east-1"
authenticationType: "instanceProfile"
secretsManager:
type: awsSecretManager
awsSecretManager:
region: "us-east-1"
authenticationType: "instanceProfile"
```
### 4. (optional for customized installations only) Customize your deployment
## Integrations
The Airbyte platform is built to integrate with your existing cloud infrastructure. You can configure various components of the platform to suit your needs. This includes an object stores, such as S3 or GCS for storing logs and state, a database for externalizing state, and a secrets manager for keep your secrets secure.
The Airbyte platform is built to integrate with your existing cloud infrastructure. You can configure various components of the platform to suit your needs. This includes an object store, such as S3 or GCS for storing logs and state, a database for externalizing state, and a secrets manager for keep your secrets secure.
Each of these integrations will require you to create a secret upfront. For instructions on how to do that check out [Creating a Secret](./creating-secrets.md)
Each of these integrations can be configured to suit your specific needs and is described in the [Integration](#integrations) section. Each of these integrations has its own section where you'll find an explanation for why it's useful to configure the integration. There, you'll also find details about how to configure the integration.
Before you can configure this stuff in a cloud provider, you need to set up your policies:
* [AWS Policies](./infrastructure/aws.md#policies)
* [Azure Policies](./infrastructure/azure.md#policies)
* [GCP Policies](./infrastructure/gcp.md#policies)
After your policies are set up, here's a list of customizations.
- [State and Logging Storage](./integrations/storage)
- [Secret Management](./integrations/secrets)
- [External Database](./integrations/database)
- [Ingress](./integrations/ingress)
## Installing Airbyte
### 5. Installing Airbyte
After you have applied your Secret values to the Cluster and you have filled out a values.yaml file appropriately for your specific configuration, you can begin a Helm Install. To do this, make sure that you have the [Helm Client](https://helm.sh/docs/intro/install/) installed and on your path.
Then you can run:
@@ -162,31 +120,28 @@ Then you can run:
```sh
helm install \
airbyte \
airbyte/airbyte
airbyte/airbyte \
--namespace airbyte \
--values ./values.yaml \
--values ./values.yaml
```
After the installation has completed, you can configure your [Ingress](./integrations/ingress) by following the directions for your specific Ingress provider.
### 6. Set up port forward for UI access
Helm install with spit out instructions for how to set up the port forward. Go ahead and run that command. It should look something like this:
```bash
export POD_NAME=$(kubectl get pods --namespace airbyte -l "app.kubernetes.io/name=webapp" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace airbyte $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace airbyte port-forward $POD_NAME 8080:$CONTAINER_PORT
```
You can now access the UI in your browser at: http://127.0.0.1:8080.
If you'd like to set up a more permanent ingress checkout our ingress customization. For a deployment to a local machine we recommend using [nginx](./integrations/ingress.md) as an easy-to-setup option.
:::note
As part of maintainging your Airbyte instance, you'll need to do periodic upgrades. See our documentation on [when and how to upgrade Airbyte](../operator-guides/upgrading-airbyte.md) for details.
:::
<!--
##TODO
## Tools
### Required Tools
Helm
Kubectl
### Optional Tools
K9s
Stern -->
:::

74
docs/deploying-airbyte/infrastructure/aws.md
View File

@@ -76,77 +76,3 @@ The [following policies](https://docs.aws.amazon.com/mediaconnect/latest/ug/iam-
]
}
```
## Using an EC2 Instance with abctl
This guide will assume that you are using the Amazon Linux distribution. However. any distribution that supports a docker engine should work with `abctl`. The launching and connecting to your EC2 Instance is outside the scope of this guide. You can find more information on how to launch and connect to EC2 Instances in the [Get started with Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html) documentation from Amazon.
:::tip
`abctl` runs by default on port 8000. You can change the port by passing the `--port` flag to the `local install` command. Make sure that the security group that you have configured for the EC2 Instance allows traffic in on the port that you deploy Airbyte on. See the [Control traffic to your AWS resources using security groups](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-security-groups.html) documentation for more information.
:::
1. Install the docker engine:
```shell
sudo yum install -y docker
```
2. Add the ec2-user (or whatever your distros default user) to the docker group:
```shell
sudo usermod -a -G docker ec2-user
```
3. Start and optionally enable (start on boot) the docker engine:
```shell
sudo systemctl start docker
sudo systemctl enable docker
```
4. Exit the shell and reconnect to the ec2 instance, an example would look like:
```shell
exit
ssh -i ec2-user-key.pem ec2-user@1.2.3.4
```
5. Download the latest version of abctl and install it in your path:
```shell
curl -LsfS https://get.airbyte.com | bash -
```
6. Run the `abctl` command and install Airbyte:
```shell
abctl local install
```
### Editing the Ingress
By default `abctl` will install and Nginx Ingress and set the host name to `localhost`. You will need to edit this to
match the host name that you have deployed Airbyte to. To do this you will need to have the `kubectl` command installed
on your EC2 Instance and available on your path.
If you do not already have the CLI tool kubectl installed, please [follow these instructions to install](https://kubernetes.io/docs/tasks/tools/).
Then you can run `kubectl edit ingress -n airbyte-abctl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig` and edit the `host`
key under the spec.rules section of the Ingress definition. The host should match the FQDN name that you are trying to
host Airbyte at, for example: `airbyte.company.example`.
## Using an ALB for Ingress
The recommended method for Cluster Ingress is an AWS ALB. The [Ingress](../integrations/ingress) section of the documentation
shows how to configure the Kubernetes Ingress using the AWS Load Balancer Controller. This assumes that you have already
correctly configured your Cluster with the AWS Load Balancer Controller. This configuration is outside the scope of this
documentation. You can find more information on how to correctly configure an ALB Ingress Controller by reading the official
[Route application and HTTP traffic with Application Load Balancers](https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html)
documentation provided by Amazon.
Once the AWS Load Balancer Controller has been correctly installed the Airbyte installation process will be able to
automatically create an ALB for you. You can combine the ALB with AWS Certificate Manager (ACM) to secure your instance
with TLS. The ACM documentation can be found here: [Getting Started with AWS Certificate Manager](https://aws.amazon.com/certificate-manager/getting-started/).
To use the ACM certificate, you can specify the certificate-arn when creating the Kubernetes Ingress. For more information
see the [Kubernetes Ingress Annotations documentation](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/guide/ingress/annotations/#certificate-arn).

2
docs/deploying-airbyte/integrations/database.md
View File

@@ -18,7 +18,7 @@ metadata:
type: Opaque
stringData:
# Database Secrets
## database-host: ## e.g. database.internla
## database-host: ## e.g. database.internal
## database-port: ## e.g. 5432
## database-name: ## e.g. airbyte
## database-user: ## e.g. airbyte

18
docs/deploying-airbyte/integrations/ingress.md
View File

@@ -11,23 +11,25 @@ import TabItem from '@theme/TabItem';
If you are using `abctl` to manage your deployment, a nginx ingress is automatically provided for you. There is no need to provision an additional ingress.
:::
To access the Airbyte UI, you will need to manually attach an ingress configuration to your deployment.
To access the Airbyte UI, you will need to manually attach an ingress configuration to your deployment. These guides assume that you have already deployed an Ingress Controller.
The following is a simplified definition of an ingress resource you could use for your Airbyte instance:
<Tabs>
<TabItem value="NGINX" label="NGINX">
If you don't already have an NGINX controller installed, you can do it by running `helm install my-release oci://ghcr.io/nginxinc/charts/nginx-ingress --version 1.3.1` or following the [instructions](https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/) from NGINX.
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: # ingress name, example: airbyte-production
name: airbyte-ingress # ingress name, example: airbyte-production-ingress
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
spec:
ingressClassName: nginx
rules:
- host: # host, example: airbyte.company.example
- host: localhost # host, example: airbyte.company.example
http:
paths:
- backend:
@@ -58,6 +60,16 @@ spec:
</TabItem>
<TabItem value="Amazon ALB" label="Amazon ALB">
First you need to have an ALB deployed. You can read more about ALBs in the detail below. Reference AWS on how to set these up.
<details>
<summary>AWS ALBs</summary>
The recommended method for Cluster Ingress is an AWS ALB. This configuration is outside the scope of this documentation. You can find more information on how to correctly configure an ALB Ingress Controller by reading the official [Route application and HTTP traffic with Application Load Balancers](https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html) documentation provided by Amazon.
Once the AWS Load Balancer Controller has been correctly installed the Airbyte installation process will be able to automatically create an ALB for you. You can combine the ALB with AWS Certificate Manager (ACM) to secure your instance with TLS. The ACM documentation can be found here: [Getting Started with AWS Certificate Manager](https://aws.amazon.com/certificate-manager/getting-started/). To use the ACM certificate, you can specify the certificate-arn when creating the Kubernetes Ingress. For more information see the [Kubernetes Ingress Annotations documentation](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/guide/ingress/annotations/#certificate-arn).
</details>
If you intend to use Amazon Application Load Balancer (ALB) for ingress, this ingress definition will be close to what's needed to get up and running:

59
docs/using-airbyte/getting-started/oss-quickstart.md
View File

@@ -213,6 +213,65 @@ If you're using a version of Airbyte that you've installed with `abctl`, you can
:::
## Using an EC2 Instance with abctl
This guide will assume that you are using the Amazon Linux distribution. However. any distribution that supports a docker engine should work with `abctl`. The launching and connecting to your EC2 Instance is outside the scope of this guide. You can find more information on how to launch and connect to EC2 Instances in the [Get started with Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html) documentation from Amazon.
:::tip
`abctl` runs by default on port 8000. You can change the port by passing the `--port` flag to the `local install` command. Make sure that the security group that you have configured for the EC2 Instance allows traffic in on the port that you deploy Airbyte on. See the [Control traffic to your AWS resources using security groups](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-security-groups.html) documentation for more information.
:::
1. Install the docker engine:
```shell
sudo yum install -y docker
```
2. Add the ec2-user (or whatever your distros default user) to the docker group:
```shell
sudo usermod -a -G docker ec2-user
```
3. Start and optionally enable (start on boot) the docker engine:
```shell
sudo systemctl start docker
sudo systemctl enable docker
```
4. Exit the shell and reconnect to the ec2 instance, an example would look like:
```shell
exit
ssh -i ec2-user-key.pem ec2-user@1.2.3.4
```
5. Download the latest version of abctl and install it in your path:
```shell
curl -LsfS https://get.airbyte.com | bash -
```
6. Run the `abctl` command and install Airbyte:
```shell
abctl local install
```
### Editing the Ingress
By default `abctl` will install and Nginx Ingress and set the host name to `localhost`. You will need to edit this to
match the host name that you have deployed Airbyte to. To do this you will need to have the `kubectl` command installed
on your EC2 Instance and available on your path.
If you do not already have the CLI tool kubectl installed, please [follow these instructions to install](https://kubernetes.io/docs/tasks/tools/).
Then you can run `kubectl edit ingress -n airbyte-abctl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig` and edit the `host`
key under the spec.rules section of the Ingress definition. The host should match the FQDN name that you are trying to
host Airbyte at, for example: `airbyte.company.example`.
## Troubleshooting
There are several channels for community support of local setup and deployment.

6
docusaurus/sidebars.js
View File

@@ -416,6 +416,12 @@ const deployAirbyte = {
"deploying-airbyte/integrations/ingress",
],
},
{
type: "doc",
label: "Creating a Secret",
id: "deploying-airbyte/creating-secrets",
},
],
};