509 lines
19 KiB
Plaintext
509 lines
19 KiB
Plaintext
---
|
|
products: oss-community
|
|
---
|
|
|
|
import Tabs from '@theme/Tabs';
|
|
import TabItem from '@theme/TabItem';
|
|
import EnvironmentVarConversion from '@site/static/_extraenv_to_helm_chart_v2.md';
|
|
import HelmSyntaxConversion from '@site/static/_helm-chart-2-syntax-changes.md';
|
|
|
|
# Upgrade to Helm chart V2 (Core)
|
|
|
|
Airbyte has upgraded its Helm chart to a new version called "V2." Using Helm chart V2 is currently optional. At some future date the V2 Helm chart will become the standard, so we advise that you upgrade your existing deployment to use the new chart before the transition. If you're a new Airbyte user, you can skip the upgrade altogether and start with the new chart.
|
|
|
|
:::important Enterprise customers
|
|
Follow the [Self-Managed Enterprise guide](../enterprise-setup/chart-v2-enterprise) instead.
|
|
:::
|
|
|
|
## Why you should upgrade
|
|
|
|
Upgrading to the new Helm chart now has the following benefits.
|
|
|
|
1. By upgrading in advance, you can schedule this upgrade for a convenient time. Avoid blocking yourself from upgrading Airbyte to a future version when the new chart is mandatory and you're busy.
|
|
|
|
2. The new Helm chart is more aligned with Helm's best practices for chart design.
|
|
|
|
3. The new Helm chart has broader and more detailed options to customize your deployment. In most cases, it's no longer necessary to specify environment variables in your `values.yaml` file because the chart offers a more detailed interface for customization. If you do need to use environment variables, you can use fewer of them.
|
|
|
|
## Which versions can upgrade to Helm chart V2
|
|
|
|
The following versions of Airbyte can use Helm chart V2:
|
|
|
|
- Airbyte version 1.6.0 and later, if installed and managed with Helm
|
|
|
|
The following versions of Airbyte _can't_ use Helm chart V2:
|
|
|
|
- Airbyte versions before 1.6.0
|
|
|
|
- Airbyte versions installed and managed with abctl
|
|
|
|
## How to upgrade
|
|
|
|
In most cases, upgrading is straightforward. To upgrade to Helm chart V2, you complete the following steps.
|
|
|
|
1. Ensure you have configured Airbyte to use an external database and external bucket storage. If you haven't, backup Airbyte's internal Postgres database.
|
|
|
|
2. Prepare to deploy a fresh installation of Airbyte in a new namespace, but only if you're using an external database. If you're using Airbyte's internal Postgres database, you'll reuse your existing namespace.
|
|
|
|
3. Create a new `values.yaml` file.
|
|
|
|
4. Deploy a new version of Airbyte using your new `values.yaml` file and the new Helm chart version.
|
|
|
|
### Backup Airbyte's internal Postgres database
|
|
|
|
<Tabs groupId="external-db">
|
|
<TabItem value='db-yes' label='External database' default>
|
|
|
|
If you have configured an external database, skip this step. However, it is a good idea to backup your database on a regular cadence.
|
|
|
|
</TabItem>
|
|
<TabItem value='db-no' label='No external database' default>
|
|
|
|
If you haven't configured an external database, you must backup Airbyte's internal Postgres database. If you fail to do this, and something goes wrong during your upgrade, you could lose all your connections and states.
|
|
|
|
1. Get your database pod name. It's probably `airbyte-db-0`, but you can check for it with the following command.
|
|
|
|
```bash
|
|
kubectl get pods --all-namespaces | grep db
|
|
```
|
|
|
|
2. Get the name of your database. It's probably `db-airbyte`, but you can check for it with the following command.
|
|
|
|
```bash
|
|
kubectl exec -n airbyte -it airbyte-db-0 -- psql -U airbyte postgres
|
|
```
|
|
|
|
3. Get your database credentials. They're probably in `airbyte-airbyte-secrets`, but you can check for them with the following command.
|
|
|
|
```bash
|
|
kubectl get secrets -n <namespace>
|
|
```
|
|
|
|
Now print them to the terminal.
|
|
|
|
```
|
|
kubectl get secret airbyte-airbyte-secrets -n airbyte -o jsonpath='{.data}'
|
|
```
|
|
|
|
You should see a result that's similar to this.
|
|
|
|
```bash
|
|
{"DATABASE_PASSWORD":"<username>","DATABASE_USER":"<password>","MINIO_ACCESS_KEY_ID":"<key>","MINIO_SECRET_ACCESS_KEY":"<key>","WORKLOAD_API_BEARER_TOKEN":"<key>"}%
|
|
```
|
|
|
|
Note your `DATABASE_PASSWORD` and `DATABASE_USER`.
|
|
|
|
4. Run `pg_dump` inside the Postgres pod and copy the dump locally.
|
|
|
|
```bash
|
|
kubectl exec -n <namespace> -it <database_pod> -- bash -c "pg_dump -U airbyte db-airbyte > /tmp/airbyte_backup.sql"
|
|
```
|
|
|
|
Copy your backup out of the pod and to the destination of your choice. In this example, you copy it to your local machine.
|
|
|
|
```bash
|
|
kubectl cp airbyte/airbyte-db-0:/tmp/airbyte_backup.sql ./airbyte_backup.sql
|
|
```
|
|
|
|
It's important you safeguard this SQL database. Although problems after upgrades are rare, if they do happen, your backup is critical to restoring Airbyte.
|
|
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
### Prepare a new namespace for Airbyte
|
|
|
|
<Tabs groupId="external-db">
|
|
<TabItem value='db-yes' label='External database' default>
|
|
|
|
When moving to Helm chart V2, deploy Airbyte with a new namespace and use a fresh values and secrets file. It is possible to do a straight upgrade, but different Airbyte users have different and sometimes complex configurations that could produce unique and unexpected situations during the upgrade. By doing a fresh install, you create a separate environment that's easier to troubleshoot if something in your values or secrets files acts unexpectedly.
|
|
|
|
```bash
|
|
kubectl create namespace airbyte-v2
|
|
```
|
|
|
|
</TabItem>
|
|
<TabItem value='db-no' label='No external database' default>
|
|
|
|
If you're not using an external database, skip this step. At deployment time, you will reinstall Airbyte in your existing namespace.
|
|
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
### Add V2 chart repo
|
|
|
|
Helm chart V2 uses a different repo URL (`/charts`) than V1 did (`/helm-charts`). In your command line tool, add this repo and index it.
|
|
|
|
```bash
|
|
helm repo add airbyte-v2 https://airbytehq.github.io/charts
|
|
helm repo update
|
|
```
|
|
|
|
You can browse all charts uploaded to your repository.
|
|
|
|
```bash
|
|
helm search repo airbyte-v2
|
|
```
|
|
|
|
### Update your values.yaml file
|
|
|
|
In most cases, the adjustments to `values.yaml` are small and involve changing keys and moving sections. This section walks you through the main updates you need to make. If you already know what to do, see [Values.yaml reference](../deploying-airbyte/values) for the full V1 and V2 interfaces.
|
|
|
|
Airbyte recommends approaching this project in this way:
|
|
|
|
1. Note the customizations in your V1 `values.yaml` file to ensure you don't forget anything.
|
|
|
|
2. Start with a basic V2 `values.yaml` to verify that it works. Map your V1 settings to V2, transferring one set of configurations at a time.
|
|
|
|
3. Don't test in production.
|
|
|
|
Follow the steps below to start generating `values.yaml`.
|
|
|
|
<details>
|
|
<summary>
|
|
Create a `values.yaml` file and a `global` configuration
|
|
</summary>
|
|
|
|
Create a new `values.yaml` file on your machine. In that file, create your basic global configuration.
|
|
|
|
```yaml title="values.yaml"
|
|
global:
|
|
edition: community
|
|
|
|
airbyteUrl: "" # The URL where Airbyte will be reached; This should match your Ingress host
|
|
```
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary>
|
|
Add your database (if applicable)
|
|
</summary>
|
|
|
|
If you're not using an external database, skip this step.
|
|
|
|
If you are using an external database, disable Airbyte's default Postgres database and add your own. The main difference in Helm chart V2 is the `global.database.database` key has changed to `global.database.name`.
|
|
|
|
```yaml title="values.yaml"
|
|
global:
|
|
database:
|
|
# -- Secret name where database credentials are stored
|
|
secretName: "" # e.g. "airbyte-config-secrets"
|
|
# -- The database host
|
|
host: ""
|
|
# -- The database port
|
|
port:
|
|
# -- The database name - this key used to be "database" in Helm chart 1.0
|
|
name: ""
|
|
|
|
# Use EITHER user or userSecretKey, but not both
|
|
# -- The database user
|
|
user: ""
|
|
# -- The key within `secretName` where the user is stored
|
|
userSecretKey: "" # e.g. "database-user"
|
|
|
|
# Use EITHER password or passwordSecretKey, but not both
|
|
# -- The database password
|
|
password: ""
|
|
# -- The key within `secretName` where the password is stored
|
|
passwordSecretKey: "" # e.g."database-password"
|
|
|
|
postgresql:
|
|
enabled: false
|
|
```
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary>
|
|
Add external log storage (if applicable)
|
|
</summary>
|
|
|
|
If you're using external log storage, implement it. If you aren't, skip this step.
|
|
|
|
```yaml
|
|
global:
|
|
storage:
|
|
secretName: ""
|
|
type: minio # default storage is minio. Set to s3, gcs, or azure, according to what you use.
|
|
|
|
bucket:
|
|
log: airbyte-bucket
|
|
state: airbyte-bucket
|
|
workloadOutput: airbyte-bucket
|
|
activityPayload: airbyte-bucket
|
|
|
|
# Set ONE OF the following storage types, according to your specification above
|
|
|
|
# S3
|
|
s3:
|
|
region: "" ## e.g. us-east-1
|
|
authenticationType: credentials ## Use "credentials" or "instanceProfile"
|
|
accessKeyId: ""
|
|
secretAccessKey: ""
|
|
|
|
# GCS
|
|
gcs:
|
|
projectId: <project-id>
|
|
credentialsJson: <base64-encoded>
|
|
credentialsJsonPath: /secrets/gcs-log-creds/gcp.json
|
|
|
|
# Azure
|
|
azure:
|
|
# one of the following: connectionString, connectionStringSecretKey
|
|
connectionString: <azure storage connection string>
|
|
connectionStringSecretKey: <secret coordinate containing an existing connection-string secret>
|
|
```
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary>
|
|
Add external connector secret management (if applicable)
|
|
</summary>
|
|
|
|
If you're using external secret management, implement it. If you aren't, skip this step.
|
|
|
|
```yaml
|
|
global:
|
|
secretsManager:
|
|
enabled: false
|
|
type: "" # one of: VAULT, GOOGLE_SECRET_MANAGER, AWS_SECRET_MANAGER, AZURE_KEY_VAULT, TESTING_CONFIG_DB_TABLE
|
|
secretName: "airbyte-config-secrets"
|
|
|
|
# Set ONE OF the following groups of configurations, based on your configuration in global.secretsManager.type.
|
|
|
|
awsSecretManager:
|
|
region: <aws-region>
|
|
authenticationType: credentials ## Use "credentials" or "instanceProfile"
|
|
tags: ## Optional - You may add tags to new secrets created by Airbyte.
|
|
- key: ## e.g. team
|
|
value: ## e.g. deployments
|
|
- key: business-unit
|
|
value: engineering
|
|
kms: ## Optional - ARN for KMS Decryption.
|
|
|
|
# OR
|
|
|
|
googleSecretManager:
|
|
projectId: <project-id>
|
|
credentialsSecretKey: gcp.json
|
|
|
|
# OR
|
|
|
|
azureKeyVault:
|
|
tenantId: ""
|
|
vaultUrl: ""
|
|
clientId: ""
|
|
clientIdSecretKey: ""
|
|
clientSecret: ""
|
|
clientSecretSecretKey: ""
|
|
tags: ""
|
|
|
|
# OR
|
|
|
|
vault:
|
|
address: ""
|
|
prefix: ""
|
|
authToken: ""
|
|
authTokenSecretKey: ""
|
|
```
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary>
|
|
Update syntax for other customizatons
|
|
</summary>
|
|
|
|
If you have further customizations in your V1 values.yaml file, move those over to your new values.yaml file, and update key names where appropriate.
|
|
|
|
- Change hyphenated V1 keys keys to camel case in V2. For example, when copying over `workload-launcher`, change it to `workloadLauncher`.
|
|
|
|
- Some keys have different names. For example, `orchestrator` is `containerOrchestrator` in V2.
|
|
|
|
Here is the full list of changes.
|
|
|
|
<HelmSyntaxConversion/>
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary>
|
|
Convert `extraEnv` variables
|
|
</summary>
|
|
|
|
In previous versions of your values.yaml file, you might have specified a number of environment variables through `extraEnv`. Many (but not all) of these variables have a dedicated interface in Helm chart V2. For example, look at the following configuration, which tells `workload-launcher` to run pods in the `jobs` node group.
|
|
|
|
```yaml title="values.yaml using Helm chart V1"
|
|
workload-launcher:
|
|
nodeSelector:
|
|
type: static
|
|
## Pods spun up by the workload launcher will run in the 'jobs' node group.
|
|
extraEnv:
|
|
- name: JOB_KUBE_NODE_SELECTORS
|
|
value: type=jobs
|
|
- name: SPEC_JOB_KUBE_NODE_SELECTORS
|
|
value: type=jobs
|
|
- name: CHECK_JOB_KUBE_NODE_SELECTORS
|
|
value: type=jobs
|
|
- name: DISCOVER_JOB_KUBE_NODE_SELECTORS
|
|
value: type=jobs
|
|
```
|
|
|
|
You can specify these values directly without using environment variables, achieving the same effect.
|
|
|
|
```yaml title="values.yaml using Helm chart V2"
|
|
global:
|
|
jobs:
|
|
kube:
|
|
nodeSelector:
|
|
type: jobs
|
|
scheduling:
|
|
check:
|
|
nodeSelectors:
|
|
type: jobs
|
|
discover:
|
|
nodeSelectors:
|
|
type: jobs
|
|
spec:
|
|
nodeSelectors:
|
|
type: jobs
|
|
|
|
workloadLauncher:
|
|
nodeSelector:
|
|
type: static
|
|
```
|
|
|
|
<EnvironmentVarConversion/>
|
|
|
|
</details>
|
|
|
|
### Deploy Airbyte {#deploy-airbyte}
|
|
|
|
<Tabs groupId="external-db">
|
|
<TabItem value='db-yes' label='External database' default>
|
|
|
|
1. Identify the Helm chart version that corresponds to the platform version you want to run. Most Helm chart versions are designed to work with one Airbyte version, and they don't necessarily have the same version number.
|
|
|
|
```bash
|
|
helm search repo airbyte-v2 --versions
|
|
```
|
|
|
|
You should see something like this:
|
|
|
|
```text
|
|
NAME CHART VERSION APP VERSION DESCRIPTION
|
|
airbyte-v2/airbyte 2.0.18 2.0.0 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.17 1.8.5 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.16 1.8.4 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.15 1.8.4 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.14 1.8.4 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.13 1.8.3 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.12 1.8.2 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.11 1.8.2 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.10 1.8.1 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.9 1.8.0 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.8 1.8.0 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.7 1.7.1 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.6 1.7.1 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.5 1.7.0 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.4 1.6.3 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.3 1.6.2 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.2 1.6.2 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.1 1.6.1 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.0 1.6.0 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte-data-plane 2.0.0 2.0.0 A Helm chart for installing an Airbyte Data Plane.
|
|
```
|
|
|
|
2. Install Airbyte into your Helm chart V2 namespace. In this example, you install Airbyte version 2.0.
|
|
|
|
```bash
|
|
helm install airbyte airbyte-v2/airbyte \
|
|
--namespace airbyte-v2 \ # Target Kubernetes namespace
|
|
--values ./values.yaml \ # Custom configuration values
|
|
--version 2.0.18 # Helm chart version to use
|
|
```
|
|
|
|
</TabItem>
|
|
<TabItem value='db-no' label='No external database' default>
|
|
|
|
:::danger
|
|
Do not proceed if you have not backed up Airbyte's internal Postgres database.
|
|
:::
|
|
|
|
1. Uninstall Airbyte. **Do not delete the Airbyte namespace**. This example assumes you installed Airbyte in the `airbyte` namespace.
|
|
|
|
```bash
|
|
helm uninstall airbyte -n airbyte
|
|
```
|
|
|
|
2. Identify the Helm chart version that corresponds to the platform version you want to run. Most Helm chart versions are designed to work with one Airbyte version, but they don't necessarily have the same version number as the Airbyte platform.
|
|
|
|
```bash
|
|
helm search repo airbyte-v2 --versions
|
|
```
|
|
|
|
You see something like this:
|
|
|
|
```text
|
|
NAME CHART VERSION APP VERSION DESCRIPTION
|
|
airbyte-v2/airbyte 2.0.18 2.0.0 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.17 1.8.5 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.16 1.8.4 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.15 1.8.4 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.14 1.8.4 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.13 1.8.3 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.12 1.8.2 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.11 1.8.2 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.10 1.8.1 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.9 1.8.0 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.8 1.8.0 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.7 1.7.1 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.6 1.7.1 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.5 1.7.0 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.4 1.6.3 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.3 1.6.2 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.2 1.6.2 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.1 1.6.1 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte 2.0.0 1.6.0 Helm chart to deploy airbyte
|
|
airbyte-v2/airbyte-data-plane 2.0.0 2.0.0 A Helm chart for installing an Airbyte Data Plane.
|
|
```
|
|
|
|
3. Reinstall Airbyte into the same namespace as before. In this example, you install Airbyte version 2.0 using the Helm chart version 2.0.18.
|
|
|
|
```bash
|
|
helm install airbyte airbyte-v2/airbyte \
|
|
--namespace airbyte \ # Target Kubernetes namespace
|
|
--values ./values.yaml \ # Custom configuration values
|
|
--version 2.0.18 \ # Helm chart version to use
|
|
```
|
|
|
|
Airbyte continues in this namespace as before, now running on Helm Chart V2.
|
|
|
|
#### Restore your database
|
|
|
|
You only need to restore your database in exceptional circumstances. Only restore your database if there is an issue with your deployment that isn't related to your configurations. The restoration process is essentially the reverse of the backup process you went through above.
|
|
|
|
These examples assume your namespace is still `airbyte` and your database pod name is still `airbyte-db-0`.
|
|
|
|
1. Copy the SQL backup file into your pod's /tmp directory.
|
|
|
|
```bash
|
|
kubectl cp ./airbyte_backup.sql airbyte/airbyte-db-0:/tmp/airbyte_backup.sql
|
|
```
|
|
|
|
2. Since your original database probably already contains data, drop and recreate the schema. This erases all data in the database.
|
|
|
|
```bash
|
|
kubectl exec -n airbyte -it airbyte-db-0 -- bash -c "psql -U airbyte -d db-airbyte -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'"
|
|
```
|
|
|
|
3. Restore the database.
|
|
|
|
```bash
|
|
kubectl exec -n airbyte -it airbyte-db-0 -- bash -c "psql -U airbyte -d db-airbyte < /tmp/airbyte_backup.sql"
|
|
```
|
|
|
|
</TabItem>
|
|
</Tabs>
|