Docs cloud enterprise (#65578)
Co-authored-by: ian-at-airbyte <ian.alton@airbyte.io> Co-authored-by: Alexandre Cuoci <Hesperide@users.noreply.github.com>
This commit is contained in:
Matteo Palarchio
@@ -218,12 +218,12 @@ Some platform features are reserved for certain Airbyte products. To avoid confu
|
||||
|
||||
To enable badges, include `products` in the Markdown metadata. The following values are possible, and you can combine them as needed.
|
||||
|
||||
- `all`: Self-Managed Community, Self-Managed Enterprise, and Cloud (doesn't include Cloud Teams, Cloud Enterprise, or Embedded)
|
||||
- `all`: Self-Managed Community, Self-Managed Enterprise, and Cloud (doesn't include Cloud Teams, Enterprise Flex, or Embedded)
|
||||
- `oss-community`: Self-Managed Community only
|
||||
- `oss-enterprise`: Self-Managed Enterprise only
|
||||
- `cloud`: Cloud only
|
||||
- `cloud-teams`: Cloud Teams only
|
||||
- `cloud-enterprise`: Cloud Enterprise only
|
||||
- `enterprise-flex`: Enterprise Flex only
|
||||
- `embedded`: Embedded only (hidden if not specified - there is no off state for the Embedded badge)
|
||||
|
||||
In this example, the Self-Managed Community badge is highlighted, and Cloud and Self-Managed Enterprise badges are grayed out.
|
||||
@@ -242,10 +242,10 @@ In this example, the Self-Managed Community badge is grayed out, but the Cloud T
|
||||
|
||||
```markdown
|
||||
---
|
||||
products: cloud-teams, oss-enterprise
|
||||
products: cloud-teams, enterprise-flex, oss-enterprise
|
||||
---
|
||||
|
||||
# This topic is only for Cloud Teams and Self-Managed Enterprise
|
||||
# This topic is only for Cloud Teams, Enterprise Flex, and Self-Managed Enterprise
|
||||
|
||||
Some text.
|
||||
```
|
||||
|
||||
451
docs/platform/enterprise-flex/data-plane.md
Normal file
451
docs/platform/enterprise-flex/data-plane.md
Normal file
@@ -0,0 +1,451 @@
|
||||
---
|
||||
products: enterprise-flex
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# Multiple region deployments
|
||||
|
||||
Airbyte Enterprise Flex customers can use Airbyte's public API to define regions and create independent data planes that operate in those regions. This ensures you're satisfying your data residency and governance requirements with a single Airbyte Cloud deployment, and it can help you reduce data egress costs with cloud providers.
|
||||
|
||||
|
||||
|
||||
## How it works
|
||||
If you're not familiar with Kubernetes, think of the control plane as the brain and data planes as the muscles doing work the brain tells them to do.
|
||||
|
||||
- The control plane is responsible for Airbyte's user interface, APIs, Terraform provider, and orchestrating work. Airbyte manages this for you in the cloud, reducing the time and resources it takes to start moving your data.
|
||||
- The data plane initiates jobs, syncs data, completes jobs, and reports its status back to the control plane. We offer [cloud regions](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-data-residency) equipped to do this for you, but you also have the flexibility to deploy your own to keep sensitive data protected or meet local data residency requirements.
|
||||
|
||||
This separation of duties is what allows a single Airbyte deployment to ensure your data remains segregated and compliant.
|
||||
|
||||
By default, Airbyte has a single data plane that any workspace in the organization can access, and it's automatically tied to the default workspace when Airbyte first starts. To configure additional data planes and regions, complete these steps.
|
||||
|
||||
If you have not already, ensure you have the [required infrastructure](https://docs.airbyte.com/platform/enterprise-flex/getting-started) to run your data plane.
|
||||
|
||||
1. [Create a region](#step-1).
|
||||
2. [Create a data plane](#step-2) in that region.
|
||||
3. [Configure Kubernetes secrets](#step-3).
|
||||
4. [Create your values.yaml file](#step-4).
|
||||
5. [Deploy your data plane](#step-5).
|
||||
6. [Associate your region to an Airbyte workspace](#step-6). You can tie each workspace to exactly one region.
|
||||
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before you begin, make sure you've completed the following:
|
||||
|
||||
- You must be an Organization Administrator to manage regions and data planes.
|
||||
- You need a Kubernetes cluster on which your data plane can run. For example, if you want your data plane to run on eu-west-1, create an EKS cluster on eu-west-1.
|
||||
- You need to use a [secrets manager](https://docs.airbyte.com/platform/deploying-airbyte/integrations/secrets) for the connections on your data plane. Modifying the configuration of connector secret storage will cause all existing connectors to fail, so we recommend only using newly created workspaces on the data plane.
|
||||
- If you haven't already, get access to Airbyte's API by creating an application and generating an access token. For help, see [Configuring API access](https://docs.airbyte.com/platform/using-airbyte/configuring-api-access).
|
||||
|
||||
## 1. Create a region {#step-1}
|
||||
|
||||
The first step is to create a region. Regions are objects that contain data planes, and which you associate to workspaces.
|
||||
|
||||
<details>
|
||||
<summary>Request</summary>
|
||||
|
||||
Send a POST request to /v1/regions/.
|
||||
|
||||
```bash
|
||||
curl --request POST \
|
||||
--url https://api.airbyte.com/v1/regions \
|
||||
--header "Authorization: Bearer $TOKEN" \
|
||||
--header "Content-Type: application/json" \
|
||||
--data '{
|
||||
"name": "aws-us-east-1",
|
||||
"organizationId": "00000000-0000-0000-0000-000000000000"
|
||||
}'
|
||||
```
|
||||
Include the following parameters in your request.
|
||||
|
||||
| Body parameter | Required? | Description |
|
||||
| ---------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `name` | Required | The name of your region in Airbyte. We reccomend as best practice that you include the cloud provider (if applicable), and actual region in the name. |
|
||||
| `organizationId` | Required | Your Airbyte organization ID. To find this in the UI, navigate to `Settings` > `General`. |
|
||||
| `enabled` | Optional | Defaults to true. Set this to `false` if you don't want this region enabled. |
|
||||
|
||||
For additional request examples, see [the API reference](https://reference.airbyte.com/reference/regions#/).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>Response</summary>
|
||||
|
||||
Make note of your `regionId`. You need it to create a data plane.
|
||||
|
||||
```json title="200 Successful operation"
|
||||
{
|
||||
//highlight-next-line
|
||||
"regionId": "uuid-string",
|
||||
"name": "region-name",
|
||||
"organizationId": "org-uuid-string",
|
||||
"enabled": true,
|
||||
"createdAt": "timestamp-string",
|
||||
"updatedAt": "timestamp-string"
|
||||
}
|
||||
```
|
||||
</details>
|
||||
|
||||
## 2. Create a data plane {#step-2}
|
||||
|
||||
Once you have a region, you create a data plane within it.
|
||||
|
||||
<details>
|
||||
<summary>Request</summary>
|
||||
|
||||
Send a POST request to /v1/dataplanes.
|
||||
|
||||
```bash
|
||||
curl -X POST https://api.airbyte.com/v1/dataplanes \
|
||||
--header "Authorization: Bearer $TOKEN" \
|
||||
--header "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "aws-us-east-1",
|
||||
"regionId": "00000000-0000-0000-0000-000000000000"
|
||||
}'
|
||||
```
|
||||
|
||||
Include the following parameters in your request.
|
||||
|
||||
| Body parameter | Required? | Description |
|
||||
| -------------- | --------- | ------------------------------------------------------------------------------------------------------------------- |
|
||||
| `name` | Required | The name of your data plane. For simplicity, you might want to name it based on the region in which you created it. |
|
||||
| `regionId` | Optional | The region this data plane belongs to. |
|
||||
|
||||
For additional request examples, see [the API reference](https://reference.airbyte.com/reference/dataplanes#/).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>Response</summary>
|
||||
|
||||
Make note of your `dataplaneId`, `clientId` and `clientSecret`. You need these values later to deploy your data plane on Kubernetes.
|
||||
|
||||
```json title="200 Successful operation"
|
||||
json
|
||||
{
|
||||
"dataplaneId": "uuid-string",
|
||||
"clientId": "client-id-string",
|
||||
"clientSecret": "client-secret-string"
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
## 3. Configure Kubernetes Secrets {#step-4}
|
||||
|
||||
Your data plane relies on Kubernetes secrets to identify itself with the control plane.
|
||||
|
||||
In step 5, you create a values.yaml file that references this Kubernetes secret store and these secret keys. Configure all required secrets before deploying your data plane.
|
||||
|
||||
You may apply your Kubernetes secrets by applying the example manifests below to your cluster, or using kubectl directly. If your Kubernetes cluster already has permissions to make requests to an external entity via an instance profile, credentials aren't required. For example, if your Amazon EKS cluster has a sufficient AWS IAM role to make requests to AWS S3, you don't need to specify access keys.
|
||||
|
||||
While you can set the name of the secret to whatever you prefer, you need to set that name in your values.yaml file. For this reason it's easiest to keep the name of airbyte-config-secrets unless you have a reason to change it.
|
||||
|
||||
<details>
|
||||
<summary>airbyte-config-secrets</summary>
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="S3" label="S3" default>
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: airbyte-config-secrets
|
||||
type: Opaque
|
||||
data:
|
||||
# Insert the data plane credentials received in step 2
|
||||
DATA_PLANE_CLIENT_ID: your-data-plane-client-id
|
||||
DATA_PLANE_CLIENT_SECRET: your-data-plane-client-id
|
||||
|
||||
# Only set these values if they are also set on your control plane
|
||||
AWS_SECRET_MANAGER_ACCESS_KEY_ID: your-aws-secret-manager-access-key
|
||||
AWS_SECRET_MANAGER_SECRET_ACCESS_KEY: your-aws-secret-manager-secret-key
|
||||
S3_ACCESS_KEY_ID: your-s3-access-key
|
||||
S3_SECRET_ACCESS_KEY: your-s3-secret-key
|
||||
```
|
||||
|
||||
Apply your secrets manifest in your command-line tool with `kubectl`: `kubectl apply -f <file>.yaml -n <namespace>`.
|
||||
|
||||
You can also use `kubectl` to create the secret directly from the command-line tool:
|
||||
|
||||
```bash
|
||||
kubectl create secret generic airbyte-config-secrets \
|
||||
--from-literal=license-key='' \
|
||||
--from-literal=data_plane_client_id='' \
|
||||
--from-literal=data_plane_client_secret='' \
|
||||
--from-literal=s3-access-key-id='' \
|
||||
--from-literal=s3-secret-access-key='' \
|
||||
--from-literal=aws-secret-manager-access-key-id='' \
|
||||
--from-literal=aws-secret-manager-secret-access-key='' \
|
||||
--namespace airbyte
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="GCS" label="GCS">
|
||||
|
||||
First, create a new file `gcp.json` containing the credentials JSON blob for the service account you are looking to assume.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: airbyte-config-secrets
|
||||
type: Opaque
|
||||
stringData:
|
||||
# Insert the data plane credentials received in step 2
|
||||
DATA_PLANE_CLIENT_ID: your-data-plane-client-id
|
||||
DATA_PLANE_CLIENT_SECRET: your-data-plane-client-id
|
||||
|
||||
# Only set these values if they are also set on your control plane
|
||||
AWS_SECRET_MANAGER_ACCESS_KEY_ID: your-aws-secret-manager-access-key
|
||||
AWS_SECRET_MANAGER_SECRET_ACCESS_KEY: your-aws-secret-manager-secret-key
|
||||
S3_ACCESS_KEY_ID: your-s3-access-key
|
||||
S3_SECRET_ACCESS_KEY: your-s3-secret-key
|
||||
|
||||
# GCP Secrets
|
||||
gcp.json: <CREDENTIALS_JSON_BLOB>
|
||||
```
|
||||
|
||||
Apply your secrets manifest in your command-line tool with `kubectl`: `kubectl apply -f <file>.yaml -n <namespace>`.
|
||||
|
||||
You can also use `kubectl` to create the secret directly from the command-line tool:
|
||||
|
||||
```bash
|
||||
kubectl create secret generic airbyte-config-secrets \
|
||||
--from-literal=license-key='' \
|
||||
--from-literal=data_plane_client_id='' \
|
||||
--from-literal=data_plane_client_secret='' \
|
||||
--from-literal=s3-access-key-id='' \
|
||||
--from-literal=s3-secret-access-key='' \
|
||||
--from-literal=aws-secret-manager-access-key-id='' \
|
||||
--from-literal=aws-secret-manager-secret-access-key='' \
|
||||
--from-file=gcp.json
|
||||
--namespace airbyte
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
</details>
|
||||
|
||||
## 5. Create your deployment values {#step-5}
|
||||
|
||||
Add the following overrides to a new `values.yaml` file.
|
||||
|
||||
```yaml title="values.yaml"
|
||||
airbyteUrl: https://airbyte.com # Base URL for the control plane so Airbyte knows where to authenticate
|
||||
# Logging:
|
||||
# level: DEBUG
|
||||
|
||||
dataPlane:
|
||||
# Used to render the data plane creds secret into the Helm chart.
|
||||
secretName: airbyte-config-secrets
|
||||
id: "preview-data-plane"
|
||||
|
||||
# Describe secret name and key where each of the client ID and secret are stored
|
||||
clientIdSecretName: airbyte-config-secrets
|
||||
clientIdSecretKey: "DATA_PLANE_CLIENT_ID"
|
||||
clientSecretSecretName: airbyte-config-secrets
|
||||
clientSecretSecretKey: "DATA_PLANE_CLIENT_SECRET"
|
||||
|
||||
|
||||
# S3 bucket secrets/config
|
||||
# Only set this section if you are using a self-managed bucket, otherwise it can be omitted.
|
||||
storage:
|
||||
secretName: airbyte-config-secrets
|
||||
type: "s3"
|
||||
bucket:
|
||||
log: my-bucket-name
|
||||
state: my-bucket-name
|
||||
workloadOutput: my-bucket-name
|
||||
s3:
|
||||
region: "us-west-2"
|
||||
authenticationType: credentials
|
||||
accessKeyIdSecretKey: S3_ACCESS_KEY_ID
|
||||
secretAccessKeySecretKey: S3_SECRET_ACCESS_KEY
|
||||
|
||||
# Secret manager secrets/config
|
||||
secretsManager:
|
||||
secretName: airbyte-config-secrets
|
||||
type: AWS_SECRET_MANAGER
|
||||
awsSecretManager:
|
||||
region: us-west-2
|
||||
authenticationType: credentials
|
||||
accessKeyIdSecretKey: AWS_SECRET_MANAGER_ACCESS_KEY_ID
|
||||
secretAccessKeySecretKey: AWS_SECRET_MANAGER_SECRET_ACCESS_KEY
|
||||
```
|
||||
|
||||
## 6. Deploy your data plane {#step-6}
|
||||
|
||||
In your command-line tool, deploy the data plane using `helm upgrade`. The examples here may not reflect your actual Airbyte version and namespace conventions, so make sure you use the settings that are appropriate for your environment.
|
||||
|
||||
```bash title="Example using the default namespace in your cluster"
|
||||
helm upgrade --install airbyte-enterprise airbyte/airbyte-data-plane --version 1.8.1 --values values.yaml
|
||||
```
|
||||
|
||||
```bash title="Example using or creating a namespace called 'airbyte-dataplane'"
|
||||
helm upgrade --install airbyte-enterprise airbyte/airbyte-data-plane --version 1.8.1 -n airbyte-dataplane --create-namespace --values values.yaml
|
||||
```
|
||||
|
||||
## 6. Associate a region to a workspace {#step-6}
|
||||
|
||||
One you have a region and a data plane, you need to associate that region to your workspace. You can associate a workspace with a region when you create that workspace or later, after it exists.
|
||||
|
||||
:::note
|
||||
You can only associate each workspace with one region.
|
||||
:::
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="workspace-association-ui" label="UI" default>
|
||||
|
||||
Follow these steps to associate your region to your current workspace using Airbyte's user interface.
|
||||
|
||||
1. In the navigation panel, click **Settings**.
|
||||
|
||||
2. Under **Workspace**, click **General**.
|
||||
|
||||
3. Under **Region**, select your region.
|
||||
|
||||
4. Click **Save changes**. Now, run any sync. You will see the workloads spin up in the new data plane you've configured.
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="workspace-association-api" label="API">
|
||||
|
||||
When creating a new workspace:
|
||||
|
||||
<details>
|
||||
<summary>Request</summary>
|
||||
|
||||
Send a POST request to /v1/workspaces/
|
||||
|
||||
```bash
|
||||
curl -X POST "api.airbyte.com/v1/workspaces" \
|
||||
--header "Authorization: Bearer $TOKEN" \
|
||||
--header "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "My New Workspace",
|
||||
"dataResidency": "auto"
|
||||
}'
|
||||
```
|
||||
|
||||
Include the following parameters in your request.
|
||||
|
||||
| Body parameter | Description |
|
||||
| --------------- | --------------------------------------------------------- |
|
||||
| `name` | The name of your workspace in Airbyte. |
|
||||
| `dataResidency` | A string with a region identifier you received in step 1. |
|
||||
|
||||
For additional request examples, see [the API reference](https://reference.airbyte.com/reference/workspaces#/).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>Response</summary>
|
||||
|
||||
```json
|
||||
{
|
||||
"workspaceId": "uuid-string",
|
||||
"name": "workspace-name",
|
||||
"dataResidency": "auto",
|
||||
"notifications": {
|
||||
"failure": {},
|
||||
"success": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
When updating a workspace:
|
||||
|
||||
<details>
|
||||
<summary>Request</summary>
|
||||
|
||||
Send a PATCH request to /v1/workspaces/`{workspaceId}`.
|
||||
|
||||
```bash
|
||||
curl -X PATCH "https://api.airbyte.com/v1/workspaces/{workspaceId}" \
|
||||
--header "Authorization: Bearer $TOKEN" \
|
||||
--header "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "Updated Workspace Name",
|
||||
"dataResidency": "us-west"
|
||||
}'
|
||||
```
|
||||
|
||||
Include the following parameters in your request.
|
||||
|
||||
| Body parameter | Description |
|
||||
| --------------- | --------------------------------------------------------- |
|
||||
| `name` | The name of your workspace in Airbyte. |
|
||||
| `dataResidency` | A string with a region identifier you received in step 1. |
|
||||
|
||||
For additional request examples, see [the API reference](https://reference.airbyte.com/reference/workspaces#/).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>Response</summary>
|
||||
|
||||
```json
|
||||
{
|
||||
"workspaceId": "uuid-string",
|
||||
"name": "updated-workspace-name",
|
||||
"dataResidency": "region-identifier",
|
||||
"notifications": {
|
||||
"failure": {},
|
||||
"success": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
## Check which region your workspaces use
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="check-regions" label="UI" default>
|
||||
|
||||
You can see a list of your workspaces and the region associated to each from Airbyte's organization settings.
|
||||
|
||||
1. In Airbyte's user interface, click **Settings**.
|
||||
|
||||
2. Under **Organization**, click **General**.
|
||||
|
||||
Airbyte displays your workspaces and each workspace region under **Regions**.
|
||||
|
||||
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="check-regions-api" label="API">
|
||||
|
||||
Request:
|
||||
|
||||
```bash
|
||||
bash
|
||||
curl -X GET "https://api.airbyte.com/v1/workspaces/{workspaceId}" \
|
||||
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
|
||||
-H "Content-Type: application/json"
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"workspaceId": "18dccc91-0ab1-4f72-9ed7-0b8fc27c5826",
|
||||
"name": "Acme Company",
|
||||
//highlight-next-line
|
||||
"dataResidency": "auto",
|
||||
}
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
86
docs/platform/enterprise-flex/getting-started.md
Normal file
86
docs/platform/enterprise-flex/getting-started.md
Normal file
@@ -0,0 +1,86 @@
|
||||
---
|
||||
products: enterprise-flex
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# Getting started
|
||||
|
||||
Any Airbyte Cloud enviornment can be easily upgraded to Enterprise Flex. To learn more about upgrading to Enterprise Flex, [talk to sales](https://airbyte.com/company/talk-to-sales).
|
||||
|
||||
You may choose to run a self-managed data plane while using Airbyte Enterprise Flex.
|
||||
|
||||
**If you are not using any self-managed data planes, then no additional infrastructure is required to begin creating connections and running syncs.**
|
||||
|
||||
The following diagram illustrates a typical Airbyte Enterpris Flex deployment running a self-managed data plane:
|
||||
|
||||
|
||||
|
||||
## Infrastructure prerequisites
|
||||
|
||||
For a production-ready deployment of self-managed data planes, various infrastructure components are required. We recommend deploying to Amazon EKS, Google Kubernetes Engine,
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="Amazon EKS" label="Amazon EKS" default>
|
||||
|
||||
| Component | Recommendation |
|
||||
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Kubernetes Cluster | Amazon EKS cluster running on EC2 instances in [2 or more availability zones](https://docs.aws.amazon.com/eks/latest/userguide/disaster-recovery-resiliency.html). |
|
||||
| External Secrets Manager | [Amazon Secrets Manager](/platform/operator-guides/configuring-airbyte#secrets) for storing connector secrets.
|
||||
| Object Storage (Optional)| Amazon S3 bucket with a directory for log storage. |
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
A few notes on Kubernetes cluster provisioning for Airbyte Self-Managed Enterprise:
|
||||
|
||||
- We support Amazon Elastic Kubernetes Service (EKS) on EC2, Google Kubernetes Engine (GKE) on Google Compute Engine (GCE), or Azure Kubernetes Service (AKS) on Azure.
|
||||
- While we support GKE Autopilot, we do not support Amazon EKS on Fargate.
|
||||
|
||||
We require you to install and configure the following Kubernetes tooling:
|
||||
|
||||
1. Install `helm` by following [these instructions](https://helm.sh/docs/intro/install/)
|
||||
2. Install `kubectl` by following [these instructions](https://kubernetes.io/docs/tasks/tools/).
|
||||
3. Configure `kubectl` to connect to your cluster by using `kubectl use-context my-cluster-name`:
|
||||
|
||||
<details>
|
||||
<summary>Configure kubectl to connect to your cluster</summary>
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="Amazon EKS" label="Amazon EKS" default>
|
||||
|
||||
1. Configure your [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) to connect to your project.
|
||||
2. Install [eksctl](https://eksctl.io/introduction/).
|
||||
3. Run `eksctl utils write-kubeconfig --cluster=$CLUSTER_NAME` to make the context available to kubectl.
|
||||
4. Use `kubectl config get-contexts` to show the available contexts.
|
||||
5. Run `kubectl config use-context $EKS_CONTEXT` to access the cluster with kubectl.
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="GKE" label="GKE">
|
||||
|
||||
1. Configure `gcloud` with `gcloud auth login`.
|
||||
2. On the Google Cloud Console, the cluster page will have a "Connect" button, with a command to run locally: `gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE_NAME --project $PROJECT_NAME`.
|
||||
3. Use `kubectl config get-contexts` to show the available contexts.
|
||||
4. Run `kubectl config use-context $EKS_CONTEXT` to access the cluster with kubectl.
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
</details>
|
||||
|
||||
We also require you to create a Kubernetes namespace for your Airbyte deployment:
|
||||
|
||||
```
|
||||
kubectl create namespace airbyte
|
||||
```
|
||||
|
||||
## Limitations and considerations
|
||||
|
||||
- While data planes process data in their respective regions, some metadata remains in the control plane.
|
||||
- Airbyte stores Cursor and Primary Key data in the control plane regardless of data plane location. If you have data that you can't store in the control plane, don't use it as a cursor or primary key.
|
||||
- The Connector Builder processes all data through the control plane, regardless of workspace settings. This limitation applies to the development and testing phase only; published connectors respect workspace data residency settings during syncs.
|
||||
- If you want to run multiple data planes in the same region for higher availability, both must be part of the same region in Airbyte and use the same secrets manager to ensure connection credentials are the same.
|
||||
- Data planes must be able to communicate with the control plane.
|
||||
- **Data planes will only ever send requests to the control plane and never require inbound requests.**
|
||||
BIN
docs/platform/enterprise-flex/img/data-planes.png
Normal file
BIN
docs/platform/enterprise-flex/img/data-planes.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 593 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 758 KiB |
BIN
docs/platform/enterprise-flex/img/flex-enterprise-example.png
Normal file
BIN
docs/platform/enterprise-flex/img/flex-enterprise-example.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 148 KiB |
53
docs/platform/enterprise-flex/readme.md
Normal file
53
docs/platform/enterprise-flex/readme.md
Normal file
@@ -0,0 +1,53 @@
|
||||
---
|
||||
products: enterprise-flex
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# Enterprise Flex
|
||||
|
||||
Airbyte Enterprise Flex is a hybrid solution that consists of a fully managed Cloud control plane that supports separate data planes that run in your own infrastructure. It's ideal for organizations with data sovereignty and compliance requirements who need to carefully manage data movement while also minimizing the time and effort needed to self-manage Airbyte.
|
||||
|
||||
## Overview
|
||||
|
||||
Many organizations collect data in all types of operational systems from users around the world. However, you might face strict requirements around how you can process and store data, as well as who can access it. These requirements comes in many forms.
|
||||
|
||||
- **Compliance**: GDPR (European Union), PIPEDA (Canada), HIPAA (USA), APPs (Australia), PIPL (China), and other frameworks govern the management of data, and can punish non-compliance with significant financial penalties.
|
||||
|
||||
- **Data sovereignty**: When sensitive data crosses borders, it can be subject to prying eyes and foreign regulations it isn't meant for. Keeping data within a specific country or geographical region helps ensure privacy and control over that data.
|
||||
|
||||
- **Security and operational policies**: Some data, like trade secrets or sensitive industry information, is so valuable that external exposure could present a major incident.
|
||||
|
||||
While these requirements are critical, organizations also have finite time and expertise. Managing these operational and compliance requirements with more infrastructure often means increased maintenance commitments, higher spend, and greater complexity.
|
||||
|
||||
Enterprise Flex addresses these needs by offering fully managed Cloud workspaces (a control plane) that connect to separate data planes you manage in your own infrastructure. You can also use fully managed data planes for less sensitive data that doesn't need to remain in your own infrastructure. Each Cloud workspace uses one region and data plane, so a single Airbyte instance with multiple workspaces is an ideal way to segregate data and connections. Enterprise Flex also supports other enterprise-grade abilities such as audit loggings, external secrets managers, and AWS PrivateLink connectivity in addition to all other features in Cloud Teams.
|
||||
|
||||
| Feature | Description |
|
||||
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| User Management | Enable multiple users to concurrently move data from a single Airbyte deployment. |
|
||||
| Single Sign-On | Manage user access to Airbyte from your Okta, Azure Entra ID or OIDC-compatible identity provider. |
|
||||
| Multiple Workspaces | Manage multiple isolated projects or teams on a single Airbyte deployment. |
|
||||
| Role-Based Access | Manage user permissions and access across workspaces from a single pane of glass. |
|
||||
| Column Hashing | Protect sensitive information by hashing personally identifiable information (PII) as it moves through your pipelines. |
|
||||
| External Secrets | Bring your own secrets manager to securely reference your credentials for data sources and destinations. |
|
||||
| Audit trail logs | Store user and platform activity in your own bucket to maintain compliance while using Airbyte . |
|
||||
| AWS PrivateLink | Connect to data sources or destinations in your VPC securely to Airbyte data planes. |
|
||||
| Support with SLAs | [Priority assistance](https://docs.airbyte.com/operator-guides/contact-support/#airbyte-enterprise-self-hosted-support) with deploying, managing and upgrading Airbyte. |
|
||||
|
||||
### Enterprise Flex versus Cloud Teams
|
||||
|
||||
Enterprise Flex includes all features that are standard in Cloud Teams with the additional capabilities of running self-managed data planes, referencing your own secrets manager, and storing audit logs.
|
||||
|
||||
Any Airbyte Cloud enviornment can be easily upgraded to Enterprise Flex. To learn more about upgrading to Enterprise Flex, [talk to sales](https://airbyte.com/company/talk-to-sales).
|
||||
|
||||
|
||||
## An example hybrid deployment
|
||||
|
||||
Every organization's precise needs differ, so you can implement Enterprise Flex in the way that suits you best. In this example, you have three workspaces.
|
||||
|
||||
- Workspace 1 contains non-sensitive data and uses Airbyte's fully managed European region.
|
||||
|
||||
- Workspaces 2 and 3, which contain sensitive data from the United States and Australia, run on your own infrastructure. Only metadata ever reaches the control plane.
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ products: oss-enterprise
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# Multiple region deployments
|
||||
# Multiple region deployments (Self-Managed Enterprise)
|
||||
|
||||
Self-Managed Enterprise customers can use Airbyte's public API to define regions and create independent data planes that operate in those regions. This ensures you're satisfying your data residency and governance requirements with a single Airbyte deployment, and it can help you reduce data egress costs with cloud providers.
|
||||
|
||||
|
||||
@@ -27,12 +27,13 @@ Airbyte is available as a Self-Managed or Cloud product, depending on your needs
|
||||
|
||||
### Cloud options
|
||||
|
||||
<Grid columns="2">
|
||||
<Grid columns="3">
|
||||
|
||||
<CardWithIcon title="Cloud" description="A Cloud-native version of Airbyte that provides a fully managed experience for data replication. Focus on moving data while Airbyte manages the infrastructure. Free 14-day trial." ctaText="Sign Up for Cloud" ctaLink="https://cloud.airbyte.com/signup" icon="fa-cloud" />
|
||||
|
||||
<CardWithIcon title="Cloud Teams" description="For organizations
|
||||
looking to scale efficiently. Role based access control, single sign on, and more ensure Cloud is a robust solution that can grow with your team." ctaText="Talk to Sales" ctaLink="https://airbyte.com/company/talk-to-sales" icon="fa-lock" />
|
||||
<CardWithIcon title="Cloud Teams" description="For organizations looking to scale efficiently. Role based access control, single sign on, and more ensure Cloud is a robust solution that can grow with your team." ctaText="Talk to Sales" ctaLink="https://airbyte.com/company/talk-to-sales" icon="fa-lock" />
|
||||
|
||||
<CardWithIcon title="Enterprise Flex" description="An enterprise-grade solution with hybrid options for organizatons seeking the convenience of a fully managed solution with separate data planes for compliance and data sovereignty." ctaText="Learn more" ctaLink="enterprise-flex" icon="fa-lock" />
|
||||
|
||||
</Grid>
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ Self-Managed Community
|
||||
Self-Managed Enterprise
|
||||
Cloud
|
||||
Cloud Teams
|
||||
Cloud Enterprise
|
||||
Enterprise Flex
|
||||
abctl
|
||||
PyAirbyte
|
||||
[Nn]amespace
|
||||
|
||||
@@ -394,6 +394,18 @@ module.exports = {
|
||||
"enterprise-setup/chart-v2-enterprise",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Enterprise Flex",
|
||||
link: {
|
||||
type: "doc",
|
||||
id: "enterprise-flex/readme",
|
||||
},
|
||||
items: [
|
||||
"enterprise-flex/getting-started",
|
||||
"enterprise-flex/data-plane"
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Upgrading Airbyte",
|
||||
|
||||
@@ -38,13 +38,13 @@ export const ProductInformation = ({ products }) => {
|
||||
const cloud = products["cloud"] || products["cloud-teams"] || products["all"];
|
||||
// cloud add-ons need to be specifically marked and are not part of the "all" shorthand
|
||||
const cloudTeams = products["cloud-teams"];
|
||||
const cloudEnterprise = products["cloud-enterprise"];
|
||||
const enterpriseFlex = products["enterprise-flex"];
|
||||
const embedded = products["embedded"];
|
||||
|
||||
return (
|
||||
<div className={styles.badges}>
|
||||
{cloudEnterprise ? (
|
||||
<Badge available={true}>Cloud Enterprise</Badge>
|
||||
{enterpriseFlex ? (
|
||||
<Badge available={true}>Enterprise Flex</Badge>
|
||||
) : (
|
||||
<Badge available={cloud}>
|
||||
Cloud{" "}
|
||||
|
||||
Reference in New Issue
Block a user