jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

role mapping & small fixes (#39605)

Browse Source
This commit is contained in:
Alexandre Cuoci
2024-06-20 11:43:43 -04:00
committed by GitHub
parent 28ceaf03ee
commit af78176e37
4 changed files with 181 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files

129
docs/access-management/role-mapping.md Normal file
View File

@@ -0,0 +1,129 @@
---
products: oss-enterprise, cloud-teams
---
# RBAC Role Mapping
Role-Based Access Control (RBAC) role mapping allows automatic assignment of specific permissions to Airbyte users based on existing roles in your organization. It ensures users have appropriate Airbyte access without manual oversight. RBAC functionality is only available in Airbyte Teams and Self-Managed Enterprise.
Enabling role mapping in Airbyte requires use of the Airbyte API. The Airbyte API exposes endpoints that allow you to retrieve and update user permissions. These endpoints can be used to build automation that manages user access to different workspaces. This functionality is currently limited to the Airbyte API, and is not available in the Terraform Provider.
To enable the Airbyte API in Airbyte Teams or Self-Managed Enterprise, follow these [prerequisites](../enterprise-setup/api-access-config.md).
## Relevant API Endpoints
Organization-wide permissions and each set of workspace permissions each count as their own permission object. For example, if an Airbyte user is an 'Organization Member' and has 'Workspace Editor' access in 3 distinct workspaces, this user has 4 permissions in total.
1. [Get a list of current Airbyte users](https://reference.airbyte.com/reference/listusers).
2. [Get a list of current Airbyte workspaces](https://reference.airbyte.com/reference/listworkspaces).
2. [Provide an Airbyte user with access to a new workspace](https://reference.airbyte.com/reference/createpermission).
3. [Get a list of a user's current permissions](https://reference.airbyte.com/reference/listpermissions).
3. [Modify permission scope or level of access](https://reference.airbyte.com/reference/updatepermission).
4. [Delete a permmission](https://reference.airbyte.com/reference/deletepermission).
## Script Example
### Prerequisites
1. A mapping of user emails to your company-specific roles (e.g. `finance-team`, `security-team`, `us-employee`, etc.):
```yaml
{
"user1@company.com": ["companyGroup1", "companyGroup2"],
"user1@company.com": ["companyGroup2", "companyGroup3"]
}
```
2. A mapping of your company-specific roles to desired Airbyte permissions:
```yaml
{
"companyGroup1": [
{
"scope": "workspace", ## Must be set to either 'workspace' or 'organization'.
"scopeId": "workspace1",
"permissionType": "workspace_admin" ## Must be set to valid value, listed https://github.com/airbytehq/airbyte-api-python-sdk/blob/main/src/airbyte_api/models/publicpermissiontype.py.
},
{
"scope": "workspace",
"scopeId": "workspace2",
"permissionType": "workspace_reader"
}
],
"companyGroup2": [
{
"scope": "workspace",
"scopeId": "workspace1",
"permissionType": "workspace_reader"
}
]
}
```
### Complete Python Script
Below is an example Python script using the above prerequisite files and the `airbyte-api` Python package to set user roles programmatically:
<details>
<summary>RBAC Role Mapping Python Example</summary>
```python
import json
import airbyte_api
from airbyte_api import api, models
usersGroupsFile = open('usersGroups.json')
usersGroups = json.load(usersGroupsFile)
groupPermissionsFile = open('groupPermissions.json')
groupPermissions = json.load(groupPermissionsFile)
# 0 - Enter your own credentials to use Airbyte API.
s = airbyte_api.AirbyteAPI(
security=models.Security(
bearer_auth='...'
),
)
# 1 - List all users in your organization. Find your organization ID in the Airbyte settings page.
res = s.users.list_users(request=api.ListUsersRequest(
api.ListUsersRequest(organization_id='00000000-00000000-00000000-00000000')
))
allAirbyteUsers = res.users_response.data
print("all users: ", allAirbyteUsers)
# 2. grant permissions
# for each user
for airbyteUserResponse in allAirbyteUsers:
if airbyteUserResponse.email in usersGroups:
userGroups = usersGroups[airbyteUserResponse.email]
# for each group where user belongs to
for group in userGroups:
if group in groupPermissions:
permissionsToGrant = groupPermissions[group]
# for each permission to create
for permission in permissionsToGrant:
print("permission to grant: ", permission)
if permission["scope"] == "workspace":
# create workspace level permission
permissionCreated = s.permissions.create_permission(
request=models.PermissionCreateRequest(
permission_type=permission["permissionType"],
user_id=airbyteUserResponse.user_id,
workspace_id=permission["scopeId"]
))
elif permission["scope"] == "organization":
# create organization permission
permissionCreated = s.permissions.create_permission(
request=models.PermissionCreateRequest(
permission_type=permission["permissionType"],
user_id=airbyteUserResponse.user_id,
organization_id=permission["scopeId"]
))
else:
print("permission scope not supported!")
```
</details>
With the script enabled, you are free to configure it on a CRON job to run at the frequency of your choice.

40
docs/enterprise-setup/scaling-airbyte.md
View File

@@ -8,15 +8,19 @@ Once you've completed the initial installation of Airbyte Self-Managed Enterpris
## Concurrent Syncs
The primary indicator of increased resource usage in Airbyte is the number of concurrent syncs running at any given time. Each concurrent sync requires at least 3 additional connector pods to be running at once (`orchestrator`, `read`, `write`). This means that 10 concurrent syncs requires 30 additional pods in your namespace. Connector pods last only for the duration of a sync, and will be appended by the ID of the ongoing job.
The primary driver of increased resource usage in Airbyte is the number of concurrent syncs running at any given time. Each concurrent sync requires at least 3 additional connector pods to be running at once (`orchestrator`, `read`, `write`). For example, 10 concurrent syncs require 30 additional pods in your namespace. Connector pods last only for the duration of a sync, and will be appended by the ID of the ongoing job.
If your deployment of Airbyte is intended to run many concurrent syncs at once (e.g. an overnight backfill), we recommend provisioning an increased number of instances. Some connectors are memory and CPU intensive, while others are not. Using an infrastructure monitoring tool, we recommend measuring the following at all times:
If your deployment of Airbyte is intended to run many concurrent syncs at once (e.g. an overnight backfill), you are likely to require an increased number of instances to run all syncs.
### Connector CPU & Memory Settings
Some connectors are memory and CPU intensive, while others are not. Using an infrastructure monitoring tool, we recommend measuring the following at all times:
* Requested CPU %
* CPU Usage %
* Requested Memory %
* Memory Usage %
With high CPU or Memory usage, we recommend scaling up your Airbyte deployment to a higher number of nodes, or reducing the maximum resource usage by any given connector pod. If high _requested_ CPU or memory usage is blocking new pods from being scheduled, while _used_ CPU or memory is low, you may modify connector pod provisioning defaults in your `values.yml` file:
If your nodes are under high CPU or Memory usage, we recommend scaling up your Airbyte deployment to a larger number of nodes, or reducing the maximum resource usage by any given connector pod. If high _requested_ CPU or memory usage is blocking new pods from being scheduled, while _used_ CPU or memory is low, you may modify connector pod provisioning defaults in your `values.yml` file:
```yaml
global:
@@ -25,14 +29,34 @@ global:
jobs:
resources:
limits:
cpu:
memory:
cpu: ## e.g. 250m
memory: ## e.g. 500m
requests:
cpu:
memory:
cpu: ## e.g. 75m
memory: ## e.g. 150m
```
If your Airbyte deployment is underprovisioned, you may often notice occasional 'stuck jobs' that remain in-progress for long periods, with eventual failures related to unavailable pods. If you begin to see such errors, we recommend you follow the troubleshooting steps above.
If your Airbyte deployment is underprovisioned, you may notice occasional 'stuck jobs' that remain in-progress for long periods, with eventual failures related to unavailable pods. Increasing job CPU and memory limits may also allow for increased sync speeds.
### Concurrent Sync Limits
To help rightsize Airbyte deployments and reduce the likelihood of stuck syncs, there are configurable limits to the number of syncs that can be run at once:
```yaml
worker:
extraEnvs: ## We recommend setting both environment variables with a single, shared value.
- name: MAX_SYNC_WORKERS
value: ## e.g. 5
- name: MAX_CHECK_WORKERS
value: ## e.g. 5
```
If you intend to run many syncs at the same time, you may also want to increase the number of worker replicas that run in your Airbyte instance:
```yaml
worker:
replicaCount: ## e.g. 2
```
## Multiple Node Groups

53
docs/enterprise-setup/upgrading-from-community.md
View File

@@ -27,43 +27,19 @@ helm upgrade [RELEASE_NAME] airbyte/airbyte
### Step 2: Configure Self-Managed Enterprise
At this step, please create and fill out the `airbyte.yml` as explained in the [Self-Managed Enterprise implementation guide](./implementation-guide.md#clone--configure-airbyte) in the `configs` directory. You should avoid making any changes to your Airbyte database or log storage at this time. When complete, you should have a completed file matching the following skeleton:
<details>
<summary>Configuring your airbyte.yml file</summary>
```yml
webapp-url: # example: localhost:8080
initial-user:
email:
first-name:
last-name:
username: # your existing Airbyte instance username
password: # your existing Airbyte instance password
license-key:
auth:
identity-providers:
- type: okta
domain:
app-name:
client-id:
client-secret:
```
</details>
Update your `values.yml` file as explained in the [Self-Managed Enterprise implementation guide](./implementation-guide.md). Avoid making any changes to your external database or log storage configuration at this time.
### Step 3: Deploy Self-Managed Enterprise
1. You can now run the following command to upgrade your instance to Self-Managed Enterprise. If you previously included additional `values` files on your existing deployment, be sure to add these here as well:
```sh
helm upgrade [RELEASE_NAME] airbyte/airbyte \
helm upgrade \
--namespace airbyte \
--values ./values.yaml \
--install [RELEASE_NAME] \
--version [RELEASE_VERSION] \
--set-file airbyteYml=./configs/airbyte.yml \
--values ./charts/airbyte/airbyte-pro-values.yaml [... additional --values]
airbyte/airbyte
```
2. Once this is complete, you will need to upgrade your ingress to include the new `/auth` path. The following is a skimmed down definition of an ingress resource you could use for Self-Managed Enterprise:
@@ -79,6 +55,7 @@ metadata:
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
spec:
ingressClassName: nginx
rules:
- host: # host, example: enterprise-demo.airbyte.com
http:
@@ -86,19 +63,27 @@ spec:
- backend:
service:
# format is ${RELEASE_NAME}-airbyte-webapp-svc
name: airbyte-pro-airbyte-webapp-svc
name: airbyte-enterprise-airbyte-webapp-svc
port:
number: # service port, example: 8080
number: 80 # service port, example: 8080
path: /
pathType: Prefix
- backend:
service:
# format is ${RELEASE_NAME}-airbyte-keycloak-svc
name: airbyte-pro-airbyte-keycloak-svc
name: airbyte-enterprise-airbyte-keycloak-svc
port:
number: # service port, example: 8180
number: 8180
path: /auth
pathType: Prefix
- backend:
service:
# format is ${RELEASE_NAME}-airbyte--server-svc
name: airbyte-enterprise-airbyte-server-svc
port:
number: 8001
path: /api/public
pathType: Prefix
```
</details>

1
docusaurus/sidebars.js
View File

@@ -585,6 +585,7 @@ module.exports = {
},
],
},
"access-management/role-mapping"
],
},
{