## Overview This PR addresses critical documentation drift between the Sonar implementation and the Airbyte documentation repository. Based on a comprehensive drift analysis performed on 2025-10-08, this update adds missing API documentation for HIGH PRIORITY features that are fully implemented in Sonar but completely undocumented. Reference: docs-drift-analysis-2025-10-08.md (in sonar repo) ## Changes Summary ### New Documentation Files (3) 1. **workspaces.md** - Workspace Management API - List workspaces with cursor-based pagination - Get, update, and delete workspaces - Workspace statistics and sync operations - Complete pagination guide and filtering examples 2. **authentication.md** - Complete Authentication Guide - All three token types (Operator Bearer, Scoped, Widget) - Region selection (US/EU) - Template filtering via tags during token generation - Token lifecycle management and security best practices - 20+ code examples 3. **tags.md** - Template Tag Management - Tag CRUD operations - Tag selection modes (ANY vs ALL) - Tagging source and connection templates - Tier-based access control examples ### Updated Documentation Files (3) 4. **source-templates.md** - Enhanced with stream management, PATCH endpoint, tag operations, source definition catalogs 5. **connection-templates.md** - Added sync_on_create, non_breaking_changes_preference, cron validation, PATCH endpoint, tag operations 6. **configuring-sources.md** - Added region selection, stream management, JMESPath querying, JSON Patch operations ## Impact - 25+ previously undocumented API endpoints now documented - ~2,500 lines of documentation added - 100+ code examples - 12 HIGH PRIORITY documentation gaps addressed ## Validation All documentation validated against: - Sonar codebase at commit 350a75fe73 - FastAPI route definitions - Pydantic schemas - Existing documentation style ## Breaking Changes None - Documentation only --------- Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
587 lines
15 KiB
Markdown
587 lines
15 KiB
Markdown
# Authentication
|
|
|
|
The Airbyte Embedded API uses a hierarchical authentication system with three types of tokens, each designed for specific use cases and security requirements.
|
|
|
|
## Token types overview
|
|
|
|
| Token Type | Use Case | Scope | Access Level |
|
|
|------------|----------|-------|--------------|
|
|
| **Operator Bearer Token** | Organization management, template creation | Organization-wide | Full access to organization resources |
|
|
| **Scoped Token** | API integration, programmatic workspace access | Single workspace | Limited to specific workspace |
|
|
| **Widget Token** | Embedded widget integration | Single workspace + origin validation | Limited to specific workspace with CORS protection |
|
|
|
|
## Operator bearer token
|
|
|
|
The Operator Bearer Token provides full organization-level access and is used for administrative operations.
|
|
|
|
### Use cases
|
|
|
|
- Creating and managing source templates
|
|
- Creating and managing connection templates
|
|
- Managing workspaces across the organization
|
|
- Generating scoped tokens and widget tokens
|
|
- Administrative API operations
|
|
|
|
### Usage
|
|
|
|
Include the operator token in the `Authorization` header:
|
|
|
|
```bash
|
|
curl https://api.airbyte.ai/api/v1/integrations/templates/sources \
|
|
|
|
-H 'Authorization: Bearer <your_operator_token>' \
|
|
-H 'Content-Type: application/json'
|
|
|
|
```
|
|
|
|
### Security best practices
|
|
|
|
- **Never expose operator tokens in client-side code**
|
|
- Store securely in secrets management system
|
|
- Use scoped tokens for end-user operations
|
|
- Rotate tokens periodically
|
|
- Limit token distribution to trusted administrators only
|
|
|
|
## Scoped token
|
|
|
|
Scoped tokens provide workspace-level access and are designed for allowing end-users to create and edit sources in their workspace.
|
|
|
|
### Use cases
|
|
|
|
- API integrations for managing sources within a specific workspace
|
|
- Multi-tenant applications with isolated workspaces
|
|
|
|
### Features
|
|
|
|
- Workspace-scoped access (cannot access other workspaces)
|
|
- Automatically creates workspace from a workspace name if it doesn't exist
|
|
- Region selection support
|
|
- Embedded in JWT with `io.airbyte.auth.workspace_scope` claim
|
|
|
|
### Generate scoped token
|
|
|
|
#### Endpoint
|
|
|
|
```bash
|
|
POST https://api.airbyte.ai/api/v1/embedded/scoped-token
|
|
|
|
```
|
|
|
|
#### Authentication
|
|
|
|
Requires **Operator Bearer Token**
|
|
|
|
#### Request body
|
|
|
|
| Field | Type | Required | Description |
|
|
|-------|------|----------|-------------|
|
|
| `workspace_name` | string | Yes | Name of the workspace to create or use |
|
|
| `region_id` | UUID | No | Region where workspace should be created (defaults to US) |
|
|
|
|
#### Region ids
|
|
|
|
| Region | Region ID |
|
|
|--------|-----------|
|
|
| US (Default) | `645a183f-b12b-4c6e-8ad3-99e165603450` |
|
|
| EU | `b9e48d61-f082-4a14-a8d0-799a907938cb` |
|
|
|
|
#### Request example
|
|
|
|
**Create scoped token for US workspace:**
|
|
|
|
```bash
|
|
curl -X POST https://api.airbyte.ai/api/v1/embedded/scoped-token \
|
|
|
|
-H 'Authorization: Bearer <your_operator_token>' \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{
|
|
|
|
"workspace_name": "customer_workspace_123"
|
|
}'
|
|
|
|
```
|
|
|
|
**Create scoped token for EU workspace:**
|
|
|
|
```bash
|
|
curl -X POST https://api.airbyte.ai/api/v1/embedded/scoped-token \
|
|
|
|
-H 'Authorization: Bearer <your_operator_token>' \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{
|
|
|
|
"workspace_name": "eu_customer_workspace",
|
|
"region_id": "b9e48d61-f082-4a14-a8d0-799a907938cb"
|
|
}'
|
|
|
|
```
|
|
|
|
#### Response example
|
|
|
|
```json
|
|
{
|
|
"token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
|
|
}
|
|
|
|
```
|
|
|
|
#### Behavior notes
|
|
|
|
- If the workspace already exists, returns a token for the existing workspace
|
|
- If the workspace doesn't exist:
|
|
- Checks if a workspace with the same name exists in Airbyte Cloud
|
|
- If found in cloud, creates local database record
|
|
- If not found, creates new workspace in both Airbyte Cloud and local database
|
|
- The `region_id` is only used when creating a new workspace
|
|
|
|
### Using scoped tokens
|
|
|
|
Once generated, use scoped tokens to access workspace-specific endpoints:
|
|
|
|
```bash
|
|
# List sources in the workspace
|
|
curl https://api.airbyte.ai/api/v1/embedded/sources \
|
|
|
|
-H 'Authorization: Bearer <scoped_token>'
|
|
|
|
# Create a source
|
|
curl -X POST https://api.airbyte.ai/api/v1/embedded/sources \
|
|
|
|
-H 'Authorization: Bearer <scoped_token>' \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{
|
|
|
|
"source_template_id": "template-123",
|
|
"name": "My Data Source"
|
|
}'
|
|
|
|
```
|
|
|
|
### Get scoped token information
|
|
|
|
Retrieve organization and workspace information from a scoped token.
|
|
|
|
#### Endpoint
|
|
|
|
```bash
|
|
GET https://api.airbyte.ai/api/v1/embedded/scoped-token/info
|
|
|
|
```
|
|
|
|
#### Authentication
|
|
|
|
Requires **Scoped Token**
|
|
|
|
#### Request example
|
|
|
|
```bash
|
|
curl https://api.airbyte.ai/api/v1/embedded/scoped-token/info \
|
|
|
|
-H 'Authorization: Bearer <scoped_token>'
|
|
|
|
```
|
|
|
|
#### Response example
|
|
|
|
```json
|
|
{
|
|
"organization_id": "12345678-1234-1234-1234-123456789012",
|
|
"workspace ID": "a1b2c3d4-e5f6-7890-ab12-cd34ef567890"
|
|
}
|
|
|
|
```
|
|
|
|
## Widget token
|
|
|
|
Widget tokens are specialized tokens designed for embedded widget integration with enhanced security features.
|
|
|
|
### Use cases
|
|
|
|
- Embedding the Airbyte configuration widget in your application
|
|
- Providing end-users with a UI to configure data sources
|
|
- Multi-tenant applications with isolated workspaces
|
|
|
|
### Features
|
|
|
|
- All features of scoped tokens
|
|
- **Origin validation** for CORS protection via `allowed_origin`
|
|
- **Template filtering** via tags (both source and connection templates)
|
|
- Base64-encoded payload containing token and pre-configured widget URL
|
|
- Tag selection modes: `any` (at least one tag) or `all` (all tags required)
|
|
|
|
### Generate widget token
|
|
|
|
#### Endpoint
|
|
|
|
```bash
|
|
POST https://api.airbyte.ai/api/v1/embedded/widget-token
|
|
|
|
```
|
|
|
|
#### Authentication
|
|
|
|
Requires **Operator Bearer Token**
|
|
|
|
#### Request body
|
|
|
|
| Field | Type | Required | Default | Description |
|
|
|-------|------|----------|---------|-------------|
|
|
| `workspace_name` | string | Yes | - | Name of the workspace to create or use |
|
|
| `allowed_origin` | string | Yes | - | The allowed origin for CORS (for example, `https://yourapp.com`) |
|
|
| `region_id` | UUID | No | US region | Region where workspace should be created |
|
|
| `selected_source_template_tags` | array of strings | No | `[]` | Tags to filter which source templates are available |
|
|
| `selected_source_template_tags_mode` | string | No | `any` | Tag selection mode: `any` or `all` |
|
|
| `selected_connection_template_tags` | array of strings | No | `[]` | Tags to filter which connection templates are available |
|
|
| `selected_connection_template_tags_mode` | string | No | `any` | Tag selection mode: `any` or `all` |
|
|
|
|
#### Tag selection modes
|
|
|
|
| Mode | Behavior | Example |
|
|
|------|----------|---------|
|
|
| `any` | Template must have **at least one** of the specified tags | Tags: `["crm", "sales"]` matches templates with either "crm" OR "sales" |
|
|
| `all` | Template must have **all** of the specified tags | Tags: `["crm", "sales"]` matches only templates with both "crm" AND "sales" |
|
|
|
|
#### Request examples
|
|
|
|
**Basic widget token:**
|
|
|
|
```bash
|
|
curl -X POST https://api.airbyte.ai/api/v1/embedded/widget-token \
|
|
|
|
-H 'Authorization: Bearer <your_operator_token>' \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{
|
|
|
|
"workspace_name": "customer_workspace_123",
|
|
"allowed_origin": "https://yourapp.com"
|
|
}'
|
|
|
|
```
|
|
|
|
**Widget token with source template filtering:**
|
|
|
|
```bash
|
|
curl -X POST https://api.airbyte.ai/api/v1/embedded/widget-token \
|
|
|
|
-H 'Authorization: Bearer <your_operator_token>' \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{
|
|
|
|
"workspace_name": "customer_workspace_123",
|
|
"allowed_origin": "https://yourapp.com",
|
|
"selected_source_template_tags": ["crm", "sales"],
|
|
"selected_source_template_tags_mode": "any"
|
|
}'
|
|
|
|
```
|
|
|
|
**Widget token with both source and connection template filtering:**
|
|
|
|
```bash
|
|
curl -X POST https://api.airbyte.ai/api/v1/embedded/widget-token \
|
|
|
|
-H 'Authorization: Bearer <your_operator_token>' \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{
|
|
|
|
"workspace_name": "customer_workspace_123",
|
|
"allowed_origin": "https://yourapp.com",
|
|
"selected_source_template_tags": ["crm"],
|
|
"selected_source_template_tags_mode": "any",
|
|
"selected_connection_template_tags": ["standard-sync"],
|
|
"selected_connection_template_tags_mode": "all",
|
|
"region_id": "b9e48d61-f082-4a14-a8d0-799a907938cb"
|
|
}'
|
|
|
|
```
|
|
|
|
#### Response example
|
|
|
|
```json
|
|
{
|
|
"token": "eyJ0b2tlbiI6ImV5SmhiR2NpT2lKSVV6STFOaUlzSW5SNWNDSTZJa3BYVkNKOS4uLiIsIndpZGdldFVybCI6Imh0dHBzOi8vYXBwLmFpcmJ5dGUuYWkvcXVpY2stc3RhcnQvd2lkZ2V0P3dvcmtzcGFjZUlkPWExYjJjM2Q0LWU1ZjYtNzg5MC1hYjEyLWNkMzRlZjU2Nzg5MCZhbGxvd2VkT3JpZ2luPWh0dHBzOi8veW91cmFwcC5jb20ifQ=="
|
|
}
|
|
|
|
```
|
|
|
|
### Using widget tokens
|
|
|
|
The widget token is a base64-encoded JSON object containing:
|
|
|
|
1. **Scoped token** - For API authentication
|
|
2. **Widget URL** - Pre-configured URL with workspace ID, origin, and template filters
|
|
|
|
#### Decode widget token
|
|
|
|
```javascript
|
|
// Example: Decode widget token in JavaScript
|
|
const decodedToken = JSON.parse(atob(widgetToken));
|
|
|
|
console.log(decodedToken.token); // Scoped token for API calls
|
|
console.log(decodedToken.widgetUrl); // URL to load the widget
|
|
|
|
```
|
|
|
|
#### Embed widget
|
|
|
|
```html
|
|
<!-- Embed the Airbyte widget in an iframe -->
|
|
<iframe
|
|
id="airbyte-widget"
|
|
src=""
|
|
width="100%"
|
|
height="600px"
|
|
frameborder="0">
|
|
</iframe>
|
|
|
|
<script>
|
|
// Decode the widget token received from your backend
|
|
const widgetToken = "eyJ0b2tlbiI6..."; // From API response
|
|
const decoded = JSON.parse(atob(widgetToken));
|
|
|
|
// Set the iframe src to the pre-configured widget URL
|
|
document.getElementById('airbyte-widget').src = decoded.widgetUrl;
|
|
</script>
|
|
|
|
```
|
|
|
|
### Template filtering with tags
|
|
|
|
Widget tokens support filtering both source templates and connection templates using tags. This allows you to customize which connectors and sync configurations are available to specific users or customer tiers.
|
|
|
|
#### Use cases
|
|
|
|
**Tier-based access:**
|
|
|
|
```json
|
|
{
|
|
"workspace_name": "free_tier_customer",
|
|
"allowed_origin": "https://yourapp.com",
|
|
"selected_source_template_tags": ["free-tier"],
|
|
"selected_source_template_tags_mode": "any"
|
|
}
|
|
|
|
```
|
|
|
|
**Industry-specific connectors:**
|
|
|
|
```json
|
|
{
|
|
"workspace_name": "healthcare_customer",
|
|
"allowed_origin": "https://yourapp.com",
|
|
"selected_source_template_tags": ["healthcare", "hipaa-compliant"],
|
|
"selected_source_template_tags_mode": "all"
|
|
}
|
|
|
|
```
|
|
|
|
**Feature gating:**
|
|
|
|
```json
|
|
{
|
|
"workspace_name": "beta_customer",
|
|
"allowed_origin": "https://yourapp.com",
|
|
"selected_source_template_tags": ["stable", "beta"],
|
|
"selected_source_template_tags_mode": "any",
|
|
"selected_connection_template_tags": ["premium-features"],
|
|
"selected_connection_template_tags_mode": "any"
|
|
}
|
|
|
|
```
|
|
|
|
## Authentication flow patterns
|
|
|
|
### Pattern 1: direct API integration
|
|
|
|
For backend services or API integrations:
|
|
|
|
1. Store **Operator Bearer Token** securely in your backend
|
|
2. Generate **Scoped Token** for each customer workspace
|
|
3. Use scoped token for all workspace-specific API calls
|
|
4. Refresh scoped tokens when they expire (they expire after 20 minutes)
|
|
|
|
```bash
|
|
# 1. Generate scoped token (once per workspace)
|
|
SCOPED_TOKEN=$(curl -X POST https://api.airbyte.ai/api/v1/embedded/scoped-token \
|
|
|
|
-H "Authorization: Bearer $OPERATOR_TOKEN" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"workspace_name": "customer_123"}' | jq -r '.token')
|
|
|
|
# 2. Use scoped token for operations
|
|
curl https://api.airbyte.ai/api/v1/embedded/sources \
|
|
|
|
-H "Authorization: Bearer $SCOPED_TOKEN"
|
|
|
|
```
|
|
|
|
### Pattern 2: embedded widget integration
|
|
|
|
For embedding the Airbyte UI in your application:
|
|
|
|
1. Store **Operator Bearer Token** in your backend
|
|
2. Create API endpoint in your backend to generate widget tokens
|
|
3. Generate **Widget Token** with appropriate tags and origin
|
|
4. Return widget token to frontend
|
|
5. Frontend decodes token and loads widget
|
|
|
|
```javascript
|
|
// Backend endpoint (Node.js example)
|
|
app.post('/api/airbyte/widget-token', async (req, res) => {
|
|
const { customerId, tier } = req.body;
|
|
|
|
// Determine tags based on customer tier
|
|
const tags = tier === 'premium' ? ['all'] : ['free-tier'];
|
|
|
|
const response = await fetch('https://api.airbyte.ai/api/v1/embedded/widget-token', {
|
|
method: 'POST',
|
|
headers: {
|
|
'Authorization': `Bearer ${process.env.AIRBYTE_OPERATOR_TOKEN}`,
|
|
'Content-Type': 'application/json'
|
|
},
|
|
body: JSON.stringify({
|
|
workspace_name: `customer_${customerId}`,
|
|
allowed_origin: 'https://yourapp.com',
|
|
selected_source_template_tags: tags,
|
|
selected_source_template_tags_mode: 'any'
|
|
})
|
|
});
|
|
|
|
const data = await response.json();
|
|
res.json({ widgetToken: data.token });
|
|
});
|
|
|
|
// Frontend code
|
|
async function loadAirbyteWidget(customerId, tier) {
|
|
// Get widget token from your backend
|
|
const response = await fetch('/api/airbyte/widget-token', {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: JSON.stringify({ customerId, tier })
|
|
});
|
|
|
|
const { widgetToken } = await response.json();
|
|
|
|
// Decode and load widget
|
|
const decoded = JSON.parse(atob(widgetToken));
|
|
document.getElementById('airbyte-widget').src = decoded.widgetUrl;
|
|
}
|
|
|
|
```
|
|
|
|
### Pattern 3: multi-region support
|
|
|
|
For applications serving users in different regions:
|
|
|
|
```bash
|
|
# Determine region based on customer location
|
|
REGION_ID="645a183f-b12b-4c6e-8ad3-99e165603450" # US
|
|
# REGION_ID="b9e48d61-f082-4a14-a8d0-799a907938cb" # EU
|
|
|
|
curl -X POST https://api.airbyte.ai/api/v1/embedded/scoped-token \
|
|
|
|
-H "Authorization: Bearer $OPERATOR_TOKEN" \
|
|
-H "Content-Type: application/json" \
|
|
-d "{
|
|
|
|
\"workspace_name\": \"customer_workspace\",
|
|
\"region_id\": \"$REGION_ID\"
|
|
}"
|
|
|
|
```
|
|
|
|
## Token lifecycle management
|
|
|
|
### Token expiration
|
|
|
|
- **Operator Bearer Tokens**: Short-lived, expires after 15 minutes
|
|
- **Scoped Tokens**: Short-lived, expires after 20 minutes
|
|
- **Widget Tokens**: Contain scoped tokens, same lifetime
|
|
|
|
## Security considerations
|
|
|
|
### CORS and origin validation
|
|
|
|
The `allowed_origin` parameter in widget tokens enforces CORS policies:
|
|
|
|
```javascript
|
|
// Correct origin format
|
|
allowed_origin: "https://yourapp.com" // ✓ Correct
|
|
allowed_origin: "https://yourapp.com:443" // ✓ Correct with port
|
|
allowed_origin: "http://localhost:3000" // ✓ For development
|
|
|
|
// Incorrect formats
|
|
allowed_origin: "https://yourapp.com/" // ✗ No trailing slash
|
|
allowed_origin: "yourapp.com" // ✗ Missing protocol
|
|
allowed_origin: "*.yourapp.com" // ✗ No wildcards
|
|
|
|
```
|
|
|
|
## Error responses
|
|
|
|
### 401 unauthorized
|
|
|
|
```json
|
|
{
|
|
"detail": "Invalid authentication credentials"
|
|
}
|
|
|
|
```
|
|
|
|
**Causes:**
|
|
|
|
- Missing Authorization header
|
|
- Invalid or expired token
|
|
- Wrong token type for endpoint
|
|
|
|
**Solutions:**
|
|
|
|
- Verify token is included in Authorization header
|
|
- Ensure using correct token type (operator vs scoped)
|
|
- Generate new token if expired
|
|
|
|
### 403 forbidden
|
|
|
|
```json
|
|
{
|
|
"detail": "Access denied to this resource"
|
|
}
|
|
|
|
```
|
|
|
|
**Causes:**
|
|
|
|
- Scoped token trying to access resources in different workspace
|
|
- Insufficient permissions for operation
|
|
|
|
**Solutions:**
|
|
|
|
- Verify token scope matches target workspace
|
|
- Use operator token for organization-level operations
|
|
|
|
### 422 validation error
|
|
|
|
```json
|
|
{
|
|
"detail": [
|
|
{
|
|
"loc": ["body", "allowed_origin"],
|
|
"msg": "field required",
|
|
"type": "value_error.missing"
|
|
}
|
|
]
|
|
}
|
|
|
|
```
|
|
|
|
**Cause:** Missing or invalid request parameters
|
|
|
|
**Solution:** Verify all required fields are included and properly formatted
|
|
|
|
## Next steps
|
|
|
|
- Learn about [Workspace Management](./workspaces.md)
|
|
- Create [Source Templates](./source-templates.md)
|
|
- Create [Connection Templates](./connection-templates.md)
|
|
- Explore [Tag Management](./tags.md) for template organization
|