# 🐙 Octavia CLI

## Disclaimer

The project is in **alpha** version.
Readers can refer to our [opened GitHub issues](https://github.com/airbytehq/airbyte/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%2Foctavia-cli) to check the ongoing work on this project.


## What is `octavia` CLI?

Octavia CLI is a tool to manage Airbyte configurations in YAML.
It has the following features:

- Scaffolding of a readable directory architecture that will host the YAML configs (`octavia init`).
- Auto-generation of YAML config file that matches the resources' schemas (`octavia generate`).
- Manage Airbyte resources with YAML config files.
- Safe resources update through diff display and validation (`octavia apply`).
- Simple secret management to avoid versioning credentials.

## Why should I use `octavia` CLI?

A CLI provides freedom to users to use the tool in whatever context and use case they have.
These are non-exhaustive use cases `octavia` can be convenient for:

- Managing Airbyte configurations with a CLI instead of a web UI.
- Versioning Airbyte configurations in Git.
- Updating of Airbyte configurations in an automated deployment pipeline.
- Integrating the Airbyte configuration deployment in a dev ops tooling stack: Helm, Ansible etc.
- Streamlining the deployment of Airbyte configurations to multiple Airbyte instance.

Feel free to share your use cases with the community in [#octavia-cli](https://airbytehq.slack.com/archives/C02RRUG9CP5) or on [Discourse](https://discuss.airbyte.io/).

## Table of content

- [Workflow](#workflow)
- [Secret management](#secret-management)
- [Install](#install)
- [Commands reference](#commands-reference)
- [Contributing](#contributing)
- [Telemetry](#telemetry)
- [Changelog](#changelog)

## Workflow

### 1. Generate local YAML files for sources or destinations

1. Retrieve the *definition id* of the connector you want to use using `octavia list command`.
2. Generate YAML configuration running `octavia generate source <DEFINITION_ID> <SOURCE_NAME>` or `octavia generate destination <DEFINITION_ID> <DESTINATION_NAME>`.

### 2. Edit your local YAML configurations

1. Edit the generated YAML configurations according to your need.
2. Use the [secret management feature](#secret-management) feature to avoid storing credentials in the YAML files.

### 3. Create the declared sources or destinations on your Airbyte instance

1. Run `octavia apply` to create the **sources** and **destinations**

### 4. Generate connections

1. Run `octavia octavia generate connection --source <PATH_TO_SOURCE_CONFIG> --destination <PATH_TO_DESTINATION_CONFIG> <CONNECTION_NAME>` to create a YAML configuration for a new connection.
2. Edit the created configuration file according to your need: change the scheduling or the replicated streams list.

### 5. Create the declared connections

1. Run `octavia apply` to create the newly declared connection on your Airbyte instance.

### 6. Update your configurations

Changes in your local configurations can be propagated to your Airbyte instance using `octavia apply`.  You will be prompted for validation of changes. You can bypass the validation step using the `--force` flag.

## Secret management

Sources and destinations configurations have credential fields that you **do not want to store as plain text in your VCS**.
`octavia` offers secret management through environment variables expansion:

```yaml
configuration:
  password: ${MY_PASSWORD}
```

If you have set a  `MY_PASSWORD` environment variable, `octavia apply` will load its value into the `password` field.

## Install

### Requirements

We decided to package the CLI in a docker image with portability in mind.
**[Please install and run Docker if you are not](https://docs.docker.com/get-docker/)**.

### As a command available in your bash profile

```bash
curl -s -o- https://raw.githubusercontent.com/airbytehq/airbyte/master/octavia-cli/install.sh | bash
```

This script:

1. Pulls the [octavia-cli image](https://hub.docker.com/r/airbyte/octavia-cli/tags) from our Docker registry.
2. Creates an `octavia` alias in your profile.
3. Creates a `~/.octavia` file whose values are mapped to the octavia container's environment variables.

### Using `docker run`

```bash
touch ~/.octavia # Create a file to store env variables that will be mapped the octavia-cli container
mkdir my_octavia_project_directory # Create your octavia project directory where YAML configurations will be stored.
docker run --name octavia-cli -i --rm -v my_octavia_project_directory:/home/octavia-project --network host --user $(id -u):$(id -g) --env-file ~/.octavia airbyte/octavia-cli:0.39.23-alpha
```

### Using `docker-compose`

Using octavia in docker-compose could be convenient for automatic `apply` on start-up.

Add another entry in the services key of your Airbyte `docker-compose.yml`

```yaml
services:
  # . . .
  octavia-cli:
    image: airbyte/octavia-cli:latest
    command: apply --force
    env_file:
      - ~/.octavia # Use a local env file to store variables that will be mapped the octavia-cli container
    volumes:
      - <path_to_your_local_octavia_project_directory>:/home/octavia-project
    depends_on:
      - webapp
```

Other commands besides `apply` can be run like so:

```bash
docker-compose run octavia-cli <command>`
```

## Commands reference

### `octavia` command flags

| **Flag**                                   | **Description**                                                                   | **Env Variable**             | **Default**                                            |
|--------------------------------------------|-----------------------------------------------------------------------------------|------------------------------|--------------------------------------------------------|
| `--airbyte-url`                            | Airbyte instance URL.                                                             | `AIRBYTE_URL`                | `http://localhost:8000`                                |
| `--workspace-id`                           | Airbyte workspace id.                                                             | `AIRBYTE_WORKSPACE_ID`       | The first workspace id found on your Airbyte instance. |
| `--enable-telemetry/--disable-telemetry`   | Enable or disable the sending of telemetry data.                                  | `OCTAVIA_ENABLE_TELEMETRY`   | True                                                   |
| `--api-http-header`                        | HTTP Header value pairs passed while calling Airbyte's API | not supported.       | None                         | None                                                   |
| `--api-http-headers-file-path`             | Path to the YAML file that contains custom HTTP Headers to send to Airbyte's API. | None                         | None                                                   |

#### Using custom HTTP headers
You can set custom HTTP headers to send to Airbyte's API with options:          
```bash
octavia --api-http-header Header-Name Header-Value --api-http-header Header-Name-2 Header-Value-2 list connectors sources
```

You can also use a custom YAML file (one is already created on init in `api_http_headers.yaml`) to declare the HTTP headers to send to the API:
```yaml
headers:
  Authorization: Basic foobar==
  User-Agent: octavia-cli/0.0.0
```
Environment variable expansion is available in this Yaml file
```yaml
headers:
  Authorization: Bearer ${MY_API_TOKEN}
```

**Options based headers are overriding file based headers if an header is declared in both.**

### `octavia` subcommands

| **Command**                             | **Usage**                                                                           |
|-----------------------------------------|-------------------------------------------------------------------------------------|
| **`octavia init`**                        | Initialize required directories for the project.                                  |
| **`octavia list connectors sources`**     | List all sources connectors available on the remote Airbyte instance.             |
| **`octavia list connectors destination`** | List all destinations connectors available on the remote Airbyte instance.        |
| **`octavia list workspace sources`**      | List existing sources in current the Airbyte workspace.                           |
| **`octavia list workspace destinations`** | List existing destinations in the current Airbyte workspace.                      |
| **`octavia list workspace connections`**  | List existing connections in the current Airbyte workspace.                       |
| **`octavia generate source`**             | Generate a local YAML configuration for a new source.                             |
| **`octavia generate destination`**        | Generate a local YAML configuration for a new destination.                        |
| **`octavia generate connection`**         | Generate a local YAML configuration for a new connection.                         |
| **`octavia apply`**                       | Create or update Airbyte remote resources according to local YAML configurations. |

#### `octavia init`

The `octavia init` commands scaffolds the required directory architecture for running `octavia generate` and `octavia apply` commands.

**Example**:

```bash
$ mkdir my_octavia_project && cd my_octavia_project
$ octavia init
🐙 - Octavia is targetting your Airbyte instance running at http://localhost:8000 on workspace e1f46f7d-5354-4200-aed6-7816015ca54b.
🐙 - Project is not yet initialized.
🔨 - Initializing the project.
✅ - Created the following directories: sources, destinations, connections.
$ ls
connections  destinations sources
```

#### `octavia list connectors sources`

List all the source connectors currently available on your Airbyte instance.

**Example**:

```bash
$ octavia list connectors sources
NAME                            DOCKER REPOSITORY                              DOCKER IMAGE TAG  SOURCE DEFINITION ID
Airtable                        airbyte/source-airtable                        0.1.1             14c6e7ea-97ed-4f5e-a7b5-25e9a80b8212
AWS CloudTrail                  airbyte/source-aws-cloudtrail                  0.1.4             6ff047c0-f5d5-4ce5-8c81-204a830fa7e1
Amazon Ads                      airbyte/source-amazon-ads                      0.1.3             c6b0a29e-1da9-4512-9002-7bfd0cba2246
Amazon Seller Partner           airbyte/source-amazon-seller-partner           0.2.16            e55879a8-0ef8-4557-abcf-ab34c53ec460
```

#### `octavia list connectors destinations`

List all the destinations connectors currently available on your Airbyte instance.

**Example**:

```bash
$ octavia list connectors destinations
NAME                                  DOCKER REPOSITORY                                 DOCKER IMAGE TAG  DESTINATION DEFINITION ID
Azure Blob Storage                    airbyte/destination-azure-blob-storage            0.1.3             b4c5d105-31fd-4817-96b6-cb923bfc04cb
Amazon SQS                            airbyte/destination-amazon-sqs                    0.1.0             0eeee7fb-518f-4045-bacc-9619e31c43ea
BigQuery                              airbyte/destination-bigquery                      0.6.11            22f6c74f-5699-40ff-833c-4a879ea40133
BigQuery (denormalized typed struct)  airbyte/destination-bigquery-denormalized         0.2.10            079d5540-f236-4294-ba7c-ade8fd918496
```

#### `octavia list workspace sources`

List all the sources existing on your targeted Airbyte instance.

**Example**:

```bash
$ octavia list workspace sources
NAME     SOURCE NAME  SOURCE ID
weather  OpenWeather  c4aa8550-2122-4a33-9a21-adbfaa638544
```

#### `octavia list workspace destinations`

List all the destinations existing on your targeted Airbyte instance.

**Example**:

```bash
$ octavia list workspace destinations
NAME   DESTINATION NAME  DESTINATION ID
my_db  Postgres          c0c977c2-48e7-46fe-9f57-576285c26d42
```

#### `octavia list workspace connections`

List all the connections existing on your targeted Airbyte instance.

**Example**:

```bash
$ octavia list workspace connections
NAME           CONNECTION ID                         STATUS  SOURCE ID                             DESTINATION ID
weather_to_pg  a4491317-153e-436f-b646-0b39338f9aab  active  c4aa8550-2122-4a33-9a21-adbfaa638544  c0c977c2-48e7-46fe-9f57-576285c26d42
```

#### `octavia generate source <DEFINITION_ID> <SOURCE_NAME>`

Generate a YAML configuration for a source.
The YAML file will be stored at `./sources/<resource_name>/configuration.yaml`.

| **Argument**    | **Description**                                                                              |
|-----------------|-----------------------------------------------------------------------------------------------|
| `DEFINITION_ID` | The source connector definition id. Can be retrieved using `octavia list connectors sources`. |
| `SOURCE_NAME`   | The name you want to give to this source in Airbyte.                                          |

**Example**:

```bash
$ octavia generate source d8540a80-6120-485d-b7d6-272bca477d9b weather
✅ - Created the source template for weather in ./sources/weather/configuration.yaml.
```

#### `octavia generate destination <DEFINITION_ID> <DESTINATION_NAME>`

Generate a YAML configuration for a destination.
The YAML file will be stored at `./destinations/<destination_name>/configuration.yaml`.

| **Argument**       | **Description**                                                                                         |
|--------------------|---------------------------------------------------------------------------------------------------------|
| `DEFINITION_ID`    | The destination connector definition id. Can be retrieved using `octavia list connectors destinations`. |
| `DESTINATION_NAME` | The name you want to give to this destination in Airbyte.                                               |

**Example**:

```bash
$ octavia generate destination 25c5221d-dce2-4163-ade9-739ef790f503 my_db
✅ - Created the destination template for my_db in ./destinations/my_db/configuration.yaml.
```

#### `octavia generate connection --source <path-to-source-configuration.yaml> --destination <path-to-destination-configuration.yaml> <CONNECTION_NAME>`

Generate a YAML configuration for a connection.
The YAML file will be stored at `./connections/<connection_name>/configuration.yaml`.

| **Option**        | **Required** | **Description**                                                                            |
|-------------------|--------------|--------------------------------------------------------------------------------------------|
| `--source`        | Yes          | Path to the YAML configuration file of the source you want to create a connection from.    |
| `--destination`   | Yes          | Path to the YAML configuration file of the destination you want to create a connection to. |

| **Argument**      | **Description**                                          |
|-------------------|----------------------------------------------------------|
| `CONNECTION_NAME` | The name you want to give to this connection in Airbyte. |

**Example**:

```bash
$ octavia generate connection --source sources/weather/configuration.yaml --destination destinations/my_db/configuration.yaml weather_to_pg
✅ - Created the connection template for weather_to_pg in ./connections/weather_to_pg/configuration.yaml.
```

#### `octavia apply`

Create or update the resource on your Airbyte instance according to local configurations found in your octavia project directory.
If the resource was not found on your Airbyte instance, **apply** will **create** the remote resource.
If the resource was found on your Airbyte instance, **apply** will prompt you for validation of the changes and will run an **update** of your resource.
Please note that if a secret field was updated on your configuration, **apply** will run this change without prompt.

| **Option**      | **Required** | **Description**                                                                            |
|-----------------|--------------|--------------------------------------------------------------------------------------------|
| `--file`        | No           | Path to the YAML configuration files you want to create or update.                         |
| `--force`       | No           | Run update without prompting for changes validation.                                       |

**Example**:

```bash
$ octavia apply
🐙 - weather exists on your Airbyte instance, let's check if we need to update it!
👀 - Here's the computed diff (🚨 remind that diff on secret fields are not displayed):
  E - Value of root['lat'] changed from "46.7603" to "45.7603".
❓ - Do you want to update weather? [y/N]: y
✍️ - Running update because a diff was detected between local and remote resource.
🎉 - Successfully updated weather on your Airbyte instance!
💾 - New state for weather stored at ./sources/weather/state_<workspace_id>.yaml.
🐙 - my_db exists on your Airbyte instance, let's check if we need to update it!
😴 - Did not update because no change detected.
🐙 - weather_to_pg exists on your Airbyte instance, let's check if we need to update it!
👀 - Here's the computed diff (🚨 remind that diff on secret fields are not displayed):
  E - Value of root['schedule']['timeUnit'] changed from "days" to "hours".
❓ - Do you want to update weather_to_pg? [y/N]: y
✍️ - Running update because a diff was detected between local and remote resource.
🎉 - Successfully updated weather_to_pg on your Airbyte instance!
💾 - New state for weather_to_pg stored at ./connections/weather_to_pg/state_<workspace_id>.yaml.
```

## Contributing

1. Please sign up to [Airbyte's Slack workspace](https://slack.airbyte.io/) and join the `#octavia-cli`. We'll sync up community efforts in this channel.
2. Pick an existing [GitHub issues](https://github.com/airbytehq/airbyte/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%2Foctavia-cli) or **open** a new one to explain what you'd like to implement.
3. Assign the GitHub issue to yourself.
4. Fork Airbyte's repo, code and test thoroughly.
5. Open a PR on our Airbyte repo from your fork.

### Developing locally

0. Build the project locally (from the root of Airbyte's repo): `SUB_BUILD=OCTAVIA_CLI ./gradlew build # from the root directory of the repo`.
1. Install Python 3.8.12. We suggest doing it through `pyenv`.
2. Create a virtualenv: `python -m venv .venv`.
3. Activate the virtualenv: `source .venv/bin/activate`.
4. Install dev dependencies: `pip install -e .\[tests\]`.
5. Install `pre-commit` hooks: `pre-commit install`.
6. Run the unittest suite: `pytest --cov=octavia_cli`.
7. Make sure the build passes (step 0) before opening a PR.

## Telemetry
This CLI has some telemetry tooling to send Airbyte some data about the usage of this tool.
We will use this data to improve the CLI and measure its adoption.
The telemetry sends data about:
* Which command was run (not the arguments or options used).
* Success or failure of the command run and the error type (not the error payload).
* The current Airbyte workspace id if the user has not set the *anonymous data collection* on their Airbyte instance.

You can disable telemetry by setting the `OCTAVIA_ENABLE_TELEMETRY` environment variable to `False` or using the `--disable-telemetry` flag.

## Changelog

| Version  | Date       | Description                                        | PR                                                       |
|----------|------------|----------------------------------------------------|----------------------------------------------------------|
| 0.39.19  | 2022-06-16 | Allow connection management on multiple workspaces | [#12727](https://github.com/airbytehq/airbyte/pull/12727)|
| 0.39.19  | 2022-06-15 | Allow users to set custom HTTP headers             | [#12893](https://github.com/airbytehq/airbyte/pull/12893)  |
| 0.39.14  | 2022-05-12 | Enable normalization on connection                 | [#12727](https://github.com/airbytehq/airbyte/pull/12727)|
| 0.37.0   | 2022-05-05 | Use snake case in connection fields                | [#12133](https://github.com/airbytehq/airbyte/pull/12133)|
| 0.35.68  | 2022-04-15 | Improve telemetry                                  | [#12072](https://github.com/airbytehq/airbyte/issues/11896)|
| 0.35.68  | 2022-04-12 | Add telemetry                                      | [#11896](https://github.com/airbytehq/airbyte/issues/11896)|
| 0.35.61  | 2022-04-07 | Alpha release                                      | [EPIC](https://github.com/airbytehq/airbyte/issues/10704)|