7cedfa48de
* script skeleton * add API call to source_definitions to fetch E2E Test Source definition ID * createSource implementation * add destination creation logic implementation * get definition IDs, catalogId, and implement connection creation * add cleanup script and write created ids to a file that can be cleaned up * make cloud header a command-line argument, other cleanup * script comments fix * remove kube references and fix indentation * temp commit - don't push * remove discover catalog function * more cleanups * more cleanups * cleanup help text * exit codes and show how many connections left * add README Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
104 lines
3.2 KiB
Markdown
104 lines
3.2 KiB
Markdown
# Load Testing Airbyte
|
|
|
|
## Overview
|
|
To perform a stress test of an Airbyte deployment, the `load_test_airbyte.sh` shell script is useful to quickly and easily create many connections.
|
|
This script creates a new E2E Test Source, E2E Test Destination, and a configurable number of connections in the indicated workspace.
|
|
|
|
## Instructions
|
|
From your top-level `/airbyte` directory, run the following to perform a load test:
|
|
|
|
```
|
|
./tools/bin/load_test/load_test_airbyte.sh -W <workspace id> -C <num_connections>
|
|
```
|
|
|
|
|
|
By default, the script assumes that the Airbyte instance's server is accessible at `localhost:8001`. This is the default server location when
|
|
deploying Airbyte with `docker-compose up`.
|
|
|
|
Additionally, the E2E Test Source created by the script will take 10 minutes to complete a sync by default.
|
|
|
|
These defaults can be overridden with flags. All available flags are described as follows:
|
|
|
|
```
|
|
-h
|
|
Display help
|
|
|
|
-W <workspace id>
|
|
Specify the workspace ID where new connectors and connections should be created.
|
|
Required.
|
|
|
|
-H <hostname>
|
|
Specify the Airbyte API server hostname that the script should call to create new connectors and connections.
|
|
Defaults to 'localhost'.
|
|
|
|
-P <port>
|
|
Specify the port for the Airbyte server.
|
|
Defaults to '8001'.
|
|
|
|
-X <header>
|
|
Specify the X-Endpoint-API-UserInfo header value for API authentication.
|
|
For Google Cloud Endpoint authentication only.
|
|
|
|
-C <count>
|
|
Specify the number of connections that should be created by the script.
|
|
Defaults to '1'.
|
|
|
|
-T <minutes>
|
|
Specify the time in minutes that each connection should sync for.
|
|
Defaults to '10'.
|
|
```
|
|
|
|
|
|
### Load Testing on Kubernetes
|
|
|
|
To load test a deployment of Airbyte running on Kubernetes, you will need to set up port-forwarding to the `airbyte-server` deployment.
|
|
This can be accomplished with the following command:
|
|
|
|
```
|
|
kubectl port-forward deployment/airbyte-server -n ab 8001:8001
|
|
```
|
|
|
|
This will make the Airbyte server available at `localhost:8001`
|
|
|
|
|
|
### Authentication
|
|
|
|
If your deployment of Airbyte happens to use Google Cloud Endpoints for authentication, you can use the `-X` option to pass
|
|
an `X-Endpoint-API-UserInfo` header value.
|
|
|
|
|
|
## Cleanup
|
|
The `load_test_airbyte.sh` script writes created IDs to files in the script's `/cleanup` directory. To delete resources that were created by the load
|
|
test script, you can run `cleanup_load_test.sh`, which reads IDs from the `/cleanup` directory and calls the Airbyte API to delete them.
|
|
|
|
|
|
### Cleanup Instructions
|
|
To run the cleanup script, from the top-level `airbyte` directory, run the following:
|
|
|
|
```
|
|
./tools/bin/load_test/cleanup_load_test.sh -W <workspace_id>
|
|
```
|
|
|
|
All available cleanup script flags are described as follows:
|
|
|
|
```
|
|
-h
|
|
Display help
|
|
|
|
-W <workspace id>
|
|
Specify the workspace ID from where connectors and connections should be deleted.
|
|
Required.
|
|
|
|
-H <hostname>
|
|
Specify the Airbyte API server hostname that the script should call to delete connectors and connections.
|
|
Defaults to 'localhost'.
|
|
|
|
-P <port>
|
|
Specify the port for the Airbyte server.
|
|
Defaults to '8001'.
|
|
|
|
-X <header>
|
|
Specify the X-Endpoint-API-UserInfo header value for API authentication.
|
|
For Google Cloud Endpoint authentication only.
|
|
```
|