c6b8279386
* Revert "Revert "Add dependencies to docker-compose (#19257)" (#19306)" This reverts commitb16f28fd4d. * Bump docker-compose version * test * Revert "test" This reverts commitc3b28c9ec3. * Bump docker compose version to 3.8 * test * test * Use a newer docker for platform build * test * Use docker compose v2 * Update docker for frontend e2e * Use docker compose v2 for e2e tests * Update tools to use docker compose v2 * Update more docker compose references to v2 * Update docker compose usage in docs * Update deploying airbyte docs * Fix permission * Case insensistive check * Add extra mention for docker compose in upgrading airbyte * Improve upgrade docker note
Load Testing Airbyte
Overview
To perform a stress test of an Airbyte deployment, the load_test_airbyte.sh shell script is useful to quickly and easily create many connections.
This script creates a new E2E Test Source, E2E Test Destination, and a configurable number of connections in the indicated workspace.
Instructions
From your top-level /airbyte directory, run the following to perform a load test:
./tools/bin/load_test/load_test_airbyte.sh -W <workspace id> -C <num_connections>
By default, the script assumes that the Airbyte instance's server is accessible at localhost:8001. This is the default server location when
deploying Airbyte with docker compose up.
Additionally, the E2E Test Source created by the script will take 10 minutes to complete a sync by default.
These defaults can be overridden with flags. All available flags are described as follows:
-h
Display help
-W <workspace id>
Specify the workspace ID where new connectors and connections should be created.
Required.
-H <hostname>
Specify the Airbyte API server hostname that the script should call to create new connectors and connections.
Defaults to 'localhost'.
-P <port>
Specify the port for the Airbyte server.
Defaults to '8001'.
-X <header>
Specify the X-Endpoint-API-UserInfo header value for API authentication.
For Google Cloud Endpoint authentication only.
-C <count>
Specify the number of connections that should be created by the script.
Defaults to '1'.
-T <minutes>
Specify the time in minutes that each connection should sync for.
Defaults to '10'.
Load Testing on Kubernetes
To load test a deployment of Airbyte running on Kubernetes, you will need to set up port-forwarding to the airbyte-server deployment.
This can be accomplished with the following command:
kubectl port-forward deployment/airbyte-server -n ab 8001:8001
This will make the Airbyte server available at localhost:8001
Authentication
If your deployment of Airbyte happens to use Google Cloud Endpoints for authentication, you can use the -X option to pass
an X-Endpoint-API-UserInfo header value.
Cleanup
The load_test_airbyte.sh script writes created IDs to files in the script's /cleanup directory. To delete resources that were created by the load
test script, you can run cleanup_load_test.sh, which reads IDs from the /cleanup directory and calls the Airbyte API to delete them.
Cleanup Instructions
To run the cleanup script, from the top-level airbyte directory, run the following:
./tools/bin/load_test/cleanup_load_test.sh -W <workspace_id>
All available cleanup script flags are described as follows:
-h
Display help
-W <workspace id>
Specify the workspace ID from where connectors and connections should be deleted.
Required.
-H <hostname>
Specify the Airbyte API server hostname that the script should call to delete connectors and connections.
Defaults to 'localhost'.
-P <port>
Specify the port for the Airbyte server.
Defaults to '8001'.
-X <header>
Specify the X-Endpoint-API-UserInfo header value for API authentication.
For Google Cloud Endpoint authentication only.