6ca4017d28
* WIP Per-stream state support for Postgres source * Fix failing test * Improve code coverage * Make global the default state manager * Add legacy adapter state manager * Formatting * Include legacy state for backwards compatibility * Add global state manager * Implement Global/CDC state handling * Fix test issues * Fix issue with updated method signature * Handle empty state case in global state manager * Adjust to protocol changes * Fix failing acceptance tests * Fix failing test * Fix unmodifiable list issue * Fix unmodifiable exception * PR feedback * Abstract global state manager selection * Handle conversion between different state types * Handle invalid conversion * Rename parameter * Refactor state manager creation * Fix failing tests * Fix failing integration tests * Add CDC test * Fix failing integration test * Revert change * Fix failing integration test * Use per-stream for postgres tests * Formatting * Correct stream descriptor validation * Correct permalink * PR feedback
MySQL Source
Documentation
This is the repository for the MySQL only source connector in Java. For information about how to use this connector within Airbyte, see User Documentation
Local development
Building via Gradle
From the Airbyte repository root, run:
./gradlew :airbyte-integrations:connectors:source-mysql:build
Locally running the connector docker image
Build
Build the connector image via Gradle:
./gradlew :airbyte-integrations:connectors:source-mysql:airbyteDocker
When building via Gradle, the docker image name and tag, respectively, are the values of the io.airbyte.name and io.airbyte.version LABELs in
the Dockerfile.
Testing
We use JUnit for Java tests.
Acceptance Tests
To run acceptance and custom integration tests:
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest
Performance Tests
To run performance tests in commandline:
./gradlew :airbyte-integrations:connectors:source-mysql:performanceTest [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
In pull request:
/test-performance connector=connectors/source-mysql [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
cpulimit: Limit the number of CPUs. The minimum is2. E.g.--cpulimit=cpulimit/2.memorylimit: Limit the size of the memory. Must include the unit at the end (e.g.MB,GB). The minimum size is6MB. E.g.--memorylimit=memorylimit/4GB.- When none of the CPU or memory limit is provided, the performance tests will run without memory or CPU limitations. The available resource will be bound that those specified in
ResourceRequirements.java.
Use MySQL script to populate the benchmark database
In order to create a database with a certain number of tables, and a certain number of records in each of them, you need to follow a few simple steps.
- Create a new database.
- Follow the TODOs in create_mysql_benchmarks.sql to change the number of tables, and the number of records of different sizes.
- Execute the script with your changes for the new database. You can run the script using the MySQL command line client:
cd airbyte-integrations/connectors/source-mysql mysql -h hostname -u user database < src/test-performance/sql/create_mysql_benchmarks.sql - After the script finishes its work, you will receive the number of tables specified in the script, with names starting with test_0 and ending with test_(the number of tables minus 1).