26f1dc3f7a
on my M2 laptop, sometimes tests fail due to timeouts or some random process getting killed (either a container, or the gradle cache inside dagger). After investigation, I realized that we were running up to 6 different containers with MSSQL db, with each taking upwards of 2GB of RAM (real, not virtual). So I'm now limiting the individual MSSQL instances to 384MB. That helps with swapping too
MsSQL (SQL Server) Source
Performance Test
To run performance tests in commandline:
./gradlew :airbyte-integrations:connectors:source-mssql:performanceTest [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
In pull request:
/test-performance connector=connectors/source-mssql [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
cpulimit: Limit the number of CPUs. The minimum is2. E.g.--cpulimit=cpulimit/2.memorylimit: Limit the size of the memory. Must include the unit at the end (e.g.MB,GB). The minimum size is6MB. E.g.--memorylimit=memorylimit/4GB.- When none of the CPU or memory limit is provided, the performance tests will run without memory or CPU limitations. The available resource will be bound that those specified in
ResourceRequirements.java.
Use MsSQL script to populate the benchmark database
In order to create a database with a certain number of tables, and a certain number of records in each of them, you need to follow a few simple steps.
- Create a new database.
- Follow the TODOs in create_mssql_benchmarks.sql to change the number of tables, and the number of records of different sizes.
- Execute the script with your changes for the new database. You can run the script with the MySQL client:
cd airbyte-integrations/connectors/source-mssql sqlcmd -S Serverinstance -E -i src/test-performance/sql/create_mssql_benchmarks.sql - After the script finishes its work, you will receive the number of tables specified in the script, with names starting with test_0 and ending with test_(the number of tables minus 1).