ebb9f3e1ac
* Prepare database access objects for dependency injection * Replace duplicate code * Remove unused imports * Remove redundant validation call * Remove unused imports * Use constants * Disable fast fail during connection pool initialization * Remove typo * Add missing test dependency * Add missing test dependency * Add missing test dependency * Fix issue caused by rebase * Add method for cloud * Autoclose DSL context during migration * Better connection close handling * Fix typo in dependency * Fix SpotBugs issue * React to rebase * Fix typo * Update JavaDoc * Fix database close calls * Pass configs to getServer * Fix typo * Fix call to removed method * Fix typo * Use catalog to manage versions * PR feedback * Centralize shutdown hook * Fix rebase issues * Document test cases * Document test cases * Formatting * Properly close database resources * Rebase cleanup
Postgres Source
Performance Test
To run performance tests in commandline:
./gradlew :airbyte-integrations:connectors:source-postgres:performanceTest [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
In pull request:
/test-performance connector=connectors/source-postgres [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
cpulimit: Limit the number of CPUs. The minimum is2. E.g.--cpulimit=cpulimit/2.memorylimit: Limit the size of the memory. Must include the unit at the end (e.g.MB,GB). The minimum size is6MB. E.g.--memorylimit=memorylimit/4GB.- When none of the CPU or memory limit is provided, the performance tests will run without memory or CPU limitations. The available resource will be bound that those specified in
ResourceRequirements.java.
Use Postgres script to populate the benchmark database
In order to create a database with a certain number of tables, and a certain number of records in each of them, you need to follow a few simple steps.
- Create a new database.
- Follow the TODOs in 3-run-script.sql to change the number of tables, and the number of records of different sizes.
- On the new database, run the following script:
cd airbyte-integrations/connectors/source-postgres psql -h <host> -d <db-name> -U <username> -p <port> -a -q -f src/test-performance/sql/1-create-copy-tables-procedure.sql psql -h <host> -d <db-name> -U <username> -p <port> -a -q -f src/test-performance/sql/2-create-insert-rows-to-table-procedure.sql psql -h <host> -d <db-name> -U <username> -p <port> -a -q -f src/test-performance/sql/3-run-script.sql - After the script finishes, you will receive the number of tables specified in the script, with names starting with test_0 and ending with test_(the number of tables minus 1).