f6bcc4914f
* Source postgres: add dd for env running locally * Source postgres: add dd for running in cloud * auto-bump connector version * Source postgres: bump postgres strict-encrypt version * Source postgres: filter datadog agent env variables just for postgres source * Source postgres: format * Source postgres: clean code * Source postgres: pass java opts for all connectors * Source postgres: temp removing dd agent from image * Source postgres: add dd agent to image * Source postgres: temp revert adding dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp removing hardcoded dd env variable * Source postgres: temp added hardcoded dd env variable * Source postgres: temp added hardcoded dd env variable * Source postgres: temp added hardcoded dd env variable * Source postgres: rename to java_opts and pass data dog host * Source postgres: add vars to kube pods * Source postgres: add vars to kube pods * Source postgres: add vars to kube pods * Source postgres: add Trace to more methods * Source postgres: add Trace to more methods * Source postgres: add Trace to more methods * Source postgres: temp reverting service name removing * Source postgres: temp reverting service name removing * Source postgres: temp reverting service name removing * Source postgres: temp adding trace to integration runner * Source postgres: temp adding trace to integration runner * Source postgres: bump postgres source dd version * Source postgres: bump postgres source dd version * Source postgres: revert temp changes * Source postgres: merge with master * Automated Commit - Formatting Changes * Source postgres: move dd java agent to base java * Source postgres: move dd java agent to base java * Source postgres: clean up * Source postgres: clean up * Automated Change * Source postgres: clean up * Source postgres: bump version * Source postgres: bump version for test * Source postgres: temp bump version * Source postgres: bump version * Automated Change * Source postgres: bump version * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: sashaNeshcheret <sashaNeshcheret@users.noreply.github.com>
Postgres Source
Performance Test
To run performance tests in commandline:
./gradlew :airbyte-integrations:connectors:source-postgres:performanceTest [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
In pull request:
/test-performance connector=connectors/source-postgres [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
cpulimit: Limit the number of CPUs. The minimum is2. E.g.--cpulimit=cpulimit/2.memorylimit: Limit the size of the memory. Must include the unit at the end (e.g.MB,GB). The minimum size is6MB. E.g.--memorylimit=memorylimit/4GB.- When none of the CPU or memory limit is provided, the performance tests will run without memory or CPU limitations. The available resource will be bound that those specified in
ResourceRequirements.java.
Use Postgres script to populate the benchmark database
In order to create a database with a certain number of tables, and a certain number of records in each of them, you need to follow a few simple steps.
- Create a new database.
- Follow the TODOs in 3-run-script.sql to change the number of tables, and the number of records of different sizes.
- On the new database, run the following script:
cd airbyte-integrations/connectors/source-postgres psql -h <host> -d <db-name> -U <username> -p <port> -a -q -f src/test-performance/sql/1-create-copy-tables-procedure.sql psql -h <host> -d <db-name> -U <username> -p <port> -a -q -f src/test-performance/sql/2-create-insert-rows-to-table-procedure.sql psql -h <host> -d <db-name> -U <username> -p <port> -a -q -f src/test-performance/sql/3-run-script.sql - After the script finishes, you will receive the number of tables specified in the script, with names starting with test_0 and ending with test_(the number of tables minus 1).