fb7cacb321
* enable ctid feature flag for testing. bump versions * Removed PostgresSourceTest pre-ctid and turn CtidEnabledPostgresSourceTest back into PostgresJdbcSourceAcceptanceTest (#28341) Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> * Update PostgresSourceTest to no longer use Legacy AirbyteStateType * Fix test. Uncovered a bug in handling of money[] and float[] * revert version change * Orderly removal of ctid flag for cursor * Handle correctly full refresh streams. (#28369) * Handle correctly full refresh streams. Handle xmin full refresh (currently blocked in ui). remove unnecessary debug logs * Test catalog with a mix of incremental and full refresh streams * Filter out full refresh streams since they are not expected to be retuned by getIncrementalIterators() * Filter out full refresh existing streams as well * revert name to reflect incremental only streams * 28505 xmin + ctid reclassify connection when ctid is not possible (#28548) * reclassify to xmin stream we cannot read with ctid * fix reclassify * improve test * Add initial load using xmin flow * typo * Added test for xmin initial load flow * Improve testing for initial load with xmin flow * sanity * sanity * bump versions and release notes --------- Co-authored-by: Duy Nguyen <duy@airbyte.io>
Postgres Source
Performance Test
To run performance tests in commandline:
./gradlew :airbyte-integrations:connectors:source-postgres:performanceTest [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
In pull request:
/test-performance connector=connectors/source-postgres [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
cpulimit: Limit the number of CPUs. The minimum is2. E.g.--cpulimit=cpulimit/2.memorylimit: Limit the size of the memory. Must include the unit at the end (e.g.MB,GB). The minimum size is6MB. E.g.--memorylimit=memorylimit/4GB.- When none of the CPU or memory limit is provided, the performance tests will run without memory or CPU limitations. The available resource will be bound that those specified in
ResourceRequirements.java.
Use Postgres script to populate the benchmark database
In order to create a database with a certain number of tables, and a certain number of records in each of them, you need to follow a few simple steps.
- Create a new database.
- Follow the TODOs in 3-run-script.sql to change the number of tables, and the number of records of different sizes.
- On the new database, run the following script:
cd airbyte-integrations/connectors/source-postgres psql -h <host> -d <db-name> -U <username> -p <port> -a -q -f src/test-performance/sql/1-create-copy-tables-procedure.sql psql -h <host> -d <db-name> -U <username> -p <port> -a -q -f src/test-performance/sql/2-create-insert-rows-to-table-procedure.sql psql -h <host> -d <db-name> -U <username> -p <port> -a -q -f src/test-performance/sql/3-run-script.sql - After the script finishes, you will receive the number of tables specified in the script, with names starting with test_0 and ending with test_(the number of tables minus 1).