1
0
mirror of synced 2026-01-08 12:03:02 -05:00
Commit Graph

60 Commits

Author SHA1 Message Date
Brian Lai
b73713e610 run source acceptance tests against image built from branch (#13401)
* run source acceptance tests against image built from branch + auto fail tests for validation

* remove auto fail tests used for validation
2022-06-02 12:22:18 -04:00
Sherif A. Nada
cc46255190 cull base-python and base-python-test (#12596) 2022-05-05 13:48:15 -07:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Sherif A. Nada
b85c51304f 🎉 Update hardway tutorial & generic source template to use SAT (#11908) 2022-04-11 20:12:06 -07:00
Jared Rhizor
526c5a0c64 automatically figure out ARM-related variables (#11450)
* automatically figure out ARM-related variables

* clean up

* explicitly use amd64-prefixed versions of alpine and postgres
2022-03-31 13:58:46 -07:00
Serhii Chvaliuk
84b2d320e6 upgrade black 22.1.0 -> 22.3.0 (#11494)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-03-29 14:13:41 +03:00
Charles
c1c8675366 Add readmes to all modules (#8893) 2022-03-13 14:45:36 -07:00
Maksym Pavlenok
61b0e9e196 🎉 Single py checker (#10246) 2022-02-21 14:01:42 +02:00
Maksym Pavlenok
b5b0976355 Fix deps for python build order (#10057) 2022-02-04 12:56:14 +02:00
Maksym Pavlenok
9aade158fc skip errors of coverage reports (#10028) 2022-02-03 11:38:48 +02:00
Maksym Pavlenok
91eff1dffd 🐛 Source S3: Loading of files' metadata (#8252) 2022-02-02 00:49:18 +02:00
Edward Gao
dad52edcc4 make airbyteDocker build cache functional (#9362) 2022-01-11 11:33:54 -08:00
Edward Gao
5dad18b377 fix destination-snowflake build for jdk17 (#8854) 2021-12-16 12:57:31 -08:00
Eugene
70d2f46bcc Source MySQL\MsSql\Postgres: added RDS base performance tests (#8215)
* Added RDS base performance tests for source-postgres, source-mssql and source-mysql
* updated perfomance test with cpu and memory limit


Co-authored-by: andriikorotkov <andrii.h.korotkov@globallogic.com>
2021-12-12 19:29:11 +02:00
Charles
e9f414d0a7 do not run excluded tags in integration tests (#8715) 2021-12-10 16:20:57 -08:00
Eugene
3a0eaafb5e 🎉 Source Mysql - added basic MySql performance tests (#7820)
* [ticket 7489, PR 7820] Source Mysql - added skeleton for MySql performance tests
2021-11-15 12:16:36 +02:00
Eugene Kulak
319492791e CDK: fix logging configuration inside source and streams (#7808) 2021-11-11 16:07:38 +02:00
Sherif A. Nada
efb5151011 🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering (#7719) 2021-11-08 19:26:32 -08:00
Jared Rhizor
1b5922896b upgrade to Gradle 7.2 (#7070)
* upgrade to gradle 7.2

* get build working with gradle 7.2

* fix build

* fix connector build?

* EXCLUDE distTar and switch includes to excludes

* switch back to includes

* temporarily try to adjust properties

* don't know why this was modified

* revert spotless upgrade

* upgrade gradle node (better arm support it says)

* add back in Xss since we are getting StackOverflowError without it

* revert bad spotless fmt

* fix acceptance tests duplicates strategy

* add zip for jdbc includes

* check if this helps for java connectors

* remove accidental commit

* remove redundant

* fix warning explosion

* oops

* override gradle.properties

* remove set-env
2021-10-15 14:03:30 -07:00
Dmytro
d767003375 Trello OAuth backend. (#6944) 2021-10-11 17:07:16 +03:00
Dmytro
7a6da86baa add coverage report (#6045)
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-09-15 10:50:07 +03:00
Sherif A. Nada
45ee6234f2 Only run pytest if test files are found in the test directories (#5012) 2021-07-27 08:33:51 -07:00
vitaliizazmic
5700645946 🎉 New Source: PrestaShop (#4465)
* Source PrestaShop #2980 - init commit

* Source PrestaShop #2980 - schemas

* Source PrestaShop #2980 - full refresh sync

* Source PrestaShop #2980 - incremental sync, acceptance tests

* Source PrestaShop #2980 - incremental acceptance tests

* Source PrestaShop #2980 - incremental acceptance tests (remove configurations from incremental)

* Source PrestaShop #2980 - change Configurations stream to full refresh

* Source PrestaShop #2980 - docs

* format

* fix schemas, fill streams with data

* fix tests and future_state

* rename presta-shop -> prestashop

* fix mounting volumes inside a nested container

* special handle for NULL dates (MySQL)

* fix build URL

* fix URL and polishing

* fix header

* fix template

* format, update pre-commit to execute in the right order

* fix scaffolding

* bump and changelog for SAT

* register as the default source

* fix test timeout

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-07-26 21:55:07 -04:00
Sherif A. Nada
a4bb30430d Python Demo Destination: KVDB (#4786) 2021-07-19 09:44:36 -07:00
Marcos Marx
6201100e72 Add entrypoint and bump version for remaining connectors. (#4539)
See the PR descriptions for all the connectors published.
2021-07-07 17:37:22 +08:00
Davin Chia
683a67c023 Upgrade Gradle Python plugin + explicitly install PyEnv to fix transient checkPython error. (#3362)
This is what the plugin's maintainer suggested: xvik/gradle-use-python-plugin#19.

I ran the build 10 times in a row without any checkPython error so this does seem to have fixed it.
2021-06-25 17:32:41 +08:00
Jared Rhizor
ef853153a5 kube zombie handling (#4137)
* working except for too much logging and bad success case

* succeeds on passing case

* completes successfully

* just doesn't kill the main

* working zombie killing

* cleanup

* more cleanup

* use correct path

* fmt

* cleanups, bugfixes, integration tests

* run worker integration tests as part of ci

* delete tester class

* fix hanging checkpoint container problem

* fix name of command

* replace todo with clarifying comment
2021-06-17 17:51:58 -07:00
Sherif A. Nada
6ec8b31bee revert change to pythonBinary (#3993) 2021-06-09 08:47:26 -07:00
Sherif A. Nada
7764fd4fab Add Gradle tasks to cleanup Python Venvs and always use python binary not python3 (#3972) 2021-06-08 16:22:34 -07:00
Eugene Kulak
c61fc3df07 SAT: migrate to CDK (#3640)
* move SAT to CDK and fix image tag

* fix publish

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-05-28 04:32:15 -04:00
Charles
8983f09aea normalize connector acceptance test names (#3539)
* Rename standard tests to acceptance tests

* Normalize the names so that the nouns are always in the same order so it is easier to find tests
2021-05-22 13:40:40 -07:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Sherif A. Nada
535d83e0af bugfix infinite pagination in CDK (#3366) 2021-05-11 14:45:27 -07:00
Sherif A. Nada
36b9b48a2e Migrate CDK sources to new Pypi package & structure (#3303) 2021-05-09 17:22:41 -07:00
Davin Chia
b5c2329748 CDK: Add OAuth and Streams/Core tests. (#3228)
Completely test the OAuth module and streams/core.py file.

Add pytest-mock as well as a convenience wrapper to run the type checker and the test together when developing.

Also configure pytest to pick up loggers at the INFO level and above, and pick up standard out for easier debugging.

Some of the script bits are unpolished - will improve them as we go along.
2021-05-06 15:52:23 +08:00
Davin Chia
fa33c9c126 Initial CDK Test set up and one test file. (#3207)
Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-05-04 14:27:58 -07:00
Eugene Kulak
88b77aa9f3 Standard tests: update template (#2937)
* update template for python sources
* update templates and their docs
* update docs, rename folders, add acceptance script
* adding hooks for custom setup and teardown
+ improve plugin behaviour, enable it explicitly
+ add acceptance.py to run acceptance tests together with integration tests
+ remove helper acceptance-test-python.sh
+ fix gradle command for acceptance tests
+ update docs

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-04-28 18:11:48 -04:00
Davin Chia
2188201c7c Upgrade to Pip 21.1. (#3070)
Upgrade to Pip 21.1 in preparation to upgrade DBT 19.

The latest Pip has

- a stricter resolver
- access to more up to date Python wheels (e.g. #2267 is solved since as any version after Pip 21 has access to ready-compiled Python wheels, see https://issues.apache.org/jira/browse/ARROW-11835).
- in theory faster pip installs (https://github.com/pypa/pip/issues/9187#issuecomment-826037738 makes me think the current Pip version is stable enough be consumed)

Also relax the constraint for `requests` and `backoff` since both were actually specifying incompatible versions that the old Pip resolver was silently swallowing.

Test this locally first by running git clean -dfx to remove all the existing venv files. The two different Pip versions do not work well.
2021-04-28 06:40:31 +08:00
Eugene Kulak
1755847c5e Standard Tests: create modular standard test suite v2 scaffolding (#2702)
* Standard Tests: Implement v2 test suites #2181 (#2792)
* rename standard tests (v2) to source_acceptance_test
2021-04-20 15:26:16 -04:00
Yury Koleda
b1061e32d9 🎉 Add MongoDB Source
Signed-off-by: fut <fut.wrk@gmail.com>
2021-03-08 14:27:14 -08:00
Sherif Nada
b6b1141e98 hotfix: use no-op pre/posthooks in standard tests 2021-03-05 07:11:06 -08:00
Sherif A. Nada
d382c876a9 Add pre/post hooks for std tests (#2233) 2021-03-03 16:34:02 -08:00
Jared Rhizor
fc94ccd6cb run isort before black (#2258) 2021-03-02 09:51:54 -08:00
Jared Rhizor
6a78cc327d add isort to format (#2085)
* add isort to format

* create airbytePythonFormat step
2021-02-16 10:51:14 -08:00
Sherif A. Nada
31208d6cc1 downgrade pip from 20.3 to 20.2 (#1658) 2021-01-14 17:26:47 -08:00
Sherif A. Nada
222413104a use standardSourceTestFile by default in python templates (#1360) 2020-12-17 13:45:17 -08:00
Sherif A. Nada
3ed5666313 Connector & standard tests documentation & add generic template (#1345) 2020-12-16 11:47:51 -08:00
Sherif A. Nada
8f30d665d1 Add Filesystem based standard test base (#1333) 2020-12-15 12:14:44 -08:00
Sherif A. Nada
894c4d5f49 Allow following symlinks when building docker images (#1332) 2020-12-15 11:53:34 -08:00
Jared Rhizor
1bd19d1bae put all integration test tasks under integrationTest (#1231)
* always re-run standardSourceTestPython

* rename and regroup to integrationTest

* add comment
2020-12-07 10:10:26 -08:00