1
0
mirror of synced 2025-12-22 19:38:29 -05:00
Commit Graph

138 Commits

Author SHA1 Message Date
vitaliizazmic
5700645946 🎉 New Source: PrestaShop (#4465)
* Source PrestaShop #2980 - init commit

* Source PrestaShop #2980 - schemas

* Source PrestaShop #2980 - full refresh sync

* Source PrestaShop #2980 - incremental sync, acceptance tests

* Source PrestaShop #2980 - incremental acceptance tests

* Source PrestaShop #2980 - incremental acceptance tests (remove configurations from incremental)

* Source PrestaShop #2980 - change Configurations stream to full refresh

* Source PrestaShop #2980 - docs

* format

* fix schemas, fill streams with data

* fix tests and future_state

* rename presta-shop -> prestashop

* fix mounting volumes inside a nested container

* special handle for NULL dates (MySQL)

* fix build URL

* fix URL and polishing

* fix header

* fix template

* format, update pre-commit to execute in the right order

* fix scaffolding

* bump and changelog for SAT

* register as the default source

* fix test timeout

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-07-26 21:55:07 -04:00
Sherif A. Nada
a4bb30430d Python Demo Destination: KVDB (#4786) 2021-07-19 09:44:36 -07:00
Marcos Marx
6201100e72 Add entrypoint and bump version for remaining connectors. (#4539)
See the PR descriptions for all the connectors published.
2021-07-07 17:37:22 +08:00
Davin Chia
683a67c023 Upgrade Gradle Python plugin + explicitly install PyEnv to fix transient checkPython error. (#3362)
This is what the plugin's maintainer suggested: xvik/gradle-use-python-plugin#19.

I ran the build 10 times in a row without any checkPython error so this does seem to have fixed it.
2021-06-25 17:32:41 +08:00
Jared Rhizor
ef853153a5 kube zombie handling (#4137)
* working except for too much logging and bad success case

* succeeds on passing case

* completes successfully

* just doesn't kill the main

* working zombie killing

* cleanup

* more cleanup

* use correct path

* fmt

* cleanups, bugfixes, integration tests

* run worker integration tests as part of ci

* delete tester class

* fix hanging checkpoint container problem

* fix name of command

* replace todo with clarifying comment
2021-06-17 17:51:58 -07:00
Sherif A. Nada
6ec8b31bee revert change to pythonBinary (#3993) 2021-06-09 08:47:26 -07:00
Sherif A. Nada
7764fd4fab Add Gradle tasks to cleanup Python Venvs and always use python binary not python3 (#3972) 2021-06-08 16:22:34 -07:00
Eugene Kulak
c61fc3df07 SAT: migrate to CDK (#3640)
* move SAT to CDK and fix image tag

* fix publish

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-05-28 04:32:15 -04:00
Charles
8983f09aea normalize connector acceptance test names (#3539)
* Rename standard tests to acceptance tests

* Normalize the names so that the nouns are always in the same order so it is easier to find tests
2021-05-22 13:40:40 -07:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Sherif A. Nada
535d83e0af bugfix infinite pagination in CDK (#3366) 2021-05-11 14:45:27 -07:00
Sherif A. Nada
36b9b48a2e Migrate CDK sources to new Pypi package & structure (#3303) 2021-05-09 17:22:41 -07:00
Davin Chia
b5c2329748 CDK: Add OAuth and Streams/Core tests. (#3228)
Completely test the OAuth module and streams/core.py file.

Add pytest-mock as well as a convenience wrapper to run the type checker and the test together when developing.

Also configure pytest to pick up loggers at the INFO level and above, and pick up standard out for easier debugging.

Some of the script bits are unpolished - will improve them as we go along.
2021-05-06 15:52:23 +08:00
Davin Chia
fa33c9c126 Initial CDK Test set up and one test file. (#3207)
Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-05-04 14:27:58 -07:00
Eugene Kulak
88b77aa9f3 Standard tests: update template (#2937)
* update template for python sources
* update templates and their docs
* update docs, rename folders, add acceptance script
* adding hooks for custom setup and teardown
+ improve plugin behaviour, enable it explicitly
+ add acceptance.py to run acceptance tests together with integration tests
+ remove helper acceptance-test-python.sh
+ fix gradle command for acceptance tests
+ update docs

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-04-28 18:11:48 -04:00
Davin Chia
2188201c7c Upgrade to Pip 21.1. (#3070)
Upgrade to Pip 21.1 in preparation to upgrade DBT 19.

The latest Pip has

- a stricter resolver
- access to more up to date Python wheels (e.g. #2267 is solved since as any version after Pip 21 has access to ready-compiled Python wheels, see https://issues.apache.org/jira/browse/ARROW-11835).
- in theory faster pip installs (https://github.com/pypa/pip/issues/9187#issuecomment-826037738 makes me think the current Pip version is stable enough be consumed)

Also relax the constraint for `requests` and `backoff` since both were actually specifying incompatible versions that the old Pip resolver was silently swallowing.

Test this locally first by running git clean -dfx to remove all the existing venv files. The two different Pip versions do not work well.
2021-04-28 06:40:31 +08:00
Eugene Kulak
1755847c5e Standard Tests: create modular standard test suite v2 scaffolding (#2702)
* Standard Tests: Implement v2 test suites #2181 (#2792)
* rename standard tests (v2) to source_acceptance_test
2021-04-20 15:26:16 -04:00
Yury Koleda
b1061e32d9 🎉 Add MongoDB Source
Signed-off-by: fut <fut.wrk@gmail.com>
2021-03-08 14:27:14 -08:00
Sherif Nada
b6b1141e98 hotfix: use no-op pre/posthooks in standard tests 2021-03-05 07:11:06 -08:00
Sherif A. Nada
d382c876a9 Add pre/post hooks for std tests (#2233) 2021-03-03 16:34:02 -08:00
Jared Rhizor
fc94ccd6cb run isort before black (#2258) 2021-03-02 09:51:54 -08:00
Jared Rhizor
6a78cc327d add isort to format (#2085)
* add isort to format

* create airbytePythonFormat step
2021-02-16 10:51:14 -08:00
Sherif A. Nada
31208d6cc1 downgrade pip from 20.3 to 20.2 (#1658) 2021-01-14 17:26:47 -08:00
Sherif A. Nada
222413104a use standardSourceTestFile by default in python templates (#1360) 2020-12-17 13:45:17 -08:00
Sherif A. Nada
3ed5666313 Connector & standard tests documentation & add generic template (#1345) 2020-12-16 11:47:51 -08:00
Sherif A. Nada
8f30d665d1 Add Filesystem based standard test base (#1333) 2020-12-15 12:14:44 -08:00
Sherif A. Nada
894c4d5f49 Allow following symlinks when building docker images (#1332) 2020-12-15 11:53:34 -08:00
Jared Rhizor
1bd19d1bae put all integration test tasks under integrationTest (#1231)
* always re-run standardSourceTestPython

* rename and regroup to integrationTest

* add comment
2020-12-07 10:10:26 -08:00
Jared Rhizor
00345c9552 fix integration tests that can't find base-python-test image (#1202) 2020-12-04 11:03:46 -08:00
Jared Rhizor
55ea0b677b make exchangerate integration test depend on docker building (#1157)
* add exchange rate dep and change behavior

* fix integration test dependencies

* revert singer helpers changes
2020-12-01 23:40:20 -08:00
Jared Rhizor
674ee2cf72 enforce order of installing python dependencies (#1149)
* fix python install order

* fix base install reqs dependencies

* don't clobber dependencies

* remove cacheIf forcing

* it wasn't clobbering actually

* fix pip version

* loosen pyyaml version

* Revert "loosen pyyaml version"

This reverts commit 5a4eb3bc08.

* move tap-adwords dependency

* downgrade shopify api version used to match

* run seed init

* add test curl

* Revert "add test curl"

This reverts commit caee9ee9ee.

* revert to simpler docker times

* rename task function so it doesn't match the task name

* turn off buildkit

* fix redshift dir for credentials

* add twilio and freshdesk to seed yaml

* fix typo

* get shopify passing similar to the adwords fix
2020-12-01 17:40:58 -08:00
Charles
31c3f06805 Install python deps declared in setup.py (#1111) 2020-11-30 00:57:07 -08:00
Jared Rhizor
5bbf459c92 depend on main dependency installation for python tests (#1039) 2020-11-20 12:00:23 -08:00
Jared Rhizor
05404bdd58 improve caching (#1005)
* python caching

* improve gradle caching

* store pip cache dir

* debugging info

* split build steps and temporarily remove testing infra from build

* fix formatting error

* fix another syntax error

* split repo caches

* switch pip dir

* use git hash instead of rehashing everything

* syntax

* list directories

* list after cache population

* list .docker

* specify custom output cache directory for docker

* only build bigquery and use buildx on ci

* add build command

* use docker driver instead of docker-container

* use correct key

* use local registry

* allow push failure

* make sure to tag first

* move part of node caching and restore rest of actual build

* cleanup

* clean up mssql gradle file

* toggle on inline cache

* try save and load method instead

* use separate tar for each image

* limit to 20

* always succeed even though disk space runs out

* try to just pull the public image to see if that's faster

* use cachefrom

* revert back to local registry version

* see how buildkit compares

* address review comments

* add comments
2020-11-19 13:14:46 -08:00
Jared Rhizor
e33f988f72 fix formatting race condition (#910) 2020-11-11 11:26:37 -08:00
Jared Rhizor
acc2cc9f57 remove all python caching (#870) 2020-11-09 23:05:28 -08:00
Jared Rhizor
adcb8b2fab add dependency on installing test reqs (#864) 2020-11-09 15:35:59 -08:00
Jared Rhizor
e1d7add50a mostly incremental builds (#817)
* support cached builds for base -> base-java -> snowflake

* use plugin for image building

* fix matching on COPY from

* remove docker.gradle

* tmp commit

* update connectors

* finish rest of build files

* fix ide errors

* more build fixes

* clean up

* clean up for new soruces

* fix spotless

* fix flake problems

* add recommended empty file

* python caching

* fixes upon review

* clean up docker and build test files

* clean up python

* clean up

* fix integration test dependencies

* fix standard tests

* fix

* remove symlink

* re-add requirements to fix normalizatioon build

* fix symlink

* fix dumbest build problem of all

* add missing integration test def

* fix missing dep

* remove class exclusion

* move trim so null source versions are allowed

* rename map

* fix hardcoded value

* remove unnecessary dep

* use dashes for salesforce package name

* fix typo

* DRY and fix test image name

* Fix edit

* assert string is not empty

* build integration test image only for integrationTest

* move code generator to tools and rename docker build tasks

* make source test depend on integration test build, not the other way

* remove guard because the docker build should exist before the integrationtest is applied

* remove comment

* DRY up airbyte-source-test

* fix plugin compilation

* add missing dependency

* rename getTaggedImage to getDevTaggedImage

* fix test vs main docker build bug
2020-11-09 11:00:11 -08:00