jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-13 00:04:47 -05:00
Code Issues Packages Projects Releases Wiki Activity
6,265 Commits 5,242 Branches 518 Tags
e4d3d60ca8b683c301dfd5408917e1bb5e261cc0
Commit Graph

40 Commits

Author SHA1 Message Date
LiRen Tu
b90e5b1cc0 Fix connector base build (#13300) 2022-05-27 23:32:06 -07:00
Denis Davydov
426e9594cb fix build: run gradlew format (#13265) 
* fix build: run gradlew format

* Revert "fix build: run gradlew format"

This reverts commit e679b4f26f.

* fix connector template to fix the build
 2022-05-27 17:36:43 +03:00
Pedro S. Lopez
6119da60cb Generated python connectors use CDK version with YAML spec support (#12827) 
* bump cdk version for generated python source connectors

* update scaffold connectors
 2022-05-13 20:58:17 -04:00
Daniel Stevenson
22cebe4b5b CDK: DRY acceptance test templates (#12067) 2022-05-02 09:08:50 +02:00
Pedro S. Lopez
0c12ad9136 python generators output spec.yaml files (#12245) 
* generators output spec.yaml files

* source-singer generator also uses spec.yaml

* update scaffold

* update python cdk tutorials to use spec.yaml

* add docs updates

* consistency
 2022-04-26 07:16:04 -07:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763) 
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
 2022-04-11 20:51:37 -07:00
Tim Roes
2855dfcdce Upgrade plop in connector generators (#10578) 
* Upgrade plop

* Remove scaffolded code

* Build fixes

* Remove scaffolded code

* Revert "Remove scaffolded code"

This reverts commit 3911f527f8.

* Revert "Remove scaffolded code"

This reverts commit 549f790e3c.

* Remove .gitignore changes

* Remove .gitignore changes

* Update scaffold generated code

* Replace titleCase with capitalCase (#10654)

* Add capitalCase helper

* Replace titleCase with capitalCase

* Update generated scaffold files

Co-authored-by: LiRen Tu <tuliren.git@outlook.com>
 2022-02-28 08:05:14 +01:00
Maksym Pavlenok
64bd0a6cd8 🐛 Source Intercom: backoff for companies' scrolling (#8395) 
* backoff for companies scroll

* remove a unused companies stream property

* fix tests

* bump version

* update source_specs
 2021-12-03 01:17:54 +02:00
Yevhenii
a39119427a 🎉 Python Connector template: Remove Deprecated Dependency for Python Sources (#7385) 
* Remove Deprecated Dependency for Python API Template
 2021-11-17 11:38:59 +02:00
Marcos Marx
655df2cf86 Fix scaffold format files (#7989) 
* run scaffold format

* format generator files

* correct singer too
 2021-11-15 17:43:09 -03:00
Sherif A. Nada
5f03d32797 fix buffered stream consumer tests (#7834) 2021-11-11 08:25:52 -08:00
Marcos Marx
10434372de Change license in connector generator base files (#6522) 
* change license in generator base files

* run gradlew scaffold

* format eof

* add eof singer base

* remove license from test_* http source

* format test_incremental http source
 2021-09-29 04:45:42 -03:00
Maksym Pavlenok
802a8184cd 🐛 Python Alpine tempales: add build-base packages (#6457) 
Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
 2021-09-28 11:43:51 +03:00
Marcos Marx
c687d4068e Connector Generator: add additional pip install for tests in docs (#6411) 
* add additional pip install for tests

* run gradlew test scaffold

* correct flake and run format

* format generator files

* correct test_source scaffold gen

Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>
 2021-09-23 22:19:16 -03:00
Maksym Pavlenok
f76ca9f215 🐛 Python tempales: add default timezone for Alpine images (#5702) 
Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
 2021-08-27 19:01:36 +03:00
Dmytro
3630cdeded 🎉 New Source: Amazon Ads (initial version) (#5023) 
* 3772: Amazon Ads source, initial version


Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-08-26 11:15:38 +03:00
Maksym Pavlenok
70713902cd 🚀 Dockerfile Python templates: migration to Alpine Docker image (#5397) 
* update Dockerfile templates

* update dockerfile for python desctinations

* add bash for all python Dockerfile templates

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
 2021-08-18 22:42:46 +03:00
LiRen Tu
da98e5784d Clean up json schema in code generators (#5383) 
Follow up on #4907.
 2021-08-12 14:40:38 -07:00
Dmytro
db6daed922 Add autogenerated uittests for python http source (#5245) 
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-08-12 10:33:50 +03:00
LiRen Tu
2906ec287a CI: Add action to check broken doc links (#5254) 
* Add action to check broken doc links

* Ignore localhost

* Update config

* Fix broken links

* Use quiet mode

* Ignore PR link

* Fix more broken links

* Fix more broken links

* Fix more broken links

* Verify pattern

* Fix more broken links

* Separate full and pr check

* Update pattern

* Test invalid link

* Remove invalid link
 2021-08-07 14:28:02 -07:00
vitaliizazmic
5700645946 🎉 New Source: PrestaShop (#4465) 
* Source PrestaShop #2980 - init commit

* Source PrestaShop #2980 - schemas

* Source PrestaShop #2980 - full refresh sync

* Source PrestaShop #2980 - incremental sync, acceptance tests

* Source PrestaShop #2980 - incremental acceptance tests

* Source PrestaShop #2980 - incremental acceptance tests (remove configurations from incremental)

* Source PrestaShop #2980 - change Configurations stream to full refresh

* Source PrestaShop #2980 - docs

* format

* fix schemas, fill streams with data

* fix tests and future_state

* rename presta-shop -> prestashop

* fix mounting volumes inside a nested container

* special handle for NULL dates (MySQL)

* fix build URL

* fix URL and polishing

* fix header

* fix template

* format, update pre-commit to execute in the right order

* fix scaffolding

* bump and changelog for SAT

* register as the default source

* fix test timeout

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-07-26 21:55:07 -04:00
Sherif A. Nada
b86ebf63fb Automatically pull latest SAT & build connector image in SAT runner (#4969) 2021-07-26 11:56:19 -07:00
Eugene Kulak
0f51edbd60 SAT: Replace validate_output_from_all_streams with empty_streams param (#4897) 
* Replace validate_output_from_all_streams with empty_streams param

* update docs

* fix comment

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-07-22 16:26:35 -04:00
Davin Chia
5f7d36efa6 🐛 Add missing dependencies for acceptance tests to run. (#4808) 2021-07-19 12:24:07 +08:00
Jared Rhizor
b4793b2510 add AIRBYTE_ENTRYPOINT for kubernetes support (#3973) 
* add AIRBYTE_ENTRYPOINT for kubernetes support

* bump versions

* bump version in seed

* Update generic template

* keep scaffold sources at 0.1.0

* add missing newline

* handle python base versions correctly

* re-bump mysql and postgres sources

* re-bump snowflake destination

* add skip tests option

* switch to running tests

* reverse conditional to make it safer

* fix publish to include the test running

* fix iterable version

* fix file generation

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-06-09 13:01:45 -07:00
vovavovavovavova
21394c6dae Source-acceptance-test: improvements (#3888) 
* rename state_path to abnormal_state_path

* make cursor_path optional

* apply 3rd fix

* format

* abnormal->future (requested)

* upd names

* upd wording (cursor absence fail message)

* bump version in dockerfile
 2021-06-08 09:09:19 +03:00
Dmytro
a1035e007b 📚 Update all Python connectors READMEs with minimum required version (#3760) 2021-06-02 12:35:29 +03:00
Davin Chia
9a92da06b9 Update CDK Tutorial to use new CDK. (#3358) 
As the title suggests.

Some intricacies around how various directories have diverged.

The HTTP templates now use integration_tests to house catalogs and config files instead of sample_files. After this PR, everything should be updated to use integration_tests for all files except config files. Config files use secrets since they often contain secrets. Anything, besides actual implementations that have yet to be updated, not following this is my mistake.

Even though it diverges from the generated templates, I've decided to leave the tutorial code and the tutorial as using sample_files for now. I think it's more straightforward than telling people to create a new secrets directory. We can change this at a later date.

Confirmed the checked in http python tutorial code works by running ./gradlew :airbyte-integrations:connectors:source-python-http-tutorial:standardSourceTestFile.
 2021-05-11 16:52:13 +08:00
Davin Chia
94f6d17796 CDK: Update conceptual docs. Update template docs. (#3352) 2021-05-11 11:02:51 +08:00
Sherif A. Nada
ce4810e6a3 Use latest SAT image by default in generated templates (#3349) 2021-05-10 18:36:12 -07:00
Sherif A. Nada
d0b8e6ddc4 Comment out optional blocks in test config YML (#3347) 2021-05-10 18:28:20 -07:00
Sherif Nada
60e6f2fa6c hotfix: format 2021-05-10 17:08:00 -07:00
Davin Chia
945e46b53e Update generator templates to use CDK. (#3316) 
* Update singer template.
* Update source python.
* Update http python.
* Fix a bug where the template tests were not previously running.
 2021-05-11 06:26:22 +08:00
Eugene Kulak
752fe9172a add colors to tests when running inside docker (#3193) 
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-05-04 08:25:14 -04:00
Eugene Kulak
41ad3a9211 Standard Tests: fix incremental test and docs (#3161) 
* fix cursor comparision in the tests

+ update docs and templates

* format

* format

* fix colors

* update acceptance-test-docker.sh script

* fix colors in docker

* Update airbyte-integrations/connector-templates/source-python-http-api/integration_tests/acceptance.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/json_schema_helper.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* address comments from @sherifnada

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-05-03 13:43:10 -04:00
Davin Chia
fd0d253373 Release CDK version. (#3130) 2021-04-29 07:27:28 -07:00
Sherif A. Nada
80ce46f357 Base-python: fix exception when reading schemas from shared/ folder (#3127) 2021-04-29 00:38:54 -07:00
Davin Chia
a187aa667a Rename unit_tests to unit_test so they are detected. Make sure the generated file does this too. (#3082) 
Fix tests for the new CDK template generator. The previous test file was not recognised as it was named unit_tests instead of unit_test. Fix this.
 2021-04-27 18:21:42 +08:00
Sherif A. Nada
4e9f4bdd06 CDK: tutorial for implementing an HTTP source (#3079) 2021-04-27 00:28:10 -07:00
Davin Chia
f660b0a946 Add template generation for Santa aka CDK. (#3034) 
Template generation for new Source using the Santa CDK - provide basic scaffolding for someone implementing a new source.

General approach is to buff up comments in the original SDK, and add TODOs with secondary comments in the generated stub methods, as well as links to existing examples (e.g. Stripe or ExchangeRate api) users can look at.

Checked in and added tests for the generated modules.
 2021-04-25 18:02:33 +08:00