jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-20 21:06:36 -05:00
Code Issues Packages Projects Releases Wiki Activity
3,680 Commits 5,351 Branches 518 Tags
b94ee00fd8e5220e3191073cd711a57e052d7fda
Commit Graph

58 Commits

Author SHA1 Message Date
Marcos Marx
10434372de Change license in connector generator base files (#6522) 
* change license in generator base files

* run gradlew scaffold

* format eof

* add eof singer base

* remove license from test_* http source

* format test_incremental http source
 2021-09-29 04:45:42 -03:00
Maksym Pavlenok
802a8184cd 🐛 Python Alpine tempales: add build-base packages (#6457) 
Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
 2021-09-28 11:43:51 +03:00
Maksym Pavlenok
f76ca9f215 🐛 Python tempales: add default timezone for Alpine images (#5702) 
Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
 2021-08-27 19:01:36 +03:00
Dmytro
3630cdeded 🎉 New Source: Amazon Ads (initial version) (#5023) 
* 3772: Amazon Ads source, initial version


Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-08-26 11:15:38 +03:00
Maksym Pavlenok
70713902cd 🚀 Dockerfile Python templates: migration to Alpine Docker image (#5397) 
* update Dockerfile templates

* update dockerfile for python desctinations

* add bash for all python Dockerfile templates

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
 2021-08-18 22:42:46 +03:00
LiRen Tu
da98e5784d Clean up json schema in code generators (#5383) 
Follow up on #4907.
 2021-08-12 14:40:38 -07:00
LiRen Tu
2906ec287a CI: Add action to check broken doc links (#5254) 
* Add action to check broken doc links

* Ignore localhost

* Update config

* Fix broken links

* Use quiet mode

* Ignore PR link

* Fix more broken links

* Fix more broken links

* Fix more broken links

* Verify pattern

* Fix more broken links

* Separate full and pr check

* Update pattern

* Test invalid link

* Remove invalid link
 2021-08-07 14:28:02 -07:00
Sherif A. Nada
b86ebf63fb Automatically pull latest SAT & build connector image in SAT runner (#4969) 2021-07-26 11:56:19 -07:00
Eugene Kulak
0f51edbd60 SAT: Replace validate_output_from_all_streams with empty_streams param (#4897) 
* Replace validate_output_from_all_streams with empty_streams param

* update docs

* fix comment

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-07-22 16:26:35 -04:00
Sherif A. Nada
a4bb30430d Python Demo Destination: KVDB (#4786) 2021-07-19 09:44:36 -07:00
Davin Chia
5f7d36efa6 🐛 Add missing dependencies for acceptance tests to run. (#4808) 2021-07-19 12:24:07 +08:00
Jared Rhizor
b4793b2510 add AIRBYTE_ENTRYPOINT for kubernetes support (#3973) 
* add AIRBYTE_ENTRYPOINT for kubernetes support

* bump versions

* bump version in seed

* Update generic template

* keep scaffold sources at 0.1.0

* add missing newline

* handle python base versions correctly

* re-bump mysql and postgres sources

* re-bump snowflake destination

* add skip tests option

* switch to running tests

* reverse conditional to make it safer

* fix publish to include the test running

* fix iterable version

* fix file generation

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-06-09 13:01:45 -07:00
vovavovavovavova
21394c6dae Source-acceptance-test: improvements (#3888) 
* rename state_path to abnormal_state_path

* make cursor_path optional

* apply 3rd fix

* format

* abnormal->future (requested)

* upd names

* upd wording (cursor absence fail message)

* bump version in dockerfile
 2021-06-08 09:09:19 +03:00
Dmytro
a1035e007b 📚 Update all Python connectors READMEs with minimum required version (#3760) 2021-06-02 12:35:29 +03:00
Davin Chia
9a92da06b9 Update CDK Tutorial to use new CDK. (#3358) 
As the title suggests.

Some intricacies around how various directories have diverged.

The HTTP templates now use integration_tests to house catalogs and config files instead of sample_files. After this PR, everything should be updated to use integration_tests for all files except config files. Config files use secrets since they often contain secrets. Anything, besides actual implementations that have yet to be updated, not following this is my mistake.

Even though it diverges from the generated templates, I've decided to leave the tutorial code and the tutorial as using sample_files for now. I think it's more straightforward than telling people to create a new secrets directory. We can change this at a later date.

Confirmed the checked in http python tutorial code works by running ./gradlew :airbyte-integrations:connectors:source-python-http-tutorial:standardSourceTestFile.
 2021-05-11 16:52:13 +08:00
Davin Chia
94f6d17796 CDK: Update conceptual docs. Update template docs. (#3352) 2021-05-11 11:02:51 +08:00
Sherif A. Nada
ce4810e6a3 Use latest SAT image by default in generated templates (#3349) 2021-05-10 18:36:12 -07:00
Sherif A. Nada
d0b8e6ddc4 Comment out optional blocks in test config YML (#3347) 2021-05-10 18:28:20 -07:00
Davin Chia
945e46b53e Update generator templates to use CDK. (#3316) 
* Update singer template.
* Update source python.
* Update http python.
* Fix a bug where the template tests were not previously running.
 2021-05-11 06:26:22 +08:00
Eugene Kulak
752fe9172a add colors to tests when running inside docker (#3193) 
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-05-04 08:25:14 -04:00
Eugene Kulak
41ad3a9211 Standard Tests: fix incremental test and docs (#3161) 
* fix cursor comparision in the tests

+ update docs and templates

* format

* format

* fix colors

* update acceptance-test-docker.sh script

* fix colors in docker

* Update airbyte-integrations/connector-templates/source-python-http-api/integration_tests/acceptance.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/json_schema_helper.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* address comments from @sherifnada

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-05-03 13:43:10 -04:00
Christophe Duong
8dff1586ab Fix spotlessApply conflict with black (#3152) 
* Fix spotlessApply conflict with black
 2021-04-30 21:09:42 +02:00
Eugene Kulak
7f66569436 Standard Tests: allow specifying exact record matches #2186 (#2960) 
* records matching

+ extra_fields
+ exact_order
+ extra_records

* fix serialization before comparision

* fix pydantic validation

* fix reporting and enable for hubspot

* format

* update docs

* update examples

* fix template

* Update docs/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-04-30 04:21:35 -04:00
Davin Chia
fd0d253373 Release CDK version. (#3130) 2021-04-29 07:27:28 -07:00
Sherif A. Nada
80ce46f357 Base-python: fix exception when reading schemas from shared/ folder (#3127) 2021-04-29 00:38:54 -07:00
Eugene Kulak
8b1ac13c7a Standard Tests: hotfix - upgrade PyYAML (#3123) 
* upgrade PyYAML

* format

* format java files too

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-04-28 19:04:25 -04:00
Eugene Kulak
88b77aa9f3 Standard tests: update template (#2937) 
* update template for python sources
* update templates and their docs
* update docs, rename folders, add acceptance script
* adding hooks for custom setup and teardown
+ improve plugin behaviour, enable it explicitly
+ add acceptance.py to run acceptance tests together with integration tests
+ remove helper acceptance-test-python.sh
+ fix gradle command for acceptance tests
+ update docs

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-04-28 18:11:48 -04:00
Davin Chia
f660b0a946 Add template generation for Santa aka CDK. (#3034) 
Template generation for new Source using the Santa CDK - provide basic scaffolding for someone implementing a new source.

General approach is to buff up comments in the original SDK, and add TODOs with secondary comments in the generated stub methods, as well as links to existing examples (e.g. Stripe or ExchangeRate api) users can look at.

Checked in and added tests for the generated modules.
 2021-04-25 18:02:33 +08:00
Sherif A. Nada
2d2d0173d6 Update codegen template READMEs (#2873) 2021-04-13 11:41:52 -07:00
Jared Rhizor
0609011e05 fix connector base versioning (#2726) 
* fix connector base versioning

* only run integration tests in non-bases directories

* upgrade datamodel-code-generator which coincidentally has already fixed this issue with generating List of Lists

* bump forgotten hubspot version

* fix build
 2021-04-02 22:46:06 -07:00
Christophe Duong
bc1547f185 Fix integration tests of connectors related to destination_sync_mode and sync_mode being required (#2688) 2021-04-01 09:58:38 +02:00
Sherif A. Nada
415f9d6d76 re-add instructions for building python packages using Gradle (#2467) 2021-03-15 13:31:57 -07:00
Sherif A. Nada
e87468d4bc improve instructions in templates & publish base docker images to dockerhub (#2441) 2021-03-15 02:16:36 -07:00
Sherif A. Nada
2139a9656b Cleanup Python packages (#2440) 2021-03-15 01:24:21 -07:00
Sherif A. Nada
ac0bcf7d01 Add comment at the top of all requirements.txt files clarifying they should not be used for dependencies (#2366) 2021-03-08 19:01:47 -08:00
Michel Tricot
9a2ec03e40 Test connector templates (#2081) 2021-02-16 22:05:29 -08:00
Michel Tricot
ac092ada6e Hide connector scaffolding behind a script (#2080) 2021-02-16 20:09:03 -08:00
Jared Rhizor
30e2e98c49 fix formatting issues (#2045) 
* fix formatting issues

* remove space

* unit tests -> format anchor
 2021-02-11 13:02:21 -08:00
Charles
be5fbc2e8a Move Python checklist to docs (#1810) 2021-01-25 16:55:55 -08:00
Charles
e3639b1a93 add import to build.gradle.hbs so that python module will build (#1556) 2021-01-06 16:04:57 -08:00
Sherif A. Nada
222413104a use standardSourceTestFile by default in python templates (#1360) 2020-12-17 13:45:17 -08:00
Christophe Duong
bc207528b3 Add docs in python source templates (#1365) 
* Add docstrings to template code (#1320)

* Make standard tests passing on a newly created python source (will make it easier to automate testing of #725)

* Add docstring in python singer source

* Add note on loggers
 2020-12-17 19:29:12 +01:00
Christophe Duong
e92e9aa8c4 Refactor python sources (#1331) 
* Make ConfigContainer internal to SingerSource only
 2020-12-16 17:52:17 +01:00
Charles
25689eea56 add incremental to jooq source (and postgres) (#1172) 2020-12-08 21:14:11 -08:00
Sherif A. Nada
e59c4994e9 Update templates to reflect standard test changes (#1214) 
* update templates for new tests

* one more

* add renamed files
 2020-12-04 20:27:24 -08:00
Charles
eb601c5be0 Add Connectors using an index yaml file, instead of individual json files (#1046) 2020-11-30 01:04:19 -08:00
Sherif A. Nada
de09b40b93 Update templates to reflect incremental model changes (#1083) 2020-11-24 22:22:03 -08:00
Sherif A. Nada
b05a2b0839 Update connector READMEs to reflect new build plugin commands (#969) 2020-11-12 20:36:18 -08:00
Charles
cf21f2a3a2 Various Docs Updates (#964) 
* Explain how to implement standard tests in python and singer sources
* Make a note that basic normalization doesn't current work with >100 many fields. There's a fix in progress for this.
* Document how to reset the connectors index in local development.
 2020-11-12 16:13:53 -08:00
Charles
54dba21b96 use hyphen instead of underscore in template (#907) 2020-11-11 12:01:20 -08:00