jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-09 15:05:02 -05:00
Code Issues Packages Projects Releases Wiki Activity
6,265 Commits 5,238 Branches 518 Tags
e4d3d60ca8b683c301dfd5408917e1bb5e261cc0
Commit Graph

68 Commits

Author SHA1 Message Date
Denis Davydov
426e9594cb fix build: run gradlew format (#13265) 
* fix build: run gradlew format

* Revert "fix build: run gradlew format"

This reverts commit e679b4f26f.

* fix connector template to fix the build
 2022-05-27 17:36:43 +03:00
Pedro S. Lopez
6119da60cb Generated python connectors use CDK version with YAML spec support (#12827) 
* bump cdk version for generated python source connectors

* update scaffold connectors
 2022-05-13 20:58:17 -04:00
Daniel Stevenson
22cebe4b5b CDK: DRY acceptance test templates (#12067) 2022-05-02 09:08:50 +02:00
Pedro S. Lopez
0c12ad9136 python generators output spec.yaml files (#12245) 
* generators output spec.yaml files

* source-singer generator also uses spec.yaml

* update scaffold

* update python cdk tutorials to use spec.yaml

* add docs updates

* consistency
 2022-04-26 07:16:04 -07:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763) 
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
 2022-04-11 20:51:37 -07:00
Tim Roes
2855dfcdce Upgrade plop in connector generators (#10578) 
* Upgrade plop

* Remove scaffolded code

* Build fixes

* Remove scaffolded code

* Revert "Remove scaffolded code"

This reverts commit 3911f527f8.

* Revert "Remove scaffolded code"

This reverts commit 549f790e3c.

* Remove .gitignore changes

* Remove .gitignore changes

* Update scaffold generated code

* Replace titleCase with capitalCase (#10654)

* Add capitalCase helper

* Replace titleCase with capitalCase

* Update generated scaffold files

Co-authored-by: LiRen Tu <tuliren.git@outlook.com>
 2022-02-28 08:05:14 +01:00
Maksym Pavlenok
64bd0a6cd8 🐛 Source Intercom: backoff for companies' scrolling (#8395) 
* backoff for companies scroll

* remove a unused companies stream property

* fix tests

* bump version

* update source_specs
 2021-12-03 01:17:54 +02:00
Yevhenii
a39119427a 🎉 Python Connector template: Remove Deprecated Dependency for Python Sources (#7385) 
* Remove Deprecated Dependency for Python API Template
 2021-11-17 11:38:59 +02:00
Marcos Marx
655df2cf86 Fix scaffold format files (#7989) 
* run scaffold format

* format generator files

* correct singer too
 2021-11-15 17:43:09 -03:00
Sherif A. Nada
5f03d32797 fix buffered stream consumer tests (#7834) 2021-11-11 08:25:52 -08:00
Marcos Marx
10434372de Change license in connector generator base files (#6522) 
* change license in generator base files

* run gradlew scaffold

* format eof

* add eof singer base

* remove license from test_* http source

* format test_incremental http source
 2021-09-29 04:45:42 -03:00
Maksym Pavlenok
802a8184cd 🐛 Python Alpine tempales: add build-base packages (#6457) 
Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
 2021-09-28 11:43:51 +03:00
Maksym Pavlenok
f76ca9f215 🐛 Python tempales: add default timezone for Alpine images (#5702) 
Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
 2021-08-27 19:01:36 +03:00
Dmytro
3630cdeded 🎉 New Source: Amazon Ads (initial version) (#5023) 
* 3772: Amazon Ads source, initial version


Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-08-26 11:15:38 +03:00
Maksym Pavlenok
70713902cd 🚀 Dockerfile Python templates: migration to Alpine Docker image (#5397) 
* update Dockerfile templates

* update dockerfile for python desctinations

* add bash for all python Dockerfile templates

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
 2021-08-18 22:42:46 +03:00
LiRen Tu
da98e5784d Clean up json schema in code generators (#5383) 
Follow up on #4907.
 2021-08-12 14:40:38 -07:00
LiRen Tu
2906ec287a CI: Add action to check broken doc links (#5254) 
* Add action to check broken doc links

* Ignore localhost

* Update config

* Fix broken links

* Use quiet mode

* Ignore PR link

* Fix more broken links

* Fix more broken links

* Fix more broken links

* Verify pattern

* Fix more broken links

* Separate full and pr check

* Update pattern

* Test invalid link

* Remove invalid link
 2021-08-07 14:28:02 -07:00
Sherif A. Nada
b86ebf63fb Automatically pull latest SAT & build connector image in SAT runner (#4969) 2021-07-26 11:56:19 -07:00
Eugene Kulak
0f51edbd60 SAT: Replace validate_output_from_all_streams with empty_streams param (#4897) 
* Replace validate_output_from_all_streams with empty_streams param

* update docs

* fix comment

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-07-22 16:26:35 -04:00
Sherif A. Nada
a4bb30430d Python Demo Destination: KVDB (#4786) 2021-07-19 09:44:36 -07:00
Davin Chia
5f7d36efa6 🐛 Add missing dependencies for acceptance tests to run. (#4808) 2021-07-19 12:24:07 +08:00
Jared Rhizor
b4793b2510 add AIRBYTE_ENTRYPOINT for kubernetes support (#3973) 
* add AIRBYTE_ENTRYPOINT for kubernetes support

* bump versions

* bump version in seed

* Update generic template

* keep scaffold sources at 0.1.0

* add missing newline

* handle python base versions correctly

* re-bump mysql and postgres sources

* re-bump snowflake destination

* add skip tests option

* switch to running tests

* reverse conditional to make it safer

* fix publish to include the test running

* fix iterable version

* fix file generation

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-06-09 13:01:45 -07:00
vovavovavovavova
21394c6dae Source-acceptance-test: improvements (#3888) 
* rename state_path to abnormal_state_path

* make cursor_path optional

* apply 3rd fix

* format

* abnormal->future (requested)

* upd names

* upd wording (cursor absence fail message)

* bump version in dockerfile
 2021-06-08 09:09:19 +03:00
Dmytro
a1035e007b 📚 Update all Python connectors READMEs with minimum required version (#3760) 2021-06-02 12:35:29 +03:00
Davin Chia
9a92da06b9 Update CDK Tutorial to use new CDK. (#3358) 
As the title suggests.

Some intricacies around how various directories have diverged.

The HTTP templates now use integration_tests to house catalogs and config files instead of sample_files. After this PR, everything should be updated to use integration_tests for all files except config files. Config files use secrets since they often contain secrets. Anything, besides actual implementations that have yet to be updated, not following this is my mistake.

Even though it diverges from the generated templates, I've decided to leave the tutorial code and the tutorial as using sample_files for now. I think it's more straightforward than telling people to create a new secrets directory. We can change this at a later date.

Confirmed the checked in http python tutorial code works by running ./gradlew :airbyte-integrations:connectors:source-python-http-tutorial:standardSourceTestFile.
 2021-05-11 16:52:13 +08:00
Davin Chia
94f6d17796 CDK: Update conceptual docs. Update template docs. (#3352) 2021-05-11 11:02:51 +08:00
Sherif A. Nada
ce4810e6a3 Use latest SAT image by default in generated templates (#3349) 2021-05-10 18:36:12 -07:00
Sherif A. Nada
d0b8e6ddc4 Comment out optional blocks in test config YML (#3347) 2021-05-10 18:28:20 -07:00
Davin Chia
945e46b53e Update generator templates to use CDK. (#3316) 
* Update singer template.
* Update source python.
* Update http python.
* Fix a bug where the template tests were not previously running.
 2021-05-11 06:26:22 +08:00
Eugene Kulak
752fe9172a add colors to tests when running inside docker (#3193) 
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-05-04 08:25:14 -04:00
Eugene Kulak
41ad3a9211 Standard Tests: fix incremental test and docs (#3161) 
* fix cursor comparision in the tests

+ update docs and templates

* format

* format

* fix colors

* update acceptance-test-docker.sh script

* fix colors in docker

* Update airbyte-integrations/connector-templates/source-python-http-api/integration_tests/acceptance.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/json_schema_helper.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* address comments from @sherifnada

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-05-03 13:43:10 -04:00
Christophe Duong
8dff1586ab Fix spotlessApply conflict with black (#3152) 
* Fix spotlessApply conflict with black
 2021-04-30 21:09:42 +02:00
Eugene Kulak
7f66569436 Standard Tests: allow specifying exact record matches #2186 (#2960) 
* records matching

+ extra_fields
+ exact_order
+ extra_records

* fix serialization before comparision

* fix pydantic validation

* fix reporting and enable for hubspot

* format

* update docs

* update examples

* fix template

* Update docs/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-04-30 04:21:35 -04:00
Davin Chia
fd0d253373 Release CDK version. (#3130) 2021-04-29 07:27:28 -07:00
Sherif A. Nada
80ce46f357 Base-python: fix exception when reading schemas from shared/ folder (#3127) 2021-04-29 00:38:54 -07:00
Eugene Kulak
8b1ac13c7a Standard Tests: hotfix - upgrade PyYAML (#3123) 
* upgrade PyYAML

* format

* format java files too

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-04-28 19:04:25 -04:00
Eugene Kulak
88b77aa9f3 Standard tests: update template (#2937) 
* update template for python sources
* update templates and their docs
* update docs, rename folders, add acceptance script
* adding hooks for custom setup and teardown
+ improve plugin behaviour, enable it explicitly
+ add acceptance.py to run acceptance tests together with integration tests
+ remove helper acceptance-test-python.sh
+ fix gradle command for acceptance tests
+ update docs

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-04-28 18:11:48 -04:00
Davin Chia
f660b0a946 Add template generation for Santa aka CDK. (#3034) 
Template generation for new Source using the Santa CDK - provide basic scaffolding for someone implementing a new source.

General approach is to buff up comments in the original SDK, and add TODOs with secondary comments in the generated stub methods, as well as links to existing examples (e.g. Stripe or ExchangeRate api) users can look at.

Checked in and added tests for the generated modules.
 2021-04-25 18:02:33 +08:00
Sherif A. Nada
2d2d0173d6 Update codegen template READMEs (#2873) 2021-04-13 11:41:52 -07:00
Jared Rhizor
0609011e05 fix connector base versioning (#2726) 
* fix connector base versioning

* only run integration tests in non-bases directories

* upgrade datamodel-code-generator which coincidentally has already fixed this issue with generating List of Lists

* bump forgotten hubspot version

* fix build
 2021-04-02 22:46:06 -07:00
Christophe Duong
bc1547f185 Fix integration tests of connectors related to destination_sync_mode and sync_mode being required (#2688) 2021-04-01 09:58:38 +02:00
Sherif A. Nada
415f9d6d76 re-add instructions for building python packages using Gradle (#2467) 2021-03-15 13:31:57 -07:00
Sherif A. Nada
e87468d4bc improve instructions in templates & publish base docker images to dockerhub (#2441) 2021-03-15 02:16:36 -07:00
Sherif A. Nada
2139a9656b Cleanup Python packages (#2440) 2021-03-15 01:24:21 -07:00
Sherif A. Nada
ac0bcf7d01 Add comment at the top of all requirements.txt files clarifying they should not be used for dependencies (#2366) 2021-03-08 19:01:47 -08:00
Michel Tricot
9a2ec03e40 Test connector templates (#2081) 2021-02-16 22:05:29 -08:00
Michel Tricot
ac092ada6e Hide connector scaffolding behind a script (#2080) 2021-02-16 20:09:03 -08:00
Jared Rhizor
30e2e98c49 fix formatting issues (#2045) 
* fix formatting issues

* remove space

* unit tests -> format anchor
 2021-02-11 13:02:21 -08:00
Charles
be5fbc2e8a Move Python checklist to docs (#1810) 2021-01-25 16:55:55 -08:00
Charles
e3639b1a93 add import to build.gradle.hbs so that python module will build (#1556) 2021-01-06 16:04:57 -08:00