1
0
mirror of synced 2025-12-21 19:11:14 -05:00
Commit Graph

41 Commits

Author SHA1 Message Date
Aaron ("AJ") Steers
c6d49bdc60 Python-CDK: Remove CDK (replaced by new repo) (#48470) 2024-11-18 17:45:18 -08:00
Natik Gadzhi
33a5cf777b [Python CDK] Delete SingerSource (#37364) 2024-04-17 07:21:22 -07:00
Natik Gadzhi
e4c942e526 [python-cdk] README cleanup (#37306) 2024-04-16 14:57:11 -07:00
Natik Gadzhi
33235c80d2 [python-cdk] readme entry about publishing new versions (#37330)
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
2024-04-16 14:55:43 -07:00
Ella Rohm-Ensing
d58466c0bb CDK: fix empty extras (#36485) 2024-03-26 15:49:47 +00:00
Ella Rohm-Ensing
195408eb15 Airbyte CDK: move from gradle to poetry (#36250) 2024-03-25 18:46:57 -05:00
Marius Posta
fdfd048225 python CDK: fix gradle task dependency (#35609) 2024-02-27 11:01:38 -08:00
Artem Inzhyyants
3355c5c432 Airbyte CDK: add filter to RemoveFields (#35326)
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
2024-02-21 13:49:10 +01:00
Natik Gadzhi
68e833104b Docs: review and update connector form scratch documentation and related guides (#33822)
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2024-01-02 16:59:51 -08:00
Marius Posta
53735b6d37 gradle: remove SUB_BUILD env var and references (#31027) 2023-10-04 14:58:22 -07:00
Ben Church
07a79897b3 Airbyte-ci: Add local cdk support (#30461) 2023-09-19 12:27:18 -07:00
Maxime Carbonneau-Leclerc
82b74cd06c Document mockserver usage for CDK development (#30202) 2023-09-07 12:31:01 -04:00
Ben Church
fb7258e2bd Move tools/ci_* projects to airbyte-ci, update to use Poetry, bump to python 3.10 (#27957)
* Move ci_connector_ops

* Move ci_credentials

* Move tools/ci_common_utils

* Rename tools to airbyte-ci

* Move to ci

* Convert ci_credentials

* Convert ci_common_utls

* Convert ci_connector_ops

* Get pipelines running

* Move pipelines to own poetry project

* Update readme

* Delete

* Add ci_code_validator

* Use pipx to install gha deps

* Fix'

* Ensure every thing is running

* Automated Commit - Formatting Changes

* Gitignore miss

* Add pipx installer

* Get local pipx dependencies

* Fix paths

* Install pipx

* ceremonial source-faker change

* Add installation step for ci_code_validator

* Add comment

* remove ci_code_validator

* Address code review comments

* add pipx install to acceptance-test-docker.sh

* Run formater

* Revert "ceremonial source-faker change"

This reverts commit 26884cd0db.

* gitignore lecacy pipeline report path

* update poetry.lock

* skip upload if logs do not exist

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
2023-07-26 15:49:59 +00:00
Alexandre Girard
97a353d5c5 Run mypy on airbyte-cdk as part of the build pipeline and fix typing issues in the file-based module (#27790)
* Try running only on modified files

* make a change

* return something with the wrong type

* Revert "return something with the wrong type"

This reverts commit 23b828371e.

* fix typing in file-based

* format

* Mypy

* fix

* leave as Mapping

* Revert "leave as Mapping"

This reverts commit 908f063f70.

* Use Dict

* update

* move dict()

* Revert "move dict()"

This reverts commit fa347a8236.

* Revert "Revert "move dict()""

This reverts commit c9237df2e4.

* Revert "Revert "Revert "move dict()"""

This reverts commit 5ac1616414.

* use Mapping

* point to config file

* comment

* strict = False

* remove --

* Revert "comment"

This reverts commit 6000814a82.

* install types

* install types in same command as mypy runs

* non-interactive

* freeze version

* pydantic plugin

* plugins

* update

* ignore missing import

* Revert "ignore missing import"

This reverts commit 1da7930fb7.

* Install pydantic instead

* fix

* this passes locally

* strict = true

* format

* explicitly import models

* Update

* remove old mypy.ini config

* temporarily disable mypy

* format

* any

* format

* fix tests

* format

* Automated Commit - Formatting Changes

* Revert "temporarily disable mypy"

This reverts commit eb8470fa3f.

* implicit reexport

* update test

* fix mypy

* Automated Commit - Formatting Changes

* fix some errors in tests

* more type fixes

* more fixes

* more

* .

* done with tests

* fix last files

* format

* Update gradle

* change source-stripe

* only run mypy on cdk

* remove strict

* Add more rules

* update

* ignore missing imports

* cast to string

* Allow untyped decorator

* reset to master

* move to the cdk

* derp

* move explicit imports around

* Automated Commit - Formatting Changes

* Revert "move explicit imports around"

This reverts commit 56e306b72f.

* move explicit imports around

* Upgrade mypy version

* point to config file

* Update readme

* Ignore errors in the models module

* Automated Commit - Formatting Changes

* move check to gradle build

* Any

* try checking out master too

* Revert "try checking out master too"

This reverts commit 8a8f3e373c.

* fetch master

* install mypy

* try without origin

* fetch from the script

* checkout master

* ls the branches

* remotes/origin/master

* remove some cruft

* comment

* remove pydantic types

* unpin mypy

* fetch from the script

* Update connectors base too

* modify a non-cdk file to confirm it doesn't get checked by mypy

* run mypy after generateComponentManifestClassFiles

* run from the venv

* pass files as arguments

* update

* fix when running without args

* with subdir

* path

* try without /

* ./

* remove filter

* try resetting

* Revert "try resetting"

This reverts commit 3a54c424de.

* exclude autogen file

* do not use the github action

* works locally

* remove extra fetch

* run on connectors base

* try bad  typing

* Revert "try bad  typing"

This reverts commit 33b512a3e4.

* reset stripe

* Revert "reset stripe"

This reverts commit 28f23fc6dd.

* Revert "Revert "reset stripe""

This reverts commit 5bf5dee371.

* missing return type

* do not ignore the autogen file

* remove extra installs

* run from venv

* Only check files modified on current branch

* Revert "Only check files modified on current branch"

This reverts commit b4b728e654.

* use merge-base

* Revert "use merge-base"

This reverts commit 3136670cbf.

* try with updated mypy

* bump

* run other steps after mypy

* reset task ordering

* run mypy though

* looser config

* tests pass

* fix mypy issues

* type: ignore

* optional

* this is always a bool

* ignore

* fix typing issues

* remove ignore

* remove mapping

* Automated Commit - Formatting Changes

* Revert "remove ignore"

This reverts commit 9ffeeb6cb1.

* update config

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
2023-07-13 16:55:48 -07:00
Alexandre Girard
f47219037e dont publish to test pypi (#23709) 2023-03-09 13:40:22 -08:00
Alexandre Girard
677fa9756d Small fixes to support python3.8 (#23653)
* Use ** instead of |

* Using typing type

* Using typing type

* Using typing type

* Revert "Merge branch 'master' into alex/support_3.8"

This reverts commit 5c7581518e, reversing
changes made to 5cb47f8c70.

* Revert "Merge branch 'master' into alex/support_3.8"

This reverts commit 5cb47f8c70, reversing
changes made to 4058fce754.

* Revert "Merge branch 'master' into alex/support_3.8"

This reverts commit e1d109905a, reversing
changes made to da881ef0d7.

* reset changes

* undo borked publish

* downgrade bumpversion.cfg and Dockerfile too

* explicitely support >=3.8

* update readme
2023-03-01 19:14:27 -08:00
Maxime Carbonneau-Leclerc
ee1ae2d4a2 Isolate our build so that the feedback loop is faster (#23411)
* Isolate our build so that the feedback loop is faster

* Improve gradle build condition

* Update documentation

* Adding auto-commit of formatting changes (as https://github.com/airbytehq/airbyte/pull/23415)
2023-03-01 10:05:50 -05:00
Catherine Noll
7da6a3bb77 Run CATs with local CDK (#23084)
Scripts to 

* Run CATs against the local CDK for one connector

* Run CATs against the local CDK for multiple connectors

* Create a connecter image with the local CDK

---------

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2023-02-24 16:13:42 -05:00
Peter Hu
4fbb567b19 cdk uses airbyte protocol from pypi (#22518)
* airbyte_protocol as dependency for cdk

download from pypi instead of using local directory in monorepo

* remove generate-protocol-files.sh

* protocol_models => models

airbyte_protocol.models more closely reflects java package name

* use published pypi package

* fix imports

* context because someone will wonder what happened here

* run formatter

---------

Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
Co-authored-by: Sherif Nada <snadalive@gmail.com>
Co-authored-by: cgardens <charles@airbyte.io>
2023-02-13 12:45:45 -06:00
Maxime Carbonneau-Leclerc
f69a128a9c [ISSUE #19981] testing version bump (#21106)
* [ISSUE #19981] testing version bump

* [ISSUE #19981] debugging part-to-bump

* [ISSUE #19981] further debugging on part-to-bump

* [ISSUE #19981] further debugging on part-to-bump yet again

* [ISSUE #19981] further debugging on part-to-bump yet again and again

* [ISSUE #19981] improving on first successful attempt

* [ISSUE #19981] adding changelog and concurrency

* [ISSUE #19981] update version in .bumpversion.cfg

* [ISSUE #19981] testing Slack notification with working channel

* [ISSUE #19981] documentation and clean up before PR

* [ISSUE #19981] make changelog-message optional

* [ISSUE #19981] align version with new release

* [ISSUE #19981] code review
2023-01-11 08:28:07 -05:00
Maxime Carbonneau-Leclerc
7685f5a3db [local airbyte cdk tests] update README.md to fix gradle command and support more docker images (#20821) 2023-01-03 10:18:23 -05:00
Joe Reuter
1a237b7a86 change readme to use python command instead of pytest to run unit tests (#20867) 2023-01-02 17:28:19 +01:00
Maxime Carbonneau-Leclerc
1fdaa1d074 [ISSUE #20044] generate pydantic models from handwritten schema (#20475)
* handwritten low code manifest example components

* add MinMaxDatetime to jsonschema

* add a basic gradle command to generate manifest components

* Add auth components to handwritten component schema

- ApiKeyAuthenticator
- BasicHttpAuthenticator
- BearerAuthenticator
- DeclarativeOauth2Authenticator
- NoAuth

* Respect optional properties in DeclarativeOauth2Authenticator

* Fix `Dict[str, Any]` mapping in auth components

* add default error handler composite error handler and http response filter components

* [low code component schema] adding backoff strategies to schema

* [low code component schema] fix float types

* [low code component schema] add RecordFilter

* Remove `config` from auth components

* [low code component schema] add Interpolation (with pending question on 'type' not being defined)

* Add CartesianProductStreamSlicer & DatetimeStreamSlicer

* Add ListStreamSlicer, and fix nesting of DatetimeStreamSlicer

* [low code component schema] add InterpolatedRequestOptionsProvider

* Add slicer components, and fix a couple of components after reviewing output

* [low code component schema] adding transformations and adding type to interpolators

* adding spec and a few small tweaks

* Add DefaultSchemaLoader

* [low code component schema] attempt on custom class

* Add descriptions for auth components

* add RequestOption

* remove interpolated objects from the schema in favor of strings only

* a few schema fixes and adding some custom pagination and stream slicer

* [low code component schema] fix CustomBackoffStrategy

* Add CustomRecordExtractor

* add some description and add additional properties

* insert a transformer to hydrate default manifest components and perform validation against the handwritten schema

* [low code component schema] validating existing schemas

* [low code component schema] clean validation script

* add manifest transformer tests and a few tweaks to the schema

* Revert "[low code component schema] clean validation script"

This reverts commit 2408f41cf6.

* Revert "[low code component schema] validating existing schemas"

This reverts commit 9d39977815.

* [low code component schema] integrate validation script to gradle

* [low code component schema] updating validation script permissions

* remove a few model gen spike files and clean up comments

* default types should take parent type into account and a few schema changes

* [ISSUE #20044] generate pydantic models from handwritten schema

* [ISSUE #20044] code review

* [ISSUE #20044] re-generating declarative component schema files

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
2022-12-20 16:53:35 -06:00
Ella Rohm-Ensing
4bef54d1b9 Add README info about using your local CDK locally and in docker (#20082)
* Add README info about installing the local CDK locally and in Docker

* Add '|| true' to validate_dockerignore()

* Update airbyte-cdk/python/README.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-12-05 17:52:35 -05:00
Tanvesh Takawale
da5aed8a31 CDK: fixed document links regarding airbyte specification (#18169) 2022-10-19 09:50:01 -07:00
Charles
22b727c0ea Update Airbyte Protocol Docs (#13709) 2022-06-14 15:04:38 -07:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Arthur Galuza
51d406d6f2 🎉 Source Github: Add MultipleTokenAuthenticator (#5223)
* Add multiple token authenticator

* Add MultipleTokenAuthenticator

* Upd docs

* Refactor

* Upd docs

* Fix merge typo

* Upd multiple token support: switch to list of tokens

* Upd multiple token support: refactoring

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Cleanup, switch to MultipleTokenAuthenticator completely

* Upd changelog

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-08-19 08:41:14 +03:00
LiRen Tu
2906ec287a CI: Add action to check broken doc links (#5254)
* Add action to check broken doc links

* Ignore localhost

* Update config

* Fix broken links

* Use quiet mode

* Ignore PR link

* Fix more broken links

* Fix more broken links

* Fix more broken links

* Verify pattern

* Fix more broken links

* Separate full and pr check

* Update pattern

* Test invalid link

* Remove invalid link
2021-08-07 14:28:02 -07:00
Sherif A. Nada
9d8b1731b0 Remove outdated note about CDK not supporting destinations (#4995) 2021-07-26 13:52:42 -07:00
Dmytro
66d96967df 🎉 Containerize connector code generator (#3763) 2021-06-08 12:25:27 +03:00
Sherif A. Nada
fc45e8f165 Fix typo in CDK publish command format 2021-05-27 14:30:49 -07:00
Sherif A. Nada
6beb4eb7e7 add npm install before all npm run generates' (#3442) 2021-05-17 09:39:37 -07:00
Sherif A. Nada
184dab77eb CDK: overhaul directory structure (#3295) 2021-05-09 15:27:38 -07:00
Abhi Vaidyanatha
4a397d2524 Rewrite and break out CDK Python overview to be more concise and digestible (#3291)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-05-07 20:30:21 -07:00
Abhi Vaidyanatha
05bdb368a0 Reorganize CDK documentation directory structure to be less nested (#3270)
* Move CDK docs down a level, consolidate directories.

* Change name of tutorial

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-05-07 12:50:33 -07:00
Davin Chia
b5c2329748 CDK: Add OAuth and Streams/Core tests. (#3228)
Completely test the OAuth module and streams/core.py file.

Add pytest-mock as well as a convenience wrapper to run the type checker and the test together when developing.

Also configure pytest to pick up loggers at the INFO level and above, and pick up standard out for easier debugging.

Some of the script bits are unpolished - will improve them as we go along.
2021-05-06 15:52:23 +08:00
Davin Chia
fa33c9c126 Initial CDK Test set up and one test file. (#3207)
Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-05-04 14:27:58 -07:00
Davin Chia
72e7fe35a3 Run MyPy on CDK/base-python and fix issues. (#3175) 2021-05-04 11:02:53 +08:00
Sherif A. Nada
dd45537239 use extras for CDK development and remove python 3.10 support (#3200) 2021-05-03 18:34:18 -07:00
Davin Chia
ced243881b Add CDK publishing setup. (#3156)
Allow us to publish CDK versions on PyPi via Github workflows.

This lets us:
- Version CDK moving forward
- Publish via Github actions to both PyPi and test PyPi for test releases.

Note, this will not work on this branch as Github only detects new workflows after they are checked into master, so I was forced to develop and test this from a fork. Browse this PR or this PR to see this command in action.

Added publishing instructions to the Release Document.
2021-05-03 08:38:47 +08:00