1
0
mirror of synced 2025-12-22 19:38:29 -05:00
Commit Graph

105 Commits

Author SHA1 Message Date
Joe Reuter
f29234a2db airbyte-lib: Refactor connectors (#34552) 2024-01-30 03:22:13 -06:00
Augustin
0b33caecda Revert "[skip ci] formatting: add missing license headers (#33250)" (#33289) 2023-12-11 11:38:37 +01:00
Augustin
60c1cc01ad [skip ci] formatting: add missing license headers (#33250) 2023-12-11 10:15:18 +01:00
Augustin
368ba78b64 🧹doc: update connectors README and remove acceptance-test-docker.sh (#32209) 2023-11-06 12:36:07 -06:00
Marius Posta
7ae97175a6 gradle: fix repo wide behaviour (#30607) 2023-09-28 05:01:13 -07:00
Ben Church
5c56ac1d84 Airbyte-ci: Remove gradle task connectorAcceptanceTest (#30326) 2023-09-19 15:16:37 -05:00
Ben Church
40781313da Update Internal Fields: update ql levels to better resemble previous high strictness (#29450) 2023-08-15 18:03:10 -05:00
Augustin
00d9462216 cat/connectors-ci: replace docker runner with dagger runner in CAT (#28000) 2023-08-11 17:58:48 +02:00
Ben Church
2f7deaee02 [skip ci] Metadata: Remove leading underscore (#29024)
* DNC

* Add test models

* Add model test

* Remove underscore from metadata files

* Regenerate models

* Add test to check for key transformation

* Allow additional fields on metadata

* Delete transform
2023-08-03 10:56:13 -07:00
Ben Church
e9490e3fb6 Connector Levels: Add new internal metadata fields (#28904)
* Add airbyte internal

* Add tests

* First pass

* Set destinations to same levels as sources

* Best guess at missing statuses

* Best guess at _ql

* Add separate enum class

* Fix support level name

* Update templates

* Add one more test
2023-08-01 18:08:33 -05:00
Ben Church
1dabc6208e Metadata: add tags field (#26320)
* Add optional tags field

* Remove duplicate icons

* Add programming tags to all

* Update docs

* supportUrl -> documentationUrl

* Ensure one language tag is applied

* Add keyvalue check

* rebase and fix tests

* Format

* Add cache buster

* Improve test

* Automated Commit - Formatting Changes

* Update error

* Fix missing tags

* Fix scaffold

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-05-26 16:13:09 -07:00
Ben Church
248bbf94c1 HACKDAY: Icon CDN (#26158)
* Move icons to connector folder

* Delete old icons

* Update upload logic

* Add icon url to definitions

* Update registry model

* Populate cdn url

* DNC butcher the pipeline

* Low hanging fruit fixes

* Fix bucket name

* Merge old and new approaches

* Fix metadata upload step

* Format

* Fix test
2023-05-24 17:25:41 -07:00
Augustin
7310494846 qa-checks: check metadata version matches dockerfile version (#25661) 2023-05-04 16:08:19 -07:00
Ben Church
5563179782 Dagster: rename catalog to registry (#25254)
* rename catalog to registry in metadata service

* rename catalog to registry in metadata files

* Run generate models

* Fix missed renames

* Add github personal access token

* Run black

* Automated Change

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-04-18 22:15:11 +02:00
Augustin
ebc907cdf7 create metadata files for all connectors (#24964) 2023-04-13 07:45:04 +02:00
Catherine Noll
7da6a3bb77 Run CATs with local CDK (#23084)
Scripts to 

* Run CATs against the local CDK for one connector

* Run CATs against the local CDK for multiple connectors

* Create a connecter image with the local CDK

---------

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2023-02-24 16:13:42 -05:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594)
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
2023-02-08 13:01:16 -08:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846)
Rename source acceptance test to connector acceptance test
2023-02-02 11:38:19 +01:00
Serhii Lazebnyi
b526aace6a 🐛Source Looker: Fix schema transformation issue (#20182)
* Fix schema transformation issue

* Updated PR number

* Added unittest

* Add a test case that uses the recursion

* Unhide from cloud

* Bumed seed version
2023-01-06 18:19:56 +01:00
Pedro S. Lopez
938436bcc9 update connector specs and definitions with new .com documentation urls (#17585)
* update definitions with new .com docs urls

* update docs urls in specs

* update generators

* regenerate scaffold connectors

* remove unrelated changes

* update more urls

* update specs

* fix tests

* run `:airbyte-config:specs:generateSeedConnectorSpecs` to fix formatting

* revert docs changes to make pr more reviewable

* revert generator readme changes to make more reviewable

* fix mysql strict encrypt expected spec

* fix postgres expected spec
2022-10-11 11:04:23 -04:00
Evan Tahler
49cb3360de Remove redundant title labels from connector specs (#17544)
* Remove redundant title labels from connector specs

* Manually update specs

* add env variable

* Remove debugging log
2022-10-05 12:58:38 -07:00
Marcos Marx
dca2256a7c Bump 2022 license version (#13233)
* Bump year in license short to 2022

* remove protocol from cdk
2022-05-26 15:00:42 -03:00
Harshith Mullapudi
657839195f Harshith/testpr 11443 (#12867)
* add unit tests

* add streams to tests

* fix: version already exists

* chore: update seed file

Co-authored-by: itaseski <ivica.taseski@seavus.com>
Co-authored-by: itaseski <ivica.taseski94@gmail.com>
2022-05-16 19:25:12 +05:30
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Maksym Pavlenok
bbd13802d8 🐛 Fix Python checker configs and Connector Base workflow (#10505) 2022-02-22 19:58:55 +02:00
VitaliiMaltsev
e30d8348b2 Change JsonSchemaPrimitive to a class (#9913)
* fix for jdk 17

* add JsonSchemaType class

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

* fix Oracle tests

* fix Redshift tests

* fix Redshift tests

* fix checkstyle

* fix MSSQL tests

* fix cockroachdb tests

* fix checkstyle

* fix checkstyle

* replace star imports

* replace star imports

* replace star imports

* update JsonSchemaType | fixed checkstyle

* Remove unused variables in test

* Fix imports

* Expand imports

* Fix more imports

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
2022-02-14 02:12:37 -08:00
Maksym Pavlenok
27b5ba3386 🎉 Source Looker: Migrate to native CDK (#9609) 2022-01-31 11:48:20 +02:00
ycherniaiev
e17fa5b040 Update fields in source-connectors specifications: kustomer-singer, lemlist, linnworks, looker, marketo, mixpanel (#8578) 2021-12-25 13:56:20 +02:00
Christophe Duong
86ca36c5c0 Format code (#7978) 2021-11-15 14:51:10 +01:00
Yevhenii
62832cdd85 🎉 Source Looker: Migrate Looker source to CDK structure, add SAT testing (#7284)
* Migrate Looker source to CDK structure, add SAT testing
2021-10-27 17:46:55 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
LiRen Tu
8954a99290 Bump source looker connector version (#4350)
* Bump looker connector version

* Update changelog

* Format code
2021-06-25 11:23:16 -07:00
Michael Irvine
842f10bd83 🎉 Source Looker: add run-look endpoint (#3911)
* looker run_looks method works with normalization

* update catalogs

* validate that look(s) entered actually exist at config time

* field type lookup works, though it is a bit messy

* don't re-run explore query if we have already run it

* add more type mappings

* add some docstrings and comments

* add title for ease of use

* Revert "update catalogs"

This reverts commit 7ecaaa4d0d.

* fix configuration file formatting

* fix some issues from code review and reorder methods

* update docs to include run_look endpoint
2021-06-25 08:31:56 -07:00
Janardhanpoola
3d44999e1c 🎉 Source Looker: enable supporting of customer-hosted (self-hosted) instances
* feature to support Looker self-hosted instance domain(#3393)

* changing self-hosted instances to any url(#3393)

* fixing build failures

* regexp for any URL

* fixing build failures

* fixing regexp pattern and adding unittests

* writing unitests using pytest

* pytest-fixture for unit-tests

* Source Looker #3587 - remove domain spec field pattern

* Source Looker #3587 - fix description of domain spec field

* Source Looker #3393 - bum version

Co-authored-by: root <root@DESKTOP-SS2MC07.localdomain>
Co-authored-by: Vitalii Vdovenko <vitalii.vdovenko@zazmic.com>
2021-06-22 16:06:36 +03:00
Jared Rhizor
b4793b2510 add AIRBYTE_ENTRYPOINT for kubernetes support (#3973)
* add AIRBYTE_ENTRYPOINT for kubernetes support

* bump versions

* bump version in seed

* Update generic template

* keep scaffold sources at 0.1.0

* add missing newline

* handle python base versions correctly

* re-bump mysql and postgres sources

* re-bump snowflake destination

* add skip tests option

* switch to running tests

* reverse conditional to make it safer

* fix publish to include the test running

* fix iterable version

* fix file generation

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-06-09 13:01:45 -07:00
Dmytro
a1035e007b 📚 Update all Python connectors READMEs with minimum required version (#3760) 2021-06-02 12:35:29 +03:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Davin Chia
f660b0a946 Add template generation for Santa aka CDK. (#3034)
Template generation for new Source using the Santa CDK - provide basic scaffolding for someone implementing a new source.

General approach is to buff up comments in the original SDK, and add TODOs with secondary comments in the generated stub methods, as well as links to existing examples (e.g. Stripe or ExchangeRate api) users can look at.

Checked in and added tests for the generated modules.
2021-04-25 18:02:33 +08:00
Charles
f445fdb5b2 match styling for spotlessApply and format (#3017)
* as a java developer I want to be able to run spotlessApply without changing styles in python code
2021-04-23 09:21:41 -07:00
Sherif A. Nada
2d2d0173d6 Update codegen template READMEs (#2873) 2021-04-13 11:41:52 -07:00
Jared Rhizor
0609011e05 fix connector base versioning (#2726)
* fix connector base versioning

* only run integration tests in non-bases directories

* upgrade datamodel-code-generator which coincidentally has already fixed this issue with generating List of Lists

* bump forgotten hubspot version

* fix build
2021-04-02 22:46:06 -07:00
Christophe Duong
bc1547f185 Fix integration tests of connectors related to destination_sync_mode and sync_mode being required (#2688) 2021-04-01 09:58:38 +02:00
Sherif A. Nada
02a5b9b8a8 depend on published base singer/python images in existing connectors (#2469) 2021-03-15 18:21:05 -07:00
Sherif A. Nada
fe22594bb9 update all python Readmes to describe dev process without gradle (#2442) 2021-03-15 18:05:38 -07:00
Sherif A. Nada
2139a9656b Cleanup Python packages (#2440) 2021-03-15 01:24:21 -07:00
Christophe Duong
070575ffdf Protocol allows future / unknown properties (#2238)
* Allow new extra properties in validation
* Create migration script to upgrade all connectors versions
* Bumpversion of all connectors
2021-03-09 13:36:36 +01:00
Sherif A. Nada
ac0bcf7d01 Add comment at the top of all requirements.txt files clarifying they should not be used for dependencies (#2366) 2021-03-08 19:01:47 -08:00
Jared Rhizor
30e2e98c49 fix formatting issues (#2045)
* fix formatting issues

* remove space

* unit tests -> format anchor
2021-02-11 13:02:21 -08:00
vitaliizazmic
1018c69c4a Looker source #1837 - CI tests failing (#1857)
Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-01-27 19:21:47 -08:00
Sherif Nada
edfb160468 hotfix formatting 2021-01-21 11:36:19 -08:00