jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-05 21:02:13 -05:00
Code Issues Packages Projects Releases Wiki Activity
5,184 Commits 5,162 Branches 518 Tags
ea92a24da35ff9a32288bd96fc00a82ff7cad481
Commit Graph

60 Commits

Author SHA1 Message Date
Greg Solovyev
e07f12dc41 Update incremental sync tutorial to match recent changes in build-a-connector tutorial (#11126) 2022-03-16 09:30:27 -07:00
Greg Solovyev
d84e852ac6 Greg/source connector guide (#11082) 
Original API used in the example is no longer free, so while following the tutorial I found a different API to use in the tutorial and updated the example code to match the new API. Also made a few small changes.

Update Build-a-connector tutorial
 * Update screenshots and instructions to match current UI
 * Update code examples to use Polygon.io API
 * Add links to language-specific guides
 * Consistently use Python (uppercase P) when referring to the language
   and python (lowercase) p in code samples
 * Consistently do not use "." in lists
 * Add an image of Airbyte startup banner
 * Add a note for M1 macs
 * Remove unused images
 * Make Dockerfile consistent with the tutorial
 2022-03-14 13:33:10 -07:00
Ramon Vermeulen
f0b0cbe191 Docs: incorrect line number (#11112) 
incorrect line number in documentation
 2022-03-14 17:14:54 -03:00
LiRen Tu
d57408269b 📝 Add connector-name examples in java profiling doc (#11006) 2022-03-09 14:38:28 -08:00
LiRen Tu
80a2b642f9 📝 Document java connector memory profiling (#10983) 
* Add doc for java connector memory profiling

* Add doc to summary

* Update doc

* Update doc

* Update doc
 2022-03-09 05:58:33 -08:00
Maksym Pavlenok
bbd13802d8 🐛 Fix Python checker configs and Connector Base workflow (#10505) 2022-02-22 19:58:55 +02:00
Maksym Pavlenok
61b0e9e196 🎉 Single py checker (#10246) 2022-02-21 14:01:42 +02:00
Eugene Kulak
d173ce741f CDK: add support for streams with state attribute (#9746) 
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2022-02-17 00:20:33 +04:00
Sherif A. Nada
032b06d2e2 Add section to UX handbook about pointing to page anchors 2022-01-17 16:53:35 -08:00
Anuj Gupta
ecfc9e1cc5 Correcting base URL for the example API (#9245) 2022-01-13 07:25:13 -03:00
Augustin
16b14c51e9 Document closure usage in transform function (#9193) 2021-12-29 18:27:06 +01:00
Edward Gao
d0f9aa8907 Update "Building a connector the hard way" doc (#9008) 2021-12-21 10:50:59 -08:00
Maksym Pavlenok
d3b0c998b8 🎉 CI secrets: full migration to GSM (#8561) 
* add python packages for CI scripts

* add tox config for all subpackages

* draft version

* init venv for scripts

* fix venv

* remove used comments

* fix run test

* change base folder

* update secret format

* update docs

* remove an unused file

* remove github secrets logic fully

* fix base_folder balue

* add functions desc
 2021-12-15 15:08:46 +02:00
Menno Hamburg
03e5160569 🎉 New CDK: Dotnet/.NET/C# implementation + free exchange rates source connector (#7945) 
* Added cdk source

* Added free exchange rates api as a dotnet example

* Corrected the use of JsonDocument, properly enabled utf-8

* Fixed error catching

* Added cli, documentation and nuget improvements

* Small improvements where needed

* Removed obsolete files

* Update README.md

* Added documentation and reference

* Update README.md

* Corrected dotnet

* Update README.md
 2021-12-13 20:24:09 -03:00
Jared Rhizor
25674fc306 upgrade to Gradle 7.3.1 / Java 17 (#7964) 
* upgrade gradle

* upgrade to Java 17 (and fix a few of the node versioning misses)

* oops

* try to run a different format version

* fix spotless by upgrading / reformatting some files

* fix ci settings

* upgrade mockito to avoid other errors

* undo bad format

* fix "incorrect" sql comments

* fmt

* add debug flag

* remove

* bump

* bump jooq to a version that has a java 17 dist

* fix

* remove logs

* oops

* revert jooq upgrade

* fix

* set up java for connector test

* fix yaml

* generate std source tests

* fail zombie job attempts and add failure reason (#8709)

* fail zombie job attempts and add failure reason

* remove failure reason

* bump gcp dependencies to pick up grpc update (#8713)

* Bump Airbyte version from 0.33.9-alpha to 0.33.10-alpha (#8714)

Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>

* Change CDK "Caching" header to "nested streams & caching"

* Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable (#8524)

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* move S3Config into destination-s3; update dependencies accordingly (#8562)

Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Iryna Grankova <87977540+igrankova@users.noreply.github.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2021-12-10 16:57:54 -08:00
Sherif A. Nada
ba09bbfe4f Change CDK "Caching" header to "nested streams & caching" 2021-12-10 15:20:54 -08:00
Vadym
f2f224fe08 Update SAT for validate_data_points Test Basic Read option. (#8342) 2021-11-30 21:06:14 +02:00
Sherif A. Nada
ad3b7be4c0 Document how to order spec fields in the UI 2021-11-18 19:14:46 -08:00
Lake Mossman
34565c6767 Add better command for generating seed specs to readme 2021-11-18 10:44:18 -08:00
Abhi Vaidyanatha
2aea2e0816 Fix a bunch of broken links. (#8083) 2021-11-18 09:55:20 -08:00
Sherif A. Nada
e15c2a925f Fix UX handbook assets (#8020) 2021-11-16 11:02:07 -08:00
Maksym Pavlenok
c0cc288f97 🎉 CI secrets: migration to GSM (#7528) 
* fix 404 responses for the ticket_comments stream

* add unit test

* add unit test

* add oauth2 access token

* Update airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py

Co-authored-by: George Claireaux <george@claireaux.co.uk>

* switching among auth methods

* update spec file

* update CI secrets logic

* update CI secrets logic

* remove debug data

* add a debug message

* fix json convertation

* fix json convertation

* support one secret by several connectors

* Update tools/bin/ci_credentials.sh

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update tools/bin/ci_credentials.sh

Co-authored-by: LiRen Tu <tuliren@gmail.com>

* update function names

* update docs

* reset failed changes

* update json set value

* update secrets json key

* Update tools/bin/ci_credentials.sh


* add JSON validation

* Update docs/connector-development/README.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* update doc

* Update README.md

* rename the function write_standard_creds

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: LiRen Tu <tuliren@gmail.com>
 2021-11-10 10:49:44 +02:00
Aaditya S
6b0e9da4bf 📚 Fix typo in the cdk documentation (#7784) 2021-11-09 07:48:22 -08:00
Lake Mossman
09fce77107 REDO: Generate seed connector specs on build (#7613) 
* add specs module with logic to fetch specs on build

* format + build and add gradle dependency for new script

* check seed file for existing specs + refactor

* add tests + a bit more refactoring

* run gw format

* update yaml config persistence to merge specs into definitions

* add comment

* add dep

* add tests for GcsBucketSpecFetcher

* get rid of static block + format

* DRY up parse call

* add GCS details to comment

* formatting + fix test

* update comment

* do not format seed specs files

* change signature of run to allow cloud to reuse this script

* run gw format

* revert commits that change signature of run

* fix comment typo

Co-authored-by: Davin Chia <davinchia@gmail.com>

* rename enum to be distinct from the enum in cloud

* add missing dependencies between modules

* add readme for seed connector spec generator

* reword

* reference readme in comment

* ignore 'spec' field in newFields logic

* rearrange dependencies so that CONNECTORS_BASE build does not depend on SeedConnectorSpecGenerator

* run format

* add some more helpful info to the GCS fetch failure message

* add more info

* get rid of unnecessary static block

* Fix publishing docs (#7589)

* Fix publishing docs

* Reorder steps and add a comment about rebuilding the platform

* Update README.md

Co-authored-by: Lake Mossman <lake@airbyte.io>

* add dependency and rebuild

* update PR template with seed connector generation steps

* revert formatting changes to PR template

* Update build.gradle

* Remove unnecessary dep

Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
 2021-11-04 17:06:16 -07:00
Sherif Nada
25c45c50f7 move handbook up one level 2021-11-01 20:14:01 -07:00
Sherif A. Nada
6b2eb71274 Add connector development UX handbook (#7553) 2021-11-01 20:07:21 -07:00
vitaliizazmic
1d922ca5fc 🐛 Source Facebook Marketing: SAT ignored fields in TestFullRefresh.test_sequential_reads 
* Source Facebook Marketing #5190 - estimate cost_per_estimated_ad_recallers for AdsInsights streams if not presented in records

* Source Facebook Marketing #5190 - add ignored fields to full refresh test

* Source Facebook Marketing #5190 - annotations

* Source Facebook Marketing #5190 - reformat

* SAT #5190 - delete remove_ignored_fields

* Source Facebook Marketing #5190 - use dpath util for excluding fields

* Facebook marketing #5190 - follow EAFP principle

* Facebook Marketing #5190 - add unit tests.

* Source Facebook Marketing #5190 - fixing according to PR

* Source Facebook Marketing #5190 - support ignored fields by stream

* Source Facebook Marketing #5190 - update docs

* Source Facebook Marketing #5190 - merge conflicts

* Source Facebook Marketing #5190 - bump SAT version

* Source Facebook Marketing #5190 - fixing unit tests

* Source Facebook Marketing #5190 - bump
 2021-11-01 17:45:06 +02:00
Marcos Marx
a82d04f688 Docs: correct youtube video link for CDK Speedrun(#7256) 2021-10-21 17:32:48 -03:00
Jared Rhizor
1affb14827 even more link fixes (#7148) 
* even more link fixes

* add remaining exchangerates items
 2021-10-20 12:56:55 -07:00
Arthur Galuza
f0debaad32 🎉 Document stream caching (#6846) 
* Upd docs: add caching section

* Upd docs

* Update docs/connector-development/cdk-python/http-streams.md

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Remove link to details

* Update docs/connector-development/cdk-python/http-streams.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/connector-development/cdk-python/http-streams.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Add demo snippet

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-10-14 16:35:29 +03:00
Abhi Vaidyanatha
ae32ecbb27 GitBook: [master] 186 pages and 77 assets modified 2021-10-08 21:17:47 +00:00
Abhi Vaidyanatha
f52b67dec9 Rich embed the CDK Speedrun video. (#6915) 
* Rich embed the CDK Speedrun video.

* Fix attempt 1.'

* Remove link.
 2021-10-08 14:12:17 -07:00
Abhi Vaidyanatha
ca61cd1dfd Add CDK video to CDK Speedrun doc. (#6914) 2021-10-08 14:00:59 -07:00
Sherif A. Nada
c5e74cf335 Make spec storybook more prominent in docs 2021-09-29 00:29:29 -07:00
Dmytro
1c5ac5b5ea 🏗️ Python CDK: add schema transformer class (#6139) 
* Python CDK: add schema transformer class
 2021-09-27 13:07:17 +03:00
Marcos Marx
b4cca602ca Source Google Search Console: correct spec.json file (#6222) 
* correct spec.json file

* bump version

* bump version

* update doc for connector spec ref

Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>
 2021-09-23 02:06:31 -03:00
Abhi Vaidyanatha
fa0bd551ef Switch out old Speedrun Tutorial. (#6236) 
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-09-17 16:30:54 -07:00
Harsha Teja Kanna
84b3fbd8db 🎉 Added optional platform flag for build image script (#6000) 2021-09-14 23:37:07 -07:00
Abhi Vaidyanatha
e7413de0dc Add a variant of the CDK speedrun that uses the PokeAPI (#5780) 
* Add a variant of the CDK speedrun that uses the PokeAPI

* Make a lot of clarifications to improve the learning experience.

* Gradle format

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-09-08 14:56:13 -07:00
Abhi Vaidyanatha
8fcaba4fab Fix CDK speedrun stream stub (#5778) 
* Fix CDK speedrun stream stub

* Add start_date back

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-09-08 14:44:34 -07:00
Sherif A. Nada
74d6c07004 📚 Link to how to preview connector specifications in the UI (#5894) 2021-09-07 17:15:13 -07:00
Dmytro
8d2cd1e798 🎉 Tool for generation catalog schema from OpenAPI definition file (#5734) 
* Add tool for generating catalog json schema from openapi definition file
 2021-09-01 21:35:43 +03:00
Dmytro
47172577d3 SAT: check records to comply jsonschema format field (#5661) 
* SAT: check records to comply jsonschema format field
 2021-09-01 13:12:31 +03:00
Abhi Vaidyanatha
516d276893 Add Faros CDK information to CDK docs (#5686) 
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-08-27 18:36:37 -07:00
Jenny Brown
6a3b594574 Clarified wording that confused a new connector developer (#5558) 2021-08-25 16:09:49 -05:00
George Claireaux
6106dc3a5b made ordering requirement more clear for stream_slices (#5627) 2021-08-25 18:40:58 +01:00
Yaroslav Dudar
8ddce6f355 🎉 Python CDK: Allow to ignore http status errors and override retry parameters (#5363) 
added auto_fail_on_errors, max_retries, retry_factor properties to python cdk
 2021-08-25 10:31:24 +03:00
Arthur Galuza
51d406d6f2 🎉 Source Github: Add MultipleTokenAuthenticator (#5223) 
* Add multiple token authenticator

* Add MultipleTokenAuthenticator

* Upd docs

* Refactor

* Upd docs

* Fix merge typo

* Upd multiple token support: switch to list of tokens

* Upd multiple token support: refactoring

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Cleanup, switch to MultipleTokenAuthenticator completely

* Upd changelog

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-08-19 08:41:14 +03:00
LiRen Tu
2906ec287a CI: Add action to check broken doc links (#5254) 
* Add action to check broken doc links

* Ignore localhost

* Update config

* Fix broken links

* Use quiet mode

* Ignore PR link

* Fix more broken links

* Fix more broken links

* Fix more broken links

* Verify pattern

* Fix more broken links

* Separate full and pr check

* Update pattern

* Test invalid link

* Remove invalid link
 2021-08-07 14:28:02 -07:00
Sherif A. Nada
f7ff9f0024 📚 Document how to obfuscate secrets & accept multiline strings in UI (#5208) 2021-08-05 14:00:52 -07:00