1
0
mirror of synced 2025-12-31 15:03:11 -05:00
Commit Graph

93 Commits

Author SHA1 Message Date
George Claireaux
e7dae06168 Enhance /publish to allow for multiple connectors and parallel execution (#13864)
* start

* revert

* azblob

* bq

* bq denorm

* megapublish baaaabyyyy

* fix needs

* matrix connectors

* auto-bump connector version

* dont failfast and max parallel 5

* multi runno

* minor

* testing matrix agents

* name

* testing multi agents

* tmp fix

* new multi agents

* multi test

* tryy

* let's do this

* magico

* fix

* label test

* couple more connector bumps

* temp

* things

* check this

* lets gooo

* more connectors

* Delete TEMP-testing-command.yml

* auto-bump connector version

* added comment describing bash part

* running single thread

* catch sentry cli

* auto-bump connector version

* destinations

* + snowflake

* saved

* auto-bump connector version

* auto-bump connector version

* java source bumps

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* remove twice-defined methods

* label things

* revert action

* using the new test action

* point at action

* wrong tag on action

* update pool label

* update to use new ec2-github-runner fork

* this needs to be more generic than publisher

* change publish to run on pool

* add comment about runner-pool usage

* updated publish command docs for multi & parallel connector runs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* unbump failed publish versions

* missed dockerfiles

* remove failed docs

* mssql fix

* overhauled the git comment output

* bumping a test connector that should work

* slight order switcheroo

* output connectors properly in first message

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-21 18:40:33 +01:00
George Claireaux
30c5d2d9a7 🎉 New Source Dockerhub (#13931)
* init

* implement working source + tests

* add docs

* add docs

* fix bad comments

* Update airbyte-integrations/connectors/source-dockerhub/acceptance-test-config.yml

* Update airbyte-integrations/connectors/source-dockerhub/Dockerfile

* Update airbyte-integrations/connectors/source-dockerhub/.dockerignore

* Apply suggestions from code review

* Update docs/integrations/sources/dockerhub.md

* Update airbyte-integrations/connectors/source-dockerhub/integration_tests/acceptance.py

Co-authored-by: George Claireaux <george@airbyte.io>

* address @Phlair's feedback

* address @Phlair's feedback

* each record is now a Docker image rather than response page

* format

* fix unit tests

* fix acceptance tests

* add icon, definition and generate seed spec

* add requests to requirements

Co-authored-by: sw-yx <shawnthe1@gmail.com>
2022-06-20 15:13:29 +01:00
Alexander Marquardt
6a78aec852 Update 4-connection-checking.md (#13812)
* Update 4-connection-checking.md

As discussed in https://airbytehq-team.slack.com/archives/C02UQ9MJ4GG/p1655312684160609 - this example is misleading, as it does not actually connect to any API to validate connectivity.

* Update 4-connection-checking.md

Added a link to OneSignal check_connection, and move the "Note" to below the example
2022-06-15 21:41:15 +02:00
Charles
22b727c0ea Update Airbyte Protocol Docs (#13709) 2022-06-14 15:04:38 -07:00
Alexander Marquardt
4ddb4f5b10 Updated to read_records instructions (#13346)
Address concerns raised in https://github.com/airbytehq/airbyte/issues/13342
2022-06-01 11:14:44 +02:00
Alexander Marquardt
4073099a8d Update building-a-python-source.md (#13348)
* Update building-a-python-source.md

Making it more clear when modifications to source_definitions.yaml will be picked up

* Update building-a-python-source.md

Fixed spelling as per @Amruta-Ranade comment
2022-06-01 11:13:08 +02:00
Evan Tahler
d7a1fe8224 Clarify what the /publish command does with the version (#13236)
I found this confusing in the past - do I need to bump the version in 2 places?
2022-05-27 09:30:27 -07:00
Topher Lubaway
3a64c25073 Fixes all broken links and enforces no broken (#13152)
* Fixes all broken links and enforces no broken

* Addresses peter's comments

removes unused summary.md: an artifact of gitbook
2022-05-25 15:59:29 -05:00
Brian Lai
13b7b1b436 improve incremental SAT testing to walk through each stream state instead of just the latest (#12802)
* improve SAT test for incrementals to walk through steam states not just the latest

* convert batch to tuple and pr feedback

* bump version and update changelog

* patch pytest so test doesn't get skipped

* update changelog and dockerfile again
2022-05-18 13:01:19 -07:00
Pedro S. Lopez
957a0be226 SAT: check that connectors emit an AirbyteTraceMessage on failure (#12796)
* add SAT case for airbyte trace message on failure

* add ability to opt-out

* add tests

* add option to docs

* bump version, update changelog

* fix type errors

* update changelog
2022-05-17 16:43:10 -04:00
wjwatkinson
39ab9a20c9 Docs: fixed dependency install command (#12922) 2022-05-17 12:29:24 -04:00
George Claireaux
2af780db3e base-java: Add utility for AirbyteTraceMessage and naively emit on any connector error (#12614)
* added AirbyteLoggedException class

* adding in int runr

* changes

* refactored to AirbyteTracedException to align with python impl.

* added catch for Exceptions that are already AirbyteTracedException

* refactor to static class & catch with UncaughtExceptionHandler

* testing ExceptionHandler

* add tests

* added docs section on using AirbyteTraceMessageUtility

* made AirbyteMessage maker methods more intuitive

* fix spotbugs errors

* format
2022-05-12 11:08:52 +01:00
Pedro S. Lopez
9126ebc4c3 SAT: add threshold_days incremental test option (#12715)
* SAT: add `threshold_days` incremental test option

* fix: support cursor values that are already dates

* dont use constant value

* update docs

* use pendulum for date parsing

* bump cdk version

* use pendulum for duration

* add support for unix timestamps

* bump version, update changelog
2022-05-11 16:05:14 -04:00
swyx
a0b7e4a09b Docs: add reminder to clone repo in CDK speedrun guide (#12761) 2022-05-11 09:08:21 -03:00
Brian Lai
ceec454e30 update python cdk tutorial with updates to exchange rates api (#12427)
* update python cdk tutorial with updates to exchange rates api

* remove extra comment in source.py tutorial sample code
2022-04-29 13:44:04 -07:00
Pedro S. Lopez
0c12ad9136 python generators output spec.yaml files (#12245)
* generators output spec.yaml files

* source-singer generator also uses spec.yaml

* update scaffold

* update python cdk tutorials to use spec.yaml

* add docs updates

* consistency
2022-04-26 07:16:04 -07:00
Sherif A. Nada
91e2e8f536 Add a note about airbyte_hidden spec fields in the connector reference docs (#12259) 2022-04-21 17:25:54 -07:00
Francisco Muniz de Paula Neto
1cd3a6bf82 Docs: Update cdk-speedrun.md (#12102)
Changes Embed of url to use simple link in markdown for Python CDK docs
2022-04-18 18:25:15 -03:00
Pedro S. Lopez
a6a6be2588 allow importing IncrementalMixin from airbyte_cdk.sources.streams (#11858)
* allow importing IncrementalMixin from airbyte_cdk.sources.streams

* format file

* update docs with hint on where to import IncrementalMixin from

* bump cdk version to 0.1.54

* update changelog

* only build python 3.9

* update changelog with python dep bump
2022-04-14 11:42:57 -07:00
Pedro S. Lopez
22ebb53ac0 📝 Update "Hard Way" incremental sync tutorial to pass SAT (#11988)
* update incremental sync tutorial to pass SAT

* add instructions for running tests

* pass `future_state_path`

* Update catalog.md
2022-04-14 07:36:22 -07:00
Jared Rhizor
14dfc1d417 fix parentheses and info/warning/danger formatting in docs (#11981)
* fix parens

* fix formatting for hints

* remove hint from non-docusaurus or gitbook doc
2022-04-13 12:08:34 -07:00
Pedro S. Lopez
24720c657d 📝 CDK tutorial: fix incorrect variable in code example (#11837) 2022-04-12 14:40:31 -07:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Sherif A. Nada
b85c51304f 🎉 Update hardway tutorial & generic source template to use SAT (#11908) 2022-04-11 20:12:06 -07:00
Topher Lubaway
c8337ea3ba Add docs sidebar (#11817)
* Brings pairity for doc checks from cloud

* WIP Sidebar config

It's sketchy JSON let's commit before we break i

* MVP sidebars

summary.md is no longer functional
it's higherarchy was translated into sidebar.js
as a JSON format that docusaurus likes

* Removes another unused readme
2022-04-07 14:16:35 -07:00
George Claireaux
ded2b65244 clarity on version bump with /publish (#11813) 2022-04-07 19:18:48 +01:00
Pedro S. Lopez
eaa8b94e45 update building connector tutorial incorrect links and typo (#11769)
* fix "Build a Connector the Hard Way" doc typos and broken links

* Update adding-incremental-sync.md
2022-04-07 08:59:02 -07:00
Serhii Chvaliuk
6092bee717 SAT: check oneOf common property has only const keyword, no default and enum keywords (#11704)
* improve test_oneof_usage

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-04-07 10:40:55 +03:00
Topher Lubaway
17407da1e6 Add docusaurus tool not pipeline logic (#11716)
* Adds docusaurs tooling

tooling works
broken links still exist

needs the pipeline to work with GH pages
merging sooner means that we won't have forked problems from
the relatively frequently updated docs folder

* Updates readme

* Update docusaurus/README.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* Update docusaurus/README.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* Update docusaurus/README.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* Update docusaurus/README.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* Update docusaurus/docusaurus.config.js

Co-authored-by: Davin Chia <davinchia@gmail.com>

Co-authored-by: Davin Chia <davinchia@gmail.com>
2022-04-06 10:32:36 -07:00
George Claireaux
42fdebd1de 📖 Added section to docs about the /publish command (#11739)
* using yq + more success/failure comments

* added gradle process + git push

* dummy bump to test publish

* dummy bump to test publish

* dummy bump to test publish

* bump version

* fix connector variable

* bump version

* only git add necessary files

* remove git config

* bump version

* making octavia user

* version bump

* auto-bump connector version

* added docs and auto-bumo flag

* bump version

* separate IMAGE_NAME and IMAGE_VERSION env vars from sentry prep

* version bump

* auto-bump connector version

* added entry to apify changelog for clarity

* added details on /publish command

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-05 17:17:17 -07:00
George Claireaux
f7d3a706b2 Publish connector command now auto bumps version (#11712)
* using yq + more success/failure comments

* added gradle process + git push

* dummy bump to test publish

* dummy bump to test publish

* dummy bump to test publish

* bump version

* fix connector variable

* bump version

* only git add necessary files

* remove git config

* bump version

* making octavia user

* version bump

* auto-bump connector version

* added docs and auto-bumo flag

* bump version

* separate IMAGE_NAME and IMAGE_VERSION env vars from sentry prep

* version bump

* auto-bump connector version

* added entry to apify changelog for clarity

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-05 23:21:55 +01:00
Jared Rhizor
526c5a0c64 automatically figure out ARM-related variables (#11450)
* automatically figure out ARM-related variables

* clean up

* explicitly use amd64-prefixed versions of alpine and postgres
2022-03-31 13:58:46 -07:00
George Claireaux
4c86007e82 SAT: local python tests log git hash with success or failure (#11497)
* added success check with git hash

* code done

* added note about local git hash test log

* format
2022-03-29 16:46:51 +01:00
Greg Solovyev
e07f12dc41 Update incremental sync tutorial to match recent changes in build-a-connector tutorial (#11126) 2022-03-16 09:30:27 -07:00
Greg Solovyev
d84e852ac6 Greg/source connector guide (#11082)
Original API used in the example is no longer free, so while following the tutorial I found a different API to use in the tutorial and updated the example code to match the new API. Also made a few small changes.

Update Build-a-connector tutorial
 * Update screenshots and instructions to match current UI
 * Update code examples to use Polygon.io API
 * Add links to language-specific guides
 * Consistently use Python (uppercase P) when referring to the language
   and python (lowercase) p in code samples
 * Consistently do not use "." in lists
 * Add an image of Airbyte startup banner
 * Add a note for M1 macs
 * Remove unused images
 * Make Dockerfile consistent with the tutorial
2022-03-14 13:33:10 -07:00
Ramon Vermeulen
f0b0cbe191 Docs: incorrect line number (#11112)
incorrect line number in documentation
2022-03-14 17:14:54 -03:00
LiRen Tu
d57408269b 📝 Add connector-name examples in java profiling doc (#11006) 2022-03-09 14:38:28 -08:00
LiRen Tu
80a2b642f9 📝 Document java connector memory profiling (#10983)
* Add doc for java connector memory profiling

* Add doc to summary

* Update doc

* Update doc

* Update doc
2022-03-09 05:58:33 -08:00
Maksym Pavlenok
bbd13802d8 🐛 Fix Python checker configs and Connector Base workflow (#10505) 2022-02-22 19:58:55 +02:00
Maksym Pavlenok
61b0e9e196 🎉 Single py checker (#10246) 2022-02-21 14:01:42 +02:00
Eugene Kulak
d173ce741f CDK: add support for streams with state attribute (#9746)
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2022-02-17 00:20:33 +04:00
Sherif A. Nada
032b06d2e2 Add section to UX handbook about pointing to page anchors 2022-01-17 16:53:35 -08:00
Anuj Gupta
ecfc9e1cc5 Correcting base URL for the example API (#9245) 2022-01-13 07:25:13 -03:00
Augustin
16b14c51e9 Document closure usage in transform function (#9193) 2021-12-29 18:27:06 +01:00
Edward Gao
d0f9aa8907 Update "Building a connector the hard way" doc (#9008) 2021-12-21 10:50:59 -08:00
Maksym Pavlenok
d3b0c998b8 🎉 CI secrets: full migration to GSM (#8561)
* add python packages for CI scripts

* add tox config for all subpackages

* draft version

* init venv for scripts

* fix venv

* remove used comments

* fix run test

* change base folder

* update secret format

* update docs

* remove an unused file

* remove github secrets logic fully

* fix base_folder balue

* add functions desc
2021-12-15 15:08:46 +02:00
Menno Hamburg
03e5160569 🎉 New CDK: Dotnet/.NET/C# implementation + free exchange rates source connector (#7945)
* Added cdk source

* Added free exchange rates api as a dotnet example

* Corrected the use of JsonDocument, properly enabled utf-8

* Fixed error catching

* Added cli, documentation and nuget improvements

* Small improvements where needed

* Removed obsolete files

* Update README.md

* Added documentation and reference

* Update README.md

* Corrected dotnet

* Update README.md
2021-12-13 20:24:09 -03:00
Jared Rhizor
25674fc306 upgrade to Gradle 7.3.1 / Java 17 (#7964)
* upgrade gradle

* upgrade to Java 17 (and fix a few of the node versioning misses)

* oops

* try to run a different format version

* fix spotless by upgrading / reformatting some files

* fix ci settings

* upgrade mockito to avoid other errors

* undo bad format

* fix "incorrect" sql comments

* fmt

* add debug flag

* remove

* bump

* bump jooq to a version that has a java 17 dist

* fix

* remove logs

* oops

* revert jooq upgrade

* fix

* set up java for connector test

* fix yaml

* generate std source tests

* fail zombie job attempts and add failure reason (#8709)

* fail zombie job attempts and add failure reason

* remove failure reason

* bump gcp dependencies to pick up grpc update (#8713)

* Bump Airbyte version from 0.33.9-alpha to 0.33.10-alpha (#8714)

Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>

* Change CDK "Caching" header to "nested streams & caching"

* Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable (#8524)

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* move S3Config into destination-s3; update dependencies accordingly (#8562)

Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Iryna Grankova <87977540+igrankova@users.noreply.github.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
2021-12-10 16:57:54 -08:00
Sherif A. Nada
ba09bbfe4f Change CDK "Caching" header to "nested streams & caching" 2021-12-10 15:20:54 -08:00
Vadym
f2f224fe08 Update SAT for validate_data_points Test Basic Read option. (#8342) 2021-11-30 21:06:14 +02:00