jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-02-01 22:01:57 -05:00
Code Issues Packages Projects Releases Wiki Activity
8,452 Commits 5,453 Branches 518 Tags
594cf29bbbfd579b32476a094faeca277ba91dfd
Commit Graph

3683 Commits

Author SHA1 Message Date
Sofie Hellmark
2b52b53ece 🎉 New Source: Visma e-conomic (#18595) 
* add visma economic as source

* fix version

* add documentation & clean up

* Update README.md

* remove non-existing stream from catalog

* fixing schema formats

* fixing acceptance tests

* add data for integration test

* add visma to builds.md

* delete integration_tests/catalog.json

* add example tokens to sample config

* fix requirements.txt

* fix accounting schema

* add title to spec

* fixed comments and test on next page token

* format visma economic

* correct source def

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-07 20:44:10 -03:00
Johan Hammarstedt
efa7d08a8a 🎉 New Source: Fastbill (#18593) 
* draft for fastbill

* adding documentation

* fixing common errors

* make requested changes according to review

* requested changes according to review

* compressing code even more

* format

* run format

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2022-11-07 19:42:39 -03:00
Albert Marrero
b7b6507164 🎉 New Source: Zapier Supported Storage (#18442) 
* generating app

* updating spec

* updating template

* initial connection confirmed

* updating configured catalog

* updating names to better reflect endpoint secret_key -> secret

* removing unnecessary schemas

* successful read from connector

* Updated Airbyte Intergrations Build, Updated Intergrations ReadME Updated, Added Zapier Supported Storage in Intergrations

* generating app

* updating spec

* updating template

* initial connection confirmed

* updating configured catalog

* updating names to better reflect endpoint secret_key -> secret

* removing unnecessary schemas

* successful read from connector

* Did all the changes the reviewer wanted.

* Update docs/integrations/README.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* updated pending errors in markdown and sample config

* including the md here as well

* updating invalid_config

* updating abnormal state

* updated configured catalogue

* updated schema among other things

* all tests passed

* Update Zapier Intergration Docs

* run format

* update doc

* add lines

* add zapier supported storage to source def

* run airbyte cnfig process

Co-authored-by: Ben <benjaminjgreene@outlook.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
 2022-11-07 19:02:11 -03:00
Artur Siepietowski
77c7d3e9b3 🎉 New source: Klarna (#18385) 
* Adding transactions for klarna source integration

* Fixing tests

* Adding payout stream

* Finalizing connector

* Adding new lines af EOF

* Applying code review suggestions. Adding doc for klarna. Adding title in spec.yaml

* run format

* format files

* add to source def

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-07 17:50:36 -03:00
Evan Tahler
e6b06a88ac AirbyteEstimateTraceMessage (#18875) 
* `AirbyteEstimateTraceMessage`

* Add PR number

* fix method name

* Lint

* Lint

* fix merge

* Update docs/understanding-airbyte/airbyte-protocol.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* `EstimateType` sub type in python

* lint

Co-authored-by: Davin Chia <davinchia@gmail.com>
 2022-11-07 12:45:39 -08:00
Ella Rohm-Ensing
e6bfe10278 [docs] Use correct header heirarchy in airbyte-protocol docs (#18917) 
* Use correct header heirarchy

* Use h2 for Actor Specification

* Revert indentation of key concepts
 2022-11-07 10:39:58 -07:00
Daemonxiao
441e927cdb Docs: update tidb.md (#19027) 2022-11-07 14:29:49 -03:00
Oleksii N
3365660a76 🐛 Source Kafka - Add missing record_count increment for JSON (#18648) 
* add missing record_count increment

* bump version

* Update kafka.md

* update version

* auto-bump connector version

Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Sajarin <sajarindider@gmail.com>
 2022-11-07 11:41:59 -05:00
antonioneto-hotmart
3ffa11cd11 🐛 Source JIRA - Field correction bug causing stream connection to fail (#18505) 
* max results field correction

* added version notes and up

* Update acceptance-test-config.yml

* auto-bump connector version

Co-authored-by: Vincent Koc <koconder@users.noreply.github.com>
Co-authored-by: Vincent Koc <koconder@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-07 10:10:01 -05:00
ireyna-modo
5846c65ee7 🐛 Source Freshdesk: Add mising / that's causing issues when building URL (#18397) 
* Add mising / that's causing issues when building URL

* attempt to align changes with tests

* Update test_300_page.py

* Update Dockerfile

* Update freshdesk.md

* auto-bump connector version

Co-authored-by: Vincent Koc <koconder@users.noreply.github.com>
Co-authored-by: Vincent Koc <koconder@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-07 10:08:24 -05:00
Augustin
7d73b08606 SAT: basic read on full catalog when test_strictness_level == high (#18937) 2022-11-07 08:09:34 +01:00
Artem Inzhyyants
a5bcea9b9e 🐛 Source Facebook Marketing: handle FacebookBadObjectError (#18971) 
* 🐛 Source Facebook Marketing: handle FacebookBadObjectError

* 🐛 Source Facebook Marketing: bump version; update docs

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-05 14:39:04 +01:00
Sophia Wiley
e298049222 Edit multi-cloud docs (#18972) 
Merging now to fix a couple of mistakes, but feel free to leave more comments on this PR.
 2022-11-04 18:33:25 -07:00
Anne
1d809a7539 Add new workspace api endpoint (#18983) 
* Add new workspace api endpoint
 2022-11-04 14:20:07 -07:00
Subodh Kant Chaturvedi
fe3f459a8f mssql-source:upgrade debezium version to 1.9.6 (#18732) 
* mssql-source:upgrade debezium version to 1.9.6

* more improvements

* upgrade version

* auto-bump connector version

* fix test

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-04 23:02:51 +05:30
Davin Chia
7bb3241781 Progress Bar API Changes. (#18953) 
API changes to support the progress bar.

- The eventual idea is for the save_stats route to be called by the workers during replication. Workers will save stats for a job id and attempt number.
- Make modifications to the /jobs/list and the /jobs/get_debug_info routes to also return estimated bytes/records.
We need both estimated metadata, as well as running states to calculate progress bar and throughput.


- add the save_stats route. This is the route that will be called by workers. I've done my best to reuse existing openapi bodies to reduce duplication.
- add the estimatedRecords and estimatedBytes fields to the AttemptStats body. This is part of the AttemptRead and the AttemptStreamStats objects. This eventually filters up to the jobs/list and jobs/get_debug_info objects. This also adds these to all the endpoints that were previously returning stats information. I think the duplicated data is a small issue and don't think it's worth splitting out a new api objects, though I will gladly do so if folks feel strongly.
minor changes to the AttemptApiController to support the new route.
- I've stubbed out the handlers for now since the backend is not yet implemented.
 2022-11-04 10:05:07 -07:00
Baz
51ac4b5238 🐛 Source Google Analytics (UA): fix discovery stage, when custom_reports is provided as single dict (#18965) 2022-11-04 18:04:14 +02:00
Artem Inzhyyants
54218f595b 🐛 Source SalesForce: fix Bulk API unsupported list (#18799) 
* 🐛 Source SalesForce: fix Bulk API unsupported list

* 🐛 Source SalesForce: bump version

* 🐛 Source Facebook Marketing: docs update

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-04 12:34:25 +01:00
Liren Tu
6967d146ec 🎉 Source redshift & bigquery: emit cursor count in state (#18954) 
* Bump version for redshift, bigquery, and snowflake

* auto-bump connector version

* auto-bump connector version

* Log failed refresh token response

* Revert snowflake version bump

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-04 01:13:02 -07:00
Sophia Wiley
77f3e3b037 Multi-cloud documentation (#18872) 
I am going to edit some of the wording in later PRs, but merging this for now. 

* added multicloud info

* Update docs/cloud/getting-started-with-airbyte-cloud.md

Co-authored-by: Joey Marshment-Howell <josephkmh@users.noreply.github.com>

* edited for clarity

* incorporated suggestions

Co-authored-by: Joey Marshment-Howell <josephkmh@users.noreply.github.com>
 2022-11-03 23:18:23 -07:00
Alexandre Girard
605fb921c4 [low-code]: Evaluate backoff strategies at runtime (#18053) 
* pass options to wait time from header

* fix constant backoff

* parameterize test

* fix tests

* missing unit tests

* eval header at runtime

* eval regex at runtime

* evaluate min_wait at runtime

* eval factor at runtime

* missing unit tests

* remove debug print

* rename

* Add tests

* Add tests

* Update docs
 2022-11-03 18:50:47 -07:00
Balasubramanian T K
838aebe322 🎉 New Source: SpaceX API [low-code cdk] (#18311) 
* Init: New Source: SpaceX API

* Chore: Removed duplicate and unwanted files

* fix: Reconfigured schema and matched the read records. chore: Acceptance test run and results passed without errors

* chore: Added documentation, updated airbyte docs specific to the connector

* fix conflict

* Chore: delete unwanted files

* chore: Created bootstrap.md

* chore: Update doc url, removed unwanted files

* fix: Fixed schema after crosscheck with several schema validators

* fix: resolved .vscode setting conflict, resolved merge conflict

* feat: Added support for all endpoints

* chore: updated documentation

* feat: changes to stream path to accept options and id

* fix: Resolve coomments

* chore: resolved comments

* chore: Delete unwanted files

* chore: Update Readme.md after resolving merge conflict

* update connector

* add spacex api to source def

* run format

* add schemaloader

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-03 22:14:39 -03:00
Eugene
5c870c1626 🐛Destination-dynamodb: enforce ssl connection (#18672) 
* [16283] Destination-dynamodb: Added strict-encrypt version and enforced ssl connection on cloud
 2022-11-03 22:15:24 +02:00
ev
f7bef97be6 New Source: Wikipedia Pageviews (#18343) 
* start adding connector

* fix check and stream slicer

* pass some tests

* fix: github url

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* add stream slicer for top and fix some tests

* remove unneeded files

* Increment the version.

* add docs and clean up unneeded comments

* Bump airbyte-cdk version to 0.2

* add schemaloader

* add wikipedia to source def

* auto-bump connector version

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-03 17:13:13 -03:00
JJ Nilbodee
c40f1b6b7f 🎉 New Source: SurveySparrow (#18395) 
* generated from template

* initial commit

* add sample_config

* add docs

* apply comments

* minor changes

* try using ListSlicer

* use ListStreamSlicer

* fix base url

* update cdk version in requirements

* add surveysparrow to source def

* add schemaloader

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-03 16:46:48 -03:00
Serhii Chvaliuk
c01b81bddb Source Mixpanel: "export" stream make line parsing more robust (#18846) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-11-03 19:46:06 +02:00
VitaliiMaltsev
8bb97010f3 MySQL Source: fixed unencrypted CDC connections (#18851) 
* MySQL Source: fixed unencrypted CDC connections

* updated changelog

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-03 19:40:20 +02:00
Sherif A. Nada
f4e3b605c9 fix typo in lowcode tutorial (#18916) 2022-11-03 09:18:37 -07:00
Zaza Javakhishvili
7693a3079a Source Amazon Seller Partner - add new reports and bump version (#18283) 
* add new amazon seller partner reports and bump version

* amazon seller partner new reports review based changes

* Amazon seller partner new reports review based changes 2

* amazon seller partner ./gradlew format

* amazon seller partner - add aditional 2 report types

* Amazon seller partner some stream fix

* amazon seller partner - ledger detailed view report data type fix

* amazon seller repport fix xml orders parse

* remove deprecated inventory helth report

* Amazon seller partner review based change 3

* Update acceptance-test-config.yml

* fix: generate source definitions for 0.2.28 version

Co-authored-by: Juozas V <monai@cure.lt>
Co-authored-by: sajarin <sajarindider@gmail.com>
 2022-11-03 11:19:10 -04:00
Tyler B
022fca824f Embed Metabase Connector Catalog (#18864) 
Replaces static connector catalog with the embedded Metabase connector catalog
 2022-11-03 09:52:47 -05:00
Tan Ho
f156be94bd New Source: Gridly (#18342) 
* Add source-gridly

* Correct current_page init value

* The first batch was fetching twice

* Remove `integration_tests/catalog.json` from source-gridly

* Allow select any view on grid to sync records

* Correct documentationUrl for source gridly

* use class property for source endpoint instead of local variable

* Add tests and format code

* Add gridly.md docs file

* add gridly to source def

* auto-bump connector version

Co-authored-by: Tan Ho <th@localizedirect.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-03 11:39:51 -03:00
Isaac Harris-Holt
606534ddbc 🎉 New Source: Alpha Vantage (#18320) 
* Initial commit

* Tidy and add custom dpath extractor

* Get acceptance tests passing

* Improve dict-checking for ObjectDpathExtractor

* Improve dict-checking for ObjectDpathExtractor

* Add more tests to ObjectDpathExtractor

* Add docs

* Address comments

* Update docs

* Add sample and invalid configs

* format

* add alpha vantage to source def

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2022-11-03 10:57:27 -03:00
Mario Molina
f49beff48d 🎉 New Source: Datadog [python cdk] (#18150) 
* New Source: Datadog

* Updating doc

* Adding unit tests

* Renaming limit var

* Updating description in spec

* add source def to seed

* add datadog to source def seed

* run format

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-03 10:46:31 -03:00
Benoit Moriceau
589f6efbfc Extract OAuth API (#18818) 
* Tmp

* Extract the Attempt API from the V1 API

* Add comments

* Move Connection API out of configuration API

* format

* format

* Rename to Controller

* Rename to Controller

* Add values to the factory

* Change the constructor to use hadler instead of objects needed by the handler

* Update with new tags.

* tmp

* Fix PMD errors

* Extract DB migrator

* Add something that I forgot

* extract destination definition api

* restore destination factory initialization

* extract destination definition specification api

* format

* format

* format

* extract health check api

* extract jobs api

* fix test

* format

* Extract logs api

* Add missing declaration

* Fix build

* Tmp

* format and PR comments

* Extract notification API

* re-organize tags

* Extract all Oauth

* Fix PMD
 2022-11-02 15:37:15 -07:00
Cole Snodgrass
9c081fb527 update images to have non-transparent background (#18874) 
* update images to have non-transparent background

* reset build.gradle
 2022-11-02 14:58:24 -07:00
Anne
d26e5bc248 DiscoverSchema endpoints calculates diff and breaking change (#18571) 
* update discover schema endpoint to calculate diff
 2022-11-02 14:10:07 -07:00
Baz
8958aa3cdc 🐛 Source Sendgrid: Skip the stream on authorization required permission error [400, 401] (#18847) 2022-11-02 20:39:58 +02:00
Amruta Ranade
fbe16d1bef edited connector docs (#18855) 2022-11-02 12:14:37 -04:00
Tim Roes
4ddd414359 🪟 🎉 Allow environment specific sections in docs (#18829) 
* Allow environment specific sections in docs

* Change syntax to lower case
 2022-11-02 08:49:14 -07:00
Subodh Kant Chaturvedi
0a37a8d352 mysql-source:fix tinyint unsigned handling (#18619) 
* mysql-source:fix tinyint unsigned handling

* update doc

* format

* upgrade version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-02 20:54:05 +05:30
Yowan Ramchoreeter
016907ed98 Add notes about EU OAUth (#18835) 
EU OAuth is not fully tested so adding a note to account for that.
 2022-11-02 11:13:30 -04:00
Tyler B
4b2d65f712 Remove "Filters and Segments" from Google Analytics v4 (#18508) 
Filters and Segments info was incorrectly added to the Google Analytics v4 connector instead of the Google Analytics (Universal Analytics) Connector.
 2022-11-02 11:12:47 -04:00
Octavia Squidington III
b299688f3b Bump Airbyte version from 0.40.17 to 0.40.18 (#18827) 
Co-authored-by: grishick <grishick@users.noreply.github.com>
 2022-11-01 19:35:23 -07:00
Greg Solovyev
7b9a097081 Add normalization changelog and bump normalization version in platform (#18813) 2022-11-01 15:36:31 -07:00
Serhii Lazebnyi
e45aec3ef5 🐛Source Exchange Rates: Fix handling error during check connection (#18726) 
* Fix handling error during check connection

* Updated PR number

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-01 23:20:20 +01:00
Serhii Lazebnyi
bc090834e0 🐛Destination Google Sheets: Fix empty headers list (#18729) 
* Fix empty headers list

* Updated PR number

* Bumped version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-01 23:19:46 +01:00
Artem Inzhyyants
88bbb5dabc 🐛 Source Facebook Marketing: reduce request limit after specific error (#18734) 
* 🐛 Source Facebook Marketing: reduce request limit after specific error

* 🐛 Source Facebook Marketing: bump version; update docs

* 🐛 Source Facebook Marketing: add test

* 🐛 Source Facebook Marketing: increase timeout
 2022-11-01 21:42:39 +01:00
Sherif A. Nada
978128afc2 Fix adjust.md header 2022-11-01 10:21:16 -07:00
Haithem SOUALA
aff9963a9f 🎉 New Source: Coinmarketcap [low-code CDK] (#18565) 
* init commit

* add docs

* add docs

* Delete logs.txt

* add items

* fix comments

* fix comment

* fix acceptance test

* remove *state.json used for incremental imports test

* Add ignored_fields on listing to get the acceptance test pass

- the crypto market is VERY volatile, the data change between 2 full imports when the test is run

* manually generate source_specs.yaml for coinmarketcap

Co-authored-by: Yiyang Li <yiyangli2010@gmail.com>
 2022-11-01 13:18:37 -03:00
Andrés Bravo
0ee930678b 🎉 New Source: Public APIs [python CDK] (#18471) 
* [ airbytehq/connector-contest#91 ] Add Source Public APIs

* [ airbytehq/connector-contest#91 ] Add Source Bootstrap.md

* [ airbytehq/connector-contest#91 ] Add Source documentation

* [ airbytehq/connector-contest#91 ] Add Source Public APIs

* [ airbytehq/connector-contest#91 ] Add Source Bootstrap.md

* [ airbytehq/connector-contest#91 ] Add Source documentation

* [ airbytehq/connector-contest#91 ] Remove uneeded incremental class and tests, updated doc

* [ airbytehq/connector-contest#91 ] Remove uneeded incremental class and tests, updated doc

* [ airbytehq/connector-contest#91 ] Fix pflake8 offenses

* run airbytePythonFormat on source-public-apis, and remove test_incremental_streams.py

* fix: generate and add source definitions

* - update documentationUrl;
- add new lint to eof

* auto-bump connector version

* correct documentationUrl in source_specs.yaml

Co-authored-by: Yiyang Li <yiyangli2010@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-01 13:18:10 -03:00