jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-01 00:02:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,968 Commits 5,135 Branches 518 Tags
9f6963ccfcddd7a573c4ca0ff94de41ce6175cc0
Commit Graph

72 Commits

Author SHA1 Message Date
Christo Grabowski
a0f74e825b 📝 Docs: Files Source setup steps update (#28090) 
* add connector setup steps

* add links to Azure Blob storage guides

* add instructions for azure blob

* expand SSH/SCP/SFTP configuration field section

* complete update of setup steps

* change limited tag for local filesystem to OSS

* minor fixes to file source page
 2023-07-12 08:31:24 -07:00
Juan
14c0212d89 Update File source doc (#27763) 
Included that AzBlob URL only needs the path to the file and not the base URL.
 2023-06-27 11:46:21 -04:00
Serhii Lazebnyi
0bdd8445cf Remove debug mode for pytest config (#27157) 
* Deleted pytest log level

* Force log level for smart open to error

* Bump version

* Bump version

* Delete extra space
 2023-06-08 19:17:15 +02:00
Augustin
c2b7043f3f connectors-ci: make source-file testable in airbyte-ci (#27107) 2023-06-08 15:03:59 +00:00
Daryna Ishchenko
2a1d7f37e2 🐛Source File: added parser error handling (#26275) 
* added parser error handling

* updated versions, added changelog

* Automated Change

---------

Co-authored-by: darynaishchenko <darynaishchenko@users.noreply.github.com>
 2023-05-25 19:02:57 +03:00
Serhii Chvaliuk
50c7cb438d Source File: add retry on SSHException('Error reading SSH protocol banner') - 2 (#26210) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2023-05-18 00:42:33 +03:00
Serhii Chvaliuk
033e62ad4c Source File Secure: bump source-file version 0.2.38 -> 0.3.7 (#26131) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2023-05-17 02:44:22 +03:00
Serhii Chvaliuk
63887f7fe0 Source File: add retry on SSHException('Error reading SSH protocol banner') (#26115) 
Signed-off-by: Serhii Chvaliuk <grubberr@gmail.com>
 2023-05-16 20:03:37 +03:00
Artem Inzhyyants
fc6165ee1c Source File: Check if reader options is JSON object (#26117) 
* Source File: check if reader options is JSON object

* Source File: bump version; update docs

* Source File: update error message

* Source File: format

* Source File: test fix

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-16 12:03:36 +00:00
Evan Tahler
3679dcd9d0 🐛 Fix date-time for airbyte types (#25965) 
* Fix airbyte types

* update firebolt test fixture

* Bump changelog PR ids

* update metadata files

* fix firebolt test

* fix file-source for date-time overload

* fix source-file changelog

* it's a date, not date-time

* another date

* auto-bump connector version

* manually bump definitions

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-12 01:15:30 +00:00
Evan Tahler
63d0402111 🐛 source-file GCP token is a secret (#25819) 
* source-file GCP token is a secret

* docs and changelog

* bump metadata

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-04 20:49:28 -07:00
Artem Inzhyyants
814b4f7f59 Source File: Handle network errors (#25641) 
* Source File: Handle network errors

* Source File: Add unit test

* Source File: bump version + docs update

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-01 15:52:05 +02:00
Artem Inzhyyants
a3fa2b15dd Source File: Fix OOM; read Excel files in chunks (#25575) 
* Source File: Use openpyxl to read excel files in chunks

* Source File: bump version

* Source File: update docs

* Source File Secure: bump version

* Source File Secure: add docstring

* Source File: use column names from reader options

* Source File: refactor; use pandas for non xlsx formats

* Source File: reformat

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-01 14:45:36 +02:00
Serhii Lazebnyi
3bfcddac2d 🎉Source File: Add datatime format parsing support for csv files (#25445) 
* Add datatime format parsing support

* Updated subject in docs

* Bump file secure

* PR number bumped

* Fix typo

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-04-25 11:57:10 +02:00
Ali Hassan
d7a3b2a04f 🐛 Source File: fix column data types for numerical values (#23759) 
* Fix column data types for numerical values

* Update unit tests

* Update Dockerfile version

* update dockerfile version for source-file-secure

* Update source-file in seed file

* Update doc

* Update source_specs.yaml

---------

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2023-04-13 18:37:39 -03:00
Ujjwal Tyagi
812453fd8e Source File: fix example in Reader Options in specification (#24525) 
* Update file.md

* Update spec.json

* Update spec.json

* update versions and changelog

* bump version number

* update changelog and rebase

* auto-bump connector version

* fix integration test spec.json file

* remove unused files

* restore acceptance test

* convert all examples to string

* convert example to string

* regenerate seed

* change to airbyte.com

* Update source_specs.yaml

* Delete source_definitions.yaml

* regenerate source spec file

* update spec for file-secure

---------

Co-authored-by: Mal Hancock <mallory@archangelic.space>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2023-04-10 17:20:56 -03:00
Daryna Ishchenko
8bb84f2912 Source File: decrease give up rate, added better error message (#24588) 
* added fields description

* removed traceback from user messages

* bump version, added changelog

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-30 18:20:10 +03:00
Denys Davydov
9fe07dcf5d Source file: do not read whole file on check and discover (#24278) 
* #1681 source file: do not read whole file on check and discover

* #1681 source file: upd changelog

* #1681 source file: add allowed hosts

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-22 10:06:02 +02:00
Daryna Ishchenko
5f0d9f0f6f Source Files: decrease give up rate (#23723) 
* updated spec, docs, error messages

* added change log

* updated error message in unit tests

* added allowedHosts

* refactored error message and url descriprion

* updated example for url field in spec

* bump source-file verion for source-file-secure

* auto-bump connector version

* updated spec.json for source-file-secure

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-10 11:13:05 +02:00
Baz
3a4c45ba4d updated name and docs (#23331) 2023-02-23 01:35:13 +02:00
Ella Rohm-Ensing
aad9d400b1 resolve connector dependency workflow missing changelogs (#23314) 
* Add changelog entry for destination-clickhouse 0.2.2

* Delete csv.md

* local-csv.md -> csv.md
 2023-02-21 17:57:19 -05:00
Jon Martin
acadee08bc Add instructions for mounting Windows directory. (#21405) 
There're currently no directions as to what a user would need to change to be able to mount a raw directory into the Docker containers to be able to use a file as a source.
 2023-01-17 07:53:59 -03:00
Simon Späti
26053ef727 📝 Add warning for working with local files and Docker Desktop (/tmp and /private) (#21004) 2023-01-05 10:54:58 +01:00
Artem Inzhyyants
10d7bea363 Source File: Fix special characters bug (#21012) 
* Source File: Fix special characters bug

* Source File: bump version; update docs

* Source File: update SSH credentials in unit tests

* Source File: bump version (secure)

* Source File: update resources manually
 2023-01-04 20:58:42 +01:00
Artem Inzhyyants
54377e4eec Source File: increase timeout OC 1197 (#20740) 
* Source File: increase timeout OC 1197

* Source File: docs update; bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-12-21 14:28:40 -06:00
Evan Tahler
a55eb7df2d Fix CI Dependency Check Failures (#20666) 
* pardot

* plaid

* quickbooks

* appfollow

* appstore

* cloudtrail

* clickup

* clockify

* coda

* Coinmarketcap

* cooper

* dixa

* dv-360

* exchange-rates

* file

* gridly

* Hellobaton

* kustomer

* mailersend

* microsoft dataverse

* n8n

* PersistIq

* Survey Sparrow

* Twilio Taskrouter

* YouTube Analytics Business

* Younium

* Yahoo Finance Price

* Yandex Metrica

* Xero

* WooCommerce

* XKCD

* Webflow

* US Census API

* Qonto

* Pivotal Tracker

* KVDB

* Firestore

* Ignore even more connectors

* test run

* SFTP JSON

* cleanup

* move pardot changelog

* update links

* remove testing HACK

* Update docs/integrations/sources/dixa.md

Co-authored-by: Augustin <augustin@airbyte.io>

* Update docs/integrations/sources/kustomer-singer.md

Co-authored-by: Augustin <augustin@airbyte.io>

* Update docs/integrations/sources/pardot.md

Co-authored-by: Augustin <augustin@airbyte.io>

* Update docs/integrations/sources/kustomer-singer.md

Co-authored-by: Augustin <augustin@airbyte.io>

* Update docs/integrations/sources/pardot.md

Co-authored-by: Augustin <augustin@airbyte.io>

Co-authored-by: Augustin <augustin@airbyte.io>
 2022-12-20 16:57:25 -06:00
resoliwan
88c9844265 Source file: fix pandas read_csv header none issue. (#18587) 
* Source file: fix pandas read_csv header none issue.

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2022-12-02 19:43:03 -03:00
Arnaud Jeannin
0164355635 🎨 Add oss/cloud tags on doc for GA connectors (#19118) 
* feat: add cloud and oss tags

* put headers back

* fix: rm prettier style

* fix: aws styles
 2022-11-17 17:01:20 +01:00
Tyler B
524e2744be Add specifics for files stored in Google Drive (#19292) 
The URL for Google Drive cannot be the URL used for sharing and must be reformatted to work
 2022-11-10 13:42:32 -05:00
Serhii Chvaliuk
fce9184d86 Source File: using AirbyteConnectionStatus for "check" command instead of Exception (#19222) 2022-11-10 14:22:52 +01:00
Baz
91a10dae57 🐛 Source File: add retry logic for Connection reset by peer - 104 error (#18428) 2022-10-27 16:05:07 +03:00
Artem Inzhyyants
0daeb8dd7a 🐛 Source File: fix check method (#18481) 
* 🐛 Source File: fix check method

* 🐛 Source File: bump version

* 🐛 Source File: test for wrong format

* 🐛 Source File: docs update

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-26 18:34:10 +02:00
Serhii Chvaliuk
5b81897b96 Source File: add validate_and_transform of config (#18116) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-10-19 22:10:51 +03:00
Arsen Losenko
a28836e0c8 Source File: Handle UnicodeDecodeError in case of incorrect filetype (#17994) 
* Handle UnicodeDecodeError in case of incorrect filetype

* Update changelog and Dockerfile

* Update changelog with PR number

* Move try/except close to the end of method

* Add test and test files
 2022-10-17 12:20:03 +03:00
Baz
8c4b61c13b 🐛 Source File: add data presence validation for HTTPS for check_connection (#17504) 2022-10-03 18:44:56 +03:00
Augustin
ff4ea3961a Republish connectors using CDK 0.1.88 to 0.1.89 (#17304) 2022-09-28 18:18:59 +02:00
Denys Davydov
d780141f89 Source File: fix schema generation for json files containing an array (#16772) 
* #547 oncall Source File: fix schema generation for json files containing arrays

* source file: upda changelog

* #547 oncall: source file - upgrade source-file-secure

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-09-19 12:18:05 +03:00
Arsen Losenko
35257a0ec1 Source File: specify pyxlsb as engine for Excel file parsing (#15568) 
* Specify engine for excel files

* Add test for the chagne related to pyxlsb

* Update changelog and bump connector version

* Update PR link and number

* Update spec to include 'excel_binary' option, formatted files

* Update spec for source-file-secure to match changes made in source-file spec
 2022-08-26 17:34:16 +03:00
Denys Davydov
81bfb5c350 Source file: fix csv schema discovery (#15870) 
* #174 source file: fix csv schema discovery

* #174 source file: upd changelog

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-08-23 12:38:08 +03:00
Serhii Chvaliuk
e9fa2c4476 🎉 Source File: Convert 'nan' to 'null' (#15768) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-08-20 09:53:36 +03:00
Serhii Chvaliuk
f4adef7927 🎉 Source File: cache binary stream to file for discover (#15698) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-08-17 15:19:30 +03:00
Serhii Chvaliuk
5eef51ae15 🎉 Source File: cache binary stream to file (#15501) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-08-11 11:56:53 +03:00
midavadim
f143c8f02a 🎉 Source File - add support for custom encoding (#15293) 
* added support for custom encoding

* fixed unit test for utf16

* updated docs

* bumped connector version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-08-10 21:13:51 +03:00
Serhii Chvaliuk
35125f8846 🎉 Source File: bump setup.py versions (#15269) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-08-06 12:28:03 +03:00
Denis Davydov
2570f9ce73 Source file: remove nested properties (#14535) 
* #13787 source file: remove nested properties

* #13787: upd changelog

* #13787 revert processing json array, add unit test

* #13787 source file: upgrade pyarrow to allow builds on aarch64

* #13787 source file: upd version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-07-13 11:29:51 +03:00
Christophe Duong
43f905b8f9 🎉 Add YAML format to source-file reader (#14588) 
* Add yaml reader

* Update docs

* Bumpversion of connector

* bump docs

* Update pyarrow dependency

* Upgrade pandas dependency

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-07-12 19:34:17 +02:00
Topher Lubaway
9c6c092a22 Revert "Improving docusaurus sidebar generation (#1927) (#14369)" (#14596) 
This reverts commit a2c194a11f.
 2022-07-11 15:27:14 -05:00
Mykyta Serbynevskiy
a2c194a11f Improving docusaurus sidebar generation (#1927) (#14369) 
* Improving docusaurus sidebar generation (#1927)

* Added "Career & open positions" folder to sidebar, adjusted "Project overview" folder

* Deleted "career-and-open-positions" folder from sidebar
 2022-07-08 14:18:27 -05:00
Serhii Chvaliuk
c021426752 🎉 Source File: Update airbyte-cdk>=0.1.47 to be compatible with latest SAT tests (#9974) 
* bump version 0.2.8 -> 0.2.9

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-02-01 19:50:47 +02:00
Augustin
14b301ce37 Update S3 and file sources docs: we do not support unstructured data (#9192) 2021-12-29 18:22:59 +01:00