jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 02:51:29 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,867 Commits 5,109 Branches 518 Tags
master
Commit Graph

54 Commits

Author SHA1 Message Date
Daryna Ishchenko
30d7b0231e feat(source-file): update to airbyte cdk version 7 (#68588) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-10-27 12:40:46 +02:00
Morgan
c8d22f9e9c fix(source-file): make client _is_zip method case-insensitive (#55205) 2025-03-06 16:30:37 +02:00
Aaron ("AJ") Steers
83ecbe0fc3 CI: apply pre-commit format fix from #49806 (#49852) 2024-12-18 14:05:43 -08:00
Alfredo Garcia
af77e73cef Adding reader options for Excel: header & skiprows (#48804) 2024-12-10 12:32:23 -06:00
Carlos Alvarenga
bc36dd2052 Source file: Fix error on zipped files (#39909) 
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2024-08-01 16:10:24 -03:00
Daryna Ishchenko
395effcf1d 🐛 Source File: add engine fastparquet to parquet reader options (#37799) 2024-05-07 17:23:40 +03:00
Julien COUTAND
a886ace52f ✨ Source File: support ZIP file (#32354) 
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
 2024-02-15 18:33:19 +01:00
mjgatz
f6b4436206 ✨ Source File: add fixed width file format support (#34678) 
Co-authored-by: mgreene <michael.greene@gravie.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
 2024-02-13 19:58:35 +01:00
Pedro S. Lopez
d4ef599838 ✨ source-file: prevent local file usage on cloud deployment mode (#30984) 
Co-authored-by: pedroslopez <pedroslopez@users.noreply.github.com>
 2023-10-16 13:00:30 -07:00
Denys Davydov
809d86bbb6 🐛 Source File: handle parse errors when running discovery (#30579) 2023-09-25 03:07:58 -05:00
Serhii Lazebnyi
0bdd8445cf Remove debug mode for pytest config (#27157) 
* Deleted pytest log level

* Force log level for smart open to error

* Bump version

* Bump version

* Delete extra space
 2023-06-08 19:17:15 +02:00
Augustin
d8820d6e80 Fix python connectors formatting errors with black / isort (#26951) 
* format source-tiktok-marketing

* format source-twilio

* format source-instgram

* format source-amazon-ads

* format source-google-search-console

* format source-stripe

* format source-google-analytics-data-api

* format source-facebook-marketing

* format source-zendesk-support

* format source-file
 2023-06-02 09:26:04 -05:00
Daryna Ishchenko
2a1d7f37e2 🐛Source File: added parser error handling (#26275) 
* added parser error handling

* updated versions, added changelog

* Automated Change

---------

Co-authored-by: darynaishchenko <darynaishchenko@users.noreply.github.com>
 2023-05-25 19:02:57 +03:00
Serhii Chvaliuk
50c7cb438d Source File: add retry on SSHException('Error reading SSH protocol banner') - 2 (#26210) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2023-05-18 00:42:33 +03:00
Serhii Chvaliuk
63887f7fe0 Source File: add retry on SSHException('Error reading SSH protocol banner') (#26115) 
Signed-off-by: Serhii Chvaliuk <grubberr@gmail.com>
 2023-05-16 20:03:37 +03:00
Evan Tahler
3679dcd9d0 🐛 Fix date-time for airbyte types (#25965) 
* Fix airbyte types

* update firebolt test fixture

* Bump changelog PR ids

* update metadata files

* fix firebolt test

* fix file-source for date-time overload

* fix source-file changelog

* it's a date, not date-time

* another date

* auto-bump connector version

* manually bump definitions

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-12 01:15:30 +00:00
Artem Inzhyyants
814b4f7f59 Source File: Handle network errors (#25641) 
* Source File: Handle network errors

* Source File: Add unit test

* Source File: bump version + docs update

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-01 15:52:05 +02:00
Artem Inzhyyants
a3fa2b15dd Source File: Fix OOM; read Excel files in chunks (#25575) 
* Source File: Use openpyxl to read excel files in chunks

* Source File: bump version

* Source File: update docs

* Source File Secure: bump version

* Source File Secure: add docstring

* Source File: use column names from reader options

* Source File: refactor; use pandas for non xlsx formats

* Source File: reformat

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-01 14:45:36 +02:00
Serhii Lazebnyi
3bfcddac2d 🎉Source File: Add datatime format parsing support for csv files (#25445) 
* Add datatime format parsing support

* Updated subject in docs

* Bump file secure

* PR number bumped

* Fix typo

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-04-25 11:57:10 +02:00
Ali Hassan
d7a3b2a04f 🐛 Source File: fix column data types for numerical values (#23759) 
* Fix column data types for numerical values

* Update unit tests

* Update Dockerfile version

* update dockerfile version for source-file-secure

* Update source-file in seed file

* Update doc

* Update source_specs.yaml

---------

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2023-04-13 18:37:39 -03:00
Daryna Ishchenko
8bb84f2912 Source File: decrease give up rate, added better error message (#24588) 
* added fields description

* removed traceback from user messages

* bump version, added changelog

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-30 18:20:10 +03:00
Denys Davydov
9fe07dcf5d Source file: do not read whole file on check and discover (#24278) 
* #1681 source file: do not read whole file on check and discover

* #1681 source file: upd changelog

* #1681 source file: add allowed hosts

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-22 10:06:02 +02:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594) 
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
 2023-02-08 13:01:16 -08:00
Artem Inzhyyants
10d7bea363 Source File: Fix special characters bug (#21012) 
* Source File: Fix special characters bug

* Source File: bump version; update docs

* Source File: update SSH credentials in unit tests

* Source File: bump version (secure)

* Source File: update resources manually
 2023-01-04 20:58:42 +01:00
Artem Inzhyyants
54377e4eec Source File: increase timeout OC 1197 (#20740) 
* Source File: increase timeout OC 1197

* Source File: docs update; bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-12-21 14:28:40 -06:00
Baz
91a10dae57 🐛 Source File: add retry logic for Connection reset by peer - 104 error (#18428) 2022-10-27 16:05:07 +03:00
Serhii Chvaliuk
5b81897b96 Source File: add validate_and_transform of config (#18116) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-10-19 22:10:51 +03:00
Marcos Marx
70dd9a85cf 🎉 Source Zendesk Chat: engagements data fix infinity looping + gradlew format (#18121) 
* fix infinity looping in chats

* added more meaninful to variable name

* bump docker version

* auto-bump connector version

* run format

Co-authored-by: Roberto Bonnet <robertojuarezwp@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-18 15:14:02 -03:00
Arsen Losenko
a28836e0c8 Source File: Handle UnicodeDecodeError in case of incorrect filetype (#17994) 
* Handle UnicodeDecodeError in case of incorrect filetype

* Update changelog and Dockerfile

* Update changelog with PR number

* Move try/except close to the end of method

* Add test and test files
 2022-10-17 12:20:03 +03:00
Denys Davydov
d780141f89 Source File: fix schema generation for json files containing an array (#16772) 
* #547 oncall Source File: fix schema generation for json files containing arrays

* source file: upda changelog

* #547 oncall: source file - upgrade source-file-secure

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-09-19 12:18:05 +03:00
Arsen Losenko
35257a0ec1 Source File: specify pyxlsb as engine for Excel file parsing (#15568) 
* Specify engine for excel files

* Add test for the chagne related to pyxlsb

* Update changelog and bump connector version

* Update PR link and number

* Update spec to include 'excel_binary' option, formatted files

* Update spec for source-file-secure to match changes made in source-file spec
 2022-08-26 17:34:16 +03:00
Denys Davydov
81bfb5c350 Source file: fix csv schema discovery (#15870) 
* #174 source file: fix csv schema discovery

* #174 source file: upd changelog

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-08-23 12:38:08 +03:00
Serhii Chvaliuk
e9fa2c4476 🎉 Source File: Convert 'nan' to 'null' (#15768) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-08-20 09:53:36 +03:00
Serhii Chvaliuk
f4adef7927 🎉 Source File: cache binary stream to file for discover (#15698) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-08-17 15:19:30 +03:00
Serhii Chvaliuk
5eef51ae15 🎉 Source File: cache binary stream to file (#15501) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-08-11 11:56:53 +03:00
midavadim
f143c8f02a 🎉 Source File - add support for custom encoding (#15293) 
* added support for custom encoding

* fixed unit test for utf16

* updated docs

* bumped connector version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-08-10 21:13:51 +03:00
Serhii Chvaliuk
35125f8846 🎉 Source File: bump setup.py versions (#15269) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-08-06 12:28:03 +03:00
Christophe Duong
25ee6f6c37 bump source-file-secure (#14704) 
* bump source-file-secure

* Bump more

* bump requirements

* Fix source-file

* auto-bump connector version

* correct unit test spec for file secure + format files

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
 2022-07-15 09:58:52 +02:00
Tim Werner
32e767c9f6 🎉 Source File: add user-agent option (#14488) 
* add user-agent option to source-file

* fix acceptance-test

* set version to 0.0 if version cannot be fetched

* vump connector version

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-07-13 13:46:05 -03:00
Denis Davydov
2570f9ce73 Source file: remove nested properties (#14535) 
* #13787 source file: remove nested properties

* #13787: upd changelog

* #13787 revert processing json array, add unit test

* #13787 source file: upgrade pyarrow to allow builds on aarch64

* #13787 source file: upd version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-07-13 11:29:51 +03:00
Christophe Duong
43f905b8f9 🎉 Add YAML format to source-file reader (#14588) 
* Add yaml reader

* Update docs

* Bumpversion of connector

* bump docs

* Update pyarrow dependency

* Upgrade pandas dependency

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-07-12 19:34:17 +02:00
Marcos Marx
dca2256a7c Bump 2022 license version (#13233) 
* Bump year in license short to 2022

* remove protocol from cdk
 2022-05-26 15:00:42 -03:00
Yevhenii
269298c854 🎉 Source File: Migrate File source to CDK structure (#7387) 
* Migrate File source to CDK structure

* fix .dockerignore file

* remove SAT requirements

* update Dockerfile

* change Dockerfile to base images python:3.7-slim

* add SAT tests

* update tests

* add secret/config.json for source-file

* update changelogs
 2021-10-29 13:15:13 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Madison Swain-Bowden
2d5d7e8e98 🎉 Allow non-default source SFTP port (#4953) 
* Allow SFTP to define port

* Define port as a string

* Provide a clearer error if port parsing fails
 2021-07-27 18:52:55 -03:00
Davin Chia
cd24ccd064 Release Source File (#3771) 2021-05-31 21:35:57 -07:00
Phlair
9360ed0da9 🎉 File source: Add support for Azure blob storage (#3660) 
Co-authored-by: Davin Chia <davinchia@gmail.com>
 2021-06-01 10:42:13 +08:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Davin Chia
f660b0a946 Add template generation for Santa aka CDK. (#3034) 
Template generation for new Source using the Santa CDK - provide basic scaffolding for someone implementing a new source.

General approach is to buff up comments in the original SDK, and add TODOs with secondary comments in the generated stub methods, as well as links to existing examples (e.g. Stripe or ExchangeRate api) users can look at.

Checked in and added tests for the generated modules.
 2021-04-25 18:02:33 +08:00
Charles
f445fdb5b2 match styling for spotlessApply and format (#3017) 
* as a java developer I want to be able to run spotlessApply without changing styles in python code
 2021-04-23 09:21:41 -07:00