jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 02:51:29 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

43 Commits

Author SHA1 Message Date
Daryna Ishchenko
30d7b0231e feat(source-file): update to airbyte cdk version 7 (#68588) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-10-27 12:40:46 +02:00
Morgan
c8d22f9e9c fix(source-file): make client _is_zip method case-insensitive (#55205) 2025-03-06 16:30:37 +02:00
Aaron ("AJ") Steers
83ecbe0fc3 CI: apply pre-commit format fix from #49806 (#49852) 2024-12-18 14:05:43 -08:00
Alfredo Garcia
af77e73cef Adding reader options for Excel: header & skiprows (#48804) 2024-12-10 12:32:23 -06:00
Carlos Alvarenga
bc36dd2052 Source file: Fix error on zipped files (#39909) 
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2024-08-01 16:10:24 -03:00
Julien COUTAND
a886ace52f ✨ Source File: support ZIP file (#32354) 
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
 2024-02-15 18:33:19 +01:00
mjgatz
f6b4436206 ✨ Source File: add fixed width file format support (#34678) 
Co-authored-by: mgreene <michael.greene@gravie.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
 2024-02-13 19:58:35 +01:00
Augustin
0b33caecda Revert "[skip ci] formatting: add missing license headers (#33250)" (#33289) 2023-12-11 11:38:37 +01:00
Augustin
60c1cc01ad [skip ci] formatting: add missing license headers (#33250) 2023-12-11 10:15:18 +01:00
Marius Posta
7ae97175a6 gradle: fix repo wide behaviour (#30607) 2023-09-28 05:01:13 -07:00
Denys Davydov
809d86bbb6 🐛 Source File: handle parse errors when running discovery (#30579) 2023-09-25 03:07:58 -05:00
Daryna Ishchenko
2a1d7f37e2 🐛Source File: added parser error handling (#26275) 
* added parser error handling

* updated versions, added changelog

* Automated Change

---------

Co-authored-by: darynaishchenko <darynaishchenko@users.noreply.github.com>
 2023-05-25 19:02:57 +03:00
Serhii Chvaliuk
50c7cb438d Source File: add retry on SSHException('Error reading SSH protocol banner') - 2 (#26210) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2023-05-18 00:42:33 +03:00
Serhii Chvaliuk
63887f7fe0 Source File: add retry on SSHException('Error reading SSH protocol banner') (#26115) 
Signed-off-by: Serhii Chvaliuk <grubberr@gmail.com>
 2023-05-16 20:03:37 +03:00
Artem Inzhyyants
fc6165ee1c Source File: Check if reader options is JSON object (#26117) 
* Source File: check if reader options is JSON object

* Source File: bump version; update docs

* Source File: update error message

* Source File: format

* Source File: test fix

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-16 12:03:36 +00:00
Evan Tahler
3679dcd9d0 🐛 Fix date-time for airbyte types (#25965) 
* Fix airbyte types

* update firebolt test fixture

* Bump changelog PR ids

* update metadata files

* fix firebolt test

* fix file-source for date-time overload

* fix source-file changelog

* it's a date, not date-time

* another date

* auto-bump connector version

* manually bump definitions

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-12 01:15:30 +00:00
Artem Inzhyyants
814b4f7f59 Source File: Handle network errors (#25641) 
* Source File: Handle network errors

* Source File: Add unit test

* Source File: bump version + docs update

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-01 15:52:05 +02:00
Artem Inzhyyants
a3fa2b15dd Source File: Fix OOM; read Excel files in chunks (#25575) 
* Source File: Use openpyxl to read excel files in chunks

* Source File: bump version

* Source File: update docs

* Source File Secure: bump version

* Source File Secure: add docstring

* Source File: use column names from reader options

* Source File: refactor; use pandas for non xlsx formats

* Source File: reformat

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-05-01 14:45:36 +02:00
Serhii Lazebnyi
3bfcddac2d 🎉Source File: Add datatime format parsing support for csv files (#25445) 
* Add datatime format parsing support

* Updated subject in docs

* Bump file secure

* PR number bumped

* Fix typo

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-04-25 11:57:10 +02:00
Ali Hassan
d7a3b2a04f 🐛 Source File: fix column data types for numerical values (#23759) 
* Fix column data types for numerical values

* Update unit tests

* Update Dockerfile version

* update dockerfile version for source-file-secure

* Update source-file in seed file

* Update doc

* Update source_specs.yaml

---------

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2023-04-13 18:37:39 -03:00
Daryna Ishchenko
5f0d9f0f6f Source Files: decrease give up rate (#23723) 
* updated spec, docs, error messages

* added change log

* updated error message in unit tests

* added allowedHosts

* refactored error message and url descriprion

* updated example for url field in spec

* bump source-file verion for source-file-secure

* auto-bump connector version

* updated spec.json for source-file-secure

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-10 11:13:05 +02:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594) 
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
 2023-02-08 13:01:16 -08:00
Baz
f1546c297c 🐛 SAT: fix minor issues with connector tests (#20237) 2022-12-15 15:49:23 +02:00
resoliwan
88c9844265 Source file: fix pandas read_csv header none issue. (#18587) 
* Source file: fix pandas read_csv header none issue.

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2022-12-02 19:43:03 -03:00
Serhii Chvaliuk
fce9184d86 Source File: using AirbyteConnectionStatus for "check" command instead of Exception (#19222) 2022-11-10 14:22:52 +01:00
Baz
91a10dae57 🐛 Source File: add retry logic for Connection reset by peer - 104 error (#18428) 2022-10-27 16:05:07 +03:00
Serhii Chvaliuk
5b81897b96 Source File: add validate_and_transform of config (#18116) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-10-19 22:10:51 +03:00
Marcos Marx
70dd9a85cf 🎉 Source Zendesk Chat: engagements data fix infinity looping + gradlew format (#18121) 
* fix infinity looping in chats

* added more meaninful to variable name

* bump docker version

* auto-bump connector version

* run format

Co-authored-by: Roberto Bonnet <robertojuarezwp@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-18 15:14:02 -03:00
Arsen Losenko
a28836e0c8 Source File: Handle UnicodeDecodeError in case of incorrect filetype (#17994) 
* Handle UnicodeDecodeError in case of incorrect filetype

* Update changelog and Dockerfile

* Update changelog with PR number

* Move try/except close to the end of method

* Add test and test files
 2022-10-17 12:20:03 +03:00
Baz
8c4b61c13b 🐛 Source File: add data presence validation for HTTPS for check_connection (#17504) 2022-10-03 18:44:56 +03:00
Denys Davydov
d780141f89 Source File: fix schema generation for json files containing an array (#16772) 
* #547 oncall Source File: fix schema generation for json files containing arrays

* source file: upda changelog

* #547 oncall: source file - upgrade source-file-secure

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-09-19 12:18:05 +03:00
Arsen Losenko
35257a0ec1 Source File: specify pyxlsb as engine for Excel file parsing (#15568) 
* Specify engine for excel files

* Add test for the chagne related to pyxlsb

* Update changelog and bump connector version

* Update PR link and number

* Update spec to include 'excel_binary' option, formatted files

* Update spec for source-file-secure to match changes made in source-file spec
 2022-08-26 17:34:16 +03:00
Arsen Losenko
7d6dfac973 🎉Source File: increase unit test coverage at least 90% (#15933) 
* Added inital test setup for client.py, added WIP tests for source.py

* Add tests for storage scheme and started tests for _open() method

* Unittest coverage increased to 90

* Small refactoring

Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
 2022-08-26 12:18:44 +02:00
Serhii Chvaliuk
e9fa2c4476 🎉 Source File: Convert 'nan' to 'null' (#15768) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-08-20 09:53:36 +03:00
midavadim
f143c8f02a 🎉 Source File - add support for custom encoding (#15293) 
* added support for custom encoding

* fixed unit test for utf16

* updated docs

* bumped connector version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-08-10 21:13:51 +03:00
Denis Davydov
2570f9ce73 Source file: remove nested properties (#14535) 
* #13787 source file: remove nested properties

* #13787: upd changelog

* #13787 revert processing json array, add unit test

* #13787 source file: upgrade pyarrow to allow builds on aarch64

* #13787 source file: upd version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-07-13 11:29:51 +03:00
Davin Chia
65c04d6b9e Fix Source-File build. (#3738) 
The main problem seems to be the newer pip resolver isn't happy with our current dependencies. Changing this was enough to fix it.

Got this version by removing the constraint, running the install step, then using the installed version from pip list.
 2021-05-29 22:39:06 +08:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Davin Chia
f660b0a946 Add template generation for Santa aka CDK. (#3034) 
Template generation for new Source using the Santa CDK - provide basic scaffolding for someone implementing a new source.

General approach is to buff up comments in the original SDK, and add TODOs with secondary comments in the generated stub methods, as well as links to existing examples (e.g. Stripe or ExchangeRate api) users can look at.

Checked in and added tests for the generated modules.
 2021-04-25 18:02:33 +08:00
Charles
f445fdb5b2 match styling for spotlessApply and format (#3017) 
* as a java developer I want to be able to run spotlessApply without changing styles in python code
 2021-04-23 09:21:41 -07:00
Eugene K
e9769c8159 Source File: fix issue with boolean types in schema (#1873) 2021-01-28 23:02:53 -08:00
Eugene K
a54dd89418 Adopt connector best practices for File Source (providers) #1584 (#1738) 
Co-authored-by: Sherif Nada <snadalive@gmail.com>
 2021-01-27 16:49:33 -08:00
Christophe Duong
daf58b2dbd Add SourceFile integration (#716) 
Implement Source for files stored in GCS/S3/SFTP/SSH/SCP
 2020-11-02 18:43:59 +01:00