jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 02:51:29 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

51 Commits

Author SHA1 Message Date
Morgan
c8d22f9e9c fix(source-file): make client _is_zip method case-insensitive (#55205) 2025-03-06 16:30:37 +02:00
Aaron ("AJ") Steers
83ecbe0fc3 CI: apply pre-commit format fix from #49806 (#49852) 2024-12-18 14:05:43 -08:00
Carlos Alvarenga
bc36dd2052 Source file: Fix error on zipped files (#39909) 
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2024-08-01 16:10:24 -03:00
Natik Gadzhi
30643ae71f chore: replace AirbyteLogger with logging.Logger in connectors (#40215) 
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
 2024-06-27 18:02:01 +02:00
Julien COUTAND
a886ace52f ✨ Source File: support ZIP file (#32354) 
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
 2024-02-15 18:33:19 +01:00
mjgatz
f6b4436206 ✨ Source File: add fixed width file format support (#34678) 
Co-authored-by: mgreene <michael.greene@gravie.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
 2024-02-13 19:58:35 +01:00
Pedro S. Lopez
d4ef599838 ✨ source-file: prevent local file usage on cloud deployment mode (#30984) 
Co-authored-by: pedroslopez <pedroslopez@users.noreply.github.com>
 2023-10-16 13:00:30 -07:00
Denys Davydov
809d86bbb6 🐛 Source File: handle parse errors when running discovery (#30579) 2023-09-25 03:07:58 -05:00
Augustin
c2b7043f3f connectors-ci: make source-file testable in airbyte-ci (#27107) 2023-06-08 15:03:59 +00:00
Daryna Ishchenko
2a1d7f37e2 🐛Source File: added parser error handling (#26275) 
* added parser error handling

* updated versions, added changelog

* Automated Change

---------

Co-authored-by: darynaishchenko <darynaishchenko@users.noreply.github.com>
 2023-05-25 19:02:57 +03:00
Serhii Lazebnyi
3bfcddac2d 🎉Source File: Add datatime format parsing support for csv files (#25445) 
* Add datatime format parsing support

* Updated subject in docs

* Bump file secure

* PR number bumped

* Fix typo

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-04-25 11:57:10 +02:00
Denys Davydov
9fe07dcf5d Source file: do not read whole file on check and discover (#24278) 
* #1681 source file: do not read whole file on check and discover

* #1681 source file: upd changelog

* #1681 source file: add allowed hosts

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-22 10:06:02 +02:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594) 
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
 2023-02-08 13:01:16 -08:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846) 
Rename source acceptance test to connector acceptance test
 2023-02-02 11:38:19 +01:00
Roman Yermilov [GL]
b039a2cdeb Source File: enable high strictness level (#21243) 
* Source File: enable high strictness level

* Addedend line

Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
 2023-01-13 19:49:18 +01:00
Artem Inzhyyants
10d7bea363 Source File: Fix special characters bug (#21012) 
* Source File: Fix special characters bug

* Source File: bump version; update docs

* Source File: update SSH credentials in unit tests

* Source File: bump version (secure)

* Source File: update resources manually
 2023-01-04 20:58:42 +01:00
resoliwan
88c9844265 Source file: fix pandas read_csv header none issue. (#18587) 
* Source file: fix pandas read_csv header none issue.

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2022-12-02 19:43:03 -03:00
Artem Inzhyyants
0daeb8dd7a 🐛 Source File: fix check method (#18481) 
* 🐛 Source File: fix check method

* 🐛 Source File: bump version

* 🐛 Source File: test for wrong format

* 🐛 Source File: docs update

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-26 18:34:10 +02:00
Serhii Chvaliuk
5b81897b96 Source File: add validate_and_transform of config (#18116) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-10-19 22:10:51 +03:00
Arsen Losenko
a28836e0c8 Source File: Handle UnicodeDecodeError in case of incorrect filetype (#17994) 
* Handle UnicodeDecodeError in case of incorrect filetype

* Update changelog and Dockerfile

* Update changelog with PR number

* Move try/except close to the end of method

* Add test and test files
 2022-10-17 12:20:03 +03:00
Pedro S. Lopez
938436bcc9 update connector specs and definitions with new .com documentation urls (#17585) 
* update definitions with new .com docs urls

* update docs urls in specs

* update generators

* regenerate scaffold connectors

* remove unrelated changes

* update more urls

* update specs

* fix tests

* run `:airbyte-config:specs:generateSeedConnectorSpecs` to fix formatting

* revert docs changes to make pr more reviewable

* revert generator readme changes to make more reviewable

* fix mysql strict encrypt expected spec

* fix postgres expected spec
 2022-10-11 11:04:23 -04:00
Arsen Losenko
35257a0ec1 Source File: specify pyxlsb as engine for Excel file parsing (#15568) 
* Specify engine for excel files

* Add test for the chagne related to pyxlsb

* Update changelog and bump connector version

* Update PR link and number

* Update spec to include 'excel_binary' option, formatted files

* Update spec for source-file-secure to match changes made in source-file spec
 2022-08-26 17:34:16 +03:00
Denys Davydov
81bfb5c350 Source file: fix csv schema discovery (#15870) 
* #174 source file: fix csv schema discovery

* #174 source file: upd changelog

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-08-23 12:38:08 +03:00
Serhii Chvaliuk
e9fa2c4476 🎉 Source File: Convert 'nan' to 'null' (#15768) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-08-20 09:53:36 +03:00
midavadim
f143c8f02a 🎉 Source File - add support for custom encoding (#15293) 
* added support for custom encoding

* fixed unit test for utf16

* updated docs

* bumped connector version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-08-10 21:13:51 +03:00
Marcos Marx
8bbe9939f6 run gradlew format (#14721) 2022-07-14 15:17:12 -03:00
Tim Werner
32e767c9f6 🎉 Source File: add user-agent option (#14488) 
* add user-agent option to source-file

* fix acceptance-test

* set version to 0.0 if version cannot be fetched

* vump connector version

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-07-13 13:46:05 -03:00
Christophe Duong
43f905b8f9 🎉 Add YAML format to source-file reader (#14588) 
* Add yaml reader

* Update docs

* Bumpversion of connector

* bump docs

* Update pyarrow dependency

* Upgrade pandas dependency

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-07-12 19:34:17 +02:00
Marcos Marx
dca2256a7c Bump 2022 license version (#13233) 
* Bump year in license short to 2022

* remove protocol from cdk
 2022-05-26 15:00:42 -03:00
Yevhenii
269298c854 🎉 Source File: Migrate File source to CDK structure (#7387) 
* Migrate File source to CDK structure

* fix .dockerignore file

* remove SAT requirements

* update Dockerfile

* change Dockerfile to base images python:3.7-slim

* add SAT tests

* update tests

* add secret/config.json for source-file

* update changelogs
 2021-10-29 13:15:13 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Marcos Marx
e269c5f976 Test/Bump Version Source File (#5006) 
* Allow SFTP to define port

* Define port as a string

* Provide a clearer error if port parsing fails

* bump source file version

* try to resolve 409 azure

Co-authored-by: Madison Swain-Bowden <bowdenm@spu.edu>
 2021-07-27 19:29:13 -03:00
Davin Chia
cd24ccd064 Release Source File (#3771) 2021-05-31 21:35:57 -07:00
Phlair
9360ed0da9 🎉 File source: Add support for Azure blob storage (#3660) 
Co-authored-by: Davin Chia <davinchia@gmail.com>
 2021-06-01 10:42:13 +08:00
Davin Chia
65c04d6b9e Fix Source-File build. (#3738) 
The main problem seems to be the newer pip resolver isn't happy with our current dependencies. Changing this was enough to fix it.

Got this version by removing the constraint, running the install step, then using the installed version from pip list.
 2021-05-29 22:39:06 +08:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Davin Chia
f660b0a946 Add template generation for Santa aka CDK. (#3034) 
Template generation for new Source using the Santa CDK - provide basic scaffolding for someone implementing a new source.

General approach is to buff up comments in the original SDK, and add TODOs with secondary comments in the generated stub methods, as well as links to existing examples (e.g. Stripe or ExchangeRate api) users can look at.

Checked in and added tests for the generated modules.
 2021-04-25 18:02:33 +08:00
Charles
f445fdb5b2 match styling for spotlessApply and format (#3017) 
* as a java developer I want to be able to run spotlessApply without changing styles in python code
 2021-04-23 09:21:41 -07:00
Christophe Duong
bc1547f185 Fix integration tests of connectors related to destination_sync_mode and sync_mode being required (#2688) 2021-04-01 09:58:38 +02:00
Sherif Nada
77b72dcf5b Revert "Revert "Merge branch 'master' of github.com:airbytehq/airbyte"" 
This reverts commit de66bf010d.
 2021-03-23 15:12:01 -07:00
Sherif Nada
de66bf010d Revert "Merge branch 'master' of github.com:airbytehq/airbyte" 
This reverts commit a3a27f2fa2, reversing
changes made to 5917feee28.
 2021-03-23 15:09:44 -07:00
Sherif A. Nada
41e14e24d3 fix source-file CI (#2563) 2021-03-22 16:00:18 -07:00
Sherif A. Nada
d4267818ae Source File: Support JsonL format (#2118) 2021-02-18 13:54:26 -08:00
vitaliizazmic
caf673dfab Test file formats #1392 (#1768) 
Co-authored-by: Sherif Nada <snadalive@gmail.com>
 2021-02-02 14:49:11 -08:00
Eugene K
e9769c8159 Source File: fix issue with boolean types in schema (#1873) 2021-01-28 23:02:53 -08:00
Eugene K
a54dd89418 Adopt connector best practices for File Source (providers) #1584 (#1738) 
Co-authored-by: Sherif Nada <snadalive@gmail.com>
 2021-01-27 16:49:33 -08:00
Sherif A. Nada
e8a332ae65 Standard source incremental tests (#1175) 2020-12-04 09:54:10 -08:00
Christophe Duong
0d7ca6f424 Fix source file comments from #935 (#944) 
Adjust Source File from manual testings in #935
 2020-11-13 13:34:34 +01:00
Christophe Duong
70f0446677 Adapt Source File connector to JSON files to test normalization (#872) 
Fix spec.json for Source File and prepare JSON loader for nested schema testing
 2020-11-11 00:33:39 +01:00
Charles
95c97cdc7a fix build: remove source-file tests that require creds (#790) 2020-11-02 14:50:54 -08:00