1
0
mirror of synced 2025-12-21 02:51:29 -05:00
Commit Graph

22 Commits

Author SHA1 Message Date
Serhii Chvaliuk
e9fa2c4476 🎉 Source File: Convert 'nan' to 'null' (#15768)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-08-20 09:53:36 +03:00
Serhii Chvaliuk
f4adef7927 🎉 Source File: cache binary stream to file for discover (#15698)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-08-17 15:19:30 +03:00
Serhii Chvaliuk
5eef51ae15 🎉 Source File: cache binary stream to file (#15501)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-08-11 11:56:53 +03:00
midavadim
f143c8f02a 🎉 Source File - add support for custom encoding (#15293)
* added support for custom encoding

* fixed unit test for utf16

* updated docs

* bumped connector version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-10 21:13:51 +03:00
Serhii Chvaliuk
35125f8846 🎉 Source File: bump setup.py versions (#15269)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-08-06 12:28:03 +03:00
Christophe Duong
25ee6f6c37 bump source-file-secure (#14704)
* bump source-file-secure

* Bump more

* bump requirements

* Fix source-file

* auto-bump connector version

* correct unit test spec for file secure + format files

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2022-07-15 09:58:52 +02:00
Tim Werner
32e767c9f6 🎉 Source File: add user-agent option (#14488)
* add user-agent option to source-file

* fix acceptance-test

* set version to 0.0 if version cannot be fetched

* vump connector version

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-13 13:46:05 -03:00
Denis Davydov
2570f9ce73 Source file: remove nested properties (#14535)
* #13787 source file: remove nested properties

* #13787: upd changelog

* #13787 revert processing json array, add unit test

* #13787 source file: upgrade pyarrow to allow builds on aarch64

* #13787 source file: upd version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-13 11:29:51 +03:00
Christophe Duong
43f905b8f9 🎉 Add YAML format to source-file reader (#14588)
* Add yaml reader

* Update docs

* Bumpversion of connector

* bump docs

* Update pyarrow dependency

* Upgrade pandas dependency

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-12 19:34:17 +02:00
Marcos Marx
dca2256a7c Bump 2022 license version (#13233)
* Bump year in license short to 2022

* remove protocol from cdk
2022-05-26 15:00:42 -03:00
Yevhenii
269298c854 🎉 Source File: Migrate File source to CDK structure (#7387)
* Migrate File source to CDK structure

* fix .dockerignore file

* remove SAT requirements

* update Dockerfile

* change Dockerfile to base images python:3.7-slim

* add SAT tests

* update tests

* add secret/config.json for source-file

* update changelogs
2021-10-29 13:15:13 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Madison Swain-Bowden
2d5d7e8e98 🎉 Allow non-default source SFTP port (#4953)
* Allow SFTP to define port

* Define port as a string

* Provide a clearer error if port parsing fails
2021-07-27 18:52:55 -03:00
Davin Chia
cd24ccd064 Release Source File (#3771) 2021-05-31 21:35:57 -07:00
Phlair
9360ed0da9 🎉 File source: Add support for Azure blob storage (#3660)
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-06-01 10:42:13 +08:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Davin Chia
f660b0a946 Add template generation for Santa aka CDK. (#3034)
Template generation for new Source using the Santa CDK - provide basic scaffolding for someone implementing a new source.

General approach is to buff up comments in the original SDK, and add TODOs with secondary comments in the generated stub methods, as well as links to existing examples (e.g. Stripe or ExchangeRate api) users can look at.

Checked in and added tests for the generated modules.
2021-04-25 18:02:33 +08:00
Charles
f445fdb5b2 match styling for spotlessApply and format (#3017)
* as a java developer I want to be able to run spotlessApply without changing styles in python code
2021-04-23 09:21:41 -07:00
oleh.zorenko
df80593108 csv discovery takes too much memory (#2089) (#2883)
* csv discovery takes too much memory (#2089)

Use `chunksize` parameter when reading csv files.

* csv discovery takes too much memory (#2089)

Handle `None` value for `fields` variable in `client.read()` function.

* csv discovery takes too much memory (#2089)

Refactor code in `client.read()` function. Remove duplicate code.

* csv discovery takes too much memory (#2089)

Implement version with yield statements.

* csv discovery takes too much memory (#2089)

Update return annotation for load_dataframes().

* csv discovery takes too much memory (#2089)

Remove unused import.

* csv discovery takes too much memory (#2089)

Update source-file connector version from `0.2.1` to `0.2.2`.
2021-04-16 19:51:53 +03:00
Sherif A. Nada
d4267818ae Source File: Support JsonL format (#2118) 2021-02-18 13:54:26 -08:00
Eugene K
e9769c8159 Source File: fix issue with boolean types in schema (#1873) 2021-01-28 23:02:53 -08:00
Eugene K
a54dd89418 Adopt connector best practices for File Source (providers) #1584 (#1738)
Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-01-27 16:49:33 -08:00