George Claireaux d9f11bcf6a 🎉 New Source: S3 (+ abstract files source) (#4990) ...

* minor line length changes

* cdk generated source + oop structure + start of implementation

* fixed some broken syntax stuff

* pre-pyarrow convert

* introducing pyarrow

* skeleton for unit tests

* read working on multiple files

* incremental first draft

* blobfile -> fileclient

* change references of 'blob' to 'file'

* minor tidy to make draft PR

* fixes

* addressed review comments + more unit tests

* finished unit tests

* bugfixes and abstract integration tests framework

* remove old commented stuff

* docstrings

* restructure as source-s3

* Delete playground.py

* integration tests

* acceptance tests and some more reshuffling

* source S3 credentials

* change _airbyte_ columns to _ab_

* update spec with better descriptions and ordering

* created s3 source docs

* source definition

* reverse docstring change in cdk

* reverse docstring change

* reverse change

* reverse docstring change

* remove TODO comments

* add PR to changelog

* removed unused libraries

* formatting & address some review comments

* rename of files/classes for clarity

* addressing review comments

* address reviews

* add s3 source

* building spec with pydantic for provider-specific inheritance

* pydantic spec and improved path pattern with wcmatch.glob

* update path patterns info in doc

* formatting

* tests gzip and bz2 compression on csv

* updated compression support in doc

* forgot to upload bz2 test file

* added pattern validation to dataset

* formatting

* Format.

* ran testScaffoldTemplates & generated this diff

* bumped version because of documentationUrl fix

Co-authored-by: Davin Chia <davinchia@gmail.com>

2021-07-30 15:06:11 +01:00

0 lines

0 B

Python

Raw Blame History

The file is empty.