George Claireaux
|
d9f11bcf6a
|
🎉 New Source: S3 (+ abstract files source) (#4990)
* minor line length changes
* cdk generated source + oop structure + start of implementation
* fixed some broken syntax stuff
* pre-pyarrow convert
* introducing pyarrow
* skeleton for unit tests
* read working on multiple files
* incremental first draft
* blobfile -> fileclient
* change references of 'blob' to 'file'
* minor tidy to make draft PR
* fixes
* addressed review comments + more unit tests
* finished unit tests
* bugfixes and abstract integration tests framework
* remove old commented stuff
* docstrings
* restructure as source-s3
* Delete playground.py
* integration tests
* acceptance tests and some more reshuffling
* source S3 credentials
* change _airbyte_ columns to _ab_
* update spec with better descriptions and ordering
* created s3 source docs
* source definition
* reverse docstring change in cdk
* reverse docstring change
* reverse change
* reverse docstring change
* remove TODO comments
* add PR to changelog
* removed unused libraries
* formatting & address some review comments
* rename of files/classes for clarity
* addressing review comments
* address reviews
* add s3 source
* building spec with pydantic for provider-specific inheritance
* pydantic spec and improved path pattern with wcmatch.glob
* update path patterns info in doc
* formatting
* tests gzip and bz2 compression on csv
* updated compression support in doc
* forgot to upload bz2 test file
* added pattern validation to dataset
* formatting
* Format.
* ran testScaffoldTemplates & generated this diff
* bumped version because of documentationUrl fix
Co-authored-by: Davin Chia <davinchia@gmail.com>
|
2021-07-30 15:06:11 +01:00 |
|