d9f11bcf6a
* minor line length changes * cdk generated source + oop structure + start of implementation * fixed some broken syntax stuff * pre-pyarrow convert * introducing pyarrow * skeleton for unit tests * read working on multiple files * incremental first draft * blobfile -> fileclient * change references of 'blob' to 'file' * minor tidy to make draft PR * fixes * addressed review comments + more unit tests * finished unit tests * bugfixes and abstract integration tests framework * remove old commented stuff * docstrings * restructure as source-s3 * Delete playground.py * integration tests * acceptance tests and some more reshuffling * source S3 credentials * change _airbyte_ columns to _ab_ * update spec with better descriptions and ordering * created s3 source docs * source definition * reverse docstring change in cdk * reverse docstring change * reverse change * reverse docstring change * remove TODO comments * add PR to changelog * removed unused libraries * formatting & address some review comments * rename of files/classes for clarity * addressing review comments * address reviews * add s3 source * building spec with pydantic for provider-specific inheritance * pydantic spec and improved path pattern with wcmatch.glob * update path patterns info in doc * formatting * tests gzip and bz2 compression on csv * updated compression support in doc * forgot to upload bz2 test file * added pattern validation to dataset * formatting * Format. * ran testScaffoldTemplates & generated this diff * bumped version because of documentationUrl fix Co-authored-by: Davin Chia <davinchia@gmail.com>
14 lines
279 B
JSON
14 lines
279 B
JSON
{
|
|
"dataset": "test",
|
|
"provider": {
|
|
"storage": "S3",
|
|
"bucket": "THIS BUCKET DOESN'T EXIST",
|
|
"aws_access_key_id": "KEY ID",
|
|
"aws_secret_access_key": "KEY",
|
|
"path_prefix": ""
|
|
},
|
|
"format": { "filetype": "csv" },
|
|
"path_pattern": "**",
|
|
"schema": "{}"
|
|
}
|