* json format support added
* json format support added
* code formatted
* format convertion changed
* format naming convertion changed
* test cased issue fixed
* test case issued resolved
* sample file and config added for integration tests
* Json doc added
Json doc added
* update
* sample file and config added for integration tests
* sample file and config added for integration tests
* update jsonl files
* review 1
* review 1
* review 1
* pyarrow version upgrade
* clean integration test folder architecture
* add timestamp record to simple_test.jsonl
* fixed integration test and parser review change
* simplify table read
* doc update
* fix specs
* user sample files
* fix sample files
* add newlines at end of files
* rename json parser
* rename jsonfile to jsonlfile
* schema inference added
* patch review fix
* Update docs/integrations/sources/s3.md
doc update
Co-authored-by: George Claireaux <george@airbyte.io>
* changing the version
* changing the title to sync with other type
* fix expected csv records
* fix expected records for avro and parquet
* review fix
* fixed master schema handling
* remove sample configs
* fix expected records
* json doc update
added more details on json parser
* fixed api name
* bump version
* auto-bump connector version [ci skip]
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: George Claireaux <george@airbyte.io>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Added history to state
* Deleted unused import
* Rollback abnormal state file
* Rollback abnormal state file
* Fixed type error issue
* Fix state issue
* Updated after review
* Bumped version
* added MVP avro parser running fine locally
* added unit tests for avro
* added wip state of avro integration test setup
* deleted unused files
* added avro specific config path
* fixed comments. Added nested record support, simplify code and minor fixes
* bumped version + docs update
* Added working acceptance tests + format
* auto-bump connector version
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Clean up and add better guidelines on how to use the Java catalogs we recently added.
Took the chance to move existing documentation to improve reading flow.
* memory & performance optimisations
* address comments
* version bump
* added advanced_options for reading csv without header, and more custom pyarrow ReadOptions
* updated to use the latest airbyte-cdk
* updated docs
* bump source-s3 to 0.1.6
* remove unneeded lines
* Use the all dep ami for python builds.
* ec2-instance-id should be ec2-image-id
* ec2-instance-id should be ec2-image-id
Co-authored-by: Jingkun Zhuang <Jingkun.Zhuang@icims.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
* infer schema in multi process
* use dill to pickle function
* moved funcs
* Revert "moved funcs"
This reverts commit c1739ad988.
* Revert "use dill to pickle function"
This reverts commit 52404a9f1b.
* Revert "infer schema in multi process"
This reverts commit f0fb6f66f9.
* multiprocess in csv schema iinfer
* simplify what happens in the multiprocess to offending code
* try this
* using tempfile
* formatting
* version bump
* changelog + formatting
* addressed review comments
* re-trigger checks
* ran testScaffoldTemplates to fix breaking check
