* destination-s3-glue: Fix precision and scale of decimal type
The decimal type in Hive will default to a precision of 10 and a scale of 0, meaning
that it cannot handle numbers larger than 10 digits. The maximum values for precision
and scale are 38 and 38 respectively. This updates the decimal type specified in
destination schemas to use the maximum values for the broadest compatibility with
numeric values.
If a table is using the default precision of 10 and a number is present which exceeds
that many digits then the query engine will throw a `fieldValue cannot be null` error
message which is very cryptic.
* run seed generator
* Automated Change
* Update connectors.md
* Automated Change
---------
Co-authored-by: Tobias Macey <tmacey@mit.edu>
* authentication documentation
* some fixes
* fix broken link
* remove session token auth
* review comments
* first version of tutorial
* Revert "first version of tutorial"
This reverts commit bbdeb997f2.
* first version of tutorial
* review comments
* improvements
* partitioning documentation
* review comments
* improvements
* record processing doc
* review comments
* Define legacy and latest registry at same time
* Fix file persist
* Get persist to work
* Expand generate models to all models
* Add new registry types
* Fix class name
* Get valid registry type definition
* Remove uuid hack
* Fix ids for json schemas
* Resolve issues
* Update legacy assets
* Add typed legacy registry
* Fix icon issue
* Regenerate models
* Update spec mask to use registry type
* Move v1 to v0
* Add json sanitized dict helper
* Fix tests and format
* Ensure we only get latest
* Code review comments
* fix missing spec error
* Move registry code to helper
* Included Cumul.io Destination connector
* add gradle file
* add dockerfile and format files
* Refactor DestinationCumulio class based on review
This commit refactors the DestinationCumulio class by removing some unused dummy variables and commented code that used to push data in the check method.
Next to that I fixed a potential bug as pointed out by @grishick who reviewed this code (thanks for that!). Previously, the state message could have been yielded before any buffers were flushed, which could result in incorrectly indicating a synchronization has been completed while data could still be inside the buffer. Now, all buffers will be flushed before yielding that message.
* Chore: add Cumul.io logo svg to airbyte-config/init/src/main/resources/icons/
* Update two unit tests that were failing due to fixing bug in a previous commit
* fix tests
---------
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
* add offset increment and page increment docs
* add cursor pagination docs
* add full example of requests submitted by connector
* improve examples and explanation of page/cursor increment
* improve descriptions
* add section about accessing last record for cursor pagination
* add recommendation about page size
* add links to low-code and python CDKs
* Fix LSN parsing from Integer to Long
* rebasing
* Rebase
* Rebase
* Other casting
* Lock the file only when reading, so the file is free when parsing the object.
Increased from 1 to 166 checkpoints, and from skipping hundreds of checkpoints to never skip a state.
* Update load function documentation
* bump mysql and mssql
* cdc: refactor to remove debezium dependency from connector packages
* use gradle's shared dependency
* more refactoring
* upgrade docker version
* resolve master merge conflicts
* Automated Change
* minor changes
* resolve merge conflicts
* avoid deserializing multiple times
* simplify
* enable checkpointing for Postgres
* more improvements
* enable assertions
* changelog + bump version
* auto-bump connector version
* auto-bump connector version
* manual bump
---------
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fixed schema type for "data_type: ENUM" and "is_repeated: true" to array of strings
* added SAT for GAQL
* updated version
* Added unittest for get_json_schema
* slight enhancement
* fixed empty lines
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* authentication documentation
* some fixes
* fix broken link
* remove session token auth
* review comments
* first version of tutorial
* Revert "first version of tutorial"
This reverts commit bbdeb997f2.
* first version of tutorial
* review comments
* improvements
* improvements
* review comments
* rename catalog to registry in metadata service
* rename catalog to registry in metadata files
* Run generate models
* Fix missed renames
* Add github personal access token
* Run black
* Automated Change
---------
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
* Pass argument along, add test that should pass and test that should fail
* Add tests with additionalProperties
* Set additionalproperties=false when not set|
* Parametrize test cases
* Make the behavior 'optional'
* Fix parametrization for all combinations
* Improve CI credentials README and rename param
* Update naming to be clearer about columns only
* record_has_unexpected_field > record_has_unexpected_column
* Automated Change
* Hacking the CAT dockerfile and run script to test my changes specifically
* First crack at running CAT on all connectors
* Write during instead of after all tests
* Async-ify it
* Add ability to define max concurrency
* Write successes
* ci_credentials: fix overwriting 'data' before getting nextPageToken
* Adjustible num_semaphores, check to make sure it's an airbyte connector first
* Automated Change
* Make create_issues and create_prs more configurable, add issue for fail_on_extra_columns
* Add ability to pass in sources as a list or from a txt file
* Add logs to issue, make project nullable
* Migrate multiple connectors
* Add cli args
* use ruamel.yaml to preserve ordering
* Separate config loading from config migration
* Add ability to pass in lists of sources to test. Sort output by exit codes. Fix max_concurrency flag
* Default to testing only beta and GA connectors
* Always write test output when available
* Revert "Add cli args"
This reverts commit b538a8c696.
* Remove slash
* Don't run on alpha connetors, handle older config style
* Don't migrate to new format, preserve quotes and long lines
* Automated Change
* Update issue, don't run for alpha connectors
* Automated Change
* Add bypass for extra fields test
* Add bypass for extra fields test
* Rename run_tests script
* Rename module
* Update args usage, small changes
* Refactor create_issues.py
* Clean up run_tests.py
* Sort out arg parsers
* Pull out get_valid_definitions_from_args
* Import definitions module instead of methods
* Use config files to provide constants for each migration
* Handle FileNotFoundError in create_issues.py, improve logging
* Rename to migrations, reference name of folder via utils
* Update readmes for migration modules, add script for getting outputs
* Use tmp dir, correct path for issue reference
* Fix bash script
* Fix create command, pull out test results insertion
* Update call to update_configuration
* add precommit to requirements
* Reorder README
* README cleanup for test and create issues
* README cleanup for create_prs and config_migration
* More readmes! Readmes galore
* allow_beta
* Restore hacky changes to dockerfile and acceptance-test-docker
* Handle 'other' release stages
* Update readme
* Remove TODO, add comments to shell script
* format according to gradle
* format
* Fix formatting
---------
Co-authored-by: marcosmarxm <marcosmarxm@users.noreply.github.com>
Co-authored-by: erohmensing <erohmensing@users.noreply.github.com>
