* map the other integer schema to long
* fix test + add test
* delete_public_access_block for bucket if public (#25663)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
* 🐛 Source Facebook Marketing: fix `expected records` for CAT (#25604)
* publish normalization (#25591)
* publish normalization
* bump normalization container version in all the destinations that use it
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: edgao <edgao@users.noreply.github.com>
* Bump Airbyte version from 0.44.2 to 0.44.3
* Destination Bigquery: update AIRBYTE_ENTRYPOINT env var for kube process (#25588)
* add AIRBYTE_ENTRYPOINT env var for kube
* amazing, absolute genius
* version bump + changelog
* derp, no need to publish denormalizeid
* fix changelog entry
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* version bumps + changelog
* whoops
* bump metadata
* bump metadatas
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
---------
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Jeff Cowan (Airbyte) <4992320+jcowanpdx@users.noreply.github.com>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Removes defunct Azure Blob Storage laoding option for Snowflake
* Bumps to major version and removes documentation that references AzureBlobStorage
* Updates the destination_definitions.yaml
* Run ProcessResources to match version of 1.0.0 mismatched spec
* Pinning urllib to older version since the 2.0 version removed classes
* destination-s3-glue: Fix decimal type syntax
The definition of the decimal type as `decimal(38,38)` was incorrect as it caused the
query engine to interpret that as meaning that all 38 digits had to be to the right of
the decimal place, so any whole number values would overflow. Setting it as
`decimal(38)` allows for the full 38 digits, but with flexible scale.
* auto-bump connector version
---------
Co-authored-by: Tobias Macey <tmacey@mit.edu>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add AIRBYTE_ENTRYPOINT env var for kube
* amazing, absolute genius
* version bump + changelog
* derp, no need to publish denormalizeid
* fix changelog entry
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* super hacky start
* also check that we're writing
* v0 convert normalization logs to airbytemessage
* add start+end logs
* aggregate errors into a single trace?
* pipefail; quick tweaks to log parser
* make spotbugs happy
* more comments, uncomment env var check
* copy in SentryExceptionHelper
* final fixes
* write tests + fix bugs
* move to base-java
* remove outdated comment
* fix spotbugs
* Automated Change
* minor version bump
* changelog
* fix behavior when env var not set
* run normalization even if destination fails
* better logic
* better logging
* oops
* move to base-java
* rebump version
* Automated Change
* auto-bump connector version
* wtf how did this work previously
* auto-bump connector version
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* destination-s3-glue: Fix precision and scale of decimal type
The decimal type in Hive will default to a precision of 10 and a scale of 0, meaning
that it cannot handle numbers larger than 10 digits. The maximum values for precision
and scale are 38 and 38 respectively. This updates the decimal type specified in
destination schemas to use the maximum values for the broadest compatibility with
numeric values.
If a table is using the default precision of 10 and a number is present which exceeds
that many digits then the query engine will throw a `fieldValue cannot be null` error
message which is very cryptic.
* run seed generator
* Automated Change
* Update connectors.md
* Automated Change
---------
Co-authored-by: Tobias Macey <tmacey@mit.edu>
* Included Cumul.io Destination connector
* add gradle file
* add dockerfile and format files
* Refactor DestinationCumulio class based on review
This commit refactors the DestinationCumulio class by removing some unused dummy variables and commented code that used to push data in the check method.
Next to that I fixed a potential bug as pointed out by @grishick who reviewed this code (thanks for that!). Previously, the state message could have been yielded before any buffers were flushed, which could result in incorrectly indicating a synchronization has been completed while data could still be inside the buffer. Now, all buffers will be flushed before yielding that message.
* Chore: add Cumul.io logo svg to airbyte-config/init/src/main/resources/icons/
* Update two unit tests that were failing due to fixing bug in a previous commit
* fix tests
---------
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
* destination-s3-glue: Use new configuration for JSONL flattening
In fafffc4607 the
configuration interface for flattening JSONL data in the S3 code changed from a boolean
to an enum. This updates the s3-glue destination connector to use that new configuration
syntax so that it gets passed through to the underlying `SerializedBufferFactory` class
properly.
* bump connector version
* Update destination_definitions.yaml
* auto-bump connector version
---------
Co-authored-by: Tobias Macey <tmacey@mit.edu>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* use standard retry mode
* Automated Change
* dest-s3 version bump + changelog
* also in redhsift + snowflake
* auto-bump connector version
* version bumps
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* snowflake-destination: allow buffer to be increased till 50
* version bump
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* destination-s3-glue: Fix struct and array schema generation
The `struct` schema type was being generated with invalid syntax due to the inclusion of
space characters around the `:` character in the type mapping of the struct field.
The `array` data type was not properly handling the optionality of the `items`
property. This lead to hard failures whenever dealing with array types that didn't set
the `items` property. This adds conditional handling of when the `items` type is present
and otherwise assumes that the type is a string.
* rollback change
* bump connector version
* return version to 0.1.3
* auto-bump connector version
---------
Co-authored-by: Tobias Macey <tmacey@mit.edu>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* always show Service Account Key JSON in BigQuery destination
* update dockerfiles and readmes
* auto-bump connector version
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* destination-s3-glue: Fix data type for emitted at column
The `_ab_emitted_at` column is actually a numeric type and not a string. This updates
the generated schema to treat it as the proper type.
* fix tests
* rollback seed dest spec update
* auto-bump connector version
---------
Co-authored-by: Tobias Macey <tmacey@mit.edu>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Added support for periodic buffer flush with tests and uses env variable
* Improves code readability and encapulates testing logic
* Removed demo changes and created const for tests
* Updated constructor to reuse method signature
* Increases Snowflake parallel integration forks
* Bumps version number, fixes linting issues and constant format
* Generate seed spec
* Redshift Destination: Add SSH Tunnelling Config Option
Issue: https://github.com/airbytehq/airbyte/issues/12131
User would like to be able to access their Redshift Destination database via an SSH Tunnelling config (SSH Bastion Host), just as in Postgres, etc. destinations.
Acceptance tests pass and exercise both Insert and S3 type loading options as well as password and key based authentication with the SSH bastion host.
Airbyters: Credentials/configs for acceptance testing against AWS Redshift are updated in Google Cloud Secret Manager. Be sure to get both the config.json and config_staging.json as well as the new PEM file
* README.md updates
** Move the pem info into the secrets config files
** Update the version id to 0.4.0
** Touch up the docs
** adding a note about S3 staging NOT using the SSH Tunnel configuration, if provided.
** make base class abstract to prevent double running the same tests
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Added SSL support for teradata connector.
Modified database size from 1GB to 120 MB for schema creating from Airbyte.
* Update TeradataDestinationSSLAcceptanceTest.java
PR Review comments addressed.
* Update TeradataDestination.java
Removed unnecessary log statement
* Deleting/Droping all tables created as part of integration testing in teardown method.
* bump version
* auto-bump connector version
---------
Co-authored-by: SatishChGit <satishchinthanippu@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Support any string based ID
* Previously string based IDs that weren't convertable to hex would
throw an exception. This fixes that issue by converting strings
to 128 bit integer and converting the md5 hex into a UUID
* Update Weaviate to 1.17.3
* Fix issue with arrays of no data type
Closes#22530
* add more testing coverage
* fix error where dataType wasn't set in weaviate
This seems to happen when records sometimes don't contain all the
properties that are defined in the schema.
This was the error observed by a user connecting Slack <> Weaviate:
property 'reactions': invalid dataType: dataType must have at least
one element, class 'Threads' not present in schema"
* add test case for additionalProperties
* Support additionalProps of object and array of obj
* Update changelog
* add weaviate icon
* fix flake tests
* add commit
* make build m1 work
* make build m1 work
* auto-bump connector version
---------
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
* Adds additional logging when flushing buffer and writing records
* Removes logging for writeRecord since this will explode log lines
* Added logging when uploading records to stage/bucket
* Fixes log lines to properly capture when records have been uploaded
* Bumps version and fixes logging message to more accurately reflect logic
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
