* Add Xata.io Destination Connector
* readme for connectir
* adjust versioning
* missing state propagation
* throw exception if inconsistentcy is found
* set version in dockerfile
* exchange mock for deprecated logger
* put invalid config to file
* changelog in docs
* add xata dest connector to builds.md
* bump dependency to newer sdk version
* bump to latest sdk version
* blank bootstrap
* use class logger
* correct way to use the logger
* unit tests
* add write integration test
* removed irrelevant log line
* raise exception on incorrect message
* e2e test for writing
* add metadata file
* remove unued file
* remove build md
* update metdata and docs
* fix formatting
---------
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
* timeplus destination connector to push data to Timeplus via REST API/SDK
* 0.2.0, support nullable columns, add more test data
boolean and pokemon
* mark the apikey as secret
* us.timeplus.cloud as sample URL
* removed the unnecessary pass
* rewrite the logic of when to delete/create stream
* format code, following the PEP-8
* add write integration test
* replacing AirbyteLogger() with logging.getLogger('airbyte')
* add logs to destination.py, refine integration test
* add docs/integrations/destinations/timeplus.md
* add icon.svg
* set to 0.4.0 version
* upgrade timeplus sdk from 1.1.1 to 1.2.1
* add metadata file
* fix docker version
---------
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
* auto-generated destination connector template
* add config and health check
* support full_refresh overwrite
* support full refresh overwrite, full refresh append, and incremental append
* write works!
* lint
* update readme
* add test messages
* list destination??
* surface error messages
* add integration test
* add destination docs
* update source docs
* change integration test to unit test
* add check unit test
* use a map for streams
* make compatible with API changes
* add indexes for primary keys
* poll for indexes to be backfilled
* simplified stream API
* support append_dedup sync mode
* cleanup
* add changelog
* fix changelog
* use indexes ready endpoint
* update types
* rename to include primary key in API, handle namespaces
* fix types
* update docs
* add better test cases
* fix docs link
* update API and headers
* fix formatting
* update API name to streaming_import
* improve request exception and add a test for error formatting
* remove secret config
* use /replace_tables endpoint instead of /clear_tables
* partial revert of replace_tables
* fix
* formatting and fix table names
* update docs
* update metadata and dockerfile
---------
Co-authored-by: Lee Danilek <lee@convex.dev>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
* Revert "Splits bases and updates build.gradle files (#25649)"
This reverts commit c673b0a692.
* Bumps branch to prevent a conflict with publishing
* Forward fixes Snowflake to use singular base-java and develop within a new package within the same module
* Forcing automated change to merge changes
* try this?
* fix tests
* assert cdc values
* handle case where we have lsn but no updated_at
* readability improvements
* tweaks to test
* version bumps + changelogs
* Automated Change
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
* add info blurb to Cloud Bucket Storage sources and destinations
* Apply suggestions from code review
Remove extra colon
Co-authored-by: Ben Church <ben@airbyte.io>
---------
Co-authored-by: Ben Church <ben@airbyte.io>
* Splits bases and updates build.gradle files
* Fixed changelog out of sync
* Bumps version number and metadata files
* auto-bump connector version
* Downgraded untouched connector bumps
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* map the other integer schema to long
* fix test + add test
* delete_public_access_block for bucket if public (#25663)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
* 🐛 Source Facebook Marketing: fix `expected records` for CAT (#25604)
* publish normalization (#25591)
* publish normalization
* bump normalization container version in all the destinations that use it
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: edgao <edgao@users.noreply.github.com>
* Bump Airbyte version from 0.44.2 to 0.44.3
* Destination Bigquery: update AIRBYTE_ENTRYPOINT env var for kube process (#25588)
* add AIRBYTE_ENTRYPOINT env var for kube
* amazing, absolute genius
* version bump + changelog
* derp, no need to publish denormalizeid
* fix changelog entry
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* version bumps + changelog
* whoops
* bump metadata
* bump metadatas
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
---------
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Jeff Cowan (Airbyte) <4992320+jcowanpdx@users.noreply.github.com>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Removes defunct Azure Blob Storage laoding option for Snowflake
* Bumps to major version and removes documentation that references AzureBlobStorage
* Updates the destination_definitions.yaml
* Run ProcessResources to match version of 1.0.0 mismatched spec
* Pinning urllib to older version since the 2.0 version removed classes
* destination-s3-glue: Fix decimal type syntax
The definition of the decimal type as `decimal(38,38)` was incorrect as it caused the
query engine to interpret that as meaning that all 38 digits had to be to the right of
the decimal place, so any whole number values would overflow. Setting it as
`decimal(38)` allows for the full 38 digits, but with flexible scale.
* auto-bump connector version
---------
Co-authored-by: Tobias Macey <tmacey@mit.edu>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add AIRBYTE_ENTRYPOINT env var for kube
* amazing, absolute genius
* version bump + changelog
* derp, no need to publish denormalizeid
* fix changelog entry
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* super hacky start
* also check that we're writing
* v0 convert normalization logs to airbytemessage
* add start+end logs
* aggregate errors into a single trace?
* pipefail; quick tweaks to log parser
* make spotbugs happy
* more comments, uncomment env var check
* copy in SentryExceptionHelper
* final fixes
* write tests + fix bugs
* move to base-java
* remove outdated comment
* fix spotbugs
* Automated Change
* minor version bump
* changelog
* fix behavior when env var not set
* run normalization even if destination fails
* better logic
* better logging
* oops
* move to base-java
* rebump version
* Automated Change
* auto-bump connector version
* wtf how did this work previously
* auto-bump connector version
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* destination-s3-glue: Fix precision and scale of decimal type
The decimal type in Hive will default to a precision of 10 and a scale of 0, meaning
that it cannot handle numbers larger than 10 digits. The maximum values for precision
and scale are 38 and 38 respectively. This updates the decimal type specified in
destination schemas to use the maximum values for the broadest compatibility with
numeric values.
If a table is using the default precision of 10 and a number is present which exceeds
that many digits then the query engine will throw a `fieldValue cannot be null` error
message which is very cryptic.
* run seed generator
* Automated Change
* Update connectors.md
* Automated Change
---------
Co-authored-by: Tobias Macey <tmacey@mit.edu>
* Included Cumul.io Destination connector
* add gradle file
* add dockerfile and format files
* Refactor DestinationCumulio class based on review
This commit refactors the DestinationCumulio class by removing some unused dummy variables and commented code that used to push data in the check method.
Next to that I fixed a potential bug as pointed out by @grishick who reviewed this code (thanks for that!). Previously, the state message could have been yielded before any buffers were flushed, which could result in incorrectly indicating a synchronization has been completed while data could still be inside the buffer. Now, all buffers will be flushed before yielding that message.
* Chore: add Cumul.io logo svg to airbyte-config/init/src/main/resources/icons/
* Update two unit tests that were failing due to fixing bug in a previous commit
* fix tests
---------
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
* destination-s3-glue: Use new configuration for JSONL flattening
In fafffc4607 the
configuration interface for flattening JSONL data in the S3 code changed from a boolean
to an enum. This updates the s3-glue destination connector to use that new configuration
syntax so that it gets passed through to the underlying `SerializedBufferFactory` class
properly.
* bump connector version
* Update destination_definitions.yaml
* auto-bump connector version
---------
Co-authored-by: Tobias Macey <tmacey@mit.edu>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* use standard retry mode
* Automated Change
* dest-s3 version bump + changelog
* also in redhsift + snowflake
* auto-bump connector version
* version bumps
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* snowflake-destination: allow buffer to be increased till 50
* version bump
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
