* rename for clarity
* fix cleanup method
* giant commit because I'm irresponsible
* rename constant
* better raw table creation
* fix build?
* move code around
* tweaks
* more code shuffling
* Automated Commit - Format and Process Resources Changes
* add tests
* minor tweak
* remove unimportant methods
* cleanup
* Automated Commit - Format and Process Resources Changes
* derp
* clean up tests
* some more fixes post-merge
* botched merge
* create NoopTyperDeduper
* try and update everything to work?
* tweak comment
* move suffix args to end of list
* fix exception message
* Automated Commit - Format and Process Resources Changes
* add sqlgenerator test for softReset
* only prepare once
* update log message
* do what intellij says
* implement one more test
* less indirection
* Automated Commit - Format and Process Resources Changes
* rename test
* use noop in test
* version bump + changelog
* use stringutils
* fix typo
* flip if-statement
* typo
* simplify logic
* fix schema change logic
* typo
* use spy for clarity
* Automated Commit - Format and Process Resources Changes
* better test teardown
* slightly better logs
* fix exception message
* softReset returns single string
* Automated Commit - Format and Process Resources Changes
* simplify if chain
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
* Add ability to detect differences in expected Schemas and perform soft resets
* Remove alter table for overwrite syncs since its unneccessary
* Updates after testing
* pr reorganize
* comments
* add collection util test
* Add Tests
* bump version
* Automated Commit - Format and Process Resources Changes
* Destination BigQuery - Reduce amount of typing and deduping for GCS staging (#28489)
* undo comment out
* centralize t&d logic for staging and standard, add valve to staging
* Share more logic for typing and deduping
* Remove record checking logic and use only time for staging inserts
* Add Javadoc
* Automated Commit - Format and Process Resources Changes
---------
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
* Change TableNotMigratedException to extend runtime exception, remove SqlGenerator interface method
* Make Lambda slightly more readable
* add test for validating v2 schemas
* change soft reset to single string
* convert back to list, update dockerfile
* remove needless default
---------
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
* `destination-redshift` will fail syncs if records or properties are too large, rather than silently skipping records and succeding
* Bump version
* remove tests that don't matter any more
* more test removal
* more test removal
---------
Co-authored-by: Augustin <augustin@airbyte.io>
* basic version
* polish
* iterate
* keep working
* fix spec
* wip
* improve destination
* basic unit tests
* move embedding dimensionality into embedder
* improve several things
* adjust documentation
* remove unnecessary call
* add some debug information
* fix local destination
* various small fixes
* bring tests into order
* document and add batching to pinecone
* checklist
* improve performance a bit and add test
* fix formatting
* fix metadata
* install C++ 11 on python base
* no more alpine for ci-connector-ops
* remove hard-to-run test
* more documentation
* better documentation
* add icon
* some small adjustments
* review comments
* format
* review comments
---------
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
* Adds data as JsonNode to pass through, running into memory issues so add JVM args to attach VisualVM
* JsonNode
* Lowers the optimal batch size to see if this improvements movement
* Fixes NPE by checking if PartialAirbyteMessage contains a PartialAirbyteRecord
* Fixes config switch when config is not explicitly set (config migration needed)
* Adds logic to check if queue has elements before getting timeOfLastMessage
* Add PartialSerialisedMessage test. (#27452)
* Test deserialise.
* Add tests.
* Simplify and fix tests.
* Format.
* Adds tests for deserializeAirbyteMessage
* Adds tests for deserializeAirbyteMessage with bad data
* Cleans up deserializeAirbyteMessage and throws Exception when invalid message
* More code cleanup
---------
Co-authored-by: ryankfu <ryan.fu@airbyte.io>
* 🤖 Auto format destination-snowflake code [skip ci]
* Cleans up code w/o JVM args & rebase
* 🤖 Auto format destination-snowflake code [skip ci]
* Adds breadcrumb on the STATE message deviation and where the deserialize/serialize is done to unpack
* 🤖 Auto format destination-snowflake code [skip ci]
* Adds back line formatter removed and comment describing rational for lower batchSize
* Bumps Snowflake version and type checks
* Added note to remove PartialAirbyteRecordMessage with low resource testing
* Automated Commit - Format and Process Resources Changes
* Fix issue with multiple namespaces in snowflake not writing to the correct staging schema
* Fix issue with multiple namespaces in snowflake not writing to the correct staging schema
* remove stage name maniuplating method
* update readme
* Source Stripe: update credit_notes expected records (#27941)
* Source Zendesk Talk: update expected records (#27942)
* Source Xero: update expected records (#27943)
* Metadata: Persist Registry entries (#27766)
* DNC
* Update poetry
* Update dagster
* Apply partition
* Get metadata entry
* Use helpers
* Write registry entry to appropriate location
* Delete when registry removed
* Update to use new file (broken)
* Render registry from registry entries
* Run format
* Fix plural issue
* Update to all metadata file blobs
* Fix test
* Update to all blobs
* Add ignore validation error for version logic
* Rename to max_run_request
* Pedros review
* Ella suggestions
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
* Update airbyte-ci/connectors/metadata_service/orchestrator/orchestrator/assets/registry_entry.py
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
* Update naming
* Add tests for connector type and deletion
* Test safe parse
* Format
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
* Fix Dagster Deploy Failure (#27955)
* Add pydantic
* Add pydantic to orchestration deploy pipeline
* 🐛 Source Jira: update expected records (#27951)
* Source Jira: update expected records
* Update issues expected records
* Source Zendesk Chat: update expected records (#27965)
* 🐛 Source Pipedrive: update expected records (#27967)
* 🐛 Source Pinterest: update expected records (#27964)
* ✨ Source Amazon-Ads: Add streams for portfolios and sponsored brands v3 (#27607)
* Add stream for sponsored brands v3
* Add new stream Portfolios
* Source Google Search Console: added discover and googleNews to searchType (#27952)
* added discover and googleNews to searchType
* updated changelog
* fixed types for streams
* 🎉 Source Instagram: Improve, refactor `STATE` management (#27908)
* add test for enabling
* update versions
* fix test
* update other snowflake loading method types
* remove standard
---------
Co-authored-by: ryankfu <ryan.fu@airbyte.io>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: ryankfu <ryankfu@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Arsen Losenko <20901439+arsenlosenko@users.noreply.github.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
* copy files from edgao branch
* start writing create table statement
* add basic unit test setup
* create a table, probably
* remove outdated todo
* derp, one more column
* ugh
* add partitioning+clustering
* use StringSubstitutor
* substitutions in updateTable
* wip generate update/insert statement
* split up into smaller methods
* handle json types correctly
* rename stuff
* more json_query vs _value stuff
* minor tweak
* super basic test setup
* laying foundation for type parsing
* more stuff
* tweaks
* more progress on type parsing
* fix json_value stuff?
* misc fixes in insert
* fix dedupFinalTable
* add testDedupRaw
* full e2e test
* type parsing: gave up and mirrored the dbt code structure to avoid bugs
* type parsing - more cleanup
* handle column name collisions
* handle tablename collisions...?
* comments
* remove original ns/name from quotedstream
* also javadoc
* remove redundant method
* fix table rename
* add incremental append test
* add full refresh append test
* comment
* call T+D sql in a reasonable location for standard inserts
* add config option
* use config option here
* type parsing - fix fromJsonSchema
* gate everything
* log query + runtime
* add spec option temporarily
* Raw Table Updates
* fix more stuff
* first big pass at toDialectType
* no quotes
* wrap everything in quotes
* resolve some TODOs
* log sql statement in tests
* overwriteFinalTable returns optional
* minor clean up
* add raw dataset override
* try to preserve the original namespace for t+d?
* write to the raw table correctly
* update todos
* write directly to raw table
this is kind of dumb because we're still trying to do tmp table operations,
and we still don't ack state until the end of the entire sync.
* standard inserts write to raw table correctly
* imports + log statements
* move logs + add comment
* explicitly create raw table
* move comment to better place
* Typing issues
* bash attempt
* formatting updates
* formatting updates
* write to the airbyte schema by default unless overriden by config options
* standard inserts truncate raw table at start of sync
* full refresh overwrite will overwrite correctly!
* fix avro record schema parsing
* better raw table recreate
* rename raw table to match standard inserts
* full refresh overwrite does tmp table things
* small clean up
* small clean up
* remove errors entry if no errors
* pull out destination config into singleton
* clean up singleton stuff
* make sure dest config exists when trying to do lookups
* avoid stringifying null
* quick thoughts on alter table
* add basic cdc testcase
* tweak cdc test setup
* rename raw table to match standard inserts
* minor tweak
* delete exact sql string assertions
* switch to JSON type
* minor cleanup
* sql whitespace changes
* explain cdc deletions
* GCS Staging Full Refresh create temp table
* assert schema
* first out of order cdc test
* add another cdc test case (currently failing)
* better test structure
* make this work
* oops, fix test
* stop trying to delete deletion records
* minor improvements to code+test
* enable concurrent test runs on integration test
* move stuff to static initializer
* extract utility method
* formatting
* Move conditional to the base java package, replace conditionals which did not use the typing and deduping flag but should have been.
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* switch back to empty list; write big assert
* minor wording tweaks
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* DestinationConfigTest
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* formatting
* remove ParsedType
* 🤖 Auto format destination-gcs code [skip ci]
* 🤖 Auto format destination-bigquery code [skip ci]
* tests verify every data type
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* full update with all data types
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* move stuff to new base lib
* 🤖 Auto format destination-gcs code [skip ci]
* Automated Commit - Formatting Changes
* 🤖 Auto format destination-bigquery code [skip ci]
* fix test
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-bigquery code [skip ci]
* 🤖 Auto format destination-gcs code [skip ci]
* asserts in dedupFinalTable
* better asserts in dedupRawTable
* [wip] test case for all data types
* 🤖 Auto format destination-gcs code [skip ci]
* 🤖 Auto format destination-bigquery code [skip ci]
* AirbyteTypeTest
* Automated Commit - Formatting Changes
* remove comments
* test chooseOneOf
* slightly better test output
* Automated Commit - Formatting Changes
* add some awful pretty print code
* more comment
* minor tweaks
* verify array/object type
* fix test
* handle deletions more correctly
* test toDialectType
* Destinations v2: better namespace handling (#27682)
* [wip] better namespace handling
* 🤖 Auto format destination-bigquery code [skip ci]
* wip also implement in gcs
* get gcs working (?)
* 🤖 Auto format destination-bigquery code [skip ci]
* remove duplicate method
* 🤖 Auto format destination-bigquery code [skip ci]
* fixed my code style settings
* make ci happy?
* 🤖 Auto format destination-bigquery code [skip ci]
* make ci happy?
* remove incorrect test
* blank line change
* initialize singleton
---------
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
* reset args correctly
* Automated Commit - Formatting Changes
* more bash stuff
* parse implicit structs
* initialize singleton in more tests
* Automated Commit - Formatting Changes
* I missed this namespace handling thing
* test more schemas
* fix singular types specified in arrays
* Automated Commit - Formatting Changes
* disable test for unimplemented feature
* initialize singleton
* remove spec options; changelogs+metadata
* randomize namespace
* also bump dockerfile
* unremove namespace sanitizing in legacy mode
* ... disable the correct test
* even more unit test fixes!
* move integration test to integration tests
---------
Co-authored-by: Cynthia Yin <cynthia@airbyte.io>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com>
* `yum clean all` after every yum install to save space
* docs and versions
* update env clean
* fix python install confusion
---------
Co-authored-by: Augustin <augustin@airbyte.io>
* add info blurb to Cloud Bucket Storage sources and destinations
* remove egress alert from destination docs
* remove egress alert from destinations, add it to sources
* revert sources to match master
* Add Xata.io Destination Connector
* readme for connectir
* adjust versioning
* missing state propagation
* throw exception if inconsistentcy is found
* set version in dockerfile
* exchange mock for deprecated logger
* put invalid config to file
* changelog in docs
* add xata dest connector to builds.md
* bump dependency to newer sdk version
* bump to latest sdk version
* blank bootstrap
* use class logger
* correct way to use the logger
* unit tests
* add write integration test
* removed irrelevant log line
* raise exception on incorrect message
* e2e test for writing
* add metadata file
* remove unued file
* remove build md
* update metdata and docs
* fix formatting
---------
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
* timeplus destination connector to push data to Timeplus via REST API/SDK
* 0.2.0, support nullable columns, add more test data
boolean and pokemon
* mark the apikey as secret
* us.timeplus.cloud as sample URL
* removed the unnecessary pass
* rewrite the logic of when to delete/create stream
* format code, following the PEP-8
* add write integration test
* replacing AirbyteLogger() with logging.getLogger('airbyte')
* add logs to destination.py, refine integration test
* add docs/integrations/destinations/timeplus.md
* add icon.svg
* set to 0.4.0 version
* upgrade timeplus sdk from 1.1.1 to 1.2.1
* add metadata file
* fix docker version
---------
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
* auto-generated destination connector template
* add config and health check
* support full_refresh overwrite
* support full refresh overwrite, full refresh append, and incremental append
* write works!
* lint
* update readme
* add test messages
* list destination??
* surface error messages
* add integration test
* add destination docs
* update source docs
* change integration test to unit test
* add check unit test
* use a map for streams
* make compatible with API changes
* add indexes for primary keys
* poll for indexes to be backfilled
* simplified stream API
* support append_dedup sync mode
* cleanup
* add changelog
* fix changelog
* use indexes ready endpoint
* update types
* rename to include primary key in API, handle namespaces
* fix types
* update docs
* add better test cases
* fix docs link
* update API and headers
* fix formatting
* update API name to streaming_import
* improve request exception and add a test for error formatting
* remove secret config
* use /replace_tables endpoint instead of /clear_tables
* partial revert of replace_tables
* fix
* formatting and fix table names
* update docs
* update metadata and dockerfile
---------
Co-authored-by: Lee Danilek <lee@convex.dev>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
* Revert "Splits bases and updates build.gradle files (#25649)"
This reverts commit c673b0a692.
* Bumps branch to prevent a conflict with publishing
* Forward fixes Snowflake to use singular base-java and develop within a new package within the same module
* Forcing automated change to merge changes
* try this?
* fix tests
* assert cdc values
* handle case where we have lsn but no updated_at
* readability improvements
* tweaks to test
* version bumps + changelogs
* Automated Change
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
* add info blurb to Cloud Bucket Storage sources and destinations
* Apply suggestions from code review
Remove extra colon
Co-authored-by: Ben Church <ben@airbyte.io>
---------
Co-authored-by: Ben Church <ben@airbyte.io>
* Splits bases and updates build.gradle files
* Fixed changelog out of sync
* Bumps version number and metadata files
* auto-bump connector version
* Downgraded untouched connector bumps
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
