As title, remove the GCS/S3 staging methods.
There isn't much usage so we can remove this. Internal Staging is also recommended by Snowflake, so using that is both cheaper and faster.
Co-authored-by: davinchia <davinchia@users.noreply.github.com>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
* handle streams with no columns
* logistics
* Automated Commit - Format and Process Resources Changes
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
* csv sheet generator supports 1s1t
* create+insert raw tables 1s1t
* add skeletons
* start writing tests
* progress in creating raw tables
* fix tests
* add s3 test; better csv generation
* handle case-sensitive column names
* also add gcs test
* hook T+D into the destination
* fix redshift; simplify
* Delete unused files?
* disable test; enable cleanup
* initialize config singleton in tests
* logistics
* header
* simplify
* fix unit tests
* correctly disable tests
* use default null for loaded_at
* fix test
* autoformat
* cython >.>
* more singleton init
* literally how?
* basic destinationhandler impl
* use raw string for type >.>
* add toDialectType
* basic createTable impl
* better sql query
* comment
* unused variables
* recorddiffer can be case-sensitive
* misc fixes
* add expected_records
* move constants to base-java
* use ternary
* fix tests
* resolve todo
* T+D can trigger on first commit
* fix test teardown
* implement softReset
* implement overwriteFinalTable
* better type stuff; check table schema
* fix
* derp
* implement updateTable?
* derp
* random wip stuff
* fix insertRaw
* theoretically implement stuff?
* stuff
* put suffix at the end
* different uuids
* fix expected records
* move tdtest resources into dat folder
* use resource files
* stuff
* move code around
* more stuff
* rename final table
* stuff
* cdc immediate deletion
* cdcComplexUpdate
* cleanup
* botched rebase
* more tests
* move back to old file
* Automated Commit - Format and Process Resources Changes
* add comments
* Automated Commit - Format and Process Resources Changes
* fix merge
* move expected_records into dat folder
* wip implement sqlgenerator test
* basic implementation
* tons of fixes, still tons more to go
* more stuff
* fix more things
* hacky convert temporal types to varchar
* test data fix
* fix variant parsing
* fix number
* fix time parsing; fix test data
* typo
* fix input data
* progress
* switch back to float
* add more test files
* swap int -> number
* fix PK null check
* fix overwriteTable
* better test
* Automated Commit - Format and Process Resources Changes
* type aliases, one more test
* also verify numeric precision/scale
* logistics
---------
Co-authored-by: edgao <edgao@users.noreply.github.com>
* Adds data as JsonNode to pass through, running into memory issues so add JVM args to attach VisualVM
* JsonNode
* Lowers the optimal batch size to see if this improvements movement
* Fixes NPE by checking if PartialAirbyteMessage contains a PartialAirbyteRecord
* Fixes config switch when config is not explicitly set (config migration needed)
* Adds logic to check if queue has elements before getting timeOfLastMessage
* Add PartialSerialisedMessage test. (#27452)
* Test deserialise.
* Add tests.
* Simplify and fix tests.
* Format.
* Adds tests for deserializeAirbyteMessage
* Adds tests for deserializeAirbyteMessage with bad data
* Cleans up deserializeAirbyteMessage and throws Exception when invalid message
* More code cleanup
---------
Co-authored-by: ryankfu <ryan.fu@airbyte.io>
* 🤖 Auto format destination-snowflake code [skip ci]
* Cleans up code w/o JVM args & rebase
* 🤖 Auto format destination-snowflake code [skip ci]
* Adds breadcrumb on the STATE message deviation and where the deserialize/serialize is done to unpack
* 🤖 Auto format destination-snowflake code [skip ci]
* Adds back line formatter removed and comment describing rational for lower batchSize
* Bumps Snowflake version and type checks
* Added note to remove PartialAirbyteRecordMessage with low resource testing
* Automated Commit - Format and Process Resources Changes
* Fix issue with multiple namespaces in snowflake not writing to the correct staging schema
* Fix issue with multiple namespaces in snowflake not writing to the correct staging schema
* remove stage name maniuplating method
* update readme
* Source Stripe: update credit_notes expected records (#27941)
* Source Zendesk Talk: update expected records (#27942)
* Source Xero: update expected records (#27943)
* Metadata: Persist Registry entries (#27766)
* DNC
* Update poetry
* Update dagster
* Apply partition
* Get metadata entry
* Use helpers
* Write registry entry to appropriate location
* Delete when registry removed
* Update to use new file (broken)
* Render registry from registry entries
* Run format
* Fix plural issue
* Update to all metadata file blobs
* Fix test
* Update to all blobs
* Add ignore validation error for version logic
* Rename to max_run_request
* Pedros review
* Ella suggestions
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
* Update airbyte-ci/connectors/metadata_service/orchestrator/orchestrator/assets/registry_entry.py
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
* Update naming
* Add tests for connector type and deletion
* Test safe parse
* Format
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
* Fix Dagster Deploy Failure (#27955)
* Add pydantic
* Add pydantic to orchestration deploy pipeline
* 🐛 Source Jira: update expected records (#27951)
* Source Jira: update expected records
* Update issues expected records
* Source Zendesk Chat: update expected records (#27965)
* 🐛 Source Pipedrive: update expected records (#27967)
* 🐛 Source Pinterest: update expected records (#27964)
* ✨ Source Amazon-Ads: Add streams for portfolios and sponsored brands v3 (#27607)
* Add stream for sponsored brands v3
* Add new stream Portfolios
* Source Google Search Console: added discover and googleNews to searchType (#27952)
* added discover and googleNews to searchType
* updated changelog
* fixed types for streams
* 🎉 Source Instagram: Improve, refactor `STATE` management (#27908)
* add test for enabling
* update versions
* fix test
* update other snowflake loading method types
* remove standard
---------
Co-authored-by: ryankfu <ryan.fu@airbyte.io>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: ryankfu <ryankfu@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Arsen Losenko <20901439+arsenlosenko@users.noreply.github.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
* `yum clean all` after every yum install to save space
* docs and versions
* update env clean
* fix python install confusion
---------
Co-authored-by: Augustin <augustin@airbyte.io>
* Revert "Splits bases and updates build.gradle files (#25649)"
This reverts commit c673b0a692.
* Bumps branch to prevent a conflict with publishing
* Forward fixes Snowflake to use singular base-java and develop within a new package within the same module
* Forcing automated change to merge changes
* Splits bases and updates build.gradle files
* Fixed changelog out of sync
* Bumps version number and metadata files
* auto-bump connector version
* Downgraded untouched connector bumps
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* map the other integer schema to long
* fix test + add test
* delete_public_access_block for bucket if public (#25663)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
* 🐛 Source Facebook Marketing: fix `expected records` for CAT (#25604)
* publish normalization (#25591)
* publish normalization
* bump normalization container version in all the destinations that use it
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: edgao <edgao@users.noreply.github.com>
* Bump Airbyte version from 0.44.2 to 0.44.3
* Destination Bigquery: update AIRBYTE_ENTRYPOINT env var for kube process (#25588)
* add AIRBYTE_ENTRYPOINT env var for kube
* amazing, absolute genius
* version bump + changelog
* derp, no need to publish denormalizeid
* fix changelog entry
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* version bumps + changelog
* whoops
* bump metadata
* bump metadatas
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
---------
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Jeff Cowan (Airbyte) <4992320+jcowanpdx@users.noreply.github.com>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
