1
0
mirror of synced 2025-12-23 03:47:05 -05:00
Commit Graph

68 Commits

Author SHA1 Message Date
Joe Reuter
e48355062e Weaviate: Normalize property names (#33218)
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
2023-12-08 16:09:14 +01:00
Joe Reuter
e7f8165683 Vector DB destinations: Add omit_raw_text flag and increase code reuse (#32697)
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
2023-12-04 17:45:04 +01:00
Joe Reuter
8f7abc2cc0 S3, Azure Blob Storage, GCS, Weviate, Milvus, Chroma, Qdrant: Bump cdk version (#32608)
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
2023-11-30 09:48:18 +01:00
Joe Reuter
6d54a5dc72 Vector DBs: Mention langchain in docs (#32501) 2023-11-17 19:18:24 +01:00
Joe Reuter
b3396626ee S3, Azure Blob Storage, GCS, Pinecone, Weaviate, Milvus, Chroma, Qdrant: Update CDK to improve spec generation (#32357)
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
2023-11-14 16:53:27 +01:00
Joe Reuter
a34e3375d7 Pinecone: Improve error messages and unit test coverage, Weaviate: Improve unit test coverage (#32292) 2023-11-09 16:59:07 +01:00
Joe Reuter
ca9a7c3835 Weaviate: Update weaviate client library (#32134) 2023-11-03 12:21:26 +01:00
Joe Reuter
402ac603f4 Weaviate: Add retry (#32038) 2023-11-02 11:50:10 +01:00
Joe Reuter
c828692b01 Weaviate: Fix memory leak (#31953) 2023-10-30 17:20:34 +01:00
Joe Reuter
3de1a05d17 Upgrade CDK for Milvus/Qdrant/Chroma/Weaviate (#31563)
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
2023-10-27 16:03:23 +02:00
Augustin
a41c4f5b3d Migrate connectors to use our python base image (Round 2) (#31599) 2023-10-19 17:58:45 +02:00
Joe Reuter
083fc2035b Vector DBs: Upgrade CDK (#31329)
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
2023-10-18 15:48:55 +02:00
Joe Reuter
64964cceb9 Vector DB Destinations: Fix OpenAI batch size and Weaviate field name collision (#31075) 2023-10-06 23:18:22 +02:00
Joe Reuter
f161e2e441 🚨 Weaviate destination: Add embedding capabilities, overwrite and dedup support, API key auth mode and available on Airbyte Cloud (#30151)
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
2023-09-28 11:09:39 +02:00
Evan Tahler
9210547af5 [Docs] No Deduped + Hostory, Append + Deduped is the future! (#29114)
* [Docs] No `Deduped + Hostory`, `Append + Deduped` is the future!

* fix links
2023-08-08 15:07:49 -07:00
Sam Stoelinga
3d737715ce 🎉 Destination Weaviate: Support any string based ID and fix issues with additionalProperties (#22527)
* Support any string based ID

* Previously string based IDs that weren't convertable to hex would
  throw an exception. This fixes that issue by converting strings
  to 128 bit integer and converting the md5 hex into a UUID
* Update Weaviate to 1.17.3

* Fix issue with arrays of no data type

Closes #22530

* add more testing coverage

* fix error where dataType wasn't set in weaviate

This seems to happen when records sometimes don't contain all the
properties that are defined in the schema.

This was the error observed by a user connecting Slack <> Weaviate:
property 'reactions': invalid dataType: dataType must have at least
one element, class 'Threads' not present in schema"

* add test case for additionalProperties

* Support additionalProps of object and array of obj

* Update changelog

* add weaviate icon

* fix flake tests

* add commit

* make build m1 work

* make build m1 work

* auto-bump connector version

---------

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2023-02-28 17:59:19 -03:00
Bob van Luijt
1c93be447d Added some reference links to Weaviate docs (#21449) 2023-01-17 10:51:24 +00:00
Sam Stoelinga
477861552c 🎉 New Destination: Weaviate (#20094)
* Add Weaviate Destination #20012

* Fix formatting and standards

* Fix flake issue

* Fix unused client variable

* Add support for int based ID fields

* Ensure stream name meets Weaviate class reqs

* add integration test for using pokemon as source

* handle nested objects by converting to json string

* create schema for transforming data to weaviate

* Add docs for weaviate destination

* Remove pokemon-schema external dependency

* Remove pikachu integration test external dep

* Add large batch test case

* add test for second sync

* Fix issue with fields starting with uppercase

* add more checks to line_break test

* Update README for Weaviate

* Make batch_size configurable with 100 as default

* Add support for providing vectors

* Update docs

* Add test for existing Weaviate class

* Add trying to create schema in check connection

* Add support for mongodb _id fields

* Add support for providing custom ID

* remove unused file

* fix flow of is_ready() check

* Move standalone functions to utils.py

* Support overwrite mode

* Add regex based stream_name_class_name conversion

* remove unneeded print statement

* Add "airbyte_secret" : true to password config

* add support for array of arrays

* remove unneeded variable declaration

* change to MutableMapping since we use del

* change name from queued_write to buffered_write

* add retry on partial batch error

* Fix partial batch retry and add tests

* fix ID generation

* Clean up recursive retry logic

* fix flake tests

* ran flake reformat

* add definitions

Co-authored-by: Ivica Taseski <ivica.taseski94@gmail.com>
Co-authored-by: itaseski <itaseski@debian-BULLSEYE-live-builder-AMD64>
2023-01-12 12:11:35 -05:00