fc12432305
## What Migrating Pydantic V2 for Protocol Messages to speed up emitting records. This gives us 2.5x boost over V1. Close https://github.com/airbytehq/airbyte-internal-issues/issues/8333 ## How - Switch to using protocol models generated for pydantic_v2, in a new (temporary) package, `airbyte-protocol-models-pdv2` . - Update pydantic dependency of the CDK accordingly to v2. - For minimal impact, still use the compatibility code `pydantic.v1` in all of our pydantic code from airbyte-cdk that does not interact with the protocol models. ## Review guide 1. Checkout the code and clear your CDK virtual env (either `rm -rf .venv && python -m venv .venv` or `poetry env list; poetry env remove <env>`. This is necessary to fully clean out the `airbyte_protocol` library, for some reason. Then: `poetry lock --no-update && poetry install --all-extras`. This should install the CDK with new models. 2. Run unit tests on the CDK 3. Take your favorite connector and point it's `pyproject.toml` on local CDK (see example in `source-s3`) and try running it's tests and it's regression tests. ## User Impact > [!warning] > This is a major CDK change due to the pydantic dependency change - if connectors use pydantic 1.10, they will break and will need to do similar `from pydantic.v1` updates to get running again. Therefore, we should release this as a major CDK version bump. ## Can this PR be safely reverted and rolled back? - [x] YES 💚 - [ ] NO ❌ Even if sources migrate to this version, state format should not change, so a revert should be possible. ## Follow up work - Ella to move into issues <details> ### Source-s3 - turn this into an issue - [ ] Update source s3 CDK version and any required code changes - [ ] Fix source-s3 unit tests - [ ] Run source-s3 regression tests - [ ] Merge and release source-s3 by June 21st ### Docs - [ ] Update documentation on how to build with CDK ### CDK pieces - [ ] Update file-based CDK format validation to use Pydantic V2 - This is doable, and requires a breaking change to change `OneOfOptionConfig`. There are a few unhandled test cases that present issues we're unsure of how to handle so far. - [ ] Update low-code component generators to use Pydantic V2 - This is doable, there are a few issues around custom component generation that are unhandled. ### Further CDK performance work - create issues for these - [ ] Research if we can replace prints with buffered output (write to byte buffer and then flush to stdout) - [ ] Replace `json` with `orjson` ... </details>
Vector DB based destinations
Note: All helpers in this directory are experimental and subject to change
This directory contains several helpers that can be used to create a destination that processes and chunks records, embeds their text part and loads them into a vector database. The specific loading behavior is defined by the destination connector itself, but chunking and embedding behavior is handled by the helpers.
To use these helpers, install the CDK with the vector-db-based extra:
pip install airbyte-cdk[vector-db-based]
The helpers can be used in the following way:
- Add the config models to the spec of the connector
- Implement the
Indexerinterface for your specific database - In the check implementation of the destination, initialize the indexer and the embedder and call
checkon them - In the write implementation of the destination, initialize the indexer, the embedder and pass them to a new instance of the writer. Then call the writers
writemethod with the iterable for incoming messages
If there are no connector-specific embedders, the airbyte_cdk.destinations.vector_db_based.embedder.create_from_config function can be used to get an embedder instance from the config.
This is how the components interact:
┌─────────────┐
│MyDestination│
└┬────────────┘
┌▽───────────────────────────────┐
│Writer │
└┬─────────┬──────────┬──────────┘
┌▽───────┐┌▽────────┐┌▽────────────────┐
│Embedder││MyIndexer││DocumentProcessor│
└────────┘└─────────┘└─────────────────┘
Normally, only the MyDestination class and the MyIndexer class has to be implemented specifically for the destination. The other classes are provided as is by the helpers.