29d74fc6fd
* convex source * fixed unit tests * use the smaller test instance * fix integration tests and pagination * fix tableName casing in api call * fix logo * more shapes -- thinking about moving this to an internal api * use new /json_schemas api * use new APIs * remove unused start_date * doc skeleton * fix deletes with incremental sync * fix test of json_schema * fix expected records * merge * version and docs * fix test * fix docs url * mention that you need to ask convex to enable it for your account * docs * two stage pagination for initial sync. some unit tests not working * fixed all the unit tests, but we need to push backends before acceptance tests work * fix snapshot pagination * fix integration tests * some checkboxes from the PR description * Update airbyte-integrations/connectors/source-convex/source_convex/source.py Co-authored-by: Sujay Jayakar <sujayakar314+github@gmail.com> * Update airbyte-integrations/connectors/source-convex/source_convex/source.py Co-authored-by: Sujay Jayakar <sujayakar314+github@gmail.com> * stop storing the delta_has_more in state * fix mypy * use UTC timestamps * rename instance_name to deployment_name * include data format request param * remove dev docker image * fix unit test * documentation * . * code review * documentation and change deployment_name to deployment_url * fix pip dependency * regenerate the spec file Co-authored-by: Sujay Jayakar <sujayakar314+github@gmail.com>
3.2 KiB
Convex
This page contains the setup guide and reference information for the Convex source connector.
Get started with Convex at the Convex website. See your data on the Convex dashboard.
Overview
The Convex connector supports Full Refresh, Incremental Append, and Incremental Dedupe with deletes.
Output schema
This source syncs each Convex table as a separate stream. Check out the list of your tables on the Convex dashboard in the "Data" view.
Types not directly supported by JSON are encoded as described in the JSONSchema for the stream.
For example, the Javascript value new Set(["a", "b"]) is encoded as {"$set": ["a", "b"]}, as described by the JSONSchema
{"type": "object", "description": "Set", "properties": {"$set": {"type": "array", "items": {"type": "string"}}}}.
Every record includes the client-defined fields for the table, for example a "messages" table may contain fields for "author" and "body".
Additionally, each document has system fields:
_iduniquely identifies the document. It is not changed by.patchor.replaceoperations._creationTimerecords a timestamp in milliseconds when the document was initially created. It is not changed by.patchor.replaceoperations._tsrecords a timestamp in nanoseconds when the document was last modified. It can be used for ordering operations in Incremental Append mode, and is automatically used in Incremental Dedupe mode._deletedidentifies whether the document was deleted. It can be used to filter deleted documents in Incremental Append mode, and is automatically used to remove documents in Incremental Dedupe mode.
Features
| Feature | Supported? |
|---|---|
| Full Refresh Sync | Yes |
| Incremental - Append Sync | Yes |
| Incremental - Dedupe Sync | Yes |
| Replicate Incremental Deletes | Yes |
| Change Data Capture | Yes |
| Namespaces | No |
Performance considerations
The Convex connector syncs all documents from the historical log. If you see performance issues due to syncing unnecessary old versions of documents, please reach out to Convex support.
Getting started
Requirements
- Convex Account
- Convex Project
- Deploy key
Setup guide
Contact Convex via email or Discord to request Airbyte support for your account.
On the Convex dashboard, navigate to the project that you want to sync. Note only "Production" deployments should be synced.
In the Data tab, you should see the tables and a sample of the data that will be synced.
- Navigate to the Settings tab.
- Copy the "Deployment URL" from the settings page to the
deployment_urlfield in Airbyte. - Click "Generate a deploy key".
- Copy the generated deploy key into the
access_keyfield in Airbyte.
Changelog
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.1.0 | 2022-10-24 | 18403 | 🎉 New Source: Convex |