29d74fc6fd
* convex source * fixed unit tests * use the smaller test instance * fix integration tests and pagination * fix tableName casing in api call * fix logo * more shapes -- thinking about moving this to an internal api * use new /json_schemas api * use new APIs * remove unused start_date * doc skeleton * fix deletes with incremental sync * fix test of json_schema * fix expected records * merge * version and docs * fix test * fix docs url * mention that you need to ask convex to enable it for your account * docs * two stage pagination for initial sync. some unit tests not working * fixed all the unit tests, but we need to push backends before acceptance tests work * fix snapshot pagination * fix integration tests * some checkboxes from the PR description * Update airbyte-integrations/connectors/source-convex/source_convex/source.py Co-authored-by: Sujay Jayakar <sujayakar314+github@gmail.com> * Update airbyte-integrations/connectors/source-convex/source_convex/source.py Co-authored-by: Sujay Jayakar <sujayakar314+github@gmail.com> * stop storing the delta_has_more in state * fix mypy * use UTC timestamps * rename instance_name to deployment_name * include data format request param * remove dev docker image * fix unit test * documentation * . * code review * documentation and change deployment_name to deployment_url * fix pip dependency * regenerate the spec file Co-authored-by: Sujay Jayakar <sujayakar314+github@gmail.com>
75 lines
3.2 KiB
Markdown
75 lines
3.2 KiB
Markdown
# Convex
|
|
|
|
This page contains the setup guide and reference information for the Convex source connector.
|
|
|
|
Get started with Convex at the [Convex website](https://convex.dev).
|
|
See your data on the [Convex dashboard](https://dashboard.convex.dev/).
|
|
|
|
## Overview
|
|
|
|
The Convex connector supports Full Refresh, Incremental Append, and Incremental Dedupe with deletes.
|
|
|
|
### Output schema
|
|
|
|
This source syncs each Convex table as a separate stream.
|
|
Check out the list of your tables on the [Convex dashboard](https://dashboard.convex.dev/) in the "Data" view.
|
|
|
|
Types not directly supported by JSON are encoded as described in the
|
|
[JSONSchema](https://json-schema.org/understanding-json-schema/reference/index.html)
|
|
for the stream.
|
|
|
|
For example, the Javascript value `new Set(["a", "b"])` is encoded as `{"$set": ["a", "b"]}`, as described by the JSONSchema
|
|
`{"type": "object", "description": "Set", "properties": {"$set": {"type": "array", "items": {"type": "string"}}}}`.
|
|
|
|
Every record includes the client-defined fields for the table, for example a `"messages"` table may contain fields for `"author"` and `"body"`.
|
|
Additionally, each document has system fields:
|
|
|
|
1. `_id` uniquely identifies the document. It is not changed by `.patch` or `.replace` operations.
|
|
2. `_creationTime` records a timestamp in milliseconds when the document was initially created. It is not changed by `.patch` or `.replace` operations.
|
|
3. `_ts` records a timestamp in nanoseconds when the document was last modified. It can be used for ordering operations in Incremental Append mode, and is automatically used in Incremental Dedupe mode.
|
|
4. `_deleted` identifies whether the document was deleted. It can be used to filter deleted documents in Incremental Append mode, and is automatically used to remove documents in Incremental Dedupe mode.
|
|
|
|
### Features
|
|
|
|
| Feature | Supported? |
|
|
| :--- | :--- |
|
|
| Full Refresh Sync | Yes |
|
|
| Incremental - Append Sync | Yes |
|
|
| Incremental - Dedupe Sync | Yes |
|
|
| Replicate Incremental Deletes | Yes |
|
|
| Change Data Capture | Yes |
|
|
| Namespaces | No |
|
|
|
|
### Performance considerations
|
|
|
|
The Convex connector syncs all documents from the historical log.
|
|
If you see performance issues due to syncing unnecessary old versions of documents,
|
|
please reach out to [Convex support](mailto:support@convex.dev).
|
|
|
|
## Getting started
|
|
|
|
### Requirements
|
|
|
|
* Convex Account
|
|
* Convex Project
|
|
* Deploy key
|
|
|
|
### Setup guide
|
|
|
|
Contact Convex via [email](mailto:support@convex.dev) or [Discord](https://convex.dev/community) to request Airbyte support for your account.
|
|
|
|
On the [Convex dashboard](https://dashboard.convex.dev/), navigate to the project that you want to sync.
|
|
Note only "Production" deployments should be synced.
|
|
|
|
In the Data tab, you should see the tables and a sample of the data that will be synced.
|
|
|
|
1. Navigate to the Settings tab.
|
|
2. Copy the "Deployment URL" from the settings page to the `deployment_url` field in Airbyte.
|
|
3. Click "Generate a deploy key".
|
|
4. Copy the generated deploy key into the `access_key` field in Airbyte.
|
|
|
|
## Changelog
|
|
|
|
| Version | Date | Pull Request | Subject |
|
|
| :--- | :--- | :--- | :--- |
|
|
| 0.1.0 | 2022-10-24 | [18403](https://github.com/airbytehq/airbyte/pull/18403) | 🎉 New Source: Convex | |