68 lines
3.2 KiB
Markdown
68 lines
3.2 KiB
Markdown
# Redis
|
|
|
|
## Sync overview
|
|
|
|
### Output schema
|
|
|
|
The incoming Airbyte data is structured depending on the target Redis cache/data type.
|
|
This connector maps an incoming data from a namespace and stream to a unique Redis key.
|
|
|
|
For the **_hash_** implementation as a Redis data type the keys and the hashes are structured in the following way:
|
|
|
|
**_key_**:
|
|
|
|
namespace:stream:id
|
|
|
|
**_hash_**:
|
|
|
|
- `_airbyte_ab_id`: Sequential id for a given key generated by using the INCR Redis command.
|
|
- `_airbyte_emitted_at`: a timestamp representing when the event was received from the data source.
|
|
- `_airbyte_data`: a json text/object representing the data that was received from the data source.
|
|
|
|
### Features
|
|
|
|
| Feature | Support | Notes |
|
|
| :----------------------------- | :-----: | :----------------------------------------------------------------------------- |
|
|
| Full Refresh Sync | ✅ | Existing keys in the Redis cache are deleted and replaced with the new keys. |
|
|
| Incremental - Append Sync | ✅ | New keys are inserted in the same keyspace without touching the existing keys. |
|
|
| Incremental - Append + Deduped | ❌ | |
|
|
| Namespaces | ✅ | Namespaces will be used to determine the correct Redis key. |
|
|
| SSH Tunnel Connection | ✅ | |
|
|
| SSL connection | ✅ | |
|
|
|
|
### Performance considerations
|
|
|
|
As long as you have the necessary memory capacity for your cache, Redis should be able to handle even millions of records without any issues since the data is stored in-memory with the option to
|
|
save snapshots periodically on disk.
|
|
|
|
## Getting started
|
|
|
|
### Requirements
|
|
|
|
- The connector is fully compatible with redis 2.8.x, 3.x.x and above
|
|
- Configuration
|
|
- **_host_**: Hostname or address of the Redis server where to connect.
|
|
- **_port_**: Port of the Redis server where to connect.
|
|
- **_username_**: Username for authenticating with the Redis server.
|
|
- **_password_**: Password for authenticating with the Redis server.
|
|
- **_cache_type_**: Redis cache/data type to use when storing the incoming messages. i.e hash,set,list,stream,etc.
|
|
- SSL toggle the switch to connect using SSL
|
|
- For SSL Modes, select:
|
|
- **disable** to disable encrypted communication between Airbyte and the source
|
|
- **verify-full** to always require encryption and verify the identity of the source
|
|
|
|
### Setup guide
|
|
|
|
######TODO: more info, screenshots?, etc...
|
|
|
|
## Changelog
|
|
|
|
<details>
|
|
<summary>Expand to review</summary>
|
|
|
|
| Version | Date | Pull Request | Subject |
|
|
| :------ | :--------- | :--------------------------------------------------------- | :-------------- |
|
|
| 0.1.4 | 2022-10-25 | [\#18358](https://github.com/airbytehq/airbyte/pull/18358) | TLS support |
|
|
| 0.1.3 | 2022-10-18 | [\#17951](https://github.com/airbytehq/airbyte/pull/17951) | Add SSH support |
|
|
|
|
</details> |