mirror of synced 2025-12-29 09:03:46 -05:00

Files

girarda 5d8cb41150 Refactor to enable support for optional JDBC parameters for all JDBC destinations (#10421 )

* refactoring to allow testing

* MySQLDestination uses connection property map instead of url arguments

* Update jdbc destinations

* A little more generic

* reset to master

* reset to master

* move to jdbcutils

* Align when multiline

* Align when multiline

* Update postgres to use property map

* Move tests to AbstractJdbcDestinationTest

* clean

* Align when multiline

* return property map

* Add postgres tests

* update clickhouse

* reformat

* reset

* reformat

* fix test

* reformat

* fix bug

* Add mssql tests

* refactor test

* fix oracle destination test

* oracle tests

* fix redshift acceptance test

* Pass string

* Revert "Pass string"

This reverts commit 697821738c.

* Double deserialization

* Revert "Double deserialization"

This reverts commit ee8d75245b.

* try updating json_operations

* Revert "try updating json_operations"

This reverts commit c8022c2994.

* json parse

* Revert "json parse"

This reverts commit 11a6725eaa.

* Revert "Revert "Double deserialization""

This reverts commit 213f47acc4.

* Revert "Revert "Revert "Double deserialization"""

This reverts commit 66822454af.

* move to constant

* Add comment

* map can be constant

* Add comment

* move map

* hide in method

* no need to create new map

* no need to create new map

* no need to create new map

* enably mysql test

* Update changelogs

* Update changelog

* update changelog

* Bump versions

* bump version

* disable dbt support

* update spec

* update other oracle tests

* update doc

* bump seed

* fix source test

* update seed spec file

* fix expected spec

2022-02-28 09:25:04 -08:00

3.7 KiB

Raw Blame History

ClickHouse

Features

Feature	Supported?(Yes/No)	Notes
Full Refresh Sync	Yes
Incremental - Append Sync	Yes
Incremental - Deduped History	Yes
Namespaces	Yes

Output Schema

Each stream will be output into its own table in ClickHouse. Each table will contain 3 columns:

_airbyte_ab_id: a uuid assigned by Airbyte to each event that is processed. The column type in ClickHouse is String.
_airbyte_emitted_at: a timestamp representing when the event was pulled from the data source. The column type in ClickHouse is DateTime64.
_airbyte_data: a json blob representing with the event data. The column type in ClickHouse is String.

Getting Started (Airbyte Cloud)

Airbyte Cloud only supports connecting to your ClickHouse instance with SSL or TLS encryption, which is supported by ClickHouse JDBC driver.

Getting Started (Airbyte Open-Source)

Requirements

To use the ClickHouse destination, you'll need:

A ClickHouse server version 21.8.10.19 or above

Configure Network Access

Make sure your ClickHouse database can be accessed by Airbyte. If your database is within a VPC, you may need to allow access from the IP you're using to expose Airbyte.

Permissions

You need a ClickHouse user with the following permissions:

can create tables and write rows.
can create databases e.g:

You can create such a user by running:

GRANT CREATE ON * TO airbyte_user;

You can also use a pre-existing user but we highly recommend creating a dedicated user for Airbyte.

Target Database

You will need to choose an existing database or create a new database that will be used to store synced data from Airbyte.

Setup the ClickHouse Destination in Airbyte

You should now have all the requirements needed to configure ClickHouse as a destination in the UI. You'll need the following information to configure the ClickHouse destination:

Host
Port (JDBC HTTP port, not the native port)
Username
Password
Database

Naming Conventions

From ClickHouse SQL Identifiers syntax:

SQL identifiers and key words must begin with a letter (a-z, but also letters with diacritical marks and non-Latin letters) or an underscore (_).
Subsequent characters in an identifier or key word can be letters, underscores, digits (0-9).
Identifiers can be quoted or non-quoted. The latter is preferred.
If you want to use identifiers the same as keywords or you want to use other symbols in identifiers, quote it using double quotes or backticks, for example, "id", id.
If you want to write portable applications you are advised to always quote a particular name or never quote it.

Therefore, Airbyte ClickHouse destination will create tables and schemas using the Unquoted identifiers when possible or fallback to Quoted Identifiers if the names are containing special characters.

Changelog

Version	Date	Pull Request	Subject
0.1.4	2022-02-25	10421	Refactor JDBC parameters handling
0.1.3	2022-02-14	10256	Add `-XX:+ExitOnOutOfMemoryError` JVM option
0.1.1	2021-12-21	#8982	Set isSchemaRequired to false
0.1.0	2021-11-04	#7620	Add ClickHouse destination

3.7 KiB Raw Blame History