jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-23 21:03:15 -05:00
Code Issues Packages Projects Releases Wiki Activity

Source postgres: Add SSL certificates to source postgres (#13840)

Browse Source 
* added ssl certificates for postgres source

* added command for remove client private key after transformation to encrypted key with .pk8 extension

* added connection with CA and client certificates for postgres destination

* updated code style

* moved common methods to the common class

* moved common methods to the common class

* fixed remarks

* updated postgres source tests

* added minor changes to spec and added fixes to password mechanism

* updated postgres source tests

* updated strict-encrypt postgres source and destination and added tests for SSL certificates for all postgres connectors

* fixed check style

* updated documentation and versions of connectors

* updated ordrs in test spec

* fixed minor remarks in specs and expected_specs

* fixed minor remarks in specs and expected_specs

* fixed Dockerfile

* fixed remarks

* fixed remarks

* fixed remarks

* fixed remarks

* fixed remarks

* rollback changes for postgres destination and move them to a new pull request

* rollback changes for postgres destination and move them to a new pull request

* fixed code style

* fixed code style

* fixed code style

* fixed code style

* fixed code style

* auto-bump connector version

* updated version of postgres strict-encrypt source

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
This commit is contained in:
andriikorotkov
2022-07-18 12:58:53 +03:00
committed by GitHub
parent 638db80acc
commit 2dc4759264
17 changed files with 980 additions and 77 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
airbyte-config/init/src/main/resources/seed/source_definitions.yaml
View File

@@ -754,7 +754,7 @@
- name: Postgres
sourceDefinitionId: decd338e-5647-4c0b-adf4-da0e75f5a750
dockerRepository: airbyte/source-postgres
dockerImageTag: 0.4.33
dockerImageTag: 0.4.34
documentationUrl: https://docs.airbyte.io/integrations/sources/postgres
icon: postgresql.svg
sourceType: database

147
airbyte-config/init/src/main/resources/seed/source_specs.yaml
View File

@@ -6978,7 +6978,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-postgres:0.4.33"
- dockerImage: "airbyte/source-postgres:0.4.34"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/postgres"
connectionSpecification:
@@ -7045,10 +7045,153 @@
order: 6
ssl:
title: "Connect using SSL"
description: "Encrypt client/server communications for increased security."
description: "Encrypt data using SSL. When activating SSL, please select\
\ one of the connection modes."
type: "boolean"
default: false
order: 7
ssl_mode:
title: "SSL modes"
description: "SSL connection modes. \n <b>disable</b> - Chose this mode\
\ to disable encryption of communication between Airbyte and source database\n\
\ <b>allow</b> - Chose this mode to enable encryption only when required\
\ by the source database\n <b>prefer</b> - Chose this mode to allow unencrypted\
\ connection only if the source database does not support encryption\n\
\ <b>require</b> - Chose this mode to always require encryption. If the\
\ source database server does not support encryption, connection will\
\ fail\n <b>verify-ca</b> - Chose this mode to always require encryption\
\ and to verify that the source database server has a valid SSL certificate\n\
\ <b>verify-full</b> - This is the most secure mode. Chose this mode\
\ to always require encryption and to verify the identity of the source\
\ database server\n See more information - <a href=\"https://jdbc.postgresql.org/documentation/head/ssl-client.html\"\
> in the docs</a>."
type: "object"
order: 7
oneOf:
- title: "disable"
additionalProperties: false
description: "Disable SSL."
required:
- "mode"
properties:
mode:
type: "string"
const: "disable"
enum:
- "disable"
default: "disable"
order: 0
- title: "allow"
additionalProperties: false
description: "Allow SSL mode."
required:
- "mode"
properties:
mode:
type: "string"
const: "allow"
enum:
- "allow"
default: "allow"
order: 0
- title: "prefer"
additionalProperties: false
description: "Prefer SSL mode."
required:
- "mode"
properties:
mode:
type: "string"
const: "prefer"
enum:
- "prefer"
default: "prefer"
order: 0
- title: "require"
additionalProperties: false
description: "Require SSL mode."
required:
- "mode"
properties:
mode:
type: "string"
const: "require"
enum:
- "require"
default: "require"
order: 0
- title: "verify-ca"
additionalProperties: false
description: "Verify-ca SSL mode."
required:
- "mode"
- "ca_certificate"
properties:
mode:
type: "string"
const: "verify-ca"
enum:
- "verify-ca"
default: "verify-ca"
order: 0
ca_certificate:
type: "string"
title: "CA certificate"
description: "CA certificate"
airbyte_secret: true
multiline: true
order: 1
client_key_password:
type: "string"
title: "Client key password (Optional)"
description: "Password for keystorage. This field is optional. If\
\ you do not add it - the password will be generated automatically."
airbyte_secret: true
order: 4
- title: "verify-full"
additionalProperties: false
description: "Verify-full SSL mode."
required:
- "mode"
- "ca_certificate"
- "client_certificate"
- "client_key"
properties:
mode:
type: "string"
const: "verify-full"
enum:
- "verify-full"
default: "verify-full"
order: 0
ca_certificate:
type: "string"
title: "CA certificate"
description: "CA certificate"
airbyte_secret: true
multiline: true
order: 1
client_certificate:
type: "string"
title: "Client certificate"
description: "Client certificate"
airbyte_secret: true
multiline: true
order: 2
client_key:
type: "string"
title: "Client key"
description: "Client key"
airbyte_secret: true
multiline: true
order: 3
client_key_password:
type: "string"
title: "Client key password (Optional)"
description: "Password for keystorage. This field is optional. If\
\ you do not add it - the password will be generated automatically."
airbyte_secret: true
order: 4
replication_method:
type: "object"
title: "Replication Method"

73
airbyte-db/db-lib/src/main/java/io/airbyte/db/PostgresUtils.java
View File

@@ -5,11 +5,14 @@
package io.airbyte.db;
import com.fasterxml.jackson.databind.JsonNode;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
import io.airbyte.db.jdbc.JdbcDatabase;
import io.airbyte.db.jdbc.JdbcUtils;
import java.io.IOException;
import java.sql.SQLException;
import java.util.List;
import org.testcontainers.containers.PostgreSQLContainer;
public class PostgresUtils {
@@ -23,4 +26,74 @@ public class PostgresUtils {
return PgLsn.fromPgString(jsonNodes.get(0).get("pg_current_wal_lsn").asText());
}
@VisibleForTesting
public static Certificate getCertificate(final PostgreSQLContainer<?> container) throws IOException, InterruptedException {
container.execInContainer("su", "-c", "psql -U test -c \"CREATE USER postgres WITH PASSWORD 'postgres';\"");
container.execInContainer("su", "-c", "psql -U test -c \"GRANT CONNECT ON DATABASE \"test\" TO postgres;\"");
container.execInContainer("su", "-c", "psql -U test -c \"ALTER USER postgres WITH SUPERUSER;\"");
container.execInContainer("su", "-c", "openssl ecparam -name prime256v1 -genkey -noout -out ca.key");
container.execInContainer("su", "-c", "openssl req -new -x509 -sha256 -key ca.key -out ca.crt -subj \"/CN=localhost\"");
container.execInContainer("su", "-c", "openssl ecparam -name prime256v1 -genkey -noout -out server.key");
container.execInContainer("su", "-c", "openssl req -new -sha256 -key server.key -out server.csr -subj \"/CN=localhost\"");
container.execInContainer("su", "-c",
"openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -sha256");
container.execInContainer("su", "-c", "cp server.key /etc/ssl/private/");
container.execInContainer("su", "-c", "cp server.crt /etc/ssl/private/");
container.execInContainer("su", "-c", "cp ca.crt /etc/ssl/private/");
container.execInContainer("su", "-c", "chmod og-rwx /etc/ssl/private/server.* /etc/ssl/private/ca.*");
container.execInContainer("su", "-c", "chown postgres:postgres /etc/ssl/private/server.crt /etc/ssl/private/server.key /etc/ssl/private/ca.crt");
container.execInContainer("su", "-c", "echo \"ssl = on\" >> /var/lib/postgresql/data/postgresql.conf");
container.execInContainer("su", "-c", "echo \"ssl_cert_file = '/etc/ssl/private/server.crt'\" >> /var/lib/postgresql/data/postgresql.conf");
container.execInContainer("su", "-c", "echo \"ssl_key_file = '/etc/ssl/private/server.key'\" >> /var/lib/postgresql/data/postgresql.conf");
container.execInContainer("su", "-c", "echo \"ssl_ca_file = '/etc/ssl/private/ca.crt'\" >> /var/lib/postgresql/data/postgresql.conf");
container.execInContainer("su", "-c", "mkdir root/.postgresql");
container.execInContainer("su", "-c",
"echo \"hostssl all all 127.0.0.1/32 cert clientcert=verify-full\" >> /var/lib/postgresql/data/pg_hba.conf");
var caCert = container.execInContainer("su", "-c", "cat ca.crt").getStdout().trim();
container.execInContainer("su", "-c", "openssl ecparam -name prime256v1 -genkey -noout -out client.key");
container.execInContainer("su", "-c", "openssl req -new -sha256 -key client.key -out client.csr -subj \"/CN=postgres\"");
container.execInContainer("su", "-c",
"openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 365 -sha256");
container.execInContainer("su", "-c", "cp client.crt ~/.postgresql/postgresql.crt");
container.execInContainer("su", "-c", "cp client.key ~/.postgresql/postgresql.key");
container.execInContainer("su", "-c", "chmod 0600 ~/.postgresql/postgresql.crt ~/.postgresql/postgresql.key");
container.execInContainer("su", "-c", "cp ca.crt root/.postgresql/ca.crt");
container.execInContainer("su", "-c", "chown postgres:postgres ~/.postgresql/ca.crt");
container.execInContainer("su", "-c", "psql -U test -c \"SELECT pg_reload_conf();\"");
var clientKey = container.execInContainer("su", "-c", "cat client.key").getStdout().trim();
var clientCert = container.execInContainer("su", "-c", "cat client.crt").getStdout().trim();
return new Certificate(caCert, clientCert, clientKey);
}
public static class Certificate {
private final String caCertificate;
private final String clientCertificate;
private final String clientKey;
public Certificate(final String caCertificate, final String clientCertificate, final String clientKey) {
this.caCertificate = caCertificate;
this.clientCertificate = clientCertificate;
this.clientKey = clientKey;
}
public String getCaCertificate() {
return caCertificate;
}
public String getClientCertificate() {
return clientCertificate;
}
public String getClientKey() {
return clientKey;
}
}
}

160
airbyte-integrations/bases/base-java/src/main/java/io/airbyte/integrations/util/PostgresSslConnectionUtils.java Normal file
View File

@@ -0,0 +1,160 @@
/*
* Copyright (c) 2022 Airbyte, Inc., all rights reserved.
*/
package io.airbyte.integrations.util;
import com.fasterxml.jackson.databind.JsonNode;
import java.io.IOException;
import java.io.PrintWriter;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import org.apache.commons.lang3.RandomStringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class PostgresSslConnectionUtils {
private static final Logger LOGGER = LoggerFactory.getLogger(PostgresSslConnectionUtils.class);
private static final String KEY_STORE_PASS = RandomStringUtils.randomAlphanumeric(10);
private static final String CA_CERTIFICATE = "ca.crt";
private static final String CLIENT_CERTIFICATE = "client.crt";
private static final String CLIENT_KEY = "client.key";
private static final String CLIENT_ENCRYPTED_KEY = "client.pk8";
public static final String PARAM_MODE = "mode";
public static final String PARAM_SSL = "ssl";
public static final String PARAM_SSL_MODE = "ssl_mode";
public static final String PARAM_SSLMODE = "sslmode";
public static final String PARAM_CLIENT_KEY_PASSWORD = "client_key_password";
public static final String PARAM_CA_CERTIFICATE = "ca_certificate";
public static final String PARAM_CLIENT_CERTIFICATE = "client_certificate";
public static final String PARAM_CLIENT_KEY = "client_key";
public static final String VERIFY_CA = "verify-ca";
public static final String VERIFY_FULL = "verify-full";
public static final String DISABLE = "disable";
public static final String TRUE_STRING_VALUE = "true";
public static final String FACTORY_VALUE = "org.postgresql.ssl.DefaultJavaSSLFactory";
public static Map<String, String> obtainConnectionOptions(final JsonNode encryption) {
final Map<String, String> additionalParameters = new HashMap<>();
if (!encryption.isNull()) {
final var method = encryption.get(PARAM_MODE).asText();
String sslPassword = encryption.has(PARAM_CLIENT_KEY_PASSWORD) ? encryption.get(PARAM_CLIENT_KEY_PASSWORD).asText() : "";
var keyStorePassword = KEY_STORE_PASS;
if (!sslPassword.isEmpty()) {
keyStorePassword = sslPassword;
}
switch (method) {
case VERIFY_CA -> {
additionalParameters.putAll(obtainConnectionCaOptions(encryption, method, keyStorePassword));
}
case VERIFY_FULL -> {
additionalParameters.putAll(obtainConnectionFullOptions(encryption, method, keyStorePassword));
}
default -> {
additionalParameters.put(PARAM_SSL, TRUE_STRING_VALUE);
additionalParameters.put(PARAM_SSLMODE, method);
}
}
}
return additionalParameters;
}
private static Map<String, String> obtainConnectionFullOptions(final JsonNode encryption,
final String method,
final String clientKeyPassword) {
final Map<String, String> additionalParameters = new HashMap<>();
try {
convertAndImportFullCertificate(encryption.get(PARAM_CA_CERTIFICATE).asText(),
encryption.get(PARAM_CLIENT_CERTIFICATE).asText(), encryption.get(PARAM_CLIENT_KEY).asText(), clientKeyPassword);
} catch (final IOException | InterruptedException e) {
throw new RuntimeException("Failed to import certificate into Java Keystore");
}
additionalParameters.put("ssl", TRUE_STRING_VALUE);
additionalParameters.put("sslmode", method);
additionalParameters.put("sslrootcert", CA_CERTIFICATE);
additionalParameters.put("sslcert", CLIENT_CERTIFICATE);
additionalParameters.put("sslkey", CLIENT_ENCRYPTED_KEY);
additionalParameters.put("sslfactory", FACTORY_VALUE);
return additionalParameters;
}
private static Map<String, String> obtainConnectionCaOptions(final JsonNode encryption,
final String method,
final String clientKeyPassword) {
final Map<String, String> additionalParameters = new HashMap<>();
try {
convertAndImportCaCertificate(encryption.get(PARAM_CA_CERTIFICATE).asText(), clientKeyPassword);
} catch (final IOException | InterruptedException e) {
throw new RuntimeException("Failed to import certificate into Java Keystore");
}
additionalParameters.put("ssl", TRUE_STRING_VALUE);
additionalParameters.put("sslmode", method);
additionalParameters.put("sslrootcert", CA_CERTIFICATE);
additionalParameters.put("sslfactory", FACTORY_VALUE);
return additionalParameters;
}
private static void convertAndImportFullCertificate(final String caCertificate,
final String clientCertificate,
final String clientKey,
final String clientKeyPassword)
throws IOException, InterruptedException {
final Runtime run = Runtime.getRuntime();
createCaCertificate(caCertificate, clientKeyPassword, run);
createCertificateFile(CLIENT_CERTIFICATE, clientCertificate);
createCertificateFile(CLIENT_KEY, clientKey);
// add client certificate to the custom keystore
runProcess("keytool -alias client-certificate -keystore customkeystore"
+ " -import -file " + CLIENT_CERTIFICATE + " -storepass " + clientKeyPassword + " -noprompt", run);
// convert client.key to client.pk8 based on the documentation
runProcess("openssl pkcs8 -topk8 -inform PEM -in " + CLIENT_KEY + " -outform DER -out "
+ CLIENT_ENCRYPTED_KEY + " -nocrypt", run);
runProcess("rm " + CLIENT_KEY, run);
updateTrustStoreSystemProperty(clientKeyPassword);
}
private static void convertAndImportCaCertificate(final String caCertificate,
final String clientKeyPassword)
throws IOException, InterruptedException {
final Runtime run = Runtime.getRuntime();
createCaCertificate(caCertificate, clientKeyPassword, run);
updateTrustStoreSystemProperty(clientKeyPassword);
}
private static void createCaCertificate(final String caCertificate,
final String clientKeyPassword,
final Runtime run)
throws IOException, InterruptedException {
createCertificateFile(CA_CERTIFICATE, caCertificate);
// add CA certificate to the custom keystore
runProcess("keytool -import -alias rds-root -keystore customkeystore"
+ " -file " + CA_CERTIFICATE + " -storepass " + clientKeyPassword + " -noprompt", run);
}
private static void updateTrustStoreSystemProperty(final String clientKeyPassword) {
String result = System.getProperty("user.dir") + "/customkeystore";
System.setProperty("javax.net.ssl.trustStore", result);
System.setProperty("javax.net.ssl.trustStorePassword", clientKeyPassword);
}
private static void createCertificateFile(String fileName, String fileValue) throws IOException {
try (final PrintWriter out = new PrintWriter(fileName, StandardCharsets.UTF_8)) {
out.print(fileValue);
}
}
private static void runProcess(final String cmd, final Runtime run) throws IOException, InterruptedException {
final Process pr = run.exec(cmd);
if (!pr.waitFor(30, TimeUnit.SECONDS)) {
pr.destroy();
throw new RuntimeException("Timeout while executing: " + cmd);
}
}
}

2
airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile
View File

@@ -16,5 +16,5 @@ ENV APPLICATION source-postgres-strict-encrypt
COPY --from=build /airbyte /airbyte
LABEL io.airbyte.version=0.4.32
LABEL io.airbyte.version=0.4.34
LABEL io.airbyte.name=airbyte/source-postgres-strict-encrypt

6
airbyte-integrations/connectors/source-postgres-strict-encrypt/src/main/java/io/airbyte/integrations/source/postgres/PostgresSourceStrictEncrypt.java
View File

@@ -4,6 +4,7 @@
package io.airbyte.integrations.source.postgres;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import io.airbyte.commons.json.Jsons;
import io.airbyte.integrations.base.IntegrationRunner;
@@ -29,6 +30,11 @@ public class PostgresSourceStrictEncrypt extends SpecModifyingSource implements
public ConnectorSpecification modifySpec(final ConnectorSpecification originalSpec) {
final ConnectorSpecification spec = Jsons.clone(originalSpec);
((ObjectNode) spec.getConnectionSpecification().get("properties")).remove("ssl");
ArrayNode modifiedSslModes = spec.getConnectionSpecification().get("properties").get("ssl_mode").get("oneOf").deepCopy();
// Assume that the first item is the "disable" option; remove it
modifiedSslModes.remove(0);
((ObjectNode) spec.getConnectionSpecification().get("properties").get("ssl_mode")).remove("oneOf");
((ObjectNode) spec.getConnectionSpecification().get("properties").get("ssl_mode")).put("oneOf", modifiedSslModes);
return spec;
}

2
airbyte-integrations/connectors/source-postgres/Dockerfile
View File

@@ -16,5 +16,5 @@ ENV APPLICATION source-postgres
COPY --from=build /airbyte /airbyte
LABEL io.airbyte.version=0.4.33
LABEL io.airbyte.version=0.4.34
LABEL io.airbyte.name=airbyte/source-postgres

27
airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java
View File

@@ -5,6 +5,13 @@
package io.airbyte.integrations.source.postgres;
import static io.airbyte.integrations.debezium.AirbyteDebeziumHandler.shouldUseCDC;
import static io.airbyte.integrations.debezium.internals.DebeziumEventUtils.CDC_DELETED_AT;
import static io.airbyte.integrations.debezium.internals.DebeziumEventUtils.CDC_UPDATED_AT;
import static io.airbyte.integrations.util.PostgresSslConnectionUtils.DISABLE;
import static io.airbyte.integrations.util.PostgresSslConnectionUtils.PARAM_MODE;
import static io.airbyte.integrations.util.PostgresSslConnectionUtils.PARAM_SSL;
import static io.airbyte.integrations.util.PostgresSslConnectionUtils.PARAM_SSL_MODE;
import static io.airbyte.integrations.util.PostgresSslConnectionUtils.obtainConnectionOptions;
import static java.util.stream.Collectors.toList;
import static java.util.stream.Collectors.toSet;
@@ -66,6 +73,7 @@ import org.slf4j.LoggerFactory;
public class PostgresSource extends AbstractJdbcSource<JDBCType> implements Source {
private static final Logger LOGGER = LoggerFactory.getLogger(PostgresSource.class);
public static final String CDC_LSN = "_ab_cdc_lsn";
public static final String DATABASE_KEY = "database";
public static final String HOST_KEY = "host";
@@ -114,9 +122,22 @@ public class PostgresSource extends AbstractJdbcSource<JDBCType> implements Sour
}
// assume ssl if not explicitly mentioned.
if (!config.has(JdbcUtils.SSL_KEY) || config.get(JdbcUtils.SSL_KEY).asBoolean()) {
additionalParameters.add("ssl=true");
additionalParameters.add("sslmode=require");
if (!config.has(PARAM_SSL) || config.get(PARAM_SSL).asBoolean()) {
if (config.has(PARAM_SSL_MODE)) {
if (DISABLE.equals(config.get(PARAM_SSL_MODE).get(PARAM_MODE).asText())) {
additionalParameters.add("sslmode=disable");
} else {
var parametersList = obtainConnectionOptions(config.get(PARAM_SSL_MODE))
.entrySet()
.stream()
.map(e -> e.getKey() + "=" + e.getValue())
.toList();
additionalParameters.addAll(parametersList);
}
} else {
additionalParameters.add("ssl=true");
additionalParameters.add("sslmode=require");
}
}
if (config.has(SCHEMAS_KEY) && config.get(SCHEMAS_KEY).isArray()) {

6
airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSourceStrictEncrypt.java
View File

@@ -4,6 +4,7 @@
package io.airbyte.integrations.source.postgres;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import io.airbyte.commons.json.Jsons;
import io.airbyte.integrations.base.IntegrationRunner;
@@ -29,6 +30,11 @@ public class PostgresSourceStrictEncrypt extends SpecModifyingSource implements
public ConnectorSpecification modifySpec(final ConnectorSpecification originalSpec) {
final ConnectorSpecification spec = Jsons.clone(originalSpec);
((ObjectNode) spec.getConnectionSpecification().get("properties")).remove("ssl");
ArrayNode modifiedSslModes = spec.getConnectionSpecification().get("properties").get("ssl_mode").get("oneOf").deepCopy();
// Assume that the first item is the "disable" option; remove it
modifiedSslModes.remove(0);
((ObjectNode) spec.getConnectionSpecification().get("properties").get("ssl_mode")).remove("oneOf");
((ObjectNode) spec.getConnectionSpecification().get("properties").get("ssl_mode")).put("oneOf", modifiedSslModes);
return spec;
}

151
airbyte-integrations/connectors/source-postgres/src/main/resources/spec.json
View File

@@ -62,11 +62,160 @@
},
"ssl": {
"title": "Connect using SSL",
"description": "Encrypt client/server communications for increased security.",
"description": "Encrypt data using SSL. When activating SSL, please select one of the connection modes.",
"type": "boolean",
"default": false,
"order": 7
},
"ssl_mode": {
"title": "SSL modes",
"description": "SSL connection modes. \n <b>disable</b> - Chose this mode to disable encryption of communication between Airbyte and source database\n <b>allow</b> - Chose this mode to enable encryption only when required by the source database\n <b>prefer</b> - Chose this mode to allow unencrypted connection only if the source database does not support encryption\n <b>require</b> - Chose this mode to always require encryption. If the source database server does not support encryption, connection will fail\n <b>verify-ca</b> - Chose this mode to always require encryption and to verify that the source database server has a valid SSL certificate\n <b>verify-full</b> - This is the most secure mode. Chose this mode to always require encryption and to verify the identity of the source database server\n See more information - <a href=\"https://jdbc.postgresql.org/documentation/head/ssl-client.html\"> in the docs</a>.",
"type": "object",
"order": 7,
"oneOf": [
{
"title": "disable",
"additionalProperties": false,
"description": "Disable SSL.",
"required": ["mode"],
"properties": {
"mode": {
"type": "string",
"const": "disable",
"enum": ["disable"],
"default": "disable",
"order": 0
}
}
},
{
"title": "allow",
"additionalProperties": false,
"description": "Allow SSL mode.",
"required": ["mode"],
"properties": {
"mode": {
"type": "string",
"const": "allow",
"enum": ["allow"],
"default": "allow",
"order": 0
}
}
},
{
"title": "prefer",
"additionalProperties": false,
"description": "Prefer SSL mode.",
"required": ["mode"],
"properties": {
"mode": {
"type": "string",
"const": "prefer",
"enum": ["prefer"],
"default": "prefer",
"order": 0
}
}
},
{
"title": "require",
"additionalProperties": false,
"description": "Require SSL mode.",
"required": ["mode"],
"properties": {
"mode": {
"type": "string",
"const": "require",
"enum": ["require"],
"default": "require",
"order": 0
}
}
},
{
"title": "verify-ca",
"additionalProperties": false,
"description": "Verify-ca SSL mode.",
"required": ["mode", "ca_certificate"],
"properties": {
"mode": {
"type": "string",
"const": "verify-ca",
"enum": ["verify-ca"],
"default": "verify-ca",
"order": 0
},
"ca_certificate": {
"type": "string",
"title": "CA certificate",
"description": "CA certificate",
"airbyte_secret": true,
"multiline": true,
"order": 1
},
"client_key_password": {
"type": "string",
"title": "Client key password (Optional)",
"description": "Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically.",
"airbyte_secret": true,
"order": 4
}
}
},
{
"title": "verify-full",
"additionalProperties": false,
"description": "Verify-full SSL mode.",
"required": [
"mode",
"ca_certificate",
"client_certificate",
"client_key"
],
"properties": {
"mode": {
"type": "string",
"const": "verify-full",
"enum": ["verify-full"],
"default": "verify-full",
"order": 0
},
"ca_certificate": {
"type": "string",
"title": "CA certificate",
"description": "CA certificate",
"airbyte_secret": true,
"multiline": true,
"order": 1
},
"client_certificate": {
"type": "string",
"title": "Client certificate",
"description": "Client certificate",
"airbyte_secret": true,
"multiline": true,
"order": 2
},
"client_key": {
"type": "string",
"title": "Client key",
"description": "Client key",
"airbyte_secret": true,
"multiline": true,
"order": 3
},
"client_key_password": {
"type": "string",
"title": "Client key password (Optional)",
"description": "Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically.",
"airbyte_secret": true,
"order": 4
}
}
}
]
},
"replication_method": {
"type": "object",
"title": "Replication Method",

153
airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/AbstractPostgresSourceSSLCertificateAcceptanceTest.java Normal file
View File

@@ -0,0 +1,153 @@
/*
* Copyright (c) 2022 Airbyte, Inc., all rights reserved.
*/
package io.airbyte.integrations.io.airbyte.integration_tests.sources;
import static io.airbyte.db.PostgresUtils.getCertificate;
import com.fasterxml.jackson.databind.JsonNode;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Lists;
import io.airbyte.commons.json.Jsons;
import io.airbyte.db.Database;
import io.airbyte.db.PostgresUtils;
import io.airbyte.db.factory.DSLContextFactory;
import io.airbyte.db.factory.DatabaseDriver;
import io.airbyte.integrations.base.ssh.SshHelpers;
import io.airbyte.integrations.standardtest.source.SourceAcceptanceTest;
import io.airbyte.integrations.standardtest.source.TestDestinationEnv;
import io.airbyte.protocol.models.CatalogHelpers;
import io.airbyte.protocol.models.ConfiguredAirbyteCatalog;
import io.airbyte.protocol.models.ConfiguredAirbyteStream;
import io.airbyte.protocol.models.ConnectorSpecification;
import io.airbyte.protocol.models.DestinationSyncMode;
import io.airbyte.protocol.models.Field;
import io.airbyte.protocol.models.JsonSchemaType;
import io.airbyte.protocol.models.SyncMode;
import java.util.HashMap;
import java.util.List;
import org.jooq.DSLContext;
import org.jooq.SQLDialect;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.utility.DockerImageName;
public abstract class AbstractPostgresSourceSSLCertificateAcceptanceTest extends SourceAcceptanceTest {
private static final String STREAM_NAME = "public.id_and_name";
private static final String STREAM_NAME2 = "public.starships";
private static final String STREAM_NAME_MATERIALIZED_VIEW = "public.testview";
private PostgreSQLContainer<?> container;
private JsonNode config;
protected static final String PASSWORD = "Passw0rd";
protected static PostgresUtils.Certificate certs;
@Override
protected void setupEnvironment(final TestDestinationEnv environment) throws Exception {
container = new PostgreSQLContainer<>(DockerImageName.parse("postgres:bullseye")
.asCompatibleSubstituteFor("postgres"));
container.start();
certs = getCertificate(container);
final JsonNode replicationMethod = Jsons.jsonNode(ImmutableMap.builder()
.put("method", "Standard")
.build());
config = Jsons.jsonNode(ImmutableMap.builder()
.put("host", container.getHost())
.put("port", container.getFirstMappedPort())
.put("database", container.getDatabaseName())
.put("schemas", Jsons.jsonNode(List.of("public")))
.put("username", container.getUsername())
.put("password", container.getPassword())
.put("ssl", true)
.put("replication_method", replicationMethod)
.put("ssl_mode", getCertificateConfiguration())
.build());
try (final DSLContext dslContext = DSLContextFactory.create(
config.get("username").asText(),
config.get("password").asText(),
DatabaseDriver.POSTGRESQL.getDriverClassName(),
String.format(DatabaseDriver.POSTGRESQL.getUrlFormatString(),
config.get("host").asText(),
config.get("port").asInt(),
config.get("database").asText()),
SQLDialect.POSTGRES)) {
final Database database = new Database(dslContext);
database.query(ctx -> {
ctx.fetch("CREATE TABLE id_and_name(id INTEGER, name VARCHAR(200));");
ctx.fetch("INSERT INTO id_and_name (id, name) VALUES (1,'picard'), (2, 'crusher'), (3, 'vash');");
ctx.fetch("CREATE TABLE starships(id INTEGER, name VARCHAR(200));");
ctx.fetch("INSERT INTO starships (id, name) VALUES (1,'enterprise-d'), (2, 'defiant'), (3, 'yamato');");
ctx.fetch("CREATE MATERIALIZED VIEW testview AS select * from id_and_name where id = '2';");
return null;
});
}
}
public abstract ImmutableMap getCertificateConfiguration();
@Override
protected void tearDown(final TestDestinationEnv testEnv) {
container.close();
}
@Override
protected String getImageName() {
return "airbyte/source-postgres:dev";
}
@Override
protected ConnectorSpecification getSpec() throws Exception {
return SshHelpers.getSpecAndInjectSsh();
}
@Override
protected JsonNode getConfig() {
return config;
}
@Override
protected ConfiguredAirbyteCatalog getConfiguredCatalog() {
return new ConfiguredAirbyteCatalog().withStreams(Lists.newArrayList(
new ConfiguredAirbyteStream()
.withSyncMode(SyncMode.INCREMENTAL)
.withCursorField(Lists.newArrayList("id"))
.withDestinationSyncMode(DestinationSyncMode.APPEND)
.withStream(CatalogHelpers.createAirbyteStream(
STREAM_NAME,
Field.of("id", JsonSchemaType.NUMBER),
Field.of("name", JsonSchemaType.STRING))
.withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL))),
new ConfiguredAirbyteStream()
.withSyncMode(SyncMode.INCREMENTAL)
.withCursorField(Lists.newArrayList("id"))
.withDestinationSyncMode(DestinationSyncMode.APPEND)
.withStream(CatalogHelpers.createAirbyteStream(
STREAM_NAME2,
Field.of("id", JsonSchemaType.NUMBER),
Field.of("name", JsonSchemaType.STRING))
.withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL))),
new ConfiguredAirbyteStream()
.withSyncMode(SyncMode.INCREMENTAL)
.withCursorField(Lists.newArrayList("id"))
.withDestinationSyncMode(DestinationSyncMode.APPEND)
.withStream(CatalogHelpers.createAirbyteStream(
STREAM_NAME_MATERIALIZED_VIEW,
Field.of("id", JsonSchemaType.NUMBER),
Field.of("name", JsonSchemaType.STRING))
.withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL)))));
}
@Override
protected JsonNode getState() {
return Jsons.jsonNode(new HashMap<>());
}
@Override
protected boolean supportsPerStream() {
return true;
}
}

20
airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/PostgresSourceSSLCaCertificateAcceptanceTest.java Normal file
View File

@@ -0,0 +1,20 @@
/*
* Copyright (c) 2022 Airbyte, Inc., all rights reserved.
*/
package io.airbyte.integrations.io.airbyte.integration_tests.sources;
import com.google.common.collect.ImmutableMap;
public class PostgresSourceSSLCaCertificateAcceptanceTest extends AbstractPostgresSourceSSLCertificateAcceptanceTest {
@Override
public ImmutableMap getCertificateConfiguration() {
return ImmutableMap.builder()
.put("mode", "verify-ca")
.put("ca_certificate", certs.getCaCertificate())
.put("client_key_password", PASSWORD)
.build();
}
}

22
airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/PostgresSourceSSLFullCertificateAcceptanceTest.java Normal file
View File

@@ -0,0 +1,22 @@
/*
* Copyright (c) 2022 Airbyte, Inc., all rights reserved.
*/
package io.airbyte.integrations.io.airbyte.integration_tests.sources;
import com.google.common.collect.ImmutableMap;
public class PostgresSourceSSLFullCertificateAcceptanceTest extends AbstractPostgresSourceSSLCertificateAcceptanceTest {
@Override
public ImmutableMap getCertificateConfiguration() {
return ImmutableMap.builder()
.put("mode", "verify-full")
.put("ca_certificate", certs.getCaCertificate())
.put("client_certificate", certs.getClientCertificate())
.put("client_key", certs.getClientKey())
.put("client_key_password", PASSWORD)
.build();
}
}

18
airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/PostgresSourceStrictEncryptAcceptanceTest.java
View File

@@ -4,12 +4,15 @@
package io.airbyte.integrations.io.airbyte.integration_tests.sources;
import static io.airbyte.db.PostgresUtils.getCertificate;
import com.fasterxml.jackson.databind.JsonNode;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Lists;
import io.airbyte.commons.json.Jsons;
import io.airbyte.commons.resources.MoreResources;
import io.airbyte.db.Database;
import io.airbyte.db.PostgresUtils;
import io.airbyte.db.factory.DSLContextFactory;
import io.airbyte.db.factory.DatabaseDriver;
import io.airbyte.integrations.base.ssh.SshHelpers;
@@ -44,11 +47,15 @@ public class PostgresSourceStrictEncryptAcceptanceTest extends SourceAcceptanceT
private PostgreSQLContainer<?> container;
private JsonNode config;
protected static final String PASSWORD = "Passw0rd";
protected static PostgresUtils.Certificate certs;
@Override
protected void setupEnvironment(final TestDestinationEnv environment) throws Exception {
container = new PostgreSQLContainer<>(DockerImageName.parse("marcosmarxm/postgres-ssl:dev").asCompatibleSubstituteFor("postgres"))
.withCommand("postgres -c ssl=on -c ssl_cert_file=/var/lib/postgresql/server.crt -c ssl_key_file=/var/lib/postgresql/server.key");
container = new PostgreSQLContainer<>(DockerImageName.parse("postgres:bullseye")
.asCompatibleSubstituteFor("postgres"));
container.start();
certs = getCertificate(container);
final JsonNode replicationMethod = Jsons.jsonNode(ImmutableMap.builder()
.put("method", "Standard")
.build());
@@ -59,6 +66,13 @@ public class PostgresSourceStrictEncryptAcceptanceTest extends SourceAcceptanceT
.put("username", container.getUsername())
.put("password", container.getPassword())
.put("replication_method", replicationMethod)
.put("ssl_mode", ImmutableMap.builder()
.put("mode", "verify-full")
.put("ca_certificate", certs.getCaCertificate())
.put("client_certificate", certs.getClientCertificate())
.put("client_key", certs.getClientKey())
.put("client_key_password", PASSWORD)
.build())
.build());
try (final DSLContext dslContext = DSLContextFactory.create(

1
airbyte-integrations/connectors/source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresSourceSSLTest.java
View File

@@ -147,6 +147,7 @@ class PostgresSourceSSLTest {
.put("username", psqlDb.getUsername())
.put("password", psqlDb.getPassword())
.put("ssl", true)
.put("ssl_mode", ImmutableMap.builder().put("mode", "require").build())
.build());
}

134
airbyte-integrations/connectors/source-postgres/src/test/resources/expected_spec.json
View File

@@ -60,6 +60,140 @@
"type": "string",
"order": 6
},
"ssl_mode": {
"title": "SSL modes",
"description": "SSL connection modes. \n <b>disable</b> - Chose this mode to disable encryption of communication between Airbyte and source database\n <b>allow</b> - Chose this mode to enable encryption only when required by the source database\n <b>prefer</b> - Chose this mode to allow unencrypted connection only if the source database does not support encryption\n <b>require</b> - Chose this mode to always require encryption. If the source database server does not support encryption, connection will fail\n <b>verify-ca</b> - Chose this mode to always require encryption and to verify that the source database server has a valid SSL certificate\n <b>verify-full</b> - This is the most secure mode. Chose this mode to always require encryption and to verify the identity of the source database server\n See more information - <a href=\"https://jdbc.postgresql.org/documentation/head/ssl-client.html\"> in the docs</a>.",
"type": "object",
"order": 7,
"oneOf": [
{
"title": "allow",
"additionalProperties": false,
"description": "Allow SSL mode.",
"required": ["mode"],
"properties": {
"mode": {
"type": "string",
"const": "allow",
"enum": ["allow"],
"default": "allow",
"order": 0
}
}
},
{
"title": "prefer",
"additionalProperties": false,
"description": "Prefer SSL mode.",
"required": ["mode"],
"properties": {
"mode": {
"type": "string",
"const": "prefer",
"enum": ["prefer"],
"default": "prefer",
"order": 0
}
}
},
{
"title": "require",
"additionalProperties": false,
"description": "Require SSL mode.",
"required": ["mode"],
"properties": {
"mode": {
"type": "string",
"const": "require",
"enum": ["require"],
"default": "require",
"order": 0
}
}
},
{
"title": "verify-ca",
"additionalProperties": false,
"description": "Verify-ca SSL mode.",
"required": ["mode", "ca_certificate"],
"properties": {
"mode": {
"type": "string",
"const": "verify-ca",
"enum": ["verify-ca"],
"default": "verify-ca",
"order": 0
},
"ca_certificate": {
"type": "string",
"title": "CA certificate",
"description": "CA certificate",
"airbyte_secret": true,
"multiline": true,
"order": 1
},
"client_key_password": {
"type": "string",
"title": "Client key password (Optional)",
"description": "Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically.",
"airbyte_secret": true,
"order": 4
}
}
},
{
"title": "verify-full",
"additionalProperties": false,
"description": "Verify-full SSL mode.",
"required": [
"mode",
"ca_certificate",
"client_certificate",
"client_key"
],
"properties": {
"mode": {
"type": "string",
"const": "verify-full",
"enum": ["verify-full"],
"default": "verify-full",
"order": 0
},
"ca_certificate": {
"type": "string",
"title": "CA certificate",
"description": "CA certificate",
"airbyte_secret": true,
"multiline": true,
"order": 1
},
"client_certificate": {
"type": "string",
"title": "Client certificate",
"description": "Client certificate",
"airbyte_secret": true,
"multiline": true,
"order": 2
},
"client_key": {
"type": "string",
"title": "Client key",
"description": "Client key",
"airbyte_secret": true,
"multiline": true,
"order": 3
},
"client_key_password": {
"type": "string",
"title": "Client key password (Optional)",
"description": "Password for keystorage. This field is optional. If you do not add it - the password will be generated automatically.",
"airbyte_secret": true,
"order": 4
}
}
}
]
},
"replication_method": {
"type": "object",
"title": "Replication Method",

133
docs/integrations/sources/postgres.md
View File

@@ -325,70 +325,71 @@ One optimization on the Airbyte side is to break one large and long sync into mu
## Changelog
| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------|
| 0.4.33 | 2022-07-14 | [14586](https://github.com/airbytehq/airbyte/pull/14586) | Validate source JDBC url parameters |
| 0.4.32 | 2022-07-07 | [14694](https://github.com/airbytehq/airbyte/pull/14694) | Force to produce LEGACY state if the use stream capable feature flag is set to false |
| 0.4.31 | 2022-07-07 | [14447](https://github.com/airbytehq/airbyte/pull/14447) | Under CDC mode, retrieve only those tables included in the publications |
| 0.4.30 | 2022-06-30 | [14251](https://github.com/airbytehq/airbyte/pull/14251) | Use more simple and comprehensive query to get selectable tables |
| 0.4.29 | 2022-06-29 | [14265](https://github.com/airbytehq/airbyte/pull/14265) | Upgrade postgresql JDBC version to 42.3.5 |
| 0.4.28 | 2022-06-23 | [14077](https://github.com/airbytehq/airbyte/pull/14077) | Use the new state management |
| 0.4.26 | 2022-06-17 | [13864](https://github.com/airbytehq/airbyte/pull/13864) | Updated stacktrace format for any trace message errors |
| 0.4.25 | 2022-06-15 | [13823](https://github.com/airbytehq/airbyte/pull/13823) | Publish adaptive postgres source that enforces ssl on cloud + Debezium version upgrade to 1.9.2 from 1.4.2 |
| 0.4.24 | 2022-06-14 | [13549](https://github.com/airbytehq/airbyte/pull/13549) | Fixed truncated precision if the value of microseconds or seconds is 0 |
| 0.4.23 | 2022-06-13 | [13655](https://github.com/airbytehq/airbyte/pull/13745) | Fixed handling datetime cursors when upgrading from older versions of the connector |
| 0.4.22 | 2022-06-09 | [13655](https://github.com/airbytehq/airbyte/pull/13655) | Fixed bug with unsupported date-time datatypes during incremental sync |
| 0.4.21 | 2022-06-06 | [13435](https://github.com/airbytehq/airbyte/pull/13435) | Adjust JDBC fetch size based on max memory and max row size |
| 0.4.20 | 2022-06-02 | [13367](https://github.com/airbytehq/airbyte/pull/13367) | Added convertion hstore to json format |
| 0.4.19 | 2022-05-25 | [13166](https://github.com/airbytehq/airbyte/pull/13166) | Added timezone awareness and handle BC dates |
| 0.4.18 | 2022-05-25 | [13083](https://github.com/airbytehq/airbyte/pull/13083) | Add support for tsquey type |
| 0.4.17 | 2022-05-19 | [13016](https://github.com/airbytehq/airbyte/pull/13016) | CDC modify schema to allow null values |
| 0.4.16 | 2022-05-14 | [12840](https://github.com/airbytehq/airbyte/pull/12840) | Added custom JDBC parameters field |
| 0.4.15 | 2022-05-13 | [12834](https://github.com/airbytehq/airbyte/pull/12834) | Fix the bug that the connector returns empty catalog for Azure Postgres database |
| 0.4.14 | 2022-05-08 | [12689](https://github.com/airbytehq/airbyte/pull/12689) | Add table retrieval according to role-based `SELECT` privilege |
| 0.4.13 | 2022-05-05 | [10230](https://github.com/airbytehq/airbyte/pull/10230) | Explicitly set null value for field in json |
| 0.4.12 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption |
| 0.4.11 | 2022-04-11 | [11729](https://github.com/airbytehq/airbyte/pull/11729) | Bump mina-sshd from 2.7.0 to 2.8.0 |
| 0.4.10 | 2022-04-08 | [11798](https://github.com/airbytehq/airbyte/pull/11798) | Fixed roles for fetching materialized view processing |
| 0.4.8 | 2022-02-21 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats |
| 0.4.7 | 2022-02-18 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Updated timestamp transformation with microseconds |
| 0.4.6 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | (unpublished) Add `-XX:+ExitOnOutOfMemoryError` JVM option |
| 0.4.5 | 2022-02-08 | [10173](https://github.com/airbytehq/airbyte/pull/10173) | Improved discovering tables in case if user does not have permissions to any table |
| 0.4.4 | 2022-01-26 | [9807](https://github.com/airbytehq/airbyte/pull/9807) | Update connector fields title/description |
| 0.4.3 | 2022-01-24 | [9554](https://github.com/airbytehq/airbyte/pull/9554) | Allow handling of java sql date in CDC |
| 0.4.2 | 2022-01-13 | [9360](https://github.com/airbytehq/airbyte/pull/9360) | Added schema selection |
| 0.4.1 | 2022-01-05 | [9116](https://github.com/airbytehq/airbyte/pull/9116) | Added materialized views processing |
| 0.4.0 | 2021-12-13 | [8726](https://github.com/airbytehq/airbyte/pull/8726) | Support all Postgres types |
| 0.3.17 | 2021-12-01 | [8371](https://github.com/airbytehq/airbyte/pull/8371) | Fixed incorrect handling "\n" in ssh key |
| 0.3.16 | 2021-11-28 | [7995](https://github.com/airbytehq/airbyte/pull/7995) | Fixed money type with amount > 1000 |
| 0.3.15 | 2021-11-26 | [8066](https://github.com/airbytehq/airbyte/pull/8266) | Fixed the case, when Views are not listed during schema discovery |
| 0.3.14 | 2021-11-17 | [8010](https://github.com/airbytehq/airbyte/pull/8010) | Added checking of privileges before table internal discovery |
| 0.3.13 | 2021-10-26 | [7339](https://github.com/airbytehq/airbyte/pull/7339) | Support or improve support for Interval, Money, Date, various geometric data types, inventory_items, and others |
| 0.3.12 | 2021-09-30 | [6585](https://github.com/airbytehq/airbyte/pull/6585) | Improved SSH Tunnel key generation steps |
| 0.3.11 | 2021-09-02 | [5742](https://github.com/airbytehq/airbyte/pull/5742) | Add SSH Tunnel support |
| 0.3.9 | 2021-08-17 | [5304](https://github.com/airbytehq/airbyte/pull/5304) | Fix CDC OOM issue |
| 0.3.8 | 2021-08-13 | [4699](https://github.com/airbytehq/airbyte/pull/4699) | Added json config validator |
| 0.3.4 | 2021-06-09 | [3973](https://github.com/airbytehq/airbyte/pull/3973) | Add `AIRBYTE_ENTRYPOINT` for Kubernetes support |
| 0.3.3 | 2021-06-08 | [3960](https://github.com/airbytehq/airbyte/pull/3960) | Add method field in specification parameters |
| 0.3.2 | 2021-05-26 | [3179](https://github.com/airbytehq/airbyte/pull/3179) | Remove `isCDC` logging |
| 0.3.1 | 2021-04-21 | [2878](https://github.com/airbytehq/airbyte/pull/2878) | Set defined cursor for CDC |
| 0.3.0 | 2021-04-21 | [2990](https://github.com/airbytehq/airbyte/pull/2990) | Support namespaces |
| 0.2.7 | 2021-04-16 | [2923](https://github.com/airbytehq/airbyte/pull/2923) | SSL spec as optional |
| 0.2.6 | 2021-04-16 | [2757](https://github.com/airbytehq/airbyte/pull/2757) | Support SSL connection |
| 0.2.5 | 2021-04-12 | [2859](https://github.com/airbytehq/airbyte/pull/2859) | CDC bugfix |
| 0.2.4 | 2021-04-09 | [2548](https://github.com/airbytehq/airbyte/pull/2548) | Support CDC |
| 0.2.3 | 2021-03-28 | [2600](https://github.com/airbytehq/airbyte/pull/2600) | Add NCHAR and NVCHAR support to DB and cursor type casting |
| 0.2.2 | 2021-03-26 | [2460](https://github.com/airbytehq/airbyte/pull/2460) | Destination supports destination sync mode |
| 0.2.1 | 2021-03-18 | [2488](https://github.com/airbytehq/airbyte/pull/2488) | Sources support primary keys |
| 0.2.0 | 2021-03-09 | [2238](https://github.com/airbytehq/airbyte/pull/2238) | Protocol allows future/unknown properties |
| 0.1.13 | 2021-02-02 | [1887](https://github.com/airbytehq/airbyte/pull/1887) | Migrate AbstractJdbcSource to use iterators |
| 0.1.12 | 2021-01-25 | [1746](https://github.com/airbytehq/airbyte/pull/1746) | Fix NPE in State Decorator |
| 0.1.11 | 2021-01-25 | [1765](https://github.com/airbytehq/airbyte/pull/1765) | Add field titles to specification |
| 0.1.10 | 2021-01-19 | [1724](https://github.com/airbytehq/airbyte/pull/1724) | Fix JdbcSource handling of tables with same names in different schemas |
| 0.1.9 | 2021-01-14 | [1655](https://github.com/airbytehq/airbyte/pull/1655) | Fix JdbcSource OOM |
| 0.1.8 | 2021-01-13 | [1588](https://github.com/airbytehq/airbyte/pull/1588) | Handle invalid numeric values in JDBC source |
| 0.1.7 | 2021-01-08 | [1307](https://github.com/airbytehq/airbyte/pull/1307) | Migrate Postgres and MySql to use new JdbcSource |
| 0.1.6 | 2020-12-09 | [1172](https://github.com/airbytehq/airbyte/pull/1172) | Support incremental sync |
| 0.1.5 | 2020-11-30 | [1038](https://github.com/airbytehq/airbyte/pull/1038) | Change JDBC sources to discover more than standard schemas |
| 0.1.4 | 2020-11-30 | [1046](https://github.com/airbytehq/airbyte/pull/1046) | Add connectors using an index YAML file |
| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------|
| 0.4.34 | 2022-07-17 | [13840](https://github.com/airbytehq/airbyte/pull/13840) | Added the ability to connect using different SSL modes and SSL certificates. |
| 0.4.33 | 2022-07-14 | [14586](https://github.com/airbytehq/airbyte/pull/14586) | Validate source JDBC url parameters |
| 0.4.32 | 2022-07-07 | [14694](https://github.com/airbytehq/airbyte/pull/14694) | Force to produce LEGACY state if the use stream capable feature flag is set to false |
| 0.4.31 | 2022-07-07 | [14447](https://github.com/airbytehq/airbyte/pull/14447) | Under CDC mode, retrieve only those tables included in the publications |
| 0.4.30 | 2022-06-30 | [14251](https://github.com/airbytehq/airbyte/pull/14251) | Use more simple and comprehensive query to get selectable tables |
| 0.4.29 | 2022-06-29 | [14265](https://github.com/airbytehq/airbyte/pull/14265) | Upgrade postgresql JDBC version to 42.3.5 |
| 0.4.28 | 2022-06-23 | [14077](https://github.com/airbytehq/airbyte/pull/14077) | Use the new state management |
| 0.4.26 | 2022-06-17 | [13864](https://github.com/airbytehq/airbyte/pull/13864) | Updated stacktrace format for any trace message errors |
| 0.4.25 | 2022-06-15 | [13823](https://github.com/airbytehq/airbyte/pull/13823) | Publish adaptive postgres source that enforces ssl on cloud + Debezium version upgrade to 1.9.2 from 1.4.2 |
| 0.4.24 | 2022-06-14 | [13549](https://github.com/airbytehq/airbyte/pull/13549) | Fixed truncated precision if the value of microseconds or seconds is 0 |
| 0.4.23 | 2022-06-13 | [13655](https://github.com/airbytehq/airbyte/pull/13745) | Fixed handling datetime cursors when upgrading from older versions of the connector |
| 0.4.22 | 2022-06-09 | [13655](https://github.com/airbytehq/airbyte/pull/13655) | Fixed bug with unsupported date-time datatypes during incremental sync |
| 0.4.21 | 2022-06-06 | [13435](https://github.com/airbytehq/airbyte/pull/13435) | Adjust JDBC fetch size based on max memory and max row size |
| 0.4.20 | 2022-06-02 | [13367](https://github.com/airbytehq/airbyte/pull/13367) | Added convertion hstore to json format |
| 0.4.19 | 2022-05-25 | [13166](https://github.com/airbytehq/airbyte/pull/13166) | Added timezone awareness and handle BC dates |
| 0.4.18 | 2022-05-25 | [13083](https://github.com/airbytehq/airbyte/pull/13083) | Add support for tsquey type |
| 0.4.17 | 2022-05-19 | [13016](https://github.com/airbytehq/airbyte/pull/13016) | CDC modify schema to allow null values |
| 0.4.16 | 2022-05-14 | [12840](https://github.com/airbytehq/airbyte/pull/12840) | Added custom JDBC parameters field |
| 0.4.15 | 2022-05-13 | [12834](https://github.com/airbytehq/airbyte/pull/12834) | Fix the bug that the connector returns empty catalog for Azure Postgres database |
| 0.4.14 | 2022-05-08 | [12689](https://github.com/airbytehq/airbyte/pull/12689) | Add table retrieval according to role-based `SELECT` privilege |
| 0.4.13 | 2022-05-05 | [10230](https://github.com/airbytehq/airbyte/pull/10230) | Explicitly set null value for field in json |
| 0.4.12 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption |
| 0.4.11 | 2022-04-11 | [11729](https://github.com/airbytehq/airbyte/pull/11729) | Bump mina-sshd from 2.7.0 to 2.8.0 |
| 0.4.10 | 2022-04-08 | [11798](https://github.com/airbytehq/airbyte/pull/11798) | Fixed roles for fetching materialized view processing |
| 0.4.8 | 2022-02-21 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats |
| 0.4.7 | 2022-02-18 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Updated timestamp transformation with microseconds |
| 0.4.6 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | (unpublished) Add `-XX:+ExitOnOutOfMemoryError` JVM option |
| 0.4.5 | 2022-02-08 | [10173](https://github.com/airbytehq/airbyte/pull/10173) | Improved discovering tables in case if user does not have permissions to any table |
| 0.4.4 | 2022-01-26 | [9807](https://github.com/airbytehq/airbyte/pull/9807) | Update connector fields title/description |
| 0.4.3 | 2022-01-24 | [9554](https://github.com/airbytehq/airbyte/pull/9554) | Allow handling of java sql date in CDC |
| 0.4.2 | 2022-01-13 | [9360](https://github.com/airbytehq/airbyte/pull/9360) | Added schema selection |
| 0.4.1 | 2022-01-05 | [9116](https://github.com/airbytehq/airbyte/pull/9116) | Added materialized views processing |
| 0.4.0 | 2021-12-13 | [8726](https://github.com/airbytehq/airbyte/pull/8726) | Support all Postgres types |
| 0.3.17 | 2021-12-01 | [8371](https://github.com/airbytehq/airbyte/pull/8371) | Fixed incorrect handling "\n" in ssh key |
| 0.3.16 | 2021-11-28 | [7995](https://github.com/airbytehq/airbyte/pull/7995) | Fixed money type with amount > 1000 |
| 0.3.15 | 2021-11-26 | [8066](https://github.com/airbytehq/airbyte/pull/8266) | Fixed the case, when Views are not listed during schema discovery |
| 0.3.14 | 2021-11-17 | [8010](https://github.com/airbytehq/airbyte/pull/8010) | Added checking of privileges before table internal discovery |
| 0.3.13 | 2021-10-26 | [7339](https://github.com/airbytehq/airbyte/pull/7339) | Support or improve support for Interval, Money, Date, various geometric data types, inventory_items, and others |
| 0.3.12 | 2021-09-30 | [6585](https://github.com/airbytehq/airbyte/pull/6585) | Improved SSH Tunnel key generation steps |
| 0.3.11 | 2021-09-02 | [5742](https://github.com/airbytehq/airbyte/pull/5742) | Add SSH Tunnel support |
| 0.3.9 | 2021-08-17 | [5304](https://github.com/airbytehq/airbyte/pull/5304) | Fix CDC OOM issue |
| 0.3.8 | 2021-08-13 | [4699](https://github.com/airbytehq/airbyte/pull/4699) | Added json config validator |
| 0.3.4 | 2021-06-09 | [3973](https://github.com/airbytehq/airbyte/pull/3973) | Add `AIRBYTE_ENTRYPOINT` for Kubernetes support |
| 0.3.3 | 2021-06-08 | [3960](https://github.com/airbytehq/airbyte/pull/3960) | Add method field in specification parameters |
| 0.3.2 | 2021-05-26 | [3179](https://github.com/airbytehq/airbyte/pull/3179) | Remove `isCDC` logging |
| 0.3.1 | 2021-04-21 | [2878](https://github.com/airbytehq/airbyte/pull/2878) | Set defined cursor for CDC |
| 0.3.0 | 2021-04-21 | [2990](https://github.com/airbytehq/airbyte/pull/2990) | Support namespaces |
| 0.2.7 | 2021-04-16 | [2923](https://github.com/airbytehq/airbyte/pull/2923) | SSL spec as optional |
| 0.2.6 | 2021-04-16 | [2757](https://github.com/airbytehq/airbyte/pull/2757) | Support SSL connection |
| 0.2.5 | 2021-04-12 | [2859](https://github.com/airbytehq/airbyte/pull/2859) | CDC bugfix |
| 0.2.4 | 2021-04-09 | [2548](https://github.com/airbytehq/airbyte/pull/2548) | Support CDC |
| 0.2.3 | 2021-03-28 | [2600](https://github.com/airbytehq/airbyte/pull/2600) | Add NCHAR and NVCHAR support to DB and cursor type casting |
| 0.2.2 | 2021-03-26 | [2460](https://github.com/airbytehq/airbyte/pull/2460) | Destination supports destination sync mode |
| 0.2.1 | 2021-03-18 | [2488](https://github.com/airbytehq/airbyte/pull/2488) | Sources support primary keys |
| 0.2.0 | 2021-03-09 | [2238](https://github.com/airbytehq/airbyte/pull/2238) | Protocol allows future/unknown properties |
| 0.1.13 | 2021-02-02 | [1887](https://github.com/airbytehq/airbyte/pull/1887) | Migrate AbstractJdbcSource to use iterators |
| 0.1.12 | 2021-01-25 | [1746](https://github.com/airbytehq/airbyte/pull/1746) | Fix NPE in State Decorator |
| 0.1.11 | 2021-01-25 | [1765](https://github.com/airbytehq/airbyte/pull/1765) | Add field titles to specification |
| 0.1.10 | 2021-01-19 | [1724](https://github.com/airbytehq/airbyte/pull/1724) | Fix JdbcSource handling of tables with same names in different schemas |
| 0.1.9 | 2021-01-14 | [1655](https://github.com/airbytehq/airbyte/pull/1655) | Fix JdbcSource OOM |
| 0.1.8 | 2021-01-13 | [1588](https://github.com/airbytehq/airbyte/pull/1588) | Handle invalid numeric values in JDBC source |
| 0.1.7 | 2021-01-08 | [1307](https://github.com/airbytehq/airbyte/pull/1307) | Migrate Postgres and MySql to use new JdbcSource |
| 0.1.6 | 2020-12-09 | [1172](https://github.com/airbytehq/airbyte/pull/1172) | Support incremental sync |
| 0.1.5 | 2020-11-30 | [1038](https://github.com/airbytehq/airbyte/pull/1038) | Change JDBC sources to discover more than standard schemas |
| 0.1.4 | 2020-11-30 | [1046](https://github.com/airbytehq/airbyte/pull/1046) | Add connectors using an index YAML file |