1
0
mirror of synced 2026-01-03 15:04:01 -05:00
Commit Graph

161 Commits

Author SHA1 Message Date
Evan Tahler
e6b06a88ac AirbyteEstimateTraceMessage (#18875)
* `AirbyteEstimateTraceMessage`

* Add PR number

* fix method name

* Lint

* Lint

* fix merge

* Update docs/understanding-airbyte/airbyte-protocol.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* `EstimateType` sub type in python

* lint

Co-authored-by: Davin Chia <davinchia@gmail.com>
2022-11-07 12:45:39 -08:00
Ella Rohm-Ensing
e6bfe10278 [docs] Use correct header heirarchy in airbyte-protocol docs (#18917)
* Use correct header heirarchy

* Use h2 for Actor Specification

* Revert indentation of key concepts
2022-11-07 10:39:58 -07:00
Greg Solovyev
7b9a097081 Add normalization changelog and bump normalization version in platform (#18813) 2022-11-01 15:36:31 -07:00
Evan Tahler
02459e8354 Protocol Change: AirbyteControlMessage.ConnectorConfig (#17907)
* Protocol Change: AirbyteConfigMessage

* update PR link in docs

* Lint

* Update python files

* Update docs/understanding-airbyte/airbyte-protocol.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/understanding-airbyte/airbyte-protocol.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* `AirbyteConfigMessage` -> `AirbyteConnectorConfigMessage`

* AirbyteOrchestratorMessage

* Update docs

* `AirbyteControlConnectorConfigMessage`

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-10-28 11:11:31 -07:00
Taras Korenko
d840a8a80d Simplify the OSS documentation deploy system (#2670) (#18598)
+ Adds Slack notification about failed Docs builds
  (slack notifications fall into "#oss-master-build-failure" for now)
+ (while here) Unbreaks docs build
2022-10-28 17:08:06 +03:00
Edward Gao
aea4c5392c Protocol change: Define a set of well-known data types (#17486)
* add WellKnownTypes.yaml

* rename to snakecase + put in airbyte-protocol

* add examples

* more descriptoins

* descriptions, more restrictions, better regex

* update documentation

* explicitly call out BC support
2022-10-27 14:04:15 -07:00
Edward Gao
bd0c2388df Destination Clickhouse: Publish normalization after removing native port (#17896)
* bump version for publish

* add link for publish pr
2022-10-12 13:51:55 -07:00
Topher Lubaway
9c46e5a560 Fixes relative links above docs to be URIs (#17850)
also adds a missing exit 1 to deploy docs
when the deploy SHOULD fail
2022-10-11 11:36:33 -05:00
Alexander Marquardt
9cc86d857f Update full-refresh-append.md (#17784)
Added a link to the new article comparing sync modes
2022-10-10 15:55:42 +02:00
Alexander Marquardt
caeb8017d3 Update full-refresh-overwrite.md (#17783)
Added a link to the new article that compares replication modes
2022-10-10 15:55:27 +02:00
Alexander Marquardt
5e30aaff14 Update incremental-append.md (#17785)
Added a link to the new article on choosing a replication mode
2022-10-10 15:55:15 +02:00
Alexander Marquardt
3ea0b5a82d Update incremental-deduped-history.md (#17786)
Add a link to new article on choosing a replication mode
2022-10-10 15:55:05 +02:00
Alexander Marquardt
f5f3e87f21 Update cdc.md (#17787)
* Update cdc.md

Added a link to the article about that explains Airbyte replication modes

* Update cdc.md

Added a link to the CDC "exploration" tutorial
2022-10-10 15:54:48 +02:00
Charles
675e153cb6 jobs db descriptions (#16543) 2022-10-08 23:16:28 -05:00
Charles
2237654392 config db data catalog (#16427) 2022-10-08 23:11:11 -05:00
Simon Späti
8a74d39ac3 📖 Removing existing Glossary of Terms and link to new Data Glossary (#17313) 2022-10-03 12:37:34 +02:00
Damilare Oyediran
33bad16267 Update supported-data-types.md (#17179)
Boolean data type was added to the official docs.
2022-09-29 15:35:13 -03:00
Liren Tu
9356d6adf5 📝 Update json avro conversion doc about logical type union (#17010)
* Update json avro doc to include logical type union

* Add a note about the potential problem

* Add issue link
2022-09-21 15:43:44 -07:00
Alexander Marquardt
2fbb85c3a0 Update incremental-append.md (#16579)
* Update incremental-append.md

Added link to the incremental sync tutorial

* Update incremental-append.md
2022-09-16 17:55:44 +02:00
Alexander Marquardt
005378626c Update incremental-deduped-history.md (#16578)
* Update incremental-deduped-history.md

Added a link to the incremental sync tutorial

* Update incremental-deduped-history.md

* Update incremental-deduped-history.md
2022-09-16 17:55:33 +02:00
Alexander Marquardt
ae85cacba5 Update full-refresh-overwrite.md (#16580) 2022-09-16 17:55:01 +02:00
Alexander Marquardt
7f2b27274b Update full-refresh-append.md (#16581) 2022-09-16 17:54:46 +02:00
Filipp Balakin
17cb363fe1 Normalization: update dbt-clickhouse version from 1.1.7 to 1.1.8 (#16339)
* Bump dbt-clickhouse version from 1.1.7 to 1.1.8

* pin dbt<1.2

* update doc

* pin dbt core lt 1.2

* bump normalization version

* loosen requirements

* fix md

* remove empty line

* bump normalization version in worker

* bump normalization version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2022-09-14 08:27:31 -03:00
Davin Chia
08090bca6b Update the worker/job documentation. Add documentation for container orchestrator. (#16575)
- Add a container orchestrator section.
- Update existing section to include details on Kubernetes vs Docker.
- Move some things around so things are smoother.
2022-09-13 13:29:53 -07:00
Evan Tahler
dcfcb75d0f AirbyteLogMessage.stack_trace for logging messages with related (non-fatal) errors (#16479)
* Test log message from faker

* AirbyteLogMessage gains stack_trace

* Fixup spacing

* bump python protocol

* fixup additionalProperties in faker spec

* bump faker version

* update docs

* use lineSeparator vs \r\n

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-12 16:01:57 -07:00
Evan Tahler
6f2fcff85c fix changelog formatting (#16618)
* fix changelog formatting

* fixup docs
2022-09-12 15:59:50 -07:00
Evan Tahler
3a1df7ea58 Update airbyte-protocol.md 2022-09-12 10:55:54 -07:00
Evan Tahler
8cf60e4ea9 Add changelog for Airbyte Protocol (#16560)
* Add changelog for Airbyte Protocol

* we are on v0.3.0 now
2022-09-12 10:30:01 -07:00
Rodi Reich Zilberman
43b8d8beaf Publish base-normalization (#16545)
* dummy

* update changelog
2022-09-09 15:10:26 -07:00
Tuhai Maksym
f81c5aa667 7198: SSH Tunnel: allow using OPENSSH key format (#15833)
* 7198: SSH Tunnel: allow using OPENSSH key format

* 7198: merge

* 7198: merge fix

* Use apache sshd lib to load private keys for tunnel

* Throw an exception in case private key failed to load

* Fix failing flow when creating NO_TUNNEL wrapper

* bump version numbers

* Address review comments. Fix test

Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
2022-09-09 10:22:03 -07:00
Daemonxiao
d4524032ae 🎉 New Destination: TiDB (#15592)
* Add new destination-tidb

* support sync

* Add normalization-tidb

* fix failed tests

* Add unnest marco

* fmt

* Add new destination-tidb

* support sync

* Add normalization-tidb

* fix failed tests

* Add unnest marco

* fmt

* fmt

* fix integration test

* Update docs/integrations/destinations/tidb.md

Co-authored-by: Xiang Zhang <angwerzx@126.com>

* Update doc

* Update doc

* Update doc

* bump normalization version

* update normalization changelog

* run format

* add dest def

* generat spec

Co-authored-by: Xiang Zhang <angwerzx@126.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2022-08-31 16:50:27 -03:00
Jimmy Ma
4fbe03bebe Add explicit tracking of Airbyte Protocol Version in ConnectorSpecification (#15340)
* Add Version to AirbyteMessage

* Move protocol version to ConnectorSpecification

* Add cdk generated protocol model

* Add protocol_version to the sample ConnectorSpec in the docs

* Update airbyte-protocol/protocol-models/src/main/resources/airbyte_protocol/airbyte_protocol.yaml

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* update doc

* Update CDK changelog

* Update CDK protocol model

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-08-29 14:46:56 -07:00
Greg Solovyev
5819733ab1 Greg/guykoh update dbt clickhouse (#14897)
* Update dbt-clickhouse version to 1.1.7 to support AirByte on ClickHouse cloud

* Fix quote handling in Clickhouse normalization tests

* Update test output for Clickhouse

* Bump version and update changelog

Co-authored-by: guykohen <guy@clickhouse.com>
2022-08-22 21:53:11 -07:00
Gergely Imreh
469a4fcf95 Update Airbyte Protocol docs self-referencing link to Namespaces (#15811) 2022-08-22 03:45:51 -03:00
Sophia Wiley
b0d13178cf Edited links to headers within doc for A Beginner's Guide to the AirbyteCatalog (#15582)
* Edited link

* Edited all links to headings
2022-08-12 10:36:26 -07:00
Tuhai Maksym
8161d8d97d 13758: SSH control host port setting not used when tunneling (#14295)
* 13758: ssh port fixed

* 13758: version updated

* 13758: changelog updated

* 13758: merge fix

* 13758: docker image version updated

* 13758: docker image version updated

* 13758: docker image version updated as deployment was unsuccesful

Co-authored-by: Greg Solovyev <grishick@users.noreply.github.com>
2022-08-05 01:02:00 +03:00
Andy Jih
8e5d2ff0f4 Typo fix (#15244) 2022-08-03 13:46:26 -04:00
George Claireaux
7ba102b441 Add and persist job failures for Normalization (#14790)
* added TracedException and uncaught exception handler

* added trace message capturing

* added tests for TRACE messages

* pre-json logging

* propagating normalization failures

* log format json & fix hang

* parsing dbt json logs

* bump normalization version

* tests

* Benoit comments

* update trace exception user message

* review comments

* bump version

* bump version

* review comments

* nit comments

* add normalization trace failure test

* version bump

* pmd

* formatto

* bump version
2022-08-02 15:37:00 +01:00
Serhii Chvaliuk
fd09c325de 🎉 Base Normalization: quote schema name to allow reserved keywords (#14683)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-07-28 00:54:40 +03:00
Edward Gao
b2dd470d3d Handle ints and longs in normalization (#14362)
* generate airbyte_type:integer

* normalization accepts `airbyte_type: integer`

* handles ints+longs

* update avro for consistency

* delete long type for now, treat all ints as longs

* update avro type mappings

{type:number, airbyte_type:integer} -> long
{type:number, airbyte_type:big_integer} -> string (i.e. "unbounded integer")

* fix test

* remove long handling

* Revert "remove long handling"

This reverts commit 33ade8d2831e675c3545ac6019d200ec312e54d9.

* Revert "update avro type mappings"

This reverts commit 5b0349badad7545efe8e1191291a628445fe1c84.

* Revert "delete long type for now, treat all ints as longs"

This reverts commit 018efd4a5d0c59f392fd8e3b0d0967c666b72947.

* Revert "update avro for consistency"

This reverts commit bcf47c6799b5906deb4f219d7f6e64ea73b41b74.

* newline@eof

* update test

* slightly better local tests

* fix test

* missed a few cases

* postgres tests use correct hostnames

* fix normalization

* fix int macro

* add test case

* normalization test output

* handle int/long correctly

* fix types for other DBs

* uint32 -> bigint; tests

* add type value assertions

* more test updates

* regenerate output

* reconcile big_integer to match docs

* update comment

* fix type

* fix mysql constructor call

* bigint only has 38 digits

* fix s3 ints, fix DAT test case

* big_integer should be string

* reduce to 28 digit big_ints

* fix test setup, mysql

* kill big_integer tests

* regenerate output

* version bumps

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-26 16:40:14 -07:00
Anna Lvova
49636982c1 🎉 Base Normalization: handle airbyte_type from stream schema in normalization (#13591)
* add datatypes

* up

* up

* add MySQL

* add MSSQL

* fix

* add macros

* add macros

* upd

* upd

* upd for clickhouse

* Return datetime2 for MS SQL

* Upd time type for mysql

* Upd datetime for MySQL

* update

* upd date type for clickhouse

* up

* auto-generate

* bump version

* bump version
2022-07-26 19:49:05 +03:00
oneshcheret
1d2c53ce2f Normalization for Snowflake destination: added support for key pair authentication (#14792)
* Normalization for Snowflake destination: added support for key pair authentication

* Normalization for Snowflake destination: updated changelogs

* Normalization for Snowflake destination: renamed property passphrase to password

* Normalization for Snowflake destination: bump normalization version in NormalizationRunnerFactory

* Normalization for Snowflake destination: added unit tests and change file creating process

* Normalization for Snowflake destination: added unit tests and change file creating process

* Normalization for Snowflake destination: added unit tests and change file creating process
2022-07-23 12:58:11 +03:00
brunofaustino
f73e0c0aea normalization: bigquery partition pruning optimization (#14485)
* bigquery partition pruning otimization

* bump version and add changelog
2022-07-19 17:54:19 -07:00
Cole Snodgrass
c622dd0c3e fix typo; linmitations -> limitations (#14695) 2022-07-14 11:37:31 -03:00
Edward Gao
89e78a6be5 🐛 Destination BIgQuery can handle nulls inside arrays (#14522) 2022-07-13 18:21:17 -07:00
Edward Gao
ddc065e9ce 🐛 normalization: Do not return NULL for MySQL column values > 512 chars (#14476)
* Revert "Revert "normalization: Do not return NULL for MySQL column values > 512 chars  (#11694)" (#14475)"

This reverts commit d268ab4140.

* better dependencies

* try internal docker host?

* Revert "try internal docker host?"

This reverts commit 709856d049.
2022-07-13 13:06:42 -07:00
Topher Lubaway
9c6c092a22 Revert "Improving docusaurus sidebar generation (#1927) (#14369)" (#14596)
This reverts commit a2c194a11f.
2022-07-11 15:27:14 -05:00
Mykyta Serbynevskiy
a2c194a11f Improving docusaurus sidebar generation (#1927) (#14369)
* Improving docusaurus sidebar generation (#1927)

* Added "Career & open positions" folder to sidebar, adjusted "Project overview" folder

* Deleted "career-and-open-positions" folder from sidebar
2022-07-08 14:18:27 -05:00
Edward Gao
d268ab4140 Revert "normalization: Do not return NULL for MySQL column values > 512 chars (#11694)" (#14475)
This reverts commit 3f001b17ca.
2022-07-06 15:22:50 -07:00
Johannes Nicolai
3f001b17ca normalization: Do not return NULL for MySQL column values > 512 chars (#11694) 2022-07-06 09:50:25 +02:00