1
0
mirror of synced 2026-01-06 06:04:16 -05:00
Commit Graph

144 Commits

Author SHA1 Message Date
Liren Tu
9356d6adf5 📝 Update json avro conversion doc about logical type union (#17010)
* Update json avro doc to include logical type union

* Add a note about the potential problem

* Add issue link
2022-09-21 15:43:44 -07:00
Alexander Marquardt
2fbb85c3a0 Update incremental-append.md (#16579)
* Update incremental-append.md

Added link to the incremental sync tutorial

* Update incremental-append.md
2022-09-16 17:55:44 +02:00
Alexander Marquardt
005378626c Update incremental-deduped-history.md (#16578)
* Update incremental-deduped-history.md

Added a link to the incremental sync tutorial

* Update incremental-deduped-history.md

* Update incremental-deduped-history.md
2022-09-16 17:55:33 +02:00
Alexander Marquardt
ae85cacba5 Update full-refresh-overwrite.md (#16580) 2022-09-16 17:55:01 +02:00
Alexander Marquardt
7f2b27274b Update full-refresh-append.md (#16581) 2022-09-16 17:54:46 +02:00
Filipp Balakin
17cb363fe1 Normalization: update dbt-clickhouse version from 1.1.7 to 1.1.8 (#16339)
* Bump dbt-clickhouse version from 1.1.7 to 1.1.8

* pin dbt<1.2

* update doc

* pin dbt core lt 1.2

* bump normalization version

* loosen requirements

* fix md

* remove empty line

* bump normalization version in worker

* bump normalization version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2022-09-14 08:27:31 -03:00
Davin Chia
08090bca6b Update the worker/job documentation. Add documentation for container orchestrator. (#16575)
- Add a container orchestrator section.
- Update existing section to include details on Kubernetes vs Docker.
- Move some things around so things are smoother.
2022-09-13 13:29:53 -07:00
Evan Tahler
dcfcb75d0f AirbyteLogMessage.stack_trace for logging messages with related (non-fatal) errors (#16479)
* Test log message from faker

* AirbyteLogMessage gains stack_trace

* Fixup spacing

* bump python protocol

* fixup additionalProperties in faker spec

* bump faker version

* update docs

* use lineSeparator vs \r\n

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-12 16:01:57 -07:00
Evan Tahler
6f2fcff85c fix changelog formatting (#16618)
* fix changelog formatting

* fixup docs
2022-09-12 15:59:50 -07:00
Evan Tahler
3a1df7ea58 Update airbyte-protocol.md 2022-09-12 10:55:54 -07:00
Evan Tahler
8cf60e4ea9 Add changelog for Airbyte Protocol (#16560)
* Add changelog for Airbyte Protocol

* we are on v0.3.0 now
2022-09-12 10:30:01 -07:00
Rodi Reich Zilberman
43b8d8beaf Publish base-normalization (#16545)
* dummy

* update changelog
2022-09-09 15:10:26 -07:00
Tuhai Maksym
f81c5aa667 7198: SSH Tunnel: allow using OPENSSH key format (#15833)
* 7198: SSH Tunnel: allow using OPENSSH key format

* 7198: merge

* 7198: merge fix

* Use apache sshd lib to load private keys for tunnel

* Throw an exception in case private key failed to load

* Fix failing flow when creating NO_TUNNEL wrapper

* bump version numbers

* Address review comments. Fix test

Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
2022-09-09 10:22:03 -07:00
Daemonxiao
d4524032ae 🎉 New Destination: TiDB (#15592)
* Add new destination-tidb

* support sync

* Add normalization-tidb

* fix failed tests

* Add unnest marco

* fmt

* Add new destination-tidb

* support sync

* Add normalization-tidb

* fix failed tests

* Add unnest marco

* fmt

* fmt

* fix integration test

* Update docs/integrations/destinations/tidb.md

Co-authored-by: Xiang Zhang <angwerzx@126.com>

* Update doc

* Update doc

* Update doc

* bump normalization version

* update normalization changelog

* run format

* add dest def

* generat spec

Co-authored-by: Xiang Zhang <angwerzx@126.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2022-08-31 16:50:27 -03:00
Jimmy Ma
4fbe03bebe Add explicit tracking of Airbyte Protocol Version in ConnectorSpecification (#15340)
* Add Version to AirbyteMessage

* Move protocol version to ConnectorSpecification

* Add cdk generated protocol model

* Add protocol_version to the sample ConnectorSpec in the docs

* Update airbyte-protocol/protocol-models/src/main/resources/airbyte_protocol/airbyte_protocol.yaml

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* update doc

* Update CDK changelog

* Update CDK protocol model

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-08-29 14:46:56 -07:00
Greg Solovyev
5819733ab1 Greg/guykoh update dbt clickhouse (#14897)
* Update dbt-clickhouse version to 1.1.7 to support AirByte on ClickHouse cloud

* Fix quote handling in Clickhouse normalization tests

* Update test output for Clickhouse

* Bump version and update changelog

Co-authored-by: guykohen <guy@clickhouse.com>
2022-08-22 21:53:11 -07:00
Gergely Imreh
469a4fcf95 Update Airbyte Protocol docs self-referencing link to Namespaces (#15811) 2022-08-22 03:45:51 -03:00
Sophia Wiley
b0d13178cf Edited links to headers within doc for A Beginner's Guide to the AirbyteCatalog (#15582)
* Edited link

* Edited all links to headings
2022-08-12 10:36:26 -07:00
Tuhai Maksym
8161d8d97d 13758: SSH control host port setting not used when tunneling (#14295)
* 13758: ssh port fixed

* 13758: version updated

* 13758: changelog updated

* 13758: merge fix

* 13758: docker image version updated

* 13758: docker image version updated

* 13758: docker image version updated as deployment was unsuccesful

Co-authored-by: Greg Solovyev <grishick@users.noreply.github.com>
2022-08-05 01:02:00 +03:00
Andy Jih
8e5d2ff0f4 Typo fix (#15244) 2022-08-03 13:46:26 -04:00
George Claireaux
7ba102b441 Add and persist job failures for Normalization (#14790)
* added TracedException and uncaught exception handler

* added trace message capturing

* added tests for TRACE messages

* pre-json logging

* propagating normalization failures

* log format json & fix hang

* parsing dbt json logs

* bump normalization version

* tests

* Benoit comments

* update trace exception user message

* review comments

* bump version

* bump version

* review comments

* nit comments

* add normalization trace failure test

* version bump

* pmd

* formatto

* bump version
2022-08-02 15:37:00 +01:00
Serhii Chvaliuk
fd09c325de 🎉 Base Normalization: quote schema name to allow reserved keywords (#14683)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-07-28 00:54:40 +03:00
Edward Gao
b2dd470d3d Handle ints and longs in normalization (#14362)
* generate airbyte_type:integer

* normalization accepts `airbyte_type: integer`

* handles ints+longs

* update avro for consistency

* delete long type for now, treat all ints as longs

* update avro type mappings

{type:number, airbyte_type:integer} -> long
{type:number, airbyte_type:big_integer} -> string (i.e. "unbounded integer")

* fix test

* remove long handling

* Revert "remove long handling"

This reverts commit 33ade8d2831e675c3545ac6019d200ec312e54d9.

* Revert "update avro type mappings"

This reverts commit 5b0349badad7545efe8e1191291a628445fe1c84.

* Revert "delete long type for now, treat all ints as longs"

This reverts commit 018efd4a5d0c59f392fd8e3b0d0967c666b72947.

* Revert "update avro for consistency"

This reverts commit bcf47c6799b5906deb4f219d7f6e64ea73b41b74.

* newline@eof

* update test

* slightly better local tests

* fix test

* missed a few cases

* postgres tests use correct hostnames

* fix normalization

* fix int macro

* add test case

* normalization test output

* handle int/long correctly

* fix types for other DBs

* uint32 -> bigint; tests

* add type value assertions

* more test updates

* regenerate output

* reconcile big_integer to match docs

* update comment

* fix type

* fix mysql constructor call

* bigint only has 38 digits

* fix s3 ints, fix DAT test case

* big_integer should be string

* reduce to 28 digit big_ints

* fix test setup, mysql

* kill big_integer tests

* regenerate output

* version bumps

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-26 16:40:14 -07:00
Anna Lvova
49636982c1 🎉 Base Normalization: handle airbyte_type from stream schema in normalization (#13591)
* add datatypes

* up

* up

* add MySQL

* add MSSQL

* fix

* add macros

* add macros

* upd

* upd

* upd for clickhouse

* Return datetime2 for MS SQL

* Upd time type for mysql

* Upd datetime for MySQL

* update

* upd date type for clickhouse

* up

* auto-generate

* bump version

* bump version
2022-07-26 19:49:05 +03:00
oneshcheret
1d2c53ce2f Normalization for Snowflake destination: added support for key pair authentication (#14792)
* Normalization for Snowflake destination: added support for key pair authentication

* Normalization for Snowflake destination: updated changelogs

* Normalization for Snowflake destination: renamed property passphrase to password

* Normalization for Snowflake destination: bump normalization version in NormalizationRunnerFactory

* Normalization for Snowflake destination: added unit tests and change file creating process

* Normalization for Snowflake destination: added unit tests and change file creating process

* Normalization for Snowflake destination: added unit tests and change file creating process
2022-07-23 12:58:11 +03:00
brunofaustino
f73e0c0aea normalization: bigquery partition pruning optimization (#14485)
* bigquery partition pruning otimization

* bump version and add changelog
2022-07-19 17:54:19 -07:00
Cole Snodgrass
c622dd0c3e fix typo; linmitations -> limitations (#14695) 2022-07-14 11:37:31 -03:00
Edward Gao
89e78a6be5 🐛 Destination BIgQuery can handle nulls inside arrays (#14522) 2022-07-13 18:21:17 -07:00
Edward Gao
ddc065e9ce 🐛 normalization: Do not return NULL for MySQL column values > 512 chars (#14476)
* Revert "Revert "normalization: Do not return NULL for MySQL column values > 512 chars  (#11694)" (#14475)"

This reverts commit d268ab4140.

* better dependencies

* try internal docker host?

* Revert "try internal docker host?"

This reverts commit 709856d049.
2022-07-13 13:06:42 -07:00
Topher Lubaway
9c6c092a22 Revert "Improving docusaurus sidebar generation (#1927) (#14369)" (#14596)
This reverts commit a2c194a11f.
2022-07-11 15:27:14 -05:00
Mykyta Serbynevskiy
a2c194a11f Improving docusaurus sidebar generation (#1927) (#14369)
* Improving docusaurus sidebar generation (#1927)

* Added "Career & open positions" folder to sidebar, adjusted "Project overview" folder

* Deleted "career-and-open-positions" folder from sidebar
2022-07-08 14:18:27 -05:00
Edward Gao
d268ab4140 Revert "normalization: Do not return NULL for MySQL column values > 512 chars (#11694)" (#14475)
This reverts commit 3f001b17ca.
2022-07-06 15:22:50 -07:00
Johannes Nicolai
3f001b17ca normalization: Do not return NULL for MySQL column values > 512 chars (#11694) 2022-07-06 09:50:25 +02:00
Serhii Chvaliuk
49d181a198 Normalization: Fix incorrect jinja2 macro json_extract_array call (#13894)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-06-19 13:13:49 +03:00
Jonathan Pearlin
abe3024511 Update airbyte-protocol.md (#13892)
* Update airbyte-protocol.md

* Fix typo

* Fix prose

* Add protocol reviewers for protocol documentation

* Remove duplicate
2022-06-17 14:06:53 -04:00
Augustin
e8146e5ec2 Normalization: Upgrade MySQL to dbt 1.0.0 (#11470) 2022-06-15 15:05:49 -07:00
Amruta Ranade
fb0b5a3694 Fix broken link (#13815)
* fixing broken links

* docusaurus updates
2022-06-15 13:57:46 -04:00
Edward Gao
897522cf51 Add some dev-facing normalization docs (#13780) 2022-06-15 08:21:14 -07:00
Charles
22b727c0ea Update Airbyte Protocol Docs (#13709) 2022-06-14 15:04:38 -07:00
Edward Gao
61ce03a436 🐛 Normalization correctly propagates deletions to the final tables (#12846) 2022-06-14 14:56:18 -07:00
Marcos Marx
18f1fb10b4 update dbt version in custom normalization (#13742) 2022-06-14 13:14:34 -03:00
Marcos Marx
6a868bec66 Docs: update normalization doc (#13739) 2022-06-14 13:12:19 -03:00
hugoJuhel
df3ebf3158 🎉 Destination mssql : adding ssh tunnel for normalisation (#11204) 2022-06-10 13:54:26 -07:00
Serhii Chvaliuk
2daaf5b4c3 Normalization - BigQuery use json_extract_string_array for array of simple types (#13289)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: andrii.leonets <aleonets@gmail.com>
Co-authored-by: Andrii Leonets <30464745+DoNotPanicUA@users.noreply.github.com>
2022-06-10 23:31:32 +03:00
Lake Mossman
fcbffe0495 Implement job state machine validations (#13213)
* add job state machine validations to persistence and tests

* update state machine diagram

* format

* remove terminal state transitions

* fail attempt before cancelling job, to respect job state machine

* update job status state machine diagram

* verify order of db calls
2022-05-31 17:21:20 -07:00
ThaliaBarrera
8f8d6cc60e Add a link to our blog post about CDC (#12913) 2022-05-26 10:45:10 -04:00
Topher Lubaway
3a64c25073 Fixes all broken links and enforces no broken (#13152)
* Fixes all broken links and enforces no broken

* Addresses peter's comments

removes unused summary.md: an artifact of gitbook
2022-05-25 15:59:29 -05:00
Serhii Chvaliuk
7401cb64c0 Normalization: Checking --event-buffer-size on old dbt crashed entrypoint.sh (#12924)
* Fixed checking --event-buffer-size on old dbt crashed entrypoint.sh

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-05-18 11:08:12 +03:00
Serhii Chvaliuk
1ea62f33f8 Normalization snowflake: add datetime without timezone (#12745)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-05-16 13:56:07 +03:00
oneshcheret
d35d7b07b8 Mssql destination: enable DAT tests, use nvarchar and datetime2 by default (#12305)
* Mssql destination: enable DAT tests for mssql destination, use nvarchar and datetime2 by default

* Mssql destination: update array handling in test

* Mssql destination: update array and JSON handling in test

* Mssql destination: remove unused method

* bugfix bigquery tests, dataset_location added

* basic-normalization.md updated

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* Mssql destination: change parent class for mssql test

Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-05-07 11:22:22 +03:00