jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-23 21:03:15 -05:00
Code Issues Packages Projects Releases Wiki Activity

Normalization: Use strict > comparison in incremental mode (#22381)

Browse Source 
* copy tests from other branch

* switch to >

* [wip] wire up tests

* make tests work

* fixes

* nicer test structure

* maybe add feature flag?

* pattern matching

* also add version check

* formatting

* refactor test also

* extract test + fix method call

* minor tweaks

* add context to log message

* put workspace id in normalization input

* use non-semver tag

* add flag for version of normalization

* also flag old version

* add test

* missed part of the commit

* format

* add test for null workspace ID

* Revert "also flag old version"

This reverts commit 3be601d16c.

* Revert "missed part of the commit"

This reverts commit 47a67b4631.

* always apply flag, even if we're behind a version

* derp

* Add more logging to the normalization activity

* Update charts and kustomize for the feature flag

* fix clickhouse integration test

* remove replace_identifiers

* Revert "remove replace_identifiers"

This reverts commit 0e7ded5a7b.

* fix replace_identifiers

* garbage debug logs

* stop trying to setup duckdb test

* wake up and choose violence

* fix mssql

* exclude duckdb from tests

* make snowflake happy

* uncomment tests

* derp

* derpderp

* format

* format

* also fix redshift???

* maybe now everything works???

* remove debug logs

* use special docker tag

* bump to new tag

* use random test schema in publish also

* properly cleanup

* remove feature flag stuff

* version bump + changelog

* Automated Commit - Formatting Changes

* bump definitions

---------

Co-authored-by: Jimmy Ma <gosusnp@users.noreply.github.com>
Co-authored-by: Jimmy Ma <jimmy@airbyte.io>
Co-authored-by: octavia-squidington-iii <octavia-bot@airbyte.io>
Co-authored-by: edgao <edgao@users.noreply.github.com>
This commit is contained in:
Edward Gao
2023-03-23 09:37:15 -07:00
committed by GitHub
parent 388e157543
commit 9b7b30f92b
21 changed files with 639 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/understanding-airbyte/basic-normalization.md
View File

@@ -353,6 +353,7 @@ Therefore, in order to "upgrade" to the desired normalization version, you need
| Airbyte Version | Normalization Version | Date | Pull Request | Subject |
|:----------------|:---------------------------|:-----------|:-------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------|
| | 0.4.0 | 2023-03-23 | [\#22381](https://github.com/airbytehq/airbyte/pull/22381) | Prevent normalization from creating unnecessary duplicates in nested tables. |
| | 0.2.27 | 2023-03-15 | [\#24077](https://github.com/airbytehq/airbyte/pull/24077) | Add more bigquery reserved words |
| | 0.2.26 | 2023-02-15 | [\#19573](https://github.com/airbytehq/airbyte/pull/19573) | Update Clickhouse dbt version to 1.4.0 |
| | 0.3.2 (broken, do not use) | 2023-01-31 | [\#22165](https://github.com/airbytehq/airbyte/pull/22165) | Fix support for non-object top-level schemas |