1
0
mirror of synced 2026-01-07 09:05:45 -05:00
Commit Graph

16 Commits

Author SHA1 Message Date
Augustin
80032f73f9 connectors-ci: deprecate slash publish (#25865) 2023-05-22 10:10:56 +02:00
Joe Bell
9a4be977c1 Destination Bigquery: stop running normalization container for DAT (#25925)
* readme update

* allow passing additional flags to test containers

* remove build dependency

* Automated Change

* versioning updates

* restore denormalized change from master

* formatting changes

* formatting

* Automated Change

* update metadata file

---------

Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
2023-05-18 00:46:32 +00:00
Edward Gao
897522cf51 Add some dev-facing normalization docs (#13780) 2022-06-15 08:21:14 -07:00
Brian Leonard
b882538147 Snowflake integration test steps (#13205)
* Destination-snowflake test config update

* Tests assume this ones doesn’t work!

* Make GCS integration

* Use existing GCS integration because tests use it

* Add comments

* Snowflake setup in base-normalization

* Markdown!

* Respect the env variable

* readme update

* Updated snapshot
2022-05-26 13:24:29 -07:00
Christophe Duong
5fc50df39d 🎉 Incremental Normalization (#7162) 2021-10-29 13:53:02 +02:00
Christophe Duong
c4620559d7 🎉 Refactor Normalization docker images and upgrade to use dbt 0.21.0 (#6959)
* Split normalization docker images for some connectors with specifics dependencies

* Regenerate (#7003)
2021-10-14 20:29:16 +02:00
Baz
e5abaeccef 🎉 Base-normalization: Implement normalization for MSSQL-destination (#6079)
See the attached PR (https://github.com/airbytehq/airbyte/pull/6079)
2021-10-07 18:46:27 +03:00
Marcos Marx
589d535a61 🎉 Oracle normalization (#5562)
* oracle normalization

* correct dbt_project function for oracle

* unit tests

* run format

* correct ephemeral tests

* add gradle dependency for oracle destination

* run int tests

* add oracle in settings.gradle for normalization run[

* use default airbyte columns

* format

* test all destinatoin ephemeral

* correct unit test

* correct unit test

* destination docs update

* correct mypy

* integration test all dest

* refactor oracle function

* merge master

* run all destinations

* flake8 escape regex

* surrogate key function

* correct few minor comments

* refactor scd sql function

* refactor scd function

* revert test

* refactor minor details

* revert tests

* revert ephemeral test

* revert unit test table_registry

* revert airbyte_protocol format

* format

* bump normalization version in worker

* minor chnages

* minor chages

* correct json_column for other destinations

* gradlew format

* revert tests

* remove comments

* add Oracle destination explicit in safe_cast_str

* add quote_in_parenthesis inside if clause

* gradlew format
2021-09-07 16:39:17 -03:00
Abhi Vaidyanatha
0d2364c1f4 Clean up connector information docs (#5086)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-29 12:17:34 -07:00
LiRen Tu
2caf3904f0 🎉 MySQL destination: normalization (#4163)
* Add mysql dbt package

* Add mysql normalization support in java

* Add mysql normalization support in python

* Fix unit tests

* Update readme

* Setup mysql container in integration test

* Add macros

* Depend on dbt-mysql from git repo

* Remove mysql limitation test

* Test normalization

* Revert protocol format change

* Fix mysel json macros

* Fix two more macros

* Fix table name length

* Fix array macro

* Fix equality test macro

* Update replace-identifiers

* Add more identifiers to replace

* Fix unnest macro

* Fix equality macro

* Check in mysql test output

* Update column limit test for mysql

* Escape parentheses

* Remove unnecessary mysql test

* Remove mysql output for easier code review

* Remove unnecessary mysql test

* Remove parentheses

* Update dependencies

* Skip mysql instead of manually write out types

* Bump version

* Check in unit test for mysql name transformer

* Fix type conversion

* Use json_value to extract scalar json fields

* Move dbt-mysql to Dockerfile (#4459)

* Format code

* Check in mysql dbt output

* Remove unnecessary quote

* Update mysql equality test to match 0.19.0

* Check in schema_test update

* Update readme

* Bump base normalization version

* Update document

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
2021-07-03 20:30:59 -07:00
Abhi Vaidyanatha
8522a1cd9f Change all instances of DBT to dbt in markdown files. (#4308)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-23 19:00:44 -07:00
Christophe Duong
bb4dcb1987 🎉 Remove hash when it is not necessary from normalization outputs (#3704)
* Refactor `generate_new_table_name` using a table name registry class instead

* update normalization docs

* Enable MyPy

* Regenerate output files

* Closes https://github.com/airbytehq/airbyte/issues/2389

* Bumpversion normalization
2021-06-01 17:07:22 +02:00
Christophe Duong
8862fba1bb 🎉 Avoid dbt runtime exception "maximum recursion depth exceeded" in ephemeral materialization
* Create new test_ephemeral and refactor with test_normalization

* Add notes in docs

* Refactor common normalization tests into DbtIntegrationTest

* Bumpversion of normalization image
2021-05-21 18:07:20 +02:00
Christophe Duong
0265012e42 Handle special characters in columns names (#3133)
* Handle special characters in columns names (add test case to integration tests)

* Add test case with column name collisions

* Bumpversion of normalization image
2021-04-30 11:59:55 +02:00
Christophe Duong
86513d6c54 Fix normalization Nesting bug (#3110)
* New test case for nested streams

* Fix filename naming (collisions and nesting)

* Update generated files from tests with new file naming

* Allow invalid json data in raw tables when normalizing on redshift

* Regenerate final sql files

* Disable unit tests on stream naming (temporarly)

* Fix unnesting bug in postgres

* Reactivate unit tests and change table registry

* Move normalization unit tests to integration tests (too slow)

* Remove heavy catalog.json used in unit_tests (actual catalog from facebook/stripe with thousands of lines)

* Bumpversion of normalization image
2021-04-29 14:32:59 +02:00
Christophe Duong
c2fa3e4c9c Introduce normalization integration tests (#3025)
* Speed normalization unit tests by dropping hubspot catalog (too heavy, will be covering it in integration tests instead

* Add integration tests for normalization

* Add dedup test case

* adjust build.gradle

* add readme for normalization

* Share PATH env variable with subprocess calls

* Handle git non-versionned tests vs versionned ones

* Format code

* Add tests check to normalization integration tests

* Add docs

* complete docs on normalization integration tests

* format code

* Normalization integration tests output (#3026)

* Version generated/output files from normalization integration tests

* simplify cast of float columns to string when used as partition key (#3027)

* bump version of normalization image

* Apply suggestions from code review

Co-authored-by: Jared Rhizor <jared@dataline.io>

* Apply suggestions from code review

Co-authored-by: Jared Rhizor <jared@dataline.io>
2021-04-27 12:01:04 +02:00