jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-01 09:02:59 -05:00
Code Issues Packages Projects Releases Wiki Activity
3,680 Commits 5,135 Branches 518 Tags
b94ee00fd8e5220e3191073cd711a57e052d7fda
Commit Graph

22 Commits

Author SHA1 Message Date
Christophe Duong
5fc50df39d 🎉 Incremental Normalization (#7162) 2021-10-29 13:53:02 +02:00
Abhi Vaidyanatha
ae32ecbb27 GitBook: [master] 186 pages and 77 assets modified 2021-10-08 21:17:47 +00:00
Anna Lvova
ec68f478ff 🐛 fix: Normalization date-time should handle empty strings "" (#6379) 
* add empty string normalization for postgres

* add empty string normalization for destinations

* fix

* fix

* fix

* fix for snowflake

* fix for mysql

* fix normalization for mysql

* upd doc

* upd doc

* Update airbyte-integrations/bases/base-normalization/integration_tests/dbt_integration_test.py

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>

* Update airbyte-integrations/bases/base-normalization/integration_tests/dbt_integration_test.py

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>

* bump version

* bump version

* add datetime normalization for mssql

* upd row count for mssql

* upd

* bump version

* upd docs for 0.1.50 normalization version

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
 2021-10-08 13:57:37 +03:00
Harshith Mullapudi
29ea7f19eb Add integration tests for Normalization - added ad_cdc_log_pos (#6799) 
* integration tests for bigquery

* added for postgres

* added tests for all the destinations

* Bump version
 2021-10-08 14:31:28 +05:30
Christophe Duong
a3196428a7 Forward destination location to dbt profiles (#6709) 
* Forward destination location to dbt profiles

* Format code

* Update version
 2021-10-06 19:20:15 +02:00
Abhi Vaidyanatha
6b19bf4089 Add high level overview to normalization doc. (#6445) 
* Add high level overview to normalization

* Address review comments

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-09-28 14:23:11 -07:00
andriikorotkov
8fa15713c3 🎉 Destination MySQl - Added support for connection via ssh (aka bastion server) (#6317) 
* updated mysql tests

* updated mysql tests

* added mysql ssh tunnel tests by key

* fixed remarks

* fixed remarks

* updated DatabricksStreamCopier

* switch to custom file for ssh config in normalization

* updated MySQL SSH tests

* bump version

* get local port properly

* updated assertSameValue for MySQL ssh tunnel

* updated image version and documentation

* updated code style

* updated CI credentials

* updated normalization documentation

Co-authored-by: George Claireaux <george@claireaux.co.uk>
 2021-09-28 13:11:32 +03:00
Yaroslav Dudar
a6ecfda2ca 🐛 Fix Snowflake destination normalization to accept any date-time format. (#6052) 
snowflake date-time format parser
 2021-09-23 11:10:12 +03:00
Christophe Duong
f9705bf731 BigQuery normalization: make credentials json optional (#5433) 
* Allow service-account-json or oauth methods for bigquery destinations
 2021-08-17 11:50:17 +02:00
Marcos Marx
e4fe62f739 Normalization: solve conflict when stream and field have same name (#4557) 
* solve conflict when stream and field have same name

* add logic to handle conflict

* change files

* change json_extract functions

* json_operations

* add normalization files

* test integration mysql

* remove table_alias

* mysql run

* json ops

* solve conflict with master

* solve mysql circle dependency dbt

* add tests for scalar and arrays

* add sql files

* bump normalization version

* format
 2021-08-11 20:18:45 -03:00
Christophe Duong
d6429a410a Normalization handles quote in column names (#5027) 
* Handle quotes in columns names
 2021-07-28 16:00:13 +02:00
Christophe Duong
5cdc7f8517 🐛 (contribution) Fix SQL model to build a Type 2 SCD to handle NULL cursor_field values correctly (#4881) 
* Update SQL model to build a Type 2 Slowly Changing Dimension (#4802)

* Make SQL more portable

* Bumpversion of normalization

Co-authored-by: Daniel Diamond <33811744+danieldiamond@users.noreply.github.com>
 2021-07-22 16:27:54 +02:00
LiRen Tu
8121171763 📝 Add server version requirement for mysql normalization (#4856) 2021-07-20 10:55:46 -07:00
LiRen Tu
2caf3904f0 🎉 MySQL destination: normalization (#4163) 
* Add mysql dbt package

* Add mysql normalization support in java

* Add mysql normalization support in python

* Fix unit tests

* Update readme

* Setup mysql container in integration test

* Add macros

* Depend on dbt-mysql from git repo

* Remove mysql limitation test

* Test normalization

* Revert protocol format change

* Fix mysel json macros

* Fix two more macros

* Fix table name length

* Fix array macro

* Fix equality test macro

* Update replace-identifiers

* Add more identifiers to replace

* Fix unnest macro

* Fix equality macro

* Check in mysql test output

* Update column limit test for mysql

* Escape parentheses

* Remove unnecessary mysql test

* Remove mysql output for easier code review

* Remove unnecessary mysql test

* Remove parentheses

* Update dependencies

* Skip mysql instead of manually write out types

* Bump version

* Check in unit test for mysql name transformer

* Fix type conversion

* Use json_value to extract scalar json fields

* Move dbt-mysql to Dockerfile (#4459)

* Format code

* Check in mysql dbt output

* Remove unnecessary quote

* Update mysql equality test to match 0.19.0

* Check in schema_test update

* Update readme

* Bump base normalization version

* Update document

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
 2021-07-03 20:30:59 -07:00
Abhi Vaidyanatha
33a80deb0a Fix broken links in docs. (#4419) 
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-06-29 14:51:46 -07:00
Abhi Vaidyanatha
8522a1cd9f Change all instances of DBT to dbt in markdown files. (#4308) 
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-06-23 19:00:44 -07:00
Christophe Duong
22b75c56ea Add docs on custom Operations (#3507) 
* Add docs on custom Operations

Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
 2021-06-14 10:31:38 +02:00
Vivek Ayer
2580a64c3c docs: link to 'Transformations with SQL' page (#3835) 
- basic-normalization markdown page was missing a link.
 2021-06-02 22:04:12 -03:00
Christophe Duong
bb4dcb1987 🎉 Remove hash when it is not necessary from normalization outputs (#3704) 
* Refactor `generate_new_table_name` using a table name registry class instead

* update normalization docs

* Enable MyPy

* Regenerate output files

* Closes https://github.com/airbytehq/airbyte/issues/2389

* Bumpversion normalization
 2021-06-01 17:07:22 +02:00
Christophe Duong
8862fba1bb 🎉 Avoid dbt runtime exception "maximum recursion depth exceeded" in ephemeral materialization 
* Create new test_ephemeral and refactor with test_normalization

* Add notes in docs

* Refactor common normalization tests into DbtIntegrationTest

* Bumpversion of normalization image
 2021-05-21 18:07:20 +02:00
Marcos Marx
7c70eb02cd GitBook: [master] 84 pages and 72 assets modified 2021-05-03 12:14:16 +00:00
Abhi Vaidyanatha
e378d40236 Documentation Reorganization (#3124) 
* First reorganization pass.

* Stop auto-generating api docs html file.

* Update spelling

* Final cleanup.

* Final changes_REAL_actual_2_thisone

* fix path for generating api docs

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: jrhizor <me@jaredrhizor.com>
 2021-04-29 15:18:52 -07:00