1
0
mirror of synced 2026-01-03 15:04:01 -05:00
Commit Graph

20 Commits

Author SHA1 Message Date
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Sherif A. Nada
49810f8c13 sweep airbyte-integrations/bases/airbyte-protocol (#12598) 2022-05-05 13:51:16 -07:00
Edward Gao
046fc5e1cc 🎉 upgrade dbt to 1.0.0 (except for oracle and mysql) (#11051) 2022-03-11 16:38:37 -08:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
LiRen Tu
2caf3904f0 🎉 MySQL destination: normalization (#4163)
* Add mysql dbt package

* Add mysql normalization support in java

* Add mysql normalization support in python

* Fix unit tests

* Update readme

* Setup mysql container in integration test

* Add macros

* Depend on dbt-mysql from git repo

* Remove mysql limitation test

* Test normalization

* Revert protocol format change

* Fix mysel json macros

* Fix two more macros

* Fix table name length

* Fix array macro

* Fix equality test macro

* Update replace-identifiers

* Add more identifiers to replace

* Fix unnest macro

* Fix equality macro

* Check in mysql test output

* Update column limit test for mysql

* Escape parentheses

* Remove unnecessary mysql test

* Remove mysql output for easier code review

* Remove unnecessary mysql test

* Remove parentheses

* Update dependencies

* Skip mysql instead of manually write out types

* Bump version

* Check in unit test for mysql name transformer

* Fix type conversion

* Use json_value to extract scalar json fields

* Move dbt-mysql to Dockerfile (#4459)

* Format code

* Check in mysql dbt output

* Remove unnecessary quote

* Update mysql equality test to match 0.19.0

* Check in schema_test update

* Update readme

* Bump base normalization version

* Update document

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
2021-07-03 20:30:59 -07:00
Christophe Duong
bb4dcb1987 🎉 Remove hash when it is not necessary from normalization outputs (#3704)
* Refactor `generate_new_table_name` using a table name registry class instead

* update normalization docs

* Enable MyPy

* Regenerate output files

* Closes https://github.com/airbytehq/airbyte/issues/2389

* Bumpversion normalization
2021-06-01 17:07:22 +02:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Christophe Duong
f666fd2f18 Upgrade normalization to use dbt from docker images (#3186) 2021-05-04 10:31:07 +02:00
Charles
9af31d0f29 downgrade dbt again (#3072) 2021-04-26 17:11:03 -07:00
Davin Chia
f660b0a946 Add template generation for Santa aka CDK. (#3034)
Template generation for new Source using the Santa CDK - provide basic scaffolding for someone implementing a new source.

General approach is to buff up comments in the original SDK, and add TODOs with secondary comments in the generated stub methods, as well as links to existing examples (e.g. Stripe or ExchangeRate api) users can look at.

Checked in and added tests for the generated modules.
2021-04-25 18:02:33 +08:00
Charles
f445fdb5b2 match styling for spotlessApply and format (#3017)
* as a java developer I want to be able to run spotlessApply without changing styles in python code
2021-04-23 09:21:41 -07:00
Christophe Duong
e500091de0 Upgrade DBT (#2987)
* Bumpversion of normalization image

* Upgraded normalization DBT to 0.19.1 (#2933)

* Updated DBT utils to 0.6.4

Co-authored-by: Marc Buchardt <37442961+MarcBuch@users.noreply.github.com>
2021-04-20 11:07:56 +02:00
Sherif Nada
77b72dcf5b Revert "Revert "Merge branch 'master' of github.com:airbytehq/airbyte""
This reverts commit de66bf010d.
2021-03-23 15:12:01 -07:00
Sherif Nada
de66bf010d Revert "Merge branch 'master' of github.com:airbytehq/airbyte"
This reverts commit a3a27f2fa2, reversing
changes made to 5917feee28.
2021-03-23 15:09:44 -07:00
Christophe Duong
4bebeed60b Harmonize /test and /publish commands on github + Fix MySQL tests in CI (#2531)
* Harmonize /test and /publish commands on github

* Make normalization build on clean system without PyICU

see https://getdbt.slack.com/archives/C50NEBJGG/p1615803416053800?thread_ts=1615511262.026400&cid=C50NEBJGG

* Follow example from https://github.com/testcontainers/testcontainers-java/pull/2689 to modify MySQL container as root

* Add docs on new syntax for /test
2021-03-23 10:47:29 +01:00
Charles
d483e3a292 freeze dbt version at 0.18.1 (#2012) 2021-02-09 10:18:54 -08:00
Jared Rhizor
e1d7add50a mostly incremental builds (#817)
* support cached builds for base -> base-java -> snowflake

* use plugin for image building

* fix matching on COPY from

* remove docker.gradle

* tmp commit

* update connectors

* finish rest of build files

* fix ide errors

* more build fixes

* clean up

* clean up for new soruces

* fix spotless

* fix flake problems

* add recommended empty file

* python caching

* fixes upon review

* clean up docker and build test files

* clean up python

* clean up

* fix integration test dependencies

* fix standard tests

* fix

* remove symlink

* re-add requirements to fix normalizatioon build

* fix symlink

* fix dumbest build problem of all

* add missing integration test def

* fix missing dep

* remove class exclusion

* move trim so null source versions are allowed

* rename map

* fix hardcoded value

* remove unnecessary dep

* use dashes for salesforce package name

* fix typo

* DRY and fix test image name

* Fix edit

* assert string is not empty

* build integration test image only for integrationTest

* move code generator to tools and rename docker build tasks

* make source test depend on integration test build, not the other way

* remove guard because the docker build should exist before the integrationtest is applied

* remove comment

* DRY up airbyte-source-test

* fix plugin compilation

* add missing dependency

* rename getTaggedImage to getDevTaggedImage

* fix test vs main docker build bug
2020-11-09 11:00:11 -08:00
Christophe Duong
d042ef524a DBT project for Airbyte normalization (#802)
Add a step that generates config files and SQL files in order to run DBT which takes care of converting JSON blob data (from catalog.json JSON Schema) into normalized tables 

Co-authored-by: cgardens <giardina.charles@gmail.com>
Co-authored-by: Sherif Nada <snadalive@gmail.com>
2020-11-06 23:02:30 +01:00
Charles
0c49634ea1 Transform Airbyte config to DBT profile (#793) 2020-11-04 08:59:41 -08:00
Charles
e5e6ce9550 normalization base project / docker image setup (#792) 2020-11-03 15:36:56 -08:00