1
0
mirror of synced 2026-01-31 19:01:59 -05:00
Commit Graph

2416 Commits

Author SHA1 Message Date
Davin Chia
1b0cf66bd3 Clean up Docker volume after running Docker Acceptance Test. (#4603) 2021-07-07 22:19:59 +08:00
Yevhenii
3dca4ee1f1 SAT: Validate config according to specification (#4565)
* Add config checking according to specification

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-07-07 16:59:08 +03:00
Davin Chia
e2074a4dc1 Logging to GCS. (#4501)
Add the ability to log to GCS.
2021-07-07 21:06:25 +08:00
oleh.zorenko
3e37e904b1 Source GitHub: few updates (#4598)
* Few updates for GitHub connector

Add `AIRBYTE_ENTRYPOINT` to `Dockerfile`.
Add `Changelog` section to `docs/integrations/sources/github.md`.
Remove `source-github/CHANGELOG.md` file.

* Bump GitHub connector version
2021-07-07 15:48:47 +03:00
Marcos Marx
6201100e72 Add entrypoint and bump version for remaining connectors. (#4539)
See the PR descriptions for all the connectors published.
2021-07-07 17:37:22 +08:00
George Claireaux
2ca4f9441b 🐛 Kubernetes: Fix server starting up before Temporal ready to operate (#4567)
* refactor waitForTemporalServerAndLog into TemporalUtils

* remove use of docker-compose-wait
2021-07-07 09:51:00 +01:00
Christophe Duong
af57170d21 Deploy a sweeper pod to clear completed pod histories (#4500)
* Deploy a sweeper pod to clear completed pod histories
2021-07-07 10:32:09 +02:00
Daniel Diamond
cf0999f13c 🐛 Source Github: Fix schema for pull_request stream (#4590)
* Cast pull_request.locked to boolean

* Bump version and update changelog
2021-07-07 11:09:12 +03:00
jonathan-duval
e2469342d9 Add operator guide for persistant volume on regional GKE cluster (#4591) 2021-07-07 14:22:00 +08:00
Marcos Marx
31555bab4d bump stripe version (#4587) 2021-07-06 21:13:24 -03:00
Sherif A. Nada
21116cad97 📚 add doc to describe how CDK handles schemas (#4537) 2021-07-06 15:58:27 -07:00
Abhi Vaidyanatha
3e3cf7b544 Fix broken link in Local CSV doc (#4578)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-06 11:24:26 -07:00
Rodrigo Menezes
17cf7bc314 🐛 Source PostgreSQL: Fixed PostgreSQL CDC numeric handling (#4568)
* 🐛 Source PostgreSQL: Fixed PostgreSQL CDC numeric handling

* Rm unnecessary tests for non-CDC

* Add more values to test in postgres cdc numeric

* Fix comment so it points to psql docs instead

* Oops, added to the wrong test file

* Add sourceType as well

* Fix precision on tests
2021-07-06 19:12:26 +01:00
irynakruk
6532774361 🎉 Source Snowflake: Added new comprehensive data type tests (#4318)
* Added SnowflakeSourceComprehensiveTest

Co-authored-by: Iryna Kruk <iryna.o.kruk@globallogic.com>
2021-07-06 17:36:36 +03:00
Marcos Marx
063e007f0f correct link in webapp (#4545) 2021-07-06 11:17:31 -03:00
oleh.zorenko
84ba3e79b3 🎉 Native GitHub connector (#4174)
* GitHub source initial version

All streams are in `full_refresh` mode only.

* Run format command

* Add handling for 404 error in Teams stream

* Update creds naming for github

* Update acceptance tests

* Update check_connection()

* Code review fixes

Remove `/integration_tests/catalog.json` file.
Use `Collaborators` class in `check_connection()` function.
Remove excessive data from GithubStream class.

* Remove unused import

* Implement requested changes

* Add incremental streams

* Remove `supports_incremental`

* Update docs

* Implement requested changes

* Implement requested changes

* Implement requested changes

* Bump connector version
2021-07-06 17:06:34 +03:00
jaimefr
2efac85c1f add deprecation info for google adwords (#4561) 2021-07-06 09:27:04 -03:00
Yevhenii
94f1d217b8 🎉 Native Gitlab connector (#4027)
* Create Airbyte-Native connector for Gitlab

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-07-06 15:25:01 +03:00
Davin Chia
dd41155cc2 Minor kube docs corrections. (#4563) 2021-07-06 17:52:08 +08:00
Davin Chia
b39670d73c Move frontend test to ec2 runner. (#4562)
* Move frontend test to run on ec2-runner.
2021-07-06 16:24:30 +08:00
Davin Chia
c83b134d1a Stablise Python Builds. (#4558)
See the PR description or issue 4559 for context.
2021-07-06 13:53:17 +08:00
John Lafleur
a9bd334ed9 Update technical-content-writer.md 2021-07-06 10:03:00 +11:00
John Lafleur
de0593e32e Update senior-software-engineer.md 2021-07-06 10:02:49 +11:00
John Lafleur
541d33f724 Update recruiter.md 2021-07-06 10:02:35 +11:00
John Lafleur
310f4665f9 Update founding-developer-evangelist.md 2021-07-06 10:02:15 +11:00
John Lafleur
fb54cb491b Update README.md 2021-07-06 09:59:35 +11:00
Eugene Kulak
15c8f8a551 SAT: Add validation of data types in the output records #3253 (#4345)
* validate that records match the schema from the catalog

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Vadym <vege1wgw@gmail.com>
2021-07-05 17:45:39 -04:00
Eugene Kulak
f4ceebe54a SAT: cursor_paths should support custom nested and absolute paths (#4552)
* extend cursor_path logic to support custom nested and absolute paths

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-07-05 16:23:50 -04:00
Sherif A. Nada
b4e55ac1b5 add acceptance test bash script wrapper for fb source (#4555) 2021-07-05 12:21:07 -07:00
Sherif Nada
d30044e91b hotfix formatting 2021-07-05 12:19:37 -07:00
Mario Molina
5c7f886aa4 🎉 API: Add an endpoint to list workspaces (#4530) 2021-07-05 12:17:40 -07:00
Artem Astapenko
fc71ab0192 Add lazy log library (#4502) 2021-07-05 23:58:55 +07:00
vovavovavovavova
08b32d18ed 🎉 New Source: SurveyMonkey (#4097)
* add base files

* upd base

* save

* save sample files

* save & todo resolve state

* save the stage

* save the stage

* pre-gradle save

* fix catalogs

* merge && format

* normal stream slices

* apply requested changes

* requested_changes

* postfix

* update comment

* expand question stream add page_id

* upd typing once + rm missed inactual todo

* upd: caching with temp file

* upd requirements (requested)

* latest requested fixes

* posttext fix retn tempfile

* apply changes && comment

* newly requested changes

* return back spec -> changes to be set in new issue

* merge && usage comment

* add unit_test for get_updated_state function

* add simple date test

* parametrized (?) unittest

* upd comment on record_mode usage

* replace config with custom var

* pytest mark parametrized use

* rm unneeded var

* upd tests (requested)

* merge && upd texts

* add env airbyte_entrypoint
2021-07-05 19:35:46 +03:00
Andrii Leonets
107f5b8d61 🎉 Abstract level for SQL relational database sources (#4123)
Abstract level for SQL relational database sources
2021-07-05 17:18:07 +03:00
Oleksandr
9517fae702 🎉 Source Shopify - change the default stream cursor_field to 'updated_at' where possible (#4472)
#4472 - Source Shopify - change the default stream cursor_field to 'updated_at' where possible

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-07-05 15:42:35 +03:00
Vadym
d382d1bb96 🎉 Publish airbyte-native twilio connector (#4529)
* Add Twilio CDK source

* Implement all streams full refresh support.

* Complete all streams and configs.

* Apply code format

* Fix streams data response.
Update streams which should not be incremental.
Fix configs.

* Fix incremental streams.
Fix configs.

* Fix acceptance tests config.

* Fix stream schemas.

* Fix messages stream schema.

* Catalogs fixed

* Update setup requirements.

* Add standard CI creds

* Update build.gradle and sample config files to remove unused references

* Updated sample config files json format

* Update streams.py

* Update incremental streams cursor_field formatting

* Update streams and test config

* Add no_empty_streams_no_usage_records_catalog.json

* Update streams code style.
Add twilio.md docs.
Other PR style fixes.

* Update twilio.md docs.

* Odd docstring removed.

* Uncomment tests.

* Add Docker AIRBYTE_ENTRYPOINT

* Add source definitions

* Update twilio source definitions
Remove twilio singer source
2021-07-05 12:36:54 +03:00
Davin Chia
71637a27c8 Improve Python Build. (#4540)
* Remove old pip build command. Deconflict Pydantic.

* Run test on ec2 instance.
2021-07-05 16:25:33 +08:00
Duk Panhavad
a7196d40c9 Add LOG_LEVEL for Temporal docker (#4532) 2021-07-05 15:26:30 +08:00
LiRen Tu
3d3988b1e3 Destination MySQL: bump version (#4531)
* Bump mysql destination version to pick up normalization

* Also change the publish command to run on ec2-runners to try and avoid build errors with gradle unable to find the right volume.

Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-07-04 21:54:01 +08:00
LiRen Tu
2caf3904f0 🎉 MySQL destination: normalization (#4163)
* Add mysql dbt package

* Add mysql normalization support in java

* Add mysql normalization support in python

* Fix unit tests

* Update readme

* Setup mysql container in integration test

* Add macros

* Depend on dbt-mysql from git repo

* Remove mysql limitation test

* Test normalization

* Revert protocol format change

* Fix mysel json macros

* Fix two more macros

* Fix table name length

* Fix array macro

* Fix equality test macro

* Update replace-identifiers

* Add more identifiers to replace

* Fix unnest macro

* Fix equality macro

* Check in mysql test output

* Update column limit test for mysql

* Escape parentheses

* Remove unnecessary mysql test

* Remove mysql output for easier code review

* Remove unnecessary mysql test

* Remove parentheses

* Update dependencies

* Skip mysql instead of manually write out types

* Bump version

* Check in unit test for mysql name transformer

* Fix type conversion

* Use json_value to extract scalar json fields

* Move dbt-mysql to Dockerfile (#4459)

* Format code

* Check in mysql dbt output

* Remove unnecessary quote

* Update mysql equality test to match 0.19.0

* Check in schema_test update

* Update readme

* Bump base normalization version

* Update document

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
2021-07-03 20:30:59 -07:00
Vadym
9531c4d4ec 🎉 New source: airbyte-native twilio connector (#4070)
* Add Twilio CDK source

* Implement all streams full refresh support.

* Complete all streams and configs.

* Apply code format

* Fix streams data response.
Update streams which should not be incremental.
Fix configs.

* Fix incremental streams.
Fix configs.

* Fix acceptance tests config.

* Fix stream schemas.

* Fix messages stream schema.

* Catalogs fixed

* Update setup requirements.

* Add standard CI creds

* Update build.gradle and sample config files to remove unused references

* Updated sample config files json format

* Update streams.py

* Update incremental streams cursor_field formatting

* Update streams and test config

* Add no_empty_streams_no_usage_records_catalog.json

* Update streams code style.
Add twilio.md docs.
Other PR style fixes.

* Update twilio.md docs.

* Odd docstring removed.

* Uncomment tests.

* Add Docker AIRBYTE_ENTRYPOINT
2021-07-03 20:59:38 +03:00
Marcos Marx
68f78d59ad Add AIRBYTE_ENTRYPOINT for Kubernetes support (#4469). 2021-07-03 22:05:32 +08:00
Marcos Marx
78acd7e97e Add Airbyte Entrypoint to Intercom (#4524) 2021-07-03 21:21:39 +08:00
Abhi Vaidyanatha
637d939d67 Update upgrading Airbyte with the new workflow and move it to Operator Guides (#4517)
* Move Upgrading Airbyte to Operator Guides now that issues will be less common

* Formatting changes

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-02 17:12:09 -07:00
Abhi Vaidyanatha
6fb2bd3401 Move deploying section above Operator guides (#4511)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-02 16:18:52 -07:00
Abhi Vaidyanatha
8332f61715 Move reset guide to Operator guides (#4510)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-02 16:12:10 -07:00
Abhi Vaidyanatha
b25051446c Add Superset installation to Airflow demo. (#4509)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-02 15:51:49 -07:00
Abhi Vaidyanatha
d725aa9fbe Add standalone getting started guides structure. (#4492)
* Add standalone getting started guides.

* Add GitHub source

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-02 13:49:26 -07:00
Sherif A. Nada
5989d19733 Disable gradle cache when publishing connectors (#4507) 2021-07-02 13:35:40 -07:00
Sherif A. Nada
9079cbdf78 Publish PR (#4121) (#4495)
Co-authored-by: Rodrigo Parra <rodpar07@gmail.com>
2021-07-02 13:34:57 -07:00