1
0
mirror of synced 2026-01-20 03:07:18 -05:00
Commit Graph

313 Commits

Author SHA1 Message Date
oleh.zorenko
3e37e904b1 Source GitHub: few updates (#4598)
* Few updates for GitHub connector

Add `AIRBYTE_ENTRYPOINT` to `Dockerfile`.
Add `Changelog` section to `docs/integrations/sources/github.md`.
Remove `source-github/CHANGELOG.md` file.

* Bump GitHub connector version
2021-07-07 15:48:47 +03:00
Marcos Marx
6201100e72 Add entrypoint and bump version for remaining connectors. (#4539)
See the PR descriptions for all the connectors published.
2021-07-07 17:37:22 +08:00
Christophe Duong
af57170d21 Deploy a sweeper pod to clear completed pod histories (#4500)
* Deploy a sweeper pod to clear completed pod histories
2021-07-07 10:32:09 +02:00
Marcos Marx
31555bab4d bump stripe version (#4587) 2021-07-06 21:13:24 -03:00
Abhi Vaidyanatha
3e3cf7b544 Fix broken link in Local CSV doc (#4578)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-06 11:24:26 -07:00
oleh.zorenko
84ba3e79b3 🎉 Native GitHub connector (#4174)
* GitHub source initial version

All streams are in `full_refresh` mode only.

* Run format command

* Add handling for 404 error in Teams stream

* Update creds naming for github

* Update acceptance tests

* Update check_connection()

* Code review fixes

Remove `/integration_tests/catalog.json` file.
Use `Collaborators` class in `check_connection()` function.
Remove excessive data from GithubStream class.

* Remove unused import

* Implement requested changes

* Add incremental streams

* Remove `supports_incremental`

* Update docs

* Implement requested changes

* Implement requested changes

* Implement requested changes

* Bump connector version
2021-07-06 17:06:34 +03:00
jaimefr
2efac85c1f add deprecation info for google adwords (#4561) 2021-07-06 09:27:04 -03:00
Yevhenii
94f1d217b8 🎉 Native Gitlab connector (#4027)
* Create Airbyte-Native connector for Gitlab

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-07-06 15:25:01 +03:00
Oleksandr
9517fae702 🎉 Source Shopify - change the default stream cursor_field to 'updated_at' where possible (#4472)
#4472 - Source Shopify - change the default stream cursor_field to 'updated_at' where possible

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-07-05 15:42:35 +03:00
LiRen Tu
3d3988b1e3 Destination MySQL: bump version (#4531)
* Bump mysql destination version to pick up normalization

* Also change the publish command to run on ec2-runners to try and avoid build errors with gradle unable to find the right volume.

Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-07-04 21:54:01 +08:00
Vadym
9531c4d4ec 🎉 New source: airbyte-native twilio connector (#4070)
* Add Twilio CDK source

* Implement all streams full refresh support.

* Complete all streams and configs.

* Apply code format

* Fix streams data response.
Update streams which should not be incremental.
Fix configs.

* Fix incremental streams.
Fix configs.

* Fix acceptance tests config.

* Fix stream schemas.

* Fix messages stream schema.

* Catalogs fixed

* Update setup requirements.

* Add standard CI creds

* Update build.gradle and sample config files to remove unused references

* Updated sample config files json format

* Update streams.py

* Update incremental streams cursor_field formatting

* Update streams and test config

* Add no_empty_streams_no_usage_records_catalog.json

* Update streams code style.
Add twilio.md docs.
Other PR style fixes.

* Update twilio.md docs.

* Odd docstring removed.

* Uncomment tests.

* Add Docker AIRBYTE_ENTRYPOINT
2021-07-03 20:59:38 +03:00
Abhi Vaidyanatha
d725aa9fbe Add standalone getting started guides structure. (#4492)
* Add standalone getting started guides.

* Add GitHub source

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-02 13:49:26 -07:00
Sherif A. Nada
9079cbdf78 Publish PR (#4121) (#4495)
Co-authored-by: Rodrigo Parra <rodpar07@gmail.com>
2021-07-02 13:34:57 -07:00
Rodrigo Parra
4b0c3494ae 🎉 Source Shopify: Add support for draft orders (#4121) 2021-07-02 13:33:47 -07:00
Abhi Vaidyanatha
3b2009723e Add note about collocating Redshift cluster and Airbyte server in the same security group (#4490)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-02 12:42:35 -07:00
Abhi Vaidyanatha
d985fe73f4 Add note about rate limits with the HubSpot API (#4491)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-01 23:04:35 -07:00
Sherif A. Nada
9708e4c8de 🐛 bugfix Source Okta: Fix endless loop while syncing data from logs stream (#4464)
Co-authored-by: sabifranjo <sabifranjo@gmail.com>
Co-authored-by: Sabolc Franjo <sabolc.franjo@ev-box.com>
2021-07-01 17:39:51 -07:00
Abhi Vaidyanatha
b94d2eb4c4 Fix pubsub description YAML frontmatter (#4475)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-01 14:32:02 -07:00
Eugene
4399656ec9 🎉 Source Cockroachdb: Added new source for Cockroachdb (#4382)
* [1705] Added cockroachdb source connector
2021-07-01 22:04:21 +03:00
Abhi Vaidyanatha
da36e22272 Add missing Sources and Destinations to Catalog Table. (#4423)
* Add missing Sources and Destinations to Catalog Table.

* Add Snowflake Source.

* Address review comments

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-30 13:27:26 -07:00
Sherif A. Nada
71c7cecd74 Source postgres: document that replicating views is supported (#4008) 2021-06-28 00:11:37 -05:00
Kirill Chernov
40aed2249d 🎉 fix Google Adwords CI & publish new version (#4205)
* added SAT tests, deleted incremental test sfrom pipeline

* pr changes and tests uncommented

* deleted facebook file from pr

* deleted facebook file from pr

* removed tap adwords from install_requires

* Update airbyte-integrations/connectors/source-google-adwords-singer/acceptance-test-config.yml

* added changelog info

* version bump

* version bump

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-06-26 01:45:22 +03:00
Sherif Nada
ea975027bc Merge branch 'master' of github.com:airbytehq/airbyte 2021-06-25 12:45:04 -07:00
Sherif Nada
10d1168a03 update docs 2021-06-25 12:44:57 -07:00
LiRen Tu
8954a99290 Bump source looker connector version (#4350)
* Bump looker connector version

* Update changelog

* Format code
2021-06-25 11:23:16 -07:00
Prasanna Ram Venkatachalam
0c25e1cca1 🎉 New Destination: Google PubSub (#4339)
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-06-25 11:09:07 -07:00
Oleksandr
0e0d0a102e 🐛 Bugfix for pagination in Shopify Source Connector #4284 (#4290)
#4284 - Bugfix for pagination in Shopify Source Connector

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-06-25 19:45:45 +03:00
Michael Irvine
842f10bd83 🎉 Source Looker: add run-look endpoint (#3911)
* looker run_looks method works with normalization

* update catalogs

* validate that look(s) entered actually exist at config time

* field type lookup works, though it is a bit messy

* don't re-run explore query if we have already run it

* add more type mappings

* add some docstrings and comments

* add title for ease of use

* Revert "update catalogs"

This reverts commit 7ecaaa4d0d.

* fix configuration file formatting

* fix some issues from code review and reorder methods

* update docs to include run_look endpoint
2021-06-25 08:31:56 -07:00
Yaroslav Dudar
2ccb1c2de0 🎉 New source: AWS CloudTrail (#4122)
AWS CloudTrail connector initial release
2021-06-25 11:10:07 +03:00
Marcos Marx
77f7096d14 🐛 Source Mailchimp: use datacenter parameter from apikey info (#4285)
* replace datacenter parameter using apikey info

* change to use dynamic property

* add dynamic property

* bump version

* change config versino
2021-06-24 21:55:13 -03:00
Sherif A. Nada
c130465048 Release Shopify source 0.1.6 (#4332) 2021-06-24 16:39:35 -07:00
jacqueskpoty
e01576d461 🎉 Source Shopify: add pages, price rules and discount codes streams (#4009)
Co-authored-by: jsewawilson <jwilson@pebblepost.com>
2021-06-24 16:18:15 -07:00
Harshith Mullapudi
f4ddf4042d 🐛 Bugfix incorrectly failing check connection operations in Google Ads and Amazon AP API connectors (#4288) 2021-06-24 15:58:37 -07:00
LiRen Tu
e2c5b1d4c7 🎉 S3 destination: Avro & Jsonl output (#4227)
* Add jsonl format to spec.json

* Implement jsonl writer

* Add documentation

* Add acceptance test

* Update document

* Bump version

* Update document example

* Implement avro writer

* Implement compression codec

* Update documentation

* Revise documentation

* Add more tests

* Add acceptance test

* Format code

* Create helper method for name updater

* Update csv doc with normalization

* Update version date
2021-06-23 14:54:39 -07:00
irynakruk
0a4712cbe7 🎉 New Source: IBM DB2 (#4197)
* Db2 source: implementation and acceptance tests

* Db2 source: added documentation, build info, Db2SourceComprehensiveTest

* Db2 source: improved Db2SourceComprehensiveTest

* Db2 source: improved Db2SourceComprehensiveTest

* Db2 source: updated docs

* Db2 source: added test for the special values in Decfloat

* Formatter changes

Co-authored-by: Iryna Kruk <iryna.o.kruk@globallogic.com>
2021-06-23 18:06:31 +03:00
Christophe Duong
75a1dda07e 🎉 New BigQuery destination with Structured/Repeated Records (#4176) 2021-06-23 16:19:36 +02:00
Eugene Kulak
2f7c15a540 Source Facebook: migrate to CDK (#3743)
* move to CDK implementation
* make insights streams configurable
* improve Async implementation using slices
* minimize set of streams in configured_catalog.json
* apply suggestions from @sherifnada
* improve async jobs handling even more

Co-authored-by: Ka-Ping Yee <zestyping@gmail.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-06-22 18:48:17 -04:00
Vadym
4c2fc3ba00 Bump harvest version (#4277)
* Bump source Harvest docker version

* Update harvest.md docs
2021-06-22 20:22:02 +03:00
Marcos Marx
b57736b0ce Documentation correct markdown list in csv and json destination (#4165)
* correct list in markdown csv and json

* add copy cmd and change variable name to used in dest settings
2021-06-21 19:53:26 -03:00
Marcos Marx
156629ca95 Bump Harvest Source connector 0.1.2 (#4223)
* fixes #4221. Fix broken spec file property and point to correct docs page

* bump harvest connector

* add changelog for harvest

* bump dockerfile version

Co-authored-by: Tyler Schroeder <tyler@tylerschroeder.com>
2021-06-21 15:22:48 -03:00
Marcos Marx
88d7953a18 Add CHANGELOG for Sources (#4215)
* add changelog for postres source

* add changelog for mysql source

* add changelog for mssql source

* add changelog for file source

* add changelog for google sheets

* add full changelog for file source

* add full changelog for google sheet source

* add full changelog for mssql source

* add changelog for http requests

* add changelog for salesforce source

* add changelog for facebook marketing source

* add changelog for google analytics source

* add changelog for shopify native source connector
2021-06-18 17:45:09 -03:00
Marcos Marx
f365c0c430 destination mssql correct spec doc info (#4185) 2021-06-17 19:51:36 -03:00
LiRen Tu
14b605e064 Bump version and fix format (#4167)
* Bump version

* Move changelog to connector doc

* Fix flakeCheck

* Fix format
2021-06-16 13:20:31 -07:00
Harshith Mullapudi
78d17bc1a4 Source Amazon Seller Partner: Error for max wait time reached when create report status is CANCELLED or FATAL (#4108)
* 1. Use pendulum - user friendly
2. Fix: Error for max wait time reached when create report status is CANCELLED or FATAL

* Write test cases and changelog
2021-06-16 11:59:40 -07:00
LiRen Tu
53b51c08d1 Bump S3 destination version (#4166) 2021-06-16 11:54:50 -07:00
Prashantha Patali
0884fa74a6 📚More clarity added for Windows based Docker Desktop environments (#4068)
* Additinal details about Windows specific long filename error during cloning

* Corrected the path

* Revert "Corrected the path"

This reverts commit bbd3b78fcb.

* Revert "Additinal details about Windows specific long filename error during cloning"

This reverts commit 0b695eea1a.

* Email is mandatory 'Specify your preferences', Link to docker inst guide

* Use relative paths while linking

* Typo in Destination section

* On Windows, fixing long filename error during cloning

* For Windows with WSL2 and Docker, clarify steps to locate destination local folder

* Link to locating local files on Windows

* Update docs/contributing-to-airbyte/updating-documentation.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/quickstart/set-up-a-connection.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/examples/postgres-replication.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/examples/postgres-replication.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/examples/postgres-replication.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/local-deployment.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/local-deployment.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

Co-authored-by: Patali, Prashanth <ppatali@hidglobal.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
2021-06-16 14:14:25 -03:00
LiRen Tu
4e449e98a5 Bump jira version (#4126) 2021-06-15 03:34:47 -07:00
Harshith Mullapudi
5a37359389 🎉 Added getting started for google ads connector (#4125) 2021-06-15 17:52:19 +08:00
LiRen Tu
87552b25e4 🎉 Destination S3: parquet output (#3908)
* Add skeleton code for parquet writer

* Refactor s3 destination code

* Add parquet to spec

* Complete parquet writer

* Change testing data from int to double

* Add acceptance test for parquet writer

* Handle special schema field names

* Format code

* Add parquet config

* Add documentation

* Add unit tests

* Fix typo

* Update document

* Bump version

* Fix date format

* Fix credential filename

* Update doc

* Update test and publish commands

* Refactor s3 format config

* Append compression codec file extension

* Update doc

* Remove compression codec file extension

* Add comments

* Add README, CHANGELOG, and sample configs

* Move changelog

* Use switch statement

* Move filename helper method to base writer

* Rename converter

* Separate test cases

* Drop union type length restriction

* Support array with multiple types

* Move comments to connector doc

* Share config between acceptance tests

* Add doc about additional properties

* Move shared code out of if branch

* Add doc about adding a new format

* Format code

* Bump version to 0.1.4

* Fix default max padding size
2021-06-14 16:49:27 -07:00
Marcos Marx
d31e245f92 Add oracle destination to config files (#4045)
* add oracle destionation to config files

* change to oracle alpha + add integration page

* minor typo
2021-06-14 17:22:51 -03:00