1
0
mirror of synced 2026-01-08 03:06:34 -05:00
Commit Graph

298 Commits

Author SHA1 Message Date
Abhi Vaidyanatha
d985fe73f4 Add note about rate limits with the HubSpot API (#4491)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-01 23:04:35 -07:00
Sherif A. Nada
9708e4c8de 🐛 bugfix Source Okta: Fix endless loop while syncing data from logs stream (#4464)
Co-authored-by: sabifranjo <sabifranjo@gmail.com>
Co-authored-by: Sabolc Franjo <sabolc.franjo@ev-box.com>
2021-07-01 17:39:51 -07:00
Abhi Vaidyanatha
b94d2eb4c4 Fix pubsub description YAML frontmatter (#4475)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-01 14:32:02 -07:00
Eugene
4399656ec9 🎉 Source Cockroachdb: Added new source for Cockroachdb (#4382)
* [1705] Added cockroachdb source connector
2021-07-01 22:04:21 +03:00
Abhi Vaidyanatha
da36e22272 Add missing Sources and Destinations to Catalog Table. (#4423)
* Add missing Sources and Destinations to Catalog Table.

* Add Snowflake Source.

* Address review comments

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-30 13:27:26 -07:00
Sherif A. Nada
71c7cecd74 Source postgres: document that replicating views is supported (#4008) 2021-06-28 00:11:37 -05:00
Kirill Chernov
40aed2249d 🎉 fix Google Adwords CI & publish new version (#4205)
* added SAT tests, deleted incremental test sfrom pipeline

* pr changes and tests uncommented

* deleted facebook file from pr

* deleted facebook file from pr

* removed tap adwords from install_requires

* Update airbyte-integrations/connectors/source-google-adwords-singer/acceptance-test-config.yml

* added changelog info

* version bump

* version bump

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-06-26 01:45:22 +03:00
Sherif Nada
ea975027bc Merge branch 'master' of github.com:airbytehq/airbyte 2021-06-25 12:45:04 -07:00
Sherif Nada
10d1168a03 update docs 2021-06-25 12:44:57 -07:00
LiRen Tu
8954a99290 Bump source looker connector version (#4350)
* Bump looker connector version

* Update changelog

* Format code
2021-06-25 11:23:16 -07:00
Prasanna Ram Venkatachalam
0c25e1cca1 🎉 New Destination: Google PubSub (#4339)
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-06-25 11:09:07 -07:00
Oleksandr
0e0d0a102e 🐛 Bugfix for pagination in Shopify Source Connector #4284 (#4290)
#4284 - Bugfix for pagination in Shopify Source Connector

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-06-25 19:45:45 +03:00
Michael Irvine
842f10bd83 🎉 Source Looker: add run-look endpoint (#3911)
* looker run_looks method works with normalization

* update catalogs

* validate that look(s) entered actually exist at config time

* field type lookup works, though it is a bit messy

* don't re-run explore query if we have already run it

* add more type mappings

* add some docstrings and comments

* add title for ease of use

* Revert "update catalogs"

This reverts commit 7ecaaa4d0d.

* fix configuration file formatting

* fix some issues from code review and reorder methods

* update docs to include run_look endpoint
2021-06-25 08:31:56 -07:00
Yaroslav Dudar
2ccb1c2de0 🎉 New source: AWS CloudTrail (#4122)
AWS CloudTrail connector initial release
2021-06-25 11:10:07 +03:00
Marcos Marx
77f7096d14 🐛 Source Mailchimp: use datacenter parameter from apikey info (#4285)
* replace datacenter parameter using apikey info

* change to use dynamic property

* add dynamic property

* bump version

* change config versino
2021-06-24 21:55:13 -03:00
Sherif A. Nada
c130465048 Release Shopify source 0.1.6 (#4332) 2021-06-24 16:39:35 -07:00
jacqueskpoty
e01576d461 🎉 Source Shopify: add pages, price rules and discount codes streams (#4009)
Co-authored-by: jsewawilson <jwilson@pebblepost.com>
2021-06-24 16:18:15 -07:00
Harshith Mullapudi
f4ddf4042d 🐛 Bugfix incorrectly failing check connection operations in Google Ads and Amazon AP API connectors (#4288) 2021-06-24 15:58:37 -07:00
LiRen Tu
e2c5b1d4c7 🎉 S3 destination: Avro & Jsonl output (#4227)
* Add jsonl format to spec.json

* Implement jsonl writer

* Add documentation

* Add acceptance test

* Update document

* Bump version

* Update document example

* Implement avro writer

* Implement compression codec

* Update documentation

* Revise documentation

* Add more tests

* Add acceptance test

* Format code

* Create helper method for name updater

* Update csv doc with normalization

* Update version date
2021-06-23 14:54:39 -07:00
irynakruk
0a4712cbe7 🎉 New Source: IBM DB2 (#4197)
* Db2 source: implementation and acceptance tests

* Db2 source: added documentation, build info, Db2SourceComprehensiveTest

* Db2 source: improved Db2SourceComprehensiveTest

* Db2 source: improved Db2SourceComprehensiveTest

* Db2 source: updated docs

* Db2 source: added test for the special values in Decfloat

* Formatter changes

Co-authored-by: Iryna Kruk <iryna.o.kruk@globallogic.com>
2021-06-23 18:06:31 +03:00
Christophe Duong
75a1dda07e 🎉 New BigQuery destination with Structured/Repeated Records (#4176) 2021-06-23 16:19:36 +02:00
Eugene Kulak
2f7c15a540 Source Facebook: migrate to CDK (#3743)
* move to CDK implementation
* make insights streams configurable
* improve Async implementation using slices
* minimize set of streams in configured_catalog.json
* apply suggestions from @sherifnada
* improve async jobs handling even more

Co-authored-by: Ka-Ping Yee <zestyping@gmail.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-06-22 18:48:17 -04:00
Vadym
4c2fc3ba00 Bump harvest version (#4277)
* Bump source Harvest docker version

* Update harvest.md docs
2021-06-22 20:22:02 +03:00
Marcos Marx
b57736b0ce Documentation correct markdown list in csv and json destination (#4165)
* correct list in markdown csv and json

* add copy cmd and change variable name to used in dest settings
2021-06-21 19:53:26 -03:00
Marcos Marx
156629ca95 Bump Harvest Source connector 0.1.2 (#4223)
* fixes #4221. Fix broken spec file property and point to correct docs page

* bump harvest connector

* add changelog for harvest

* bump dockerfile version

Co-authored-by: Tyler Schroeder <tyler@tylerschroeder.com>
2021-06-21 15:22:48 -03:00
Marcos Marx
88d7953a18 Add CHANGELOG for Sources (#4215)
* add changelog for postres source

* add changelog for mysql source

* add changelog for mssql source

* add changelog for file source

* add changelog for google sheets

* add full changelog for file source

* add full changelog for google sheet source

* add full changelog for mssql source

* add changelog for http requests

* add changelog for salesforce source

* add changelog for facebook marketing source

* add changelog for google analytics source

* add changelog for shopify native source connector
2021-06-18 17:45:09 -03:00
Marcos Marx
f365c0c430 destination mssql correct spec doc info (#4185) 2021-06-17 19:51:36 -03:00
LiRen Tu
14b605e064 Bump version and fix format (#4167)
* Bump version

* Move changelog to connector doc

* Fix flakeCheck

* Fix format
2021-06-16 13:20:31 -07:00
Harshith Mullapudi
78d17bc1a4 Source Amazon Seller Partner: Error for max wait time reached when create report status is CANCELLED or FATAL (#4108)
* 1. Use pendulum - user friendly
2. Fix: Error for max wait time reached when create report status is CANCELLED or FATAL

* Write test cases and changelog
2021-06-16 11:59:40 -07:00
LiRen Tu
53b51c08d1 Bump S3 destination version (#4166) 2021-06-16 11:54:50 -07:00
Prashantha Patali
0884fa74a6 📚More clarity added for Windows based Docker Desktop environments (#4068)
* Additinal details about Windows specific long filename error during cloning

* Corrected the path

* Revert "Corrected the path"

This reverts commit bbd3b78fcb.

* Revert "Additinal details about Windows specific long filename error during cloning"

This reverts commit 0b695eea1a.

* Email is mandatory 'Specify your preferences', Link to docker inst guide

* Use relative paths while linking

* Typo in Destination section

* On Windows, fixing long filename error during cloning

* For Windows with WSL2 and Docker, clarify steps to locate destination local folder

* Link to locating local files on Windows

* Update docs/contributing-to-airbyte/updating-documentation.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/quickstart/set-up-a-connection.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/examples/postgres-replication.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/examples/postgres-replication.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/examples/postgres-replication.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/local-deployment.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/local-deployment.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

Co-authored-by: Patali, Prashanth <ppatali@hidglobal.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
2021-06-16 14:14:25 -03:00
LiRen Tu
4e449e98a5 Bump jira version (#4126) 2021-06-15 03:34:47 -07:00
Harshith Mullapudi
5a37359389 🎉 Added getting started for google ads connector (#4125) 2021-06-15 17:52:19 +08:00
LiRen Tu
87552b25e4 🎉 Destination S3: parquet output (#3908)
* Add skeleton code for parquet writer

* Refactor s3 destination code

* Add parquet to spec

* Complete parquet writer

* Change testing data from int to double

* Add acceptance test for parquet writer

* Handle special schema field names

* Format code

* Add parquet config

* Add documentation

* Add unit tests

* Fix typo

* Update document

* Bump version

* Fix date format

* Fix credential filename

* Update doc

* Update test and publish commands

* Refactor s3 format config

* Append compression codec file extension

* Update doc

* Remove compression codec file extension

* Add comments

* Add README, CHANGELOG, and sample configs

* Move changelog

* Use switch statement

* Move filename helper method to base writer

* Rename converter

* Separate test cases

* Drop union type length restriction

* Support array with multiple types

* Move comments to connector doc

* Share config between acceptance tests

* Add doc about additional properties

* Move shared code out of if branch

* Add doc about adding a new format

* Format code

* Bump version to 0.1.4

* Fix default max padding size
2021-06-14 16:49:27 -07:00
Marcos Marx
d31e245f92 Add oracle destination to config files (#4045)
* add oracle destionation to config files

* change to oracle alpha + add integration page

* minor typo
2021-06-14 17:22:51 -03:00
Duk Panhavad
d2f9643650 🎉 Destination S3: alternative S3 support (#4038)
Authored by @panhavad

* base from prev PR
* add s3 alternative destination connector feature
* fix testGetOutputFilename
* default using aws
* Update airbyte-integrations/connectors/destination-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/copy/s3/S3Config.java
2021-06-13 00:57:19 -07:00
Abhi Vaidyanatha
e26f813536 Remove Postgres references in OracleDB Destination Docs (#4043)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-10 15:29:48 -06:00
irynakruk
446ec0407f 🎉 New Source: Snowflake DB (#3844) 2021-06-10 12:52:47 -07:00
vovavovavovavova
07b09a9a12 🎉 Posthog: New Source (#3768)
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-06-08 16:55:05 -07:00
Oleksandr
01dc7bbde9 🎉 Add Airbyte-native shopify connector (replaces Singer-based) (#3787)
* first commit

* Delete source copy.py

File doesn't need

* IAdded Incremental Refresh + stream_slice for Refunds, Transactions

* small changes to source.py

* fixed Incremental Refresh

* Fixed acceptence-tests

* Edited no-refunds-catalog for acceptence-tests

* changed the state

* Commit for Review

* removed spaces

* added parameter to orders stream to read all the records by default

* added parameter to abandoned_checkouts stream to read all the records by default

* minor fix

* minor fix

* changed authentication method from basic http > header based auth, removed usage of api_key

* reformated source.py

* improved source.py, removed dependencies block from build.gradle

* removed unneeded dependencies + formated source.py

* changed Shopify in source_definitions.yaml from singer-based to cdk version, changed years in the documents

* added source_definition for source-shopify cdk json file

* Delete airbyte-config/init/bin/main/config/STANDARD_SOURCE_DEFINITION directory

* Add files via upload

* added secret credentials for source-shopify

* Delete 9da77001-af33-4bcd-be46-6252bf9342b9.json

* re-added source_definition file

* Delete airbyte-config/init/bin/main/config/STANDARD_SOURCE_DEFINITION directory

* re-added source definition

* Delete airbyte-config/init/bin/main/config/STANDARD_SOURCE_DEFINITION directory

* edit source definitions

* source definition change: removed icon block

* ...

* changed source_definition.json

* changed source_definition.json

* Added ability to specify the start_date of data replica.

* changed docker version to 0.1.1 - bumped version on image, aedited request_params for metafields

* changed docker version to 0.1.1 - bumped version on image, aedited request_params for metafields

* changed docker version to 0.1.1 - bumped version on image, aedited request_params for metafields

* changed collects to full_refresh by default

* changed collects to full_refresh by default

* Delete airbyte-config/.settings directory

* Delete .classpath

* Delete .project

* Delete .project

* Delete .classpath

* Delete airbyte-config/init/.settings directory

* Delete airbyte-config/init/bin directory

* Delete airbyte-config/models/.settings directory

* Delete airbyte-config/models/bin directory

* Delete .classpath

* Delete .project

* Delete airbyte-config/persistence/.settings directory

* Delete airbyte-config/persistence/bin directory

* Delete .classpath

* Delete .project

* changed python to python3 in README.md

* small changes to code source.py, bump the version, edited .gitignore

* edited spec.json with pattern for start_date

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-06-08 12:46:35 +03:00
Yevhenii
a90e5f0eca 🎉 New Source: Amplitude (#3664)
* New Amplitude Source: Full Refresh and Incremental Sync

* update api.py, add CHANGELOG.md file

* bump version

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-06-08 10:36:31 +03:00
jacqueskpoty
8c9bf60bd8 updating integrations destinations local files doccs (#3910)
* updating integrations destinations local files doccs

* updating integrations destinations local files doccs

* Update docs/integrations/destinations/local-json.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* updating integrations destinations local files doccs

* updating integrations destinations local files doccs

* Update docs/integrations/destinations/local-csv.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/integrations/destinations/local-csv.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/integrations/destinations/local-csv.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/integrations/destinations/local-csv.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* updating integrations destinations local files doccs

Co-authored-by: jsewawilson <jwilson@pebblepost.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
2021-06-07 21:34:35 -03:00
Christophe Duong
913c55df6b 🎉 Add dataset location to BigQuery connector (issue #3277) (#3889)
Authored-by: sabifranjo <sabifranjo@gmail.com>
Authored-by: Sabolc Franjo <sabolc.franjo@ev-box.com>
2021-06-07 15:33:47 +02:00
Vadym
e699f39740 🎉 New Source: Harvest (#3709)
* Add harvest source.
Add Clients, Contacts, Company, Invoices, InvoiceMessages, InvoicePayments streams full_refresh support.

* Add all Harvest source streams.
Add config and schema files for all the streams.

* Complete streams implementation.
Add config and schemas.

* Fix expenses stream calls.
Fix ExpensesBase request_params method.
Add source docs and definitions.

* Update harvest.md docs.

* Comment source_definitions.yaml harvest definition.

* Delete Harvest standard source definitions.

* Add Harvest creds to commands.

* Update Harvest build.gradle.

* Add extra report streams schemas.
Update config files.
Update incremental streams mixin.

* Fix HarvestStreamWithPaginationSliced stream_slices method for incremental stream support

* Remove source-acceptance-test test requirement

* * Fix schema.
* Update streams
* Update docs

* Update billable_rates and cost_rates to full_refresh only

* Fix HarvestStreamIncrementalMixin request_params method

* Fix BillableRates and CostRates streams initialization

* Refactor streams.py code common usages.
Update HarvestTokenAuthenticator class definition.

* Add streams docstring.
Fix sub streams slicing.
Update spec and schemas.
Update md docs.

* Update report streams to be support incremental sync.
Update report schemas.

* Removed Uninvoiced and ProjectBudget incremental sync support.
Fix configs and configured catalog.

* Fix streams cursor value.
Update IncrementalReportsBase request_params.
Update schemas and config.

* Add harvest source definitions.
2021-06-07 15:11:30 +03:00
masonwheeler
8dadd1cebd Oracle destination implementation (#3498)
Working implementation of Oracle destination

Co-authored-by: cgardens <giardina.charles@gmail.com>
2021-06-03 16:27:09 -06:00
Abhi Vaidyanatha
a262f46e42 Add changelog for Stripe connector (#3832)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-03 15:40:55 -06:00
LiRen Tu
c13b9883e8 🎉 New destination: S3 (#3672)
* Update README icon links

* Update airbyte-specification doc

* Extend base connector

* Remove redundant region

* Separate warning from info

* Implement s3 destination

* Run format

* Clarify logging message

* Rename variables and functions

* Update documentation

* Rename and annotate interface

* Inject formatter factory

* Remove part size

* Fix spec field names and add unit tests

* Add unit tests for csv output formatter

* Format code

* Complete acceptance test and fix bugs

* Fix uuid

* Remove generator template files

They belong to another PR.

* Add unhappy test case

* Checkin airbyte state message

* Adjust stream transfer manager parameters

* Use underscore in filename

* Create csv sheet generator to handle data processing

* Format code

* Add partition id to filename

* Rename date format variable
2021-06-03 09:40:51 -07:00
vitaliizazmic
2072b77e97 🎉 Source Jira: Support all available entities in Jira Cloud
* Jira source #1389  - adding schemas for streams

* Jira source #1389  - supporting streams

* Jira source #1389  - creating_project script

* Jira source #1389  - updating docs

* Jira source #1389  - fixing check method

* Jira source #1389  - uploading missing schemes

* Jira source #1389  - disabling JQL and Server info streams

* Jira source #1389 - fixing according to PR comments

* Jira source #1389 - fixing filter_sharing and screen_tab_fields streams

* Update airbyte-integrations/connectors/source-jira/source_jira/client.py

* Google search console source #2257 - improving configured catalog(adding sync_mode and destination_sync_mode to streams)

* Jira Source - incremental sync

* Jira source #1390 - issues incremental sync

* Jira source #1390 - issue worklogs incremental sync

* Source Jira #1390 - incremental sync improving

* Source Jira #1390 - migrating to airbyte-cdk, creating CHANGELOG.md

* Source Jira #1389 - reformat

* Jira Source HTTP CDK

* Source Jira #3453 - cleaning branch

* Source Jira #3453 - cleaning branch (fix)

* Source Jira #3453 - abstractmethod get_updated_state

* Jira dummy data #2100 #2101

* Jira source #2100 - data generator

* Jira source #2100 - issue related streams populating

* Jira source #2100 - project related streams populating

* Jira source #2101 - populating data for non issue or project related streams

* Source Jira #2100 - improving according to comments

* Source Jira #2100 - format

* Source Jira #1389 - bump version

* Source Jira #1389 - enabling base_read acceptance test divided by stream groups

* Source Jira #1389 - bump version

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-06-02 22:13:36 +03:00
Gabriel Rebessi
eade88efa7 New Source: Okta (#3563)
* Add source-okta

* Add PR suggestions

* Remove typo

* Remove unused file

* hard code page size

* add missing fields to catalog.json

* Add missing fields to catalog

* Add docs to Okta

* Add credential for okta

* Update docs/integrations/README.md

* Fix linting

* fix dependency conflict

* Fix integration tests

* update ci_credentials

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-06-02 14:03:00 -03:00
oleh.zorenko
4b3e345866 Source Amazon SP: Add missing files (#3782)
* Add missing files

Add files for publishing the connector.
Update typing in few files.
Add `amazon-seller-partner.md` file.
Add `Amazon Seller Partner` to `builds.md` and to `README.md` files.

* Remove release files

* Comment out tests in `acceptance-test-config.yml`

* Update few files

Remove `!Dockerfile.test` from `.dockerignore` file.
Add `dependencies` to `build.gradle` file.

* Update `amazon-seller-partner.md` file

* Add stream to docs

Add `GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL` stream.

* Add release info

Add connector to `source_definitions.yaml` file.
Add connector to `STANDARD_SOURCE_DEFINITION` folder.
2021-06-01 22:55:38 +03:00