1
0
mirror of synced 2025-12-21 11:01:41 -05:00
Commit Graph

28471 Commits

Author SHA1 Message Date
LiRen Tu
14b605e064 Bump version and fix format (#4167)
* Bump version

* Move changelog to connector doc

* Fix flakeCheck

* Fix format
2021-06-16 13:20:31 -07:00
Harshith Mullapudi
78d17bc1a4 Source Amazon Seller Partner: Error for max wait time reached when create report status is CANCELLED or FATAL (#4108)
* 1. Use pendulum - user friendly
2. Fix: Error for max wait time reached when create report status is CANCELLED or FATAL

* Write test cases and changelog
2021-06-16 11:59:40 -07:00
LiRen Tu
53b51c08d1 Bump S3 destination version (#4166) 2021-06-16 11:54:50 -07:00
Prashantha Patali
0884fa74a6 📚More clarity added for Windows based Docker Desktop environments (#4068)
* Additinal details about Windows specific long filename error during cloning

* Corrected the path

* Revert "Corrected the path"

This reverts commit bbd3b78fcb.

* Revert "Additinal details about Windows specific long filename error during cloning"

This reverts commit 0b695eea1a.

* Email is mandatory 'Specify your preferences', Link to docker inst guide

* Use relative paths while linking

* Typo in Destination section

* On Windows, fixing long filename error during cloning

* For Windows with WSL2 and Docker, clarify steps to locate destination local folder

* Link to locating local files on Windows

* Update docs/contributing-to-airbyte/updating-documentation.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/quickstart/set-up-a-connection.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/examples/postgres-replication.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/examples/postgres-replication.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/examples/postgres-replication.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/local-deployment.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/local-deployment.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

Co-authored-by: Patali, Prashanth <ppatali@hidglobal.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
2021-06-16 14:14:25 -03:00
LiRen Tu
4e449e98a5 Bump jira version (#4126) 2021-06-15 03:34:47 -07:00
Harshith Mullapudi
5a37359389 🎉 Added getting started for google ads connector (#4125) 2021-06-15 17:52:19 +08:00
LiRen Tu
87552b25e4 🎉 Destination S3: parquet output (#3908)
* Add skeleton code for parquet writer

* Refactor s3 destination code

* Add parquet to spec

* Complete parquet writer

* Change testing data from int to double

* Add acceptance test for parquet writer

* Handle special schema field names

* Format code

* Add parquet config

* Add documentation

* Add unit tests

* Fix typo

* Update document

* Bump version

* Fix date format

* Fix credential filename

* Update doc

* Update test and publish commands

* Refactor s3 format config

* Append compression codec file extension

* Update doc

* Remove compression codec file extension

* Add comments

* Add README, CHANGELOG, and sample configs

* Move changelog

* Use switch statement

* Move filename helper method to base writer

* Rename converter

* Separate test cases

* Drop union type length restriction

* Support array with multiple types

* Move comments to connector doc

* Share config between acceptance tests

* Add doc about additional properties

* Move shared code out of if branch

* Add doc about adding a new format

* Format code

* Bump version to 0.1.4

* Fix default max padding size
2021-06-14 16:49:27 -07:00
Marcos Marx
d31e245f92 Add oracle destination to config files (#4045)
* add oracle destionation to config files

* change to oracle alpha + add integration page

* minor typo
2021-06-14 17:22:51 -03:00
Duk Panhavad
d2f9643650 🎉 Destination S3: alternative S3 support (#4038)
Authored by @panhavad

* base from prev PR
* add s3 alternative destination connector feature
* fix testGetOutputFilename
* default using aws
* Update airbyte-integrations/connectors/destination-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/copy/s3/S3Config.java
2021-06-13 00:57:19 -07:00
Abhi Vaidyanatha
e26f813536 Remove Postgres references in OracleDB Destination Docs (#4043)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-10 15:29:48 -06:00
irynakruk
446ec0407f 🎉 New Source: Snowflake DB (#3844) 2021-06-10 12:52:47 -07:00
vovavovavovavova
07b09a9a12 🎉 Posthog: New Source (#3768)
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-06-08 16:55:05 -07:00
Oleksandr
01dc7bbde9 🎉 Add Airbyte-native shopify connector (replaces Singer-based) (#3787)
* first commit

* Delete source copy.py

File doesn't need

* IAdded Incremental Refresh + stream_slice for Refunds, Transactions

* small changes to source.py

* fixed Incremental Refresh

* Fixed acceptence-tests

* Edited no-refunds-catalog for acceptence-tests

* changed the state

* Commit for Review

* removed spaces

* added parameter to orders stream to read all the records by default

* added parameter to abandoned_checkouts stream to read all the records by default

* minor fix

* minor fix

* changed authentication method from basic http > header based auth, removed usage of api_key

* reformated source.py

* improved source.py, removed dependencies block from build.gradle

* removed unneeded dependencies + formated source.py

* changed Shopify in source_definitions.yaml from singer-based to cdk version, changed years in the documents

* added source_definition for source-shopify cdk json file

* Delete airbyte-config/init/bin/main/config/STANDARD_SOURCE_DEFINITION directory

* Add files via upload

* added secret credentials for source-shopify

* Delete 9da77001-af33-4bcd-be46-6252bf9342b9.json

* re-added source_definition file

* Delete airbyte-config/init/bin/main/config/STANDARD_SOURCE_DEFINITION directory

* re-added source definition

* Delete airbyte-config/init/bin/main/config/STANDARD_SOURCE_DEFINITION directory

* edit source definitions

* source definition change: removed icon block

* ...

* changed source_definition.json

* changed source_definition.json

* Added ability to specify the start_date of data replica.

* changed docker version to 0.1.1 - bumped version on image, aedited request_params for metafields

* changed docker version to 0.1.1 - bumped version on image, aedited request_params for metafields

* changed docker version to 0.1.1 - bumped version on image, aedited request_params for metafields

* changed collects to full_refresh by default

* changed collects to full_refresh by default

* Delete airbyte-config/.settings directory

* Delete .classpath

* Delete .project

* Delete .project

* Delete .classpath

* Delete airbyte-config/init/.settings directory

* Delete airbyte-config/init/bin directory

* Delete airbyte-config/models/.settings directory

* Delete airbyte-config/models/bin directory

* Delete .classpath

* Delete .project

* Delete airbyte-config/persistence/.settings directory

* Delete airbyte-config/persistence/bin directory

* Delete .classpath

* Delete .project

* changed python to python3 in README.md

* small changes to code source.py, bump the version, edited .gitignore

* edited spec.json with pattern for start_date

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2021-06-08 12:46:35 +03:00
Yevhenii
a90e5f0eca 🎉 New Source: Amplitude (#3664)
* New Amplitude Source: Full Refresh and Incremental Sync

* update api.py, add CHANGELOG.md file

* bump version

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-06-08 10:36:31 +03:00
jacqueskpoty
8c9bf60bd8 updating integrations destinations local files doccs (#3910)
* updating integrations destinations local files doccs

* updating integrations destinations local files doccs

* Update docs/integrations/destinations/local-json.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* updating integrations destinations local files doccs

* updating integrations destinations local files doccs

* Update docs/integrations/destinations/local-csv.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/integrations/destinations/local-csv.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/integrations/destinations/local-csv.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/integrations/destinations/local-csv.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* updating integrations destinations local files doccs

Co-authored-by: jsewawilson <jwilson@pebblepost.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
2021-06-07 21:34:35 -03:00
Christophe Duong
913c55df6b 🎉 Add dataset location to BigQuery connector (issue #3277) (#3889)
Authored-by: sabifranjo <sabifranjo@gmail.com>
Authored-by: Sabolc Franjo <sabolc.franjo@ev-box.com>
2021-06-07 15:33:47 +02:00
Vadym
e699f39740 🎉 New Source: Harvest (#3709)
* Add harvest source.
Add Clients, Contacts, Company, Invoices, InvoiceMessages, InvoicePayments streams full_refresh support.

* Add all Harvest source streams.
Add config and schema files for all the streams.

* Complete streams implementation.
Add config and schemas.

* Fix expenses stream calls.
Fix ExpensesBase request_params method.
Add source docs and definitions.

* Update harvest.md docs.

* Comment source_definitions.yaml harvest definition.

* Delete Harvest standard source definitions.

* Add Harvest creds to commands.

* Update Harvest build.gradle.

* Add extra report streams schemas.
Update config files.
Update incremental streams mixin.

* Fix HarvestStreamWithPaginationSliced stream_slices method for incremental stream support

* Remove source-acceptance-test test requirement

* * Fix schema.
* Update streams
* Update docs

* Update billable_rates and cost_rates to full_refresh only

* Fix HarvestStreamIncrementalMixin request_params method

* Fix BillableRates and CostRates streams initialization

* Refactor streams.py code common usages.
Update HarvestTokenAuthenticator class definition.

* Add streams docstring.
Fix sub streams slicing.
Update spec and schemas.
Update md docs.

* Update report streams to be support incremental sync.
Update report schemas.

* Removed Uninvoiced and ProjectBudget incremental sync support.
Fix configs and configured catalog.

* Fix streams cursor value.
Update IncrementalReportsBase request_params.
Update schemas and config.

* Add harvest source definitions.
2021-06-07 15:11:30 +03:00
masonwheeler
8dadd1cebd Oracle destination implementation (#3498)
Working implementation of Oracle destination

Co-authored-by: cgardens <giardina.charles@gmail.com>
2021-06-03 16:27:09 -06:00
Abhi Vaidyanatha
a262f46e42 Add changelog for Stripe connector (#3832)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-06-03 15:40:55 -06:00
LiRen Tu
c13b9883e8 🎉 New destination: S3 (#3672)
* Update README icon links

* Update airbyte-specification doc

* Extend base connector

* Remove redundant region

* Separate warning from info

* Implement s3 destination

* Run format

* Clarify logging message

* Rename variables and functions

* Update documentation

* Rename and annotate interface

* Inject formatter factory

* Remove part size

* Fix spec field names and add unit tests

* Add unit tests for csv output formatter

* Format code

* Complete acceptance test and fix bugs

* Fix uuid

* Remove generator template files

They belong to another PR.

* Add unhappy test case

* Checkin airbyte state message

* Adjust stream transfer manager parameters

* Use underscore in filename

* Create csv sheet generator to handle data processing

* Format code

* Add partition id to filename

* Rename date format variable
2021-06-03 09:40:51 -07:00
vitaliizazmic
2072b77e97 🎉 Source Jira: Support all available entities in Jira Cloud
* Jira source #1389  - adding schemas for streams

* Jira source #1389  - supporting streams

* Jira source #1389  - creating_project script

* Jira source #1389  - updating docs

* Jira source #1389  - fixing check method

* Jira source #1389  - uploading missing schemes

* Jira source #1389  - disabling JQL and Server info streams

* Jira source #1389 - fixing according to PR comments

* Jira source #1389 - fixing filter_sharing and screen_tab_fields streams

* Update airbyte-integrations/connectors/source-jira/source_jira/client.py

* Google search console source #2257 - improving configured catalog(adding sync_mode and destination_sync_mode to streams)

* Jira Source - incremental sync

* Jira source #1390 - issues incremental sync

* Jira source #1390 - issue worklogs incremental sync

* Source Jira #1390 - incremental sync improving

* Source Jira #1390 - migrating to airbyte-cdk, creating CHANGELOG.md

* Source Jira #1389 - reformat

* Jira Source HTTP CDK

* Source Jira #3453 - cleaning branch

* Source Jira #3453 - cleaning branch (fix)

* Source Jira #3453 - abstractmethod get_updated_state

* Jira dummy data #2100 #2101

* Jira source #2100 - data generator

* Jira source #2100 - issue related streams populating

* Jira source #2100 - project related streams populating

* Jira source #2101 - populating data for non issue or project related streams

* Source Jira #2100 - improving according to comments

* Source Jira #2100 - format

* Source Jira #1389 - bump version

* Source Jira #1389 - enabling base_read acceptance test divided by stream groups

* Source Jira #1389 - bump version

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-06-02 22:13:36 +03:00
Gabriel Rebessi
eade88efa7 New Source: Okta (#3563)
* Add source-okta

* Add PR suggestions

* Remove typo

* Remove unused file

* hard code page size

* add missing fields to catalog.json

* Add missing fields to catalog

* Add docs to Okta

* Add credential for okta

* Update docs/integrations/README.md

* Fix linting

* fix dependency conflict

* Fix integration tests

* update ci_credentials

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-06-02 14:03:00 -03:00
oleh.zorenko
4b3e345866 Source Amazon SP: Add missing files (#3782)
* Add missing files

Add files for publishing the connector.
Update typing in few files.
Add `amazon-seller-partner.md` file.
Add `Amazon Seller Partner` to `builds.md` and to `README.md` files.

* Remove release files

* Comment out tests in `acceptance-test-config.yml`

* Update few files

Remove `!Dockerfile.test` from `.dockerignore` file.
Add `dependencies` to `build.gradle` file.

* Update `amazon-seller-partner.md` file

* Add stream to docs

Add `GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL` stream.

* Add release info

Add connector to `source_definitions.yaml` file.
Add connector to `STANDARD_SOURCE_DEFINITION` folder.
2021-06-01 22:55:38 +03:00
etsybaev
1fed4b0e2c 🐛 Destination MSSQL: fix issue with unicode symbols handling (ex.\u2028 showed as ?) (#3671)
* [3552] Fixed MSSQL _airbyte_data field's type to support unicode
2021-06-01 09:38:13 +03:00
Phlair
9360ed0da9 🎉 File source: Add support for Azure blob storage (#3660)
Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-06-01 10:42:13 +08:00
Marcos Marx
769d8b1216 removing <details> gitbook doenst support it (#3749) 2021-05-31 19:14:40 -03:00
Marcos Marx
2435012a46 Update exchange source doc and add changelog exp (#3682)
* update exchange source doc and add changelog exp

* add break line and 0.1.0 commit
2021-05-30 20:06:59 -03:00
Sherif A. Nada
248c285c6b remove connector health page and display all connectors on the root connectors page (#3478) 2021-05-29 09:09:58 -07:00
John Lafleur
a0a6d02824 Update asana.md 2021-05-27 16:38:23 +11:00
oleh.zorenko
5aa03643cf Update source-exchange-rates connector (#3566) 2021-05-26 21:59:42 +03:00
oleh.zorenko
61cb866faa Fix typo in docs/integrations/sources/asana.md (#3589) 2021-05-25 20:10:30 +03:00
oleh.zorenko
c8711abbd6 🎉 New Source: Asana (#3510)
* Asana source

* Fix creds for CI.

* Update connection status in acceptance test config

Change status from `exception` to `failed`.

* Implement change request.

Remove few files from /integration_tests folder.
Use `stream_slices` and/or `request_params` functions instear of `read_records` function

* Update sample_config.json file.

* Update airbyte-integrations/connectors/source-asana/CHANGELOG.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update `stream_slices` using.

Create reneric `read_stream` function in AsanaStream class and move there logic from `stream_slices` function.

* Rename functions.

rename `read_stream` to `read_slices_from_records`.

* Changes about publishing.

Add asana source to `source_definitions.yaml`.
Add `asana.svg`.
Create connector related file in `STANDARD_SOURCE_DEFINITION` folder.

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-05-25 18:55:46 +03:00
Subodh Kant Chaturvedi
7ccc4fafe8 🎉 source: implementation for mysql cdc (#3505)
* source: implementation for mysql cdc

* add target file and position

* dont want to add file in this PR

* refine tests + add comments

* fix typo

* address review comments

* fix formatting error

* resolve conflicts

* update docs + bump docker minor version

* remove un-necessary new lines + add multiple checks for cdc

* address review comments from Davin

* increase the version in source_definitions.yaml

* rebuild seed
2021-05-25 00:31:57 +05:30
Marcos Marx
df3d37650d correct smarthsheet markdown (#3568) 2021-05-24 15:19:33 -03:00
John Lafleur
e563936c2c Updated description of Clickhouse source connector (#3556)
* Updated description of Clickhouse source connector

The first sentence of the overview seemed to indicate it only supported Full Refresh, while it supports both full refresh and incremental

* Update clickhouse.md

* Update clickhouse.md
2021-05-24 20:26:52 +11:00
Yevhenii
3647c3cf1d 🎉 New Source: Recharge (#3466)
* Recharge Connector: Full Refresh add Incremental syncs, Acceptance tests, docs

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-05-20 14:45:52 +03:00
Vadym
f9feb2e903 🎉 Source Slack bugfix: don't overwrite thread_ts in threads stream (#3483)
* setup.py install_requirements updated

* setup.py formatting fix.
slack.md docks update.

* Bump source-slack-singer docker version
2021-05-20 12:38:05 +03:00
Eugene Kulak
d2eb5c0564 🎉 New Source: Klaviyo (#3463)
* Klaviyo source

* add test creds

* format and docs

* remove accidentally committed files

* fix build

* format

* restore page size

* add creds to CI

* update configured_catalog.json

* register in default sources

* fix spec

* Update airbyte-integrations/connectors/source-klaviyo/source_klaviyo/source.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* address comments from @sherifnada

* dump spec and enable spec test

* fix acceptance-test-config.yml

* fix configured_catalog.json

* fix configured_catalog.json

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-05-19 18:36:09 -04:00
masonwheeler
a67c769980 MS SQL Server Destination implementation
Fixes issue #613.

Normalization is not yet enabled.  This will have to be added at a later point.
2021-05-17 09:54:52 -06:00
Marcos Marx
a18b0e80d0 GitBook: [master] 161 pages and 75 assets modified 2021-05-16 22:51:02 +00:00
Davin Chia
fbbd25069c Correct GA readme error. (#3407) 2021-05-14 09:58:18 +08:00
Yevhenii
d44fc16e65 Source Stripe: Add Acceptance Tests to Stripe Connector (#3367)
* Add Acceptance Tests to Stripe Connector

* move configured_catalog.json to sample_files

* bump version

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-05-13 18:40:24 +03:00
Subodh Kant Chaturvedi
69125cfbeb source: clickhouse setup docs (#3375)
* clickhouse source setup docs

* fix typo

* address review comment
2021-05-13 13:35:10 +05:30
Abhi Vaidyanatha
40e664fcec Add guide on how to upgrade a connector (#3369)
* Add guide on how to upgrade a connector.

* Update intrusctions on navigating to DockerHub

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-05-11 16:03:04 -07:00
vitaliizazmic
735b0783dc 🎉 New Source: google search console
* Google search console source #2257 - new source

* Google search console source #2257 - reformat

* Google search console source #2257 - adding gcc to docker container

* Google search console source #2257 - remove unused files, update acceptance test configs, change tap repo to airbyte

* Google search console source #2257 - updating acceptance tests configs

* Google search console source #2257 - updating acceptance cursor_paths

* Google search console source #2257 - temporary disable tests

* Google search console source #2257 - disable performance_report_date stream

* Google search console source #2257 - disable performance_report_date stream (update docs)

* Google search console source #2257 - disable performance_report_date stream for tests

* Google search console source #2257 - updating singer tap fork
2021-05-10 09:59:44 +03:00
Sherif A. Nada
32773cb7f8 update FB marketing rate limiting link (#3294) 2021-05-07 17:32:36 -07:00
Subodh Kant Chaturvedi
a2ec9ee1e5 mysql destination docs (#3275)
* setup doc for mysql destination

* update the documentation in right places
2021-05-07 13:38:07 +05:30
Jared Rhizor
beeee785f2 clarify publication and replication identity handling in cdc docs (#3258)
* clarify publication and replication identity handling in cdc docs

* remove character
2021-05-06 17:34:42 -07:00
Abhi Vaidyanatha
b215733e22 Add source connector for the PokéAPI (#3149)
* Add toy source for the PokeAPI

* Address review comments.

* Run gradlew format

* Move Pokemon list constant to its own file

* Fix gradle formatting issues

* Prepare connector for publishing.

* Fix some naming bugs

* Update config file path

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-05-04 19:18:35 -07:00
Jared Rhizor
32ee10e76d snowflake gcs staging (#3183)
* spec

* snowflake gcs

* spec comment

* fmt

* remove sout

* add docs

* format

* respond to comments

* fix

* bump to 0.3.3
2021-05-04 09:59:32 -07:00