jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-27 07:02:03 -05:00
Code Issues Packages Projects Releases Wiki Activity
2,152 Commits 5,392 Branches 518 Tags
2c480e1ca705edfda992fe298f3f71a0ab4e02ae
Commit Graph

261 Commits

Author SHA1 Message Date
irynakruk
446ec0407f 🎉 New Source: Snowflake DB (#3844) 2021-06-10 12:52:47 -07:00
vovavovavovavova
07b09a9a12 🎉 Posthog: New Source (#3768) 
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-06-08 16:55:05 -07:00
Oleksandr
01dc7bbde9 🎉 Add Airbyte-native shopify connector (replaces Singer-based) (#3787) 
* first commit

* Delete source copy.py

File doesn't need

* IAdded Incremental Refresh + stream_slice for Refunds, Transactions

* small changes to source.py

* fixed Incremental Refresh

* Fixed acceptence-tests

* Edited no-refunds-catalog for acceptence-tests

* changed the state

* Commit for Review

* removed spaces

* added parameter to orders stream to read all the records by default

* added parameter to abandoned_checkouts stream to read all the records by default

* minor fix

* minor fix

* changed authentication method from basic http > header based auth, removed usage of api_key

* reformated source.py

* improved source.py, removed dependencies block from build.gradle

* removed unneeded dependencies + formated source.py

* changed Shopify in source_definitions.yaml from singer-based to cdk version, changed years in the documents

* added source_definition for source-shopify cdk json file

* Delete airbyte-config/init/bin/main/config/STANDARD_SOURCE_DEFINITION directory

* Add files via upload

* added secret credentials for source-shopify

* Delete 9da77001-af33-4bcd-be46-6252bf9342b9.json

* re-added source_definition file

* Delete airbyte-config/init/bin/main/config/STANDARD_SOURCE_DEFINITION directory

* re-added source definition

* Delete airbyte-config/init/bin/main/config/STANDARD_SOURCE_DEFINITION directory

* edit source definitions

* source definition change: removed icon block

* ...

* changed source_definition.json

* changed source_definition.json

* Added ability to specify the start_date of data replica.

* changed docker version to 0.1.1 - bumped version on image, aedited request_params for metafields

* changed docker version to 0.1.1 - bumped version on image, aedited request_params for metafields

* changed docker version to 0.1.1 - bumped version on image, aedited request_params for metafields

* changed collects to full_refresh by default

* changed collects to full_refresh by default

* Delete airbyte-config/.settings directory

* Delete .classpath

* Delete .project

* Delete .project

* Delete .classpath

* Delete airbyte-config/init/.settings directory

* Delete airbyte-config/init/bin directory

* Delete airbyte-config/models/.settings directory

* Delete airbyte-config/models/bin directory

* Delete .classpath

* Delete .project

* Delete airbyte-config/persistence/.settings directory

* Delete airbyte-config/persistence/bin directory

* Delete .classpath

* Delete .project

* changed python to python3 in README.md

* small changes to code source.py, bump the version, edited .gitignore

* edited spec.json with pattern for start_date

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
 2021-06-08 12:46:35 +03:00
Yevhenii
a90e5f0eca 🎉 New Source: Amplitude (#3664) 
* New Amplitude Source: Full Refresh and Incremental Sync

* update api.py, add CHANGELOG.md file

* bump version

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
 2021-06-08 10:36:31 +03:00
jacqueskpoty
8c9bf60bd8 updating integrations destinations local files doccs (#3910) 
* updating integrations destinations local files doccs

* updating integrations destinations local files doccs

* Update docs/integrations/destinations/local-json.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* updating integrations destinations local files doccs

* updating integrations destinations local files doccs

* Update docs/integrations/destinations/local-csv.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/integrations/destinations/local-csv.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/integrations/destinations/local-csv.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/integrations/destinations/local-csv.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* updating integrations destinations local files doccs

Co-authored-by: jsewawilson <jwilson@pebblepost.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
 2021-06-07 21:34:35 -03:00
Christophe Duong
913c55df6b 🎉 Add dataset location to BigQuery connector (issue #3277) (#3889) 
Authored-by: sabifranjo <sabifranjo@gmail.com>
Authored-by: Sabolc Franjo <sabolc.franjo@ev-box.com>
 2021-06-07 15:33:47 +02:00
Vadym
e699f39740 🎉 New Source: Harvest (#3709) 
* Add harvest source.
Add Clients, Contacts, Company, Invoices, InvoiceMessages, InvoicePayments streams full_refresh support.

* Add all Harvest source streams.
Add config and schema files for all the streams.

* Complete streams implementation.
Add config and schemas.

* Fix expenses stream calls.
Fix ExpensesBase request_params method.
Add source docs and definitions.

* Update harvest.md docs.

* Comment source_definitions.yaml harvest definition.

* Delete Harvest standard source definitions.

* Add Harvest creds to commands.

* Update Harvest build.gradle.

* Add extra report streams schemas.
Update config files.
Update incremental streams mixin.

* Fix HarvestStreamWithPaginationSliced stream_slices method for incremental stream support

* Remove source-acceptance-test test requirement

* * Fix schema.
* Update streams
* Update docs

* Update billable_rates and cost_rates to full_refresh only

* Fix HarvestStreamIncrementalMixin request_params method

* Fix BillableRates and CostRates streams initialization

* Refactor streams.py code common usages.
Update HarvestTokenAuthenticator class definition.

* Add streams docstring.
Fix sub streams slicing.
Update spec and schemas.
Update md docs.

* Update report streams to be support incremental sync.
Update report schemas.

* Removed Uninvoiced and ProjectBudget incremental sync support.
Fix configs and configured catalog.

* Fix streams cursor value.
Update IncrementalReportsBase request_params.
Update schemas and config.

* Add harvest source definitions.
 2021-06-07 15:11:30 +03:00
masonwheeler
8dadd1cebd Oracle destination implementation (#3498) 
Working implementation of Oracle destination

Co-authored-by: cgardens <giardina.charles@gmail.com>
 2021-06-03 16:27:09 -06:00
Abhi Vaidyanatha
a262f46e42 Add changelog for Stripe connector (#3832) 
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-06-03 15:40:55 -06:00
LiRen Tu
c13b9883e8 🎉 New destination: S3 (#3672) 
* Update README icon links

* Update airbyte-specification doc

* Extend base connector

* Remove redundant region

* Separate warning from info

* Implement s3 destination

* Run format

* Clarify logging message

* Rename variables and functions

* Update documentation

* Rename and annotate interface

* Inject formatter factory

* Remove part size

* Fix spec field names and add unit tests

* Add unit tests for csv output formatter

* Format code

* Complete acceptance test and fix bugs

* Fix uuid

* Remove generator template files

They belong to another PR.

* Add unhappy test case

* Checkin airbyte state message

* Adjust stream transfer manager parameters

* Use underscore in filename

* Create csv sheet generator to handle data processing

* Format code

* Add partition id to filename

* Rename date format variable
 2021-06-03 09:40:51 -07:00
vitaliizazmic
2072b77e97 🎉 Source Jira: Support all available entities in Jira Cloud 
* Jira source #1389  - adding schemas for streams

* Jira source #1389  - supporting streams

* Jira source #1389  - creating_project script

* Jira source #1389  - updating docs

* Jira source #1389  - fixing check method

* Jira source #1389  - uploading missing schemes

* Jira source #1389  - disabling JQL and Server info streams

* Jira source #1389 - fixing according to PR comments

* Jira source #1389 - fixing filter_sharing and screen_tab_fields streams

* Update airbyte-integrations/connectors/source-jira/source_jira/client.py

* Google search console source #2257 - improving configured catalog(adding sync_mode and destination_sync_mode to streams)

* Jira Source - incremental sync

* Jira source #1390 - issues incremental sync

* Jira source #1390 - issue worklogs incremental sync

* Source Jira #1390 - incremental sync improving

* Source Jira #1390 - migrating to airbyte-cdk, creating CHANGELOG.md

* Source Jira #1389 - reformat

* Jira Source HTTP CDK

* Source Jira #3453 - cleaning branch

* Source Jira #3453 - cleaning branch (fix)

* Source Jira #3453 - abstractmethod get_updated_state

* Jira dummy data #2100 #2101

* Jira source #2100 - data generator

* Jira source #2100 - issue related streams populating

* Jira source #2100 - project related streams populating

* Jira source #2101 - populating data for non issue or project related streams

* Source Jira #2100 - improving according to comments

* Source Jira #2100 - format

* Source Jira #1389 - bump version

* Source Jira #1389 - enabling base_read acceptance test divided by stream groups

* Source Jira #1389 - bump version

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-06-02 22:13:36 +03:00
Gabriel Rebessi
eade88efa7 New Source: Okta (#3563) 
* Add source-okta

* Add PR suggestions

* Remove typo

* Remove unused file

* hard code page size

* add missing fields to catalog.json

* Add missing fields to catalog

* Add docs to Okta

* Add credential for okta

* Update docs/integrations/README.md

* Fix linting

* fix dependency conflict

* Fix integration tests

* update ci_credentials

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-06-02 14:03:00 -03:00
oleh.zorenko
4b3e345866 Source Amazon SP: Add missing files (#3782) 
* Add missing files

Add files for publishing the connector.
Update typing in few files.
Add `amazon-seller-partner.md` file.
Add `Amazon Seller Partner` to `builds.md` and to `README.md` files.

* Remove release files

* Comment out tests in `acceptance-test-config.yml`

* Update few files

Remove `!Dockerfile.test` from `.dockerignore` file.
Add `dependencies` to `build.gradle` file.

* Update `amazon-seller-partner.md` file

* Add stream to docs

Add `GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL` stream.

* Add release info

Add connector to `source_definitions.yaml` file.
Add connector to `STANDARD_SOURCE_DEFINITION` folder.
 2021-06-01 22:55:38 +03:00
etsybaev
1fed4b0e2c 🐛 Destination MSSQL: fix issue with unicode symbols handling (ex.\u2028 showed as ?) (#3671) 
* [3552] Fixed MSSQL _airbyte_data field's type to support unicode
 2021-06-01 09:38:13 +03:00
Phlair
9360ed0da9 🎉 File source: Add support for Azure blob storage (#3660) 
Co-authored-by: Davin Chia <davinchia@gmail.com>
 2021-06-01 10:42:13 +08:00
Marcos Marx
769d8b1216 removing <details> gitbook doenst support it (#3749) 2021-05-31 19:14:40 -03:00
Marcos Marx
2435012a46 Update exchange source doc and add changelog exp (#3682) 
* update exchange source doc and add changelog exp

* add break line and 0.1.0 commit
 2021-05-30 20:06:59 -03:00
Sherif A. Nada
248c285c6b remove connector health page and display all connectors on the root connectors page (#3478) 2021-05-29 09:09:58 -07:00
John Lafleur
a0a6d02824 Update asana.md 2021-05-27 16:38:23 +11:00
oleh.zorenko
5aa03643cf Update source-exchange-rates connector (#3566) 2021-05-26 21:59:42 +03:00
oleh.zorenko
61cb866faa Fix typo in docs/integrations/sources/asana.md (#3589) 2021-05-25 20:10:30 +03:00
oleh.zorenko
c8711abbd6 🎉 New Source: Asana (#3510) 
* Asana source

* Fix creds for CI.

* Update connection status in acceptance test config

Change status from `exception` to `failed`.

* Implement change request.

Remove few files from /integration_tests folder.
Use `stream_slices` and/or `request_params` functions instear of `read_records` function

* Update sample_config.json file.

* Update airbyte-integrations/connectors/source-asana/CHANGELOG.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update `stream_slices` using.

Create reneric `read_stream` function in AsanaStream class and move there logic from `stream_slices` function.

* Rename functions.

rename `read_stream` to `read_slices_from_records`.

* Changes about publishing.

Add asana source to `source_definitions.yaml`.
Add `asana.svg`.
Create connector related file in `STANDARD_SOURCE_DEFINITION` folder.

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-05-25 18:55:46 +03:00
Subodh Kant Chaturvedi
7ccc4fafe8 🎉 source: implementation for mysql cdc (#3505) 
* source: implementation for mysql cdc

* add target file and position

* dont want to add file in this PR

* refine tests + add comments

* fix typo

* address review comments

* fix formatting error

* resolve conflicts

* update docs + bump docker minor version

* remove un-necessary new lines + add multiple checks for cdc

* address review comments from Davin

* increase the version in source_definitions.yaml

* rebuild seed
 2021-05-25 00:31:57 +05:30
Marcos Marx
df3d37650d correct smarthsheet markdown (#3568) 2021-05-24 15:19:33 -03:00
John Lafleur
e563936c2c Updated description of Clickhouse source connector (#3556) 
* Updated description of Clickhouse source connector

The first sentence of the overview seemed to indicate it only supported Full Refresh, while it supports both full refresh and incremental

* Update clickhouse.md

* Update clickhouse.md
 2021-05-24 20:26:52 +11:00
Yevhenii
3647c3cf1d 🎉 New Source: Recharge (#3466) 
* Recharge Connector: Full Refresh add Incremental syncs, Acceptance tests, docs

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
 2021-05-20 14:45:52 +03:00
Vadym
f9feb2e903 🎉 Source Slack bugfix: don't overwrite thread_ts in threads stream (#3483) 
* setup.py install_requirements updated

* setup.py formatting fix.
slack.md docks update.

* Bump source-slack-singer docker version
 2021-05-20 12:38:05 +03:00
Eugene Kulak
d2eb5c0564 🎉 New Source: Klaviyo (#3463) 
* Klaviyo source

* add test creds

* format and docs

* remove accidentally committed files

* fix build

* format

* restore page size

* add creds to CI

* update configured_catalog.json

* register in default sources

* fix spec

* Update airbyte-integrations/connectors/source-klaviyo/source_klaviyo/source.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* address comments from @sherifnada

* dump spec and enable spec test

* fix acceptance-test-config.yml

* fix configured_catalog.json

* fix configured_catalog.json

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-05-19 18:36:09 -04:00
masonwheeler
a67c769980 MS SQL Server Destination implementation 
Fixes issue #613.

Normalization is not yet enabled.  This will have to be added at a later point.
 2021-05-17 09:54:52 -06:00
Marcos Marx
a18b0e80d0 GitBook: [master] 161 pages and 75 assets modified 2021-05-16 22:51:02 +00:00
Davin Chia
fbbd25069c Correct GA readme error. (#3407) 2021-05-14 09:58:18 +08:00
Yevhenii
d44fc16e65 Source Stripe: Add Acceptance Tests to Stripe Connector (#3367) 
* Add Acceptance Tests to Stripe Connector

* move configured_catalog.json to sample_files

* bump version

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
 2021-05-13 18:40:24 +03:00
Subodh Kant Chaturvedi
69125cfbeb source: clickhouse setup docs (#3375) 
* clickhouse source setup docs

* fix typo

* address review comment
 2021-05-13 13:35:10 +05:30
Abhi Vaidyanatha
40e664fcec Add guide on how to upgrade a connector (#3369) 
* Add guide on how to upgrade a connector.

* Update intrusctions on navigating to DockerHub

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-05-11 16:03:04 -07:00
vitaliizazmic
735b0783dc 🎉 New Source: google search console 
* Google search console source #2257 - new source

* Google search console source #2257 - reformat

* Google search console source #2257 - adding gcc to docker container

* Google search console source #2257 - remove unused files, update acceptance test configs, change tap repo to airbyte

* Google search console source #2257 - updating acceptance tests configs

* Google search console source #2257 - updating acceptance cursor_paths

* Google search console source #2257 - temporary disable tests

* Google search console source #2257 - disable performance_report_date stream

* Google search console source #2257 - disable performance_report_date stream (update docs)

* Google search console source #2257 - disable performance_report_date stream for tests

* Google search console source #2257 - updating singer tap fork
 2021-05-10 09:59:44 +03:00
Sherif A. Nada
32773cb7f8 update FB marketing rate limiting link (#3294) 2021-05-07 17:32:36 -07:00
Subodh Kant Chaturvedi
a2ec9ee1e5 mysql destination docs (#3275) 
* setup doc for mysql destination

* update the documentation in right places
 2021-05-07 13:38:07 +05:30
Jared Rhizor
beeee785f2 clarify publication and replication identity handling in cdc docs (#3258) 
* clarify publication and replication identity handling in cdc docs

* remove character
 2021-05-06 17:34:42 -07:00
Abhi Vaidyanatha
b215733e22 Add source connector for the PokéAPI (#3149) 
* Add toy source for the PokeAPI

* Address review comments.

* Run gradlew format

* Move Pokemon list constant to its own file

* Fix gradle formatting issues

* Prepare connector for publishing.

* Fix some naming bugs

* Update config file path

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-05-04 19:18:35 -07:00
Jared Rhizor
32ee10e76d snowflake gcs staging (#3183) 
* spec

* snowflake gcs

* spec comment

* fmt

* remove sout

* add docs

* format

* respond to comments

* fix

* bump to 0.3.3
 2021-05-04 09:59:32 -07:00
Yevhenii
6bc63264e7 🎉 New Source: Zendesk Chat (#3088) 
* Zendesk Chat connector: implement full_refresh sync

* format

* Source Zendesk Chat: Implement Incremental sync (#3157)

* Zendesk Chat: Implement Incremental sync

* Roll back the stream Chats to the previous endpoint

* update integration-base-python to v0.1.6

* update limit to 100

* add acceptence tests

* update test and setup configs

* remove unknown file

* update naming of classes

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>

* register Zendesk Chat as source

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-05-03 16:02:00 -04:00
Marcos Marx
7c70eb02cd GitBook: [master] 84 pages and 72 assets modified 2021-05-03 12:14:16 +00:00
Marcos Marx
237b99e774 correct typo in mysql source docs (#3143) 
* correct mysql source docs

* change bigint order
 2021-04-30 18:45:40 -03:00
Yevhenii
672ebb49a3 Source MongoDB: add supporting TLS and Replica Sets (#3111) 
* MongoDB Source: add config for TLS and Replica Sets

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
 2021-04-30 22:57:34 +03:00
Christophe Duong
77ffd74b32 Ignore records that are too big in Redshift destinations (instead of failing) (#2988) 
* Abort sync if one of the part fails to copy to temp table

* Check for record size when copying data from s3 to redshift

* Handle big record in RedshiftInsertDestination too
 2021-04-30 21:04:03 +02:00
Abhi Vaidyanatha
e378d40236 Documentation Reorganization (#3124) 
* First reorganization pass.

* Stop auto-generating api docs html file.

* Update spelling

* Final cleanup.

* Final changes_REAL_actual_2_thisone

* fix path for generating api docs

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: jrhizor <me@jaredrhizor.com>
 2021-04-29 15:18:52 -07:00
Marcos Marx
e7007268f1 add some connectors to connector health page (#3099) 
* add oracle to connector health page

* add appstore and appsflyer
 2021-04-28 22:03:04 -03:00
Davin Chia
e63ab8490a Make Redshift part size configurable. (#3053) 
The Redshift Copy strategy currently has it's part size set to 10 MB. Since S3 allows a file to be broken into max of 10k parts, this results in a 100GB table limit. A user is trying to sync a table of 115GB and running into this issue.

This makes the part size configurable so users can increase this size if needed.
 2021-04-27 13:35:35 +08:00
vitaliizazmic
68d87c283d Smartsheets source setup CI 
* Smartsheets source #2902 - setup CI

* Smartsheets source #2902 - configured_catalog.json

* Smartsheets source #2902 - update documentation
 2021-04-22 17:56:15 +03:00
John Lafleur
3401d1e491 Update quickbooks.md 2021-04-22 11:38:47 +11:00