1
0
mirror of synced 2025-12-21 11:01:41 -05:00
Commit Graph

851 Commits

Author SHA1 Message Date
Jared Rhizor
32ee10e76d snowflake gcs staging (#3183)
* spec

* snowflake gcs

* spec comment

* fmt

* remove sout

* add docs

* format

* respond to comments

* fix

* bump to 0.3.3
2021-05-04 09:59:32 -07:00
Yevhenii
6bc63264e7 🎉 New Source: Zendesk Chat (#3088)
* Zendesk Chat connector: implement full_refresh sync

* format

* Source Zendesk Chat: Implement Incremental sync (#3157)

* Zendesk Chat: Implement Incremental sync

* Roll back the stream Chats to the previous endpoint

* update integration-base-python to v0.1.6

* update limit to 100

* add acceptence tests

* update test and setup configs

* remove unknown file

* update naming of classes

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>

* register Zendesk Chat as source

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-05-03 16:02:00 -04:00
Sherif A. Nada
97fefde8dc fix stripe cursor comparison bug and add dockerfile and build.gradle (#3096) 2021-04-27 21:21:03 -07:00
Christophe Duong
c2fa3e4c9c Introduce normalization integration tests (#3025)
* Speed normalization unit tests by dropping hubspot catalog (too heavy, will be covering it in integration tests instead

* Add integration tests for normalization

* Add dedup test case

* adjust build.gradle

* add readme for normalization

* Share PATH env variable with subprocess calls

* Handle git non-versionned tests vs versionned ones

* Format code

* Add tests check to normalization integration tests

* Add docs

* complete docs on normalization integration tests

* format code

* Normalization integration tests output (#3026)

* Version generated/output files from normalization integration tests

* simplify cast of float columns to string when used as partition key (#3027)

* bump version of normalization image

* Apply suggestions from code review

Co-authored-by: Jared Rhizor <jared@dataline.io>

* Apply suggestions from code review

Co-authored-by: Jared Rhizor <jared@dataline.io>
2021-04-27 12:01:04 +02:00
nclsbayona
636a21e15e Add check_requirements.sh #3028 (#3054) 2021-04-26 13:38:33 -07:00
Jared Rhizor
a3b4444372 snowflake s3 copy & redshift s3 refactor (#2921)
* snowflake s3 copy

* refactor (some tests still need updating)

* revert accidentally removing files

* re-add purge

* use baseconnector

* getconnection logs error

* use generic configs for copiers/suppliers/consumers

* use stream copier terminology

* remove weird delegate generics

* some test changes

* remove non-ci test that doesn't have a good equivalent atm

* misc

* finally fixed

* tests and fix

* add credentials

* fix redshift build

* respond to comments

* fix check

* bump versions for redshift and snowflake

* fix creds
2021-04-26 09:41:53 -07:00
vitaliizazmic
68d87c283d Smartsheets source setup CI
* Smartsheets source #2902 - setup CI

* Smartsheets source #2902 - configured_catalog.json

* Smartsheets source #2902 - update documentation
2021-04-22 17:56:15 +03:00
Davin Chia
e6d412eae3 Release 0.21.0-alpha. (#3001)
* Remove no longer needed test setup.

* Bump version: 0.20.0-alpha → 0.21.0-alpha
2021-04-21 12:15:41 +08:00
Yevhenii
f8286e4da3 🎉 New Source: Iterable (#2895)
* Source Iterable: support full_refresh/incremental sync (#2412)

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-04-20 16:21:17 +03:00
Davin Chia
b468c39481 Explicitly build the images to get the tests to pass for now. (#2961)
* Explicitly build the images to get the tests to pass for now.

* Fix NPE and add better logging.
2021-04-20 10:58:43 +08:00
Marcos Marx
ca8f304f90 Add SSL option for Postgres source/destination (#2757)
* add ssl for source-postgres

* add config in utf8 test

* correct comments from @jrhizor and @sherifnada

* correct config get

* add ssl test postgres

* add sh generate ssl files

* change pg ssl test

* use custom image

* correct spec.json

* correc tests

* remove unecessary config

* add config and correct spec.json

* add ssl to postgres destination

* add tools to generate custom dockers images and correct spec.json

* change how additional parameter is append

* add logic ssl for postgres destination

* remove if for append add params

* gradlew format
2021-04-16 15:37:55 -07:00
vitaliizazmic
aec897867f 🎉 New Source: Quickbooks
* Google audit logs #2694 - incremental sync

* Google audit logs #2694 - configured catalog

* Google audit logs #2694 - new source

* Quickbooks #2309 - adding singer tap version

* Quickbooks #2309 - update documentation

* Quickbooks #2309 - remove unnecessary files

* Quickbooks #2309 - reformat

* Quickbooks #2309 - install gcc in docker container

* Quickbooks #2309 - adding source to IMAGES_TO_SKIP_SECOND_INCREMENTAL_READ

* Quickbooks #2309 - update sample_state.json

* Update docs/integrations/sources/quickbooks.md

* Quickbooks source #2309 - adding to health page

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-04-16 19:13:21 +03:00
Yevhenii
1294e41d76 🎉 New source: Zendesk Talk (#2775)
* Zendesk Talk #2346: full refresh/incremental sync connector with adopt best practices

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-04-13 19:50:01 +03:00
Jared Rhizor
0609011e05 fix connector base versioning (#2726)
* fix connector base versioning

* only run integration tests in non-bases directories

* upgrade datamodel-code-generator which coincidentally has already fixed this issue with generating List of Lists

* bump forgotten hubspot version

* fix build
2021-04-02 22:46:06 -07:00
vitaliizazmic
2327b353aa 🎉 New Source: Google Workspace Admin audit logs
* Google audit logs #2456 - Create new Google Workspace Admin Reports source for full refresh sync audit logs

* Update docs/integrations/sources/google-workspace-admin-reports.md

* Google audit logs #2456 - reformat

* Google audit logs #2456 - increase result per page

* Google audit logs #2456 - fix schema

* Google audit logs #2456 - add schemas package data

* Google audit logs #2456 - fixing timeout error

* Google audit logs #2456 - fixing configured_catalog.json

* Google audit logs #2456 - lookback spec parameter

* Google audit logs #2456 - remove end time

* skip testIdenticalFullRefreshes for source-google-workspace-admin-reports

* Tempo Source #2295 - reformat

* Update airbyte-integrations/connectors/source-google-workspace-admin-reports/source_google_workspace_admin_reports/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-04-01 02:27:38 +03:00
Sherif Nada
77b72dcf5b Revert "Revert "Merge branch 'master' of github.com:airbytehq/airbyte""
This reverts commit de66bf010d.
2021-03-23 15:12:01 -07:00
Sherif Nada
de66bf010d Revert "Merge branch 'master' of github.com:airbytehq/airbyte"
This reverts commit a3a27f2fa2, reversing
changes made to 5917feee28.
2021-03-23 15:09:44 -07:00
Christophe Duong
4bebeed60b Harmonize /test and /publish commands on github + Fix MySQL tests in CI (#2531)
* Harmonize /test and /publish commands on github

* Make normalization build on clean system without PyICU

see https://getdbt.slack.com/archives/C50NEBJGG/p1615803416053800?thread_ts=1615511262.026400&cid=C50NEBJGG

* Follow example from https://github.com/testcontainers/testcontainers-java/pull/2689 to modify MySQL container as root

* Add docs on new syntax for /test
2021-03-23 10:47:29 +01:00
Yevhenii
e3c9853b2e 🎉 New Source: Gitlab (#2518)
* #2305 Issue: Gitlab connector implement

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
2021-03-22 15:36:04 +02:00
Eugene K
0b9b038ca4 🎉 New source: Native Hubspot connector (#2215)
* #2150 Issue: created native connector with schema folder populated

* #2150 Issue: make format code

* first version

* fix few issues

* fix issues

* fix read issue

* format

* docs

* docker tags

* extend configured catalog for testing

* fix source definitions

* format

* fix call rate issue, add backoff for retry after

* add general backoff

* write secrets for new connector

* drop singer connector registration

* refactor streams, resolve properties in schemas at runtime

* replace deprecated endpoint for company contacts

* replace deprecated pipeline endpoint

* update comments

* update docs

* fix typo

* fix stream contact lists

* fix pagination and forms result fetching

* fix health_check

* format and update catalog

* revert changes

* drop singer based hubspot

* fix company contacts substream

* move deals to separate test

* fix deals tests

* remove dynamic fields from records

* move deals to catalog again

* extend CRMObjectStream with associations

* format

* update schemas with updated field, change engagement layout

* fix Campaign stream

* remove custom tests

* remove dependency

* remove oauth

* Source Hubspot: add incremental streams (#2425)

* add incremental

* add incremental

* polishing

* update docs

* fix docstring

* clean up

* fix incremental bookmark access

* fix incremental tests

* clean up

* add custom test for incremental, improve logging

* format

* Update airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Source Hubspot: best practices (#2537)

* fix error reporting and add unit tests

* fix test and refactor cursor fields

* format

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>

* restored configured_catalog.json

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-03-20 06:18:55 -03:00
Yevhenii
2b0f8d556f #1814 Issue: Implement Instagram connector (#2226)
* #2166 Issue: create Instagram connector and implement all relevant streams as full refresh

* #2166 Issue: add Insights streams

* #2273 Issue: add Incremental for streams

* #2273 Issue: code clean up

* update code after review

* add check on error for Story Insight

* add comments to code

* Source Instagram: adopt best practices, add docs, pull data from all IG business accounts (#2373)

* #2276 Issue: adopt best practices, add separate integration test for Insight's streams, create docs, update version of SDK library

* #2304 Issue: pull data from all IG business accounts

* add BASE_DIRECTORY to integration_test.py

* format configured_catalog(s)

* add credentials variables

* implement separated incremental states for different account_id, update docs

* Update instagram.md

* simplify state format

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* add sample_config.json file

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-03-12 15:49:16 -03:00
vitaliizazmic
393e8fad25 New Source: Google Directory
* Google directory source #2110 - creating new source

* Google Directory #2110 - implementing new source

* Google directory #2110 - handling rate limit

* Google Directory #2110 - handling errors and rate limits

* Google Directory #2110 - reformat

* Google Directory #2110 - adding CI credentials

* Google Directory #2110 - adding to the source definition registry

* Google Directory #2110 - adding to the source definition registry(fix)

* Google Directory #2110 - injecting the config into the build environment

* Update google-directory.md

* Update google-directory.md

* Google directory #2110 - rename max_results to results_per_page and increase it to 100, fixing setup.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-03-12 00:59:45 +02:00
Artem Astapenko
29482430ea Jamakase/add cypress (#2199)
* Add data-id to controls

* Add tests

* Add e2e test starting

* Add skip onboarding test
2021-03-03 07:26:21 +07:00
Jared Rhizor
dfe1451aba add cli (#1922)
* add cli

* clean up

* generate

* fix dependency

* add go so we can install restish in build

* final fix to build

* rest of final fix to build
2021-02-19 14:56:13 -08:00
Sherif A. Nada
d2218b131a Source Github: Bugfix rate limiting (#2051) 2021-02-16 21:58:21 -08:00
Michel Tricot
ac092ada6e Hide connector scaffolding behind a script (#2080) 2021-02-16 20:09:03 -08:00
Jared Rhizor
5e0ec04418 delete all images (#2032) 2021-02-10 09:15:52 -08:00
Sherif A. Nada
9d24b80992 Allow publishing a connector image via github comment (#1892) 2021-02-01 14:11:21 -08:00
Eugene K
a54dd89418 Adopt connector best practices for File Source (providers) #1584 (#1738)
Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-01-27 16:49:33 -08:00
vitaliizazmic
143f880c78 Tempo source #1764 - enable standard tests (#1829) 2021-01-27 16:44:11 -08:00
Michel Tricot
d9da1172a2 Quick & dirty setup of a read only airbyte instance (#1802) 2021-01-26 18:54:12 -08:00
Jared Rhizor
dc12475da4 fix blue color (#1822) 2021-01-25 11:23:08 -08:00
Eugene K
8d765319ef remove Facebook Marketing source (singer-based) (#1794) 2021-01-22 15:12:13 -08:00
Charles
6fc99a8a52 push migrate app as part of airbyte core (#1748) 2021-01-21 14:05:53 -08:00
Jared Rhizor
e2970933a6 remove old build status dashboard (#1713) 2021-01-19 10:40:44 -08:00
Jared Rhizor
90e6505efe add s3 status reporting for integration test builds (#1679)
* add initialization for status dashboard static api

* working status reporter for integration tests

* try different formatting for date

* actually run tests

* only report build status for master

* some level of functions + datetime switching by architecture
2021-01-19 09:34:17 -08:00
Eugene K
fb7262dfa8 Add Airbyte-native facebook marketing source (#1552)
Co-authored-by: Jared Rhizor <me@jaredrhizor.com>
Co-authored-by: Sherif Nada <snadalive@gmail.com>
2021-01-14 22:20:41 -08:00
Jared Rhizor
b1a4196212 kube ci testing (#1492)
* add to ci pipeline

* wait BEFORE port forwarding

* fix kube testing

* change order

* pare tests down to just acceptance tests

* seconds

* longer sleep

* multiple waits

* add back java support

* fix folder

* show logs

* show logs no matter what

* see if it passes again

* rename

* finish cleaning up script

* revert rest of removals

* format

* clean up

* resplit

* try kind

* again

* describe pods on failure instead

* run as a separate step because we're running into disk space issues

* indentation

* wait longer for port to be usable

* start minikube build

* install socat

* don't bother trapping port forward

* try to use local machine in minikube

* fix

* fix

* try ip

* show more logs

* try 0.0.0.0

* try inet

* try external socket ip

* Revert "try external socket ip"

This reverts commit 7cf976aa10.

* try formatting mount differently

* show all paths in airbyte local

* recursive listing

* don't test for file existence on Kube

* revert changes that were made for testing

* use different html client for kube

* just skip single failing test
2021-01-07 11:29:37 -08:00
Sherif A. Nada
6a6daa4956 Test connectors only once when building their image (#1555) 2021-01-06 16:06:54 -08:00
Tommy Giardina
92fa63fb33 Plaid connector (#1476) 2020-12-31 09:22:20 -08:00
Sherif A. Nada
2f22a7e088 Y/looker (#1441)
Co-authored-by: ykurochkin <zhenia.kurochkin@gmail.com>
2020-12-24 20:04:40 -08:00
Sherif A. Nada
5b69b76024 Add microsoft teams source (#1422) 2020-12-21 22:33:19 -08:00
Jared Rhizor
6ef6f46382 use bumpversion for releasing (#1398)
* add support for bumpversion

* use separate bump versions

* temporarily disable building artifacts for testing

* Bump version: 0.8.0-alpha → 0.8.1-alpha

* remove sed call

* Bump version: 0.8.1-alpha → 0.8.2-alpha

* update message at end of script

* re-enable building artifacts

* manually revert bumping

* improve instructions

* fix

* after writing the PR desc this should just be in docs
2020-12-18 11:02:41 -08:00
Sherif A. Nada
10b90b2cb5 Add source drift (#1393)
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Eugene K <34233075+eugene-kulak@users.noreply.github.com>
2020-12-17 22:44:24 -08:00
jrhizor
a4c05f2d56 Revert "remove file creation hack"
This reverts commit f5f58e2f64.
2020-12-16 16:01:23 -08:00
jrhizor
f5f58e2f64 remove file creation hack 2020-12-16 13:54:42 -08:00
Sherif A. Nada
894c4d5f49 Allow following symlinks when building docker images (#1332) 2020-12-15 11:53:34 -08:00
Sherif A. Nada
c48129f728 Move a number of singer sources to incremental (#1290) 2020-12-15 11:28:44 -08:00
Sherif A. Nada
4369cbc636 Add Zoom Source Connector (#1300)
Co-authored-by: ykurochkin <zhenia.kurochkin@gmail.com>
2020-12-11 15:53:09 -08:00
Sherif A. Nada
0e4e4fd66e Add Jira Source Connector (#1287) 2020-12-10 14:50:45 -08:00