1
0
mirror of synced 2025-12-21 11:01:41 -05:00
Commit Graph

851 Commits

Author SHA1 Message Date
Jared Rhizor
d3c7270d13 rename mixpanel creds (#1286) 2020-12-10 11:40:03 -08:00
Eugene K
1db1e9267d Source mixpanel (#1195)
* mixpanel integration

* add gcc to build signer tap

* fix source

* update catalogs

* fix registration name

* use base classes for source and tests

+ fix issues with creds and incremental in configured_catalog.json

* restore json config file

* set default start_date to 1 year ago

* format

* fix logging of errors in BaseSingerSource
2020-12-10 11:16:08 -08:00
Charles
c3d27bd212 hotfix: run normalization AFTER source and destination are closed (release v0.7.1-alpha) (#1263) 2020-12-09 10:59:05 -08:00
Sherif A. Nada
3ab8c26a6f Add intercom source (#1255) 2020-12-08 14:42:58 -08:00
Jared Rhizor
1bd19d1bae put all integration test tasks under integrationTest (#1231)
* always re-run standardSourceTestPython

* rename and regroup to integrationTest

* add comment
2020-12-07 10:10:26 -08:00
Sherif A. Nada
d291e5d1fa Add braintree source (#1221)
Co-authored-by: ykurochkin <zhenia.kurochkin@gmail.com>
2020-12-06 20:11:51 -08:00
Charles
b5977bae17 Add Zendesk Source (#1227) 2020-12-06 19:39:15 -08:00
Sherif A. Nada
93ad0e9551 Add slack source (#1222)
Co-authored-by: ykurochkin <zhenia.kurochkin@gmail.com>
2020-12-06 13:17:18 -08:00
Jared Rhizor
62d060cb7d add build status dashboard (#1201)
* add build status dashboard

* exclude tools from styling

* fix sorting

* respond to comments
2020-12-04 20:55:36 -08:00
Sherif A. Nada
9633230a07 merge greenhouse PR (#1215) 2020-12-04 19:29:37 -08:00
Sherif A. Nada
9e7884eeb5 DRY CI script (#1207) 2020-12-04 15:35:27 -08:00
Sherif A. Nada
7e773a4f3d Fix twilio & sendgrid CI (#1206) 2020-12-04 14:49:41 -08:00
Jared Rhizor
2ac26e3285 prevent launching duplicate integration tests from master (#1205) 2020-12-04 13:00:22 -08:00
Jared Rhizor
62bf4b1934 fix freshdesk, recurly, and twilio integraton tests (#1204) 2020-12-04 12:21:23 -08:00
jrhizor
7d83799271 hotfix: add logging and failure handling for build stat reporting 2020-12-04 09:16:21 -08:00
jrhizor
cdafce149f hotfix: use basic auth for stats 2020-12-04 08:55:27 -08:00
jrhizor
45197b498a hotfix: use ref instead of branch 2020-12-04 08:40:32 -08:00
Jared Rhizor
41c7e53062 report state to kvdb for master builds (#1197)
* write to kv bucket

* add secrets to github action
2020-12-04 08:27:58 -08:00
Jared Rhizor
784a28547a show workflow url in comment requesting integration test (#1186)
* move integration test launching after caching

* run in separate job so it's not blocking

* move if to job level

* edit comment

* fix yaml error

* switch to using workflow path

* dump github context

* add comment id propagation

* add comment updates when things finish

* only have reactions for slash command dispatch, not the actual worker

* fix syntax

* fix syntax better

* yaml is terrible

* remove todo comments

* add error message and use not success

* final tweaks

* oops

* remove context dump
2020-12-03 16:31:19 -08:00
jrhizor
f88a1c4fac hotfix: use inputs for connector workflow dispatch 2020-12-03 15:11:43 -08:00
Jared Rhizor
b10d46cc05 add ability to launch all integration/standard tests from slash commands (#1169)
* add shopify singer

* temporarily run exchangerate on push

* depend on base singer docker output, not just base

* add static arg ref

* add integration test runner

* fmt

* add echos to show what is happening in script

* remove integration test and standard test building as part of main workflow

* remove empty integration test launcher script and re-enable checking for image existing

* format

* automatically run everything on master

* test running all on branch

* use list

* add required regex flag

* move inputs up to root of action

* move back to workflow dispath

* handle default value in non-dispatch workflows

* fix typo

* add exit 1 on failure to find connector

* add launcher

* make launcher executable

* add workflow launcher script
2020-12-03 15:05:50 -08:00
Christophe Duong
daac59b43d Add incremental support to BQ (#995) (#1108)
* Add incremental support to BQ (#995)
2020-12-03 20:03:25 +01:00
Charles
d6cf060093 turn off segment for acceptance test script (#1177) 2020-12-02 20:11:28 -08:00
Charles
4b8efa1c14 use correct path for twilio in ci (#1163) 2020-12-02 08:20:15 -08:00
Sherif A. Nada
d5c53102f6 Fix typo in CI script (#1159) 2020-12-02 00:53:14 -08:00
Sherif A. Nada
a19d914b91 Remove Braintree (#1158) 2020-12-02 00:41:43 -08:00
jrhizor
92c76501c8 hotfix: correct braintree ci directory creation 2020-12-01 23:42:00 -08:00
Sherif A. Nada
75f411d80d braintree (#1156) 2020-12-01 23:24:05 -08:00
Sherif A. Nada
241f1238d1 add CI creds (#1148) 2020-12-01 23:16:13 -08:00
Jared Rhizor
674ee2cf72 enforce order of installing python dependencies (#1149)
* fix python install order

* fix base install reqs dependencies

* don't clobber dependencies

* remove cacheIf forcing

* it wasn't clobbering actually

* fix pip version

* loosen pyyaml version

* Revert "loosen pyyaml version"

This reverts commit 5a4eb3bc08.

* move tap-adwords dependency

* downgrade shopify api version used to match

* run seed init

* add test curl

* Revert "add test curl"

This reverts commit caee9ee9ee.

* revert to simpler docker times

* rename task function so it doesn't match the task name

* turn off buildkit

* fix redshift dir for credentials

* add twilio and freshdesk to seed yaml

* fix typo

* get shopify passing similar to the adwords fix
2020-12-01 17:40:58 -08:00
Christophe Duong
bbaee23a83 Impl of source-redshift (#1009) 2020-11-30 18:01:44 +01:00
Sherif A. Nada
130b0fa311 add recurly creds to ci (#1085) 2020-11-24 22:59:38 -08:00
jrhizor
3115f4fb06 hotfix: rerun tasks for integration and accceptance tests 2020-11-20 12:07:51 -08:00
Jared Rhizor
05404bdd58 improve caching (#1005)
* python caching

* improve gradle caching

* store pip cache dir

* debugging info

* split build steps and temporarily remove testing infra from build

* fix formatting error

* fix another syntax error

* split repo caches

* switch pip dir

* use git hash instead of rehashing everything

* syntax

* list directories

* list after cache population

* list .docker

* specify custom output cache directory for docker

* only build bigquery and use buildx on ci

* add build command

* use docker driver instead of docker-container

* use correct key

* use local registry

* allow push failure

* make sure to tag first

* move part of node caching and restore rest of actual build

* cleanup

* clean up mssql gradle file

* toggle on inline cache

* try save and load method instead

* use separate tar for each image

* limit to 20

* always succeed even though disk space runs out

* try to just pull the public image to see if that's faster

* use cachefrom

* revert back to local registry version

* see how buildkit compares

* address review comments

* add comments
2020-11-19 13:14:46 -08:00
Charles
d1fd379938 add handling for different test types in manage.sh (#1013) 2020-11-18 13:54:37 -08:00
Charles
deea76b3fb Update Mailchimp Gradle and CI configs (#952) 2020-11-12 11:31:05 -08:00
Jared Rhizor
97e30c4ee7 check for image versions automatically in build (#948)
* add script

* remove useless cat warning

* remove unnecessary endline

* support checking core images

* add to github actions
2020-11-12 11:01:28 -08:00
Charles
017739248c bump google analytics version (#931) 2020-11-11 22:07:42 -08:00
Jared Rhizor
dde61db796 add shopify integration (#861)
* shopify integration

* fix for spotless

* gcc

* get syncs working

* don't include pii in replication for testing

* add documetation about private app api keys

* switch to api password

* fix credential checking

* remove unused gitignore

* finish configuring standard test

* remove orders from integration test catalog

* Update airbyte-integrations/connectors/source-shopify-singer/setup.py

Co-authored-by: Charles <giardina.charles@gmail.com>

* add secret for CI

* allow py files to start with def to fix generated test case

* fmt

* add gcc to test image

* limit catalog for faster tests

* fmt

Co-authored-by: Charles <giardina.charles@gmail.com>
2020-11-10 15:49:52 -08:00
Christophe Duong
70f0446677 Adapt Source File connector to JSON files to test normalization (#872)
Fix spec.json for Source File and prepare JSON loader for nested schema testing
2020-11-11 00:33:39 +01:00
Jared Rhizor
e1d7add50a mostly incremental builds (#817)
* support cached builds for base -> base-java -> snowflake

* use plugin for image building

* fix matching on COPY from

* remove docker.gradle

* tmp commit

* update connectors

* finish rest of build files

* fix ide errors

* more build fixes

* clean up

* clean up for new soruces

* fix spotless

* fix flake problems

* add recommended empty file

* python caching

* fixes upon review

* clean up docker and build test files

* clean up python

* clean up

* fix integration test dependencies

* fix standard tests

* fix

* remove symlink

* re-add requirements to fix normalizatioon build

* fix symlink

* fix dumbest build problem of all

* add missing integration test def

* fix missing dep

* remove class exclusion

* move trim so null source versions are allowed

* rename map

* fix hardcoded value

* remove unnecessary dep

* use dashes for salesforce package name

* fix typo

* DRY and fix test image name

* Fix edit

* assert string is not empty

* build integration test image only for integrationTest

* move code generator to tools and rename docker build tasks

* make source test depend on integration test build, not the other way

* remove guard because the docker build should exist before the integrationtest is applied

* remove comment

* DRY up airbyte-source-test

* fix plugin compilation

* add missing dependency

* rename getTaggedImage to getDevTaggedImage

* fix test vs main docker build bug
2020-11-09 11:00:11 -08:00
Michel Tricot
d9a8c69406 Fix build failure on master due to docs check (#849) 2020-11-06 20:07:30 -08:00
Michel Tricot
65d535f1af Detect broken links and fix remaining broken links (#797) 2020-11-06 16:57:57 -08:00
Sherif A. Nada
7b88a050dd Add Marketo source (#826) 2020-11-05 10:43:36 -08:00
Charles
b4a0d03ad1 Google Adwords Source (singer) (#818) 2020-11-04 16:56:56 -08:00
Sherif A. Nada
f7b9bda81f Add Singer Facebook Source (#808) 2020-11-04 16:38:54 -08:00
Sherif A. Nada
0cf060961f remove generate-python-models (#814) 2020-11-04 10:26:52 -08:00
Sherif A. Nada
94059e5606 remove postgres singer source (#810) 2020-11-04 09:57:12 -08:00
Sherif A. Nada
7059825804 Add postgres JDBC source (#794) 2020-11-03 10:56:36 -08:00
Jared Rhizor
5f0b0a4101 add snowflake destination (#784)
* add snowflake destination

* dry

* temporarily run in PR build

* fmt

* fix nits

* finish fixing nits

* propate timestamp

* stop running snowflake tests in prs

* Update docs/integrations/destinations/snowflake.md

Co-authored-by: Charles <giardina.charles@gmail.com>

* use non-transient tables

* separate knowledge of integration name from individual destination tests

Co-authored-by: Charles <giardina.charles@gmail.com>
2020-11-02 11:51:15 -08:00