1
0
mirror of synced 2025-12-21 11:01:41 -05:00
Commit Graph

851 Commits

Author SHA1 Message Date
Christophe Duong
daf58b2dbd Add SourceFile integration (#716)
Implement Source for files stored in GCS/S3/SFTP/SSH/SCP
2020-11-02 18:43:59 +01:00
Sherif A. Nada
fd6b0d7e48 remove gsheets standard test from PRs (#773) 2020-11-01 10:26:43 -08:00
Jared Rhizor
73446b2f48 google analytics source + regex standard testing (#715)
* save GA state

* bugfix, documentation, ci creds

* fmt

* set up standard testing

* set up config for check

* oops

* fix path for standard tests

* add to registry

* add documentation and env var handling

* working with new standard source test

* send pageview on setup

* use file for tracker code

* add regexes

* add logging and remove intentionally failing test case

* fmt

* Update airbyte-integrations/bases/standard-source-test/src/main/java/io/airbyte/integrations/base/TestSource.java

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/bases/standard-source-test/src/main/java/io/airbyte/integrations/base/TestSource.java

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* switch to start_date for connection test

* add comment and use future date

* use dashes for packages

* better way to call GA than hardcoded

* restructure for more consistent naming

* fmt

* update ordering

* fix dep from merge

* add regex tests to jdbc

* fmt

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2020-10-30 13:48:30 -07:00
Charles
edc49b89d9 Hubspot Source (#732) 2020-10-29 22:05:49 -07:00
Sherif A. Nada
f408bce2dc Add gsheets source (#695) 2020-10-29 08:50:12 -07:00
Charles
a0003dac82 Write python to implement standard source tests (#719) 2020-10-27 20:16:48 -07:00
Charles
d4f663f2d5 only run integration tests for a subset of integrations during PR builds (#710) 2020-10-26 17:16:41 -07:00
Charles
ef629aabfe salesforce source (#698) 2020-10-26 10:39:04 -07:00
Jared Rhizor
b34d559df3 Python Developer Experience (#684)
* add symlinks for intellij includes

* add python support for exchangerate

* add python support for other modules

* get working across all projects

* add pydantic deps

* revert testing code

* mypy

* use python 3.7 in github actions

* remove unused envs plugin

* increase line length

* increase line length to 140 (run formatting)

* don't ignore most init.py

* does this happen to work on github actions?

* try to fix generation

* try to have GA chown

* exclude re-generation
2020-10-23 12:43:18 -07:00
Sherif A. Nada
e20cf1a6f4 publish build scans in CI (#691) 2020-10-22 22:47:26 -07:00
Charles
4d1a788177 standard source tests (#686) 2020-10-22 20:06:33 -07:00
Charles
4316b34ec5 Source Github Singer (#680)
* note: still needs to be added to integration index

* note: still needs docs
2020-10-22 14:59:46 -07:00
Charles
8f7b559550 Integrations Reorganization: Connectors (#676) 2020-10-21 13:05:25 -07:00
Charles
eef954224a remove singer style integrations (#669) 2020-10-20 23:04:36 -07:00
Sherif A. Nada
41ce2b96dc fix image push script (#633) 2020-10-19 16:23:45 -07:00
Michel Tricot
b955a5ac74 gitignore secrets at top level (#606) 2020-10-17 22:04:46 -07:00
Jared Rhizor
8d165a7ea2 working python source (#533)
* use log messages for check and add todos

* working state

* handle stderr in discoer and use airbyte catalog as output

* misc restructure

* inject logger again

* use config container (still at the base level)

* add tests and fix logging for check command

* move michel's branch into this one

* fmt

* fix invalid credential test

* restructure catalog helpers

* terrible hack to get stripe to not fail completely

* fix discovery test now that a single message contains the catalog

* all working tests except sync

* partially working mapping (in wrong location)

* working but untested reads

* split logic out

* fmt

* fix deadlock bug

* convert catalog in test to airbyte catalog format

* fix stripe tests

* fmt

* remoce unused log command

remove config

* output credentials

* clean up optional state command

* use integration launcher

* DRY up read eval

* logging fixes and remove break

* rename

* fmt

* fix inverted if statement
2020-10-14 12:14:15 -07:00
Michel Tricot
5e2ccaa49b DRY connector gradle files (#562) 2020-10-14 10:59:47 -07:00
Charles
ff3a2dba09 bigquery destination (java) (#551) 2020-10-13 16:11:50 -07:00
Charles
93bcfa7ea0 hotfix: docker-compose 1.27.3 volume issue (#452)
* https://github.com/airbytehq/airbyte/issues/451
2020-09-24 15:11:45 -07:00
Jared Rhizor
8a10251b29 add gradle to version release script (#429) 2020-09-23 15:33:00 -07:00
Jared Rhizor
32ceb77b1c add script for releasing a version of airbyte (#409)
* add alpha flag to docker version

* add deploy script

* actually push tag

* rename to release_version.sh

* never .. only ...
2020-09-22 22:51:44 -07:00
Michel Tricot
e3737a2c06 Introducing Airbyte (#348) 2020-09-18 20:41:09 -07:00
Jared Rhizor
0659d29e69 add and test stripe source (#255)
* add docker build and empty build framework for tests

* add source and source connection specs

* register stripe tap in integrations enum

* add tmp

* add testing

* fmt

* force check connection

* update tests

* update gradle and remove unused seed script

* update readme

* fmt

* load credentials for build

* fix build based off of merge conflicts

* use valid mountable path for mac

* fmt

* cleanup

* use a single script for writing credentials

* re-add host

* chmod +x tools/bin/ci_credentials.sh

* fmt

* change file name
2020-09-17 11:23:58 -07:00
Sherif A. Nada
45a706434a display logs after running acceptance tests (#314) 2020-09-16 14:43:42 -07:00
Charles
2022cc39d1 set acceptance tests to not use a daemon (#313) 2020-09-16 14:39:20 -07:00
Charles
17764d8ffb explicitly specify nodejs version (#287)
* explicitly set the version of nodejs that gradle uses, instead of just using whatever it already finds on the local machine

* remove dataline/build-base (unused)
2020-09-16 10:24:09 -07:00
Jared Rhizor
c87cfd3991 move labels to bottom of dockerfiles for better caching (#233) 2020-09-11 13:46:33 -07:00
Michel Tricot
5c5185f1b4 First process to push integrations (#215) 2020-09-10 15:36:43 -07:00
Michel Tricot
e43af2dcf2 Always push the latest master images to dockerhub (#207) 2020-09-10 09:47:32 -07:00
Sherif A. Nada
b33980536b Add acceptance tests to run through most critical features (#165) 2020-09-09 17:50:54 -07:00
Michel Tricot
1cc0859a29 Add tests for config models & persistence (#159) 2020-09-02 09:43:35 -07:00
Sherif A. Nada
7424ac3bbb Add initial acceptance test structure and first test (#150) 2020-09-01 14:44:47 -07:00
Michel Tricot
73567dd73f Enables PMD (#142) 2020-08-31 22:19:25 -07:00
Michel Tricot
098a121c24 Changes project codestyle (#139) 2020-08-31 13:49:40 -07:00
Michel Tricot
f344ecbbb1 Prepare clean mount management (#126) 2020-08-28 14:04:51 -07:00
Sherif A. Nada
6f3e670c05 Fix sync worker deps & tests (#120)
* save

* working sync tests

* nullcheck

* disable test until docker is fixed
2020-08-27 12:59:57 -07:00
Michel Tricot
d31b0b5407 Improve build speed + Prepare fixing docker tests (#112) 2020-08-26 10:02:49 -07:00
Sherif A. Nada
0feb7672d9 Add sync worker (#84) 2020-08-25 14:58:58 -07:00
Sherif A. Nada
dc8138d753 correctly run gradle tests in build script (#96) 2020-08-24 10:20:31 -07:00
Jared Rhizor
511c28ed25 split scheduler into its own container (#92)
* split out scheduler

* move entrypoints into the dockerfiles

* update test docker image name
2020-08-21 11:26:55 -07:00
Michel Tricot
d408239029 Use docker for integrations (#87) 2020-08-20 19:31:39 -07:00
Jared Rhizor
84ceb51bf8 improve build speed (#82)
* remove all building from start script

* add sherif's change of not caching gradle deps first

* prefer cached dependencies

* use mount for gradle cache and stop running the server-dist build twice

* only use docker compose

* use build file again

* separate tests

* add test entrypoint

* make testable by pushing image

* get build working with only docker compose

* simplify name

* create webapp_base so we can run tests on it

* run tests

* separate base image builds

* correct tag for webapp-base

* make names line up

* another one

* make tests non-interactive

* final

* use CI mode for the test run

* readable logs in CI

* update documentation

* use auto instead of rich

* add note to readme about running tests before pushing a public docker image
2020-08-20 16:36:26 -07:00
Sherif A. Nada
636c5dda3a Add Singer infra (#58) 2020-08-14 16:32:06 -07:00
Jared Rhizor
cb3814fbb2 initial db setup (#18)
* add sqlite support

* remove no-op test

* remove empty scheduler class checked in

* add prod env var to start script

* switch to using data dir in docker

* add stopgap data dir

* fix merge err

* move database operations out of commons into separate module

* fix package paths

* get tests working

* revert gradle properties

* rename to server uuid

* describe ServerUuid

* fix mount folder creation
2020-08-11 09:04:04 -07:00
Michel Tricot
b91aa2c86c Conduit is dead, long live dataline (#33) 2020-08-10 19:44:10 -07:00
Michel Tricot
5f3c82d6f5 Prepare project for Webapp + Support multiple images with Docker compose (#27) 2020-08-10 17:53:19 -07:00
Charles
be11cdfae6 remove oss tooling (#11) 2020-08-06 14:42:12 -07:00
Michel Tricot
0b4595292d Configure logging (#10) 2020-08-05 19:48:06 -07:00
Michel Tricot
7491c94704 Generate API code + bootstrap webserver + simplify launch (#8) 2020-08-05 12:44:52 -07:00