1
0
mirror of synced 2025-12-19 18:14:56 -05:00
Commit Graph

41480 Commits

Author SHA1 Message Date
Michel Tricot
d408239029 Use docker for integrations (#87) 2020-08-20 19:31:39 -07:00
Charles
c77cc75fbf cors (#88) 2020-08-20 18:12:12 -07:00
Jared Rhizor
84ceb51bf8 improve build speed (#82)
* remove all building from start script

* add sherif's change of not caching gradle deps first

* prefer cached dependencies

* use mount for gradle cache and stop running the server-dist build twice

* only use docker compose

* use build file again

* separate tests

* add test entrypoint

* make testable by pushing image

* get build working with only docker compose

* simplify name

* create webapp_base so we can run tests on it

* run tests

* separate base image builds

* correct tag for webapp-base

* make names line up

* another one

* make tests non-interactive

* final

* use CI mode for the test run

* readable logs in CI

* update documentation

* use auto instead of rich

* add note to readme about running tests before pushing a public docker image
2020-08-20 16:36:26 -07:00
Charles
db9a984864 Enforce Input Contract on Workers (#83) 2020-08-20 15:41:50 -07:00
Sherif A. Nada
1b348f5a18 update readme to indicate new build directions (#81) 2020-08-20 10:05:26 -07:00
Charles
ce5db3112b Add Dataline types to interface for check connection and discovery workers (#79) 2020-08-20 10:04:41 -07:00
Charles
d3cdd05aa1 move initial source config to correct directory (#80) 2020-08-20 09:58:25 -07:00
Jared Rhizor
f499d6d956 discovery demo (#72)
* start sharing logic with persistence class

* get discovery worker working in a hacky way

* misc fixes

* working with nothign running

* fix

* log errors in discovery workers for now

* use singer names for the fields and the correct level of config json

* add logging config and .configuration selection in the correct place

* temporarily commit worthless db creds

* add filter_dbs to speed up discovery

* increase discovery timeout to one minute

* switch back to using actual output files

* missed files

* fix whitespace
2020-08-19 17:20:54 -07:00
Sherif A. Nada
66281d3afb remove version and tables from standard sync summary (#78) 2020-08-19 16:47:29 -07:00
Charles
adc8dc80ee Config Fetching Helpers (#73) 2020-08-19 16:36:23 -07:00
Sherif A. Nada
9a9439fa72 Add Singer CheckConnection Worker (#75) 2020-08-19 10:58:08 -07:00
Charles
cf10405c82 add tests (#71) 2020-08-19 10:42:54 -07:00
Charles
1d8e4ee2b8 Fix Enum Namespacing (#76) 2020-08-19 10:39:23 -07:00
Michel Tricot
63d6fe5f55 Reorder spotless steps (#77) 2020-08-19 10:39:07 -07:00
Sherif A. Nada
389a101b76 use waitFor instead of wait() to avoid IllegalMonitorException (#69) 2020-08-19 10:33:54 -07:00
Charles
c7424e89e0 Trigger Manual Jobs from API (#70) 2020-08-18 18:11:35 -07:00
Jared Rhizor
45ae34d9b0 basic scheduling (#37)
* wip

* run scheduler with ServerApp

* use .env for environment

* working scheduling end to end

* use new db schema / tooling

* tmp commit

* complete postgres switchover

* reduce debug logging

* fix bug preventing repeat runs of a job

* refactor

* more refactoring

* add comments to job submitter

* use same log4j config

* don't use star import

* move echoworker

* treat cancelled as completed

* rename

* use scheduled executor

* use jackson instead of gson

* commit autogenerated
2020-08-18 14:42:38 -07:00
Charles
8ebec16d91 Trigger Check Connection from API (#68) 2020-08-18 13:44:05 -07:00
Artem Astapenko
6c03c5f2b9 Web App Initial Components (#61)
* Add project structure. Add markup for 1st screen

* Get rid of yarn

* Update react-scripts version. Move types to dev-deps

* Enable more checks for ts

* Fix types errors. Add initial markup for onboarding

* Add dropdown component. Add part of onboarding form

* Remove pro-fonts. Edit frequency form

* Add markup for create source

* Edit ServiceForm components

* Add destination page

* Update index and readme files
2020-08-18 17:18:48 +07:00
Charles
90864b6f9f Seed Config Persistence (#67) 2020-08-17 17:20:04 -07:00
Charles
7390f681a6 pass in persistence paths as env variables (#60) 2020-08-17 15:03:27 -07:00
Sherif A. Nada
af601cdb0a hotfix docker compose (#65) 2020-08-17 12:44:45 -07:00
Jared Rhizor
7fc9f1ee1e re-add license file (#64) 2020-08-17 11:52:57 -07:00
Michel Tricot
35233266b3 GitBook: [master] one page modified 2020-08-17 18:29:16 +00:00
Michel Tricot
9c06d75860 GitBook: [master] 6 pages modified 2020-08-17 18:28:57 +00:00
Michel Tricot
87e9c99aef Adds License to java code (#62) 2020-08-17 11:14:01 -07:00
Michel Tricot
97a9dc2303 Remove boilerplate code for enum conversion (#59) 2020-08-17 09:41:38 -07:00
Sherif A. Nada
636c5dda3a Add Singer infra (#58) 2020-08-14 16:32:06 -07:00
Charles
9203edfff2 replace wild card java imports (#56) 2020-08-14 11:30:40 -07:00
Jared Rhizor
ef0db383f7 use postgres / jooq (without code generation) / test containers (#57)
* add database image to docker compose

* add database setup

* allow connetions from a connect string in env variables

* add a test for the docker case

* add pg init script in docker-compose

* fix env and add server dependency on db

* convert schema to postgres

* use postgres driver

* fix build

* fix edge cases (dependencies between containers, data dirs, etc)

* use docker wait

* remove debugging logging

* add grant

* working test and working docker-compose

* use beforeall

* use key instead of id for k-v metadata

* clean up env names

* fix typo
2020-08-14 11:02:10 -07:00
Charles
f10fd2537e add connectionsHandler (#53) 2020-08-14 08:16:25 -07:00
Charles
a070aaf6eb add destinations, destinationSpecifications, and destinationImplementations resources to API (#49)
* add destinationsHandler

* add destinationSpecificationsHandler

* add destinationImplementationsHandler
2020-08-14 07:32:21 -07:00
Charles
781b39bdfa Handler Unit Tests and remaining SourceImplementation API endpoints (#48) 2020-08-14 07:21:07 -07:00
Michel Tricot
687e345140 GitBook: [master] one page modified 2020-08-14 03:01:23 +00:00
Michel Tricot
6f86016d51 GitBook: [master] 4 pages modified 2020-08-14 02:51:44 +00:00
Michel Tricot
df5982e347 GitBook: [master] one page modified 2020-08-14 01:46:51 +00:00
Michel Tricot
6f73d923de Delete gitbook file 2020-08-13 18:03:37 -07:00
Michel Tricot
6a7d507511 GitBook: [master] 24 pages and 2 assets modified 2020-08-14 00:47:05 +00:00
Jared Rhizor
6a5617451d update job schema to allow for oneoff jobs (#50)
* update job schema to allow for oneoff jobs

* unrelated fix required for build
2020-08-13 13:56:49 -07:00
Michel Tricot
1ed129426d GitBook: [master] 7 pages modified 2020-08-13 18:09:34 +00:00
Michel Tricot
f2936362bd GitBook: [master] 5 pages modified 2020-08-13 18:08:37 +00:00
Sherif A. Nada
20783b6382 Add Worker Skeleton, BaseSingerWorker, and SingerDiscoveryWorker (#36) 2020-08-12 18:03:37 -07:00
Sherif A. Nada
dfebfa4415 save (#47) 2020-08-12 14:06:40 -07:00
Charles
b60233610d Implement Workspace and Source* API endpoints (#42) 2020-08-12 13:44:48 -07:00
Jared Rhizor
a5bd32931d fix version selection from env file (#46)
* demonstrate failing build

* fix version selection from env file
2020-08-12 11:24:21 -07:00
Michel Tricot
e4b5f37245 Update PR template (#45) 2020-08-12 10:42:10 -07:00
Charles
fba9e6eeae JSON configuration persistence 2 (#39)
In response to feedback on this PR #17 and after fiddling around with what implementing some endpoints looks like, I've made two updates to the configuration persistence:

* The whole class is synchronized, to try to avoid any concurrency issues in the MVP.
* Added a couple exception to the persistence interface. The philosophy I used here was that
    * we expose named exceptions that are actionable to the user. e.g. config not found, or you are trying to insert or access a config that does not pass validation.
    * we do NOT expose named exceptions related to the internal of the database (e.g. data corruption or we can't access the disk). we let these get handled as runtime exceptions.
2020-08-12 08:54:21 -07:00
Charles
93a1fdd2f0 Add json schemas to source (#41) 2020-08-12 08:50:06 -07:00
Charles
922acfc5fa API infra (#40)
* error handling

* request logging

* fix root path
2020-08-12 08:49:02 -07:00
Michel Tricot
6f35679f2b Github adjustments (#43)
* add extra step

* test

* cache

* cache

* simplify

* better bug report

* better content

* add comment

* fix PR template
2020-08-12 00:25:29 -07:00