1
0
mirror of synced 2025-12-21 11:01:41 -05:00
Commit Graph

71 Commits

Author SHA1 Message Date
LiRen Tu
81417e6728 Add connector metadata as sentry tags (#10475)
* Pass worker metadata to connector

* Fix compilation

* Pass in job id and image from worker

* Remove application version

* Add default job environment variables

* Add back removed comment

* Rename env map to job metadata

* Fix env configs

* Read connector from application

* Use empty string

* Remove println

* Fix unit test

* Fix compilation error

* Introduce constants for worker env

* Add worker env to ENV_VARS_TO_TRANSFER

* Pass into getWorkerMetadata map to all constructions

* Format code

* Format octavia cli

* Fix test compilation

* Fix typos
2022-03-09 07:36:03 -08:00
VitaliiMaltsev
e30d8348b2 Change JsonSchemaPrimitive to a class (#9913)
* fix for jdk 17

* add JsonSchemaType class

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

* fix Oracle tests

* fix Redshift tests

* fix Redshift tests

* fix checkstyle

* fix MSSQL tests

* fix cockroachdb tests

* fix checkstyle

* fix checkstyle

* replace star imports

* replace star imports

* replace star imports

* update JsonSchemaType | fixed checkstyle

* Remove unused variables in test

* Fix imports

* Expand imports

* Fix more imports

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
2022-02-14 02:12:37 -08:00
LiRen Tu
39049cbf24 🎉 End-to-end test source: support stream duplication (#10298)
* Add support to duplicate one stream multiple times

* Update cloud version

* Bump version

* Remove unused logger

* Add trace for each stream in mock source

* Fix cloud version

* Bump version in seed
2022-02-12 00:50:48 -08:00
LiRen Tu
8518767f81 🐛 Mock source: fix schema validation and json generation (#9954)
* Remove legacy modes

* Use patched json generator library

* Update doc

* Fix unit test

* Resolve merge conflict

* Add unit tests for the generator

* Bump version

* Go back to master branch

* Fix cloud version

* Format code

* Bump version in seed
2022-02-01 03:31:51 -08:00
LiRen Tu
a094142825 Format code (#9892) 2022-01-29 19:56:00 -08:00
Alexander Tsukanov
479f0d7c8d [MVP] Integrate sentry to all java-based connectors (#9745)
* airbyte-9328: Added Sentry integration to BigQuery and BigQuery denormalized connector.

* airbyte-5050: Added strategy for INSERT ROW.

* airbyte-9328: Added Sentry integration to Snowflake.

* airbyte-9328: Fix Sentry config.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Fix PR comments.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Small changes.

* airbyte-9328: Small changes.

* airbyte-9328: Move SENTRY DSN keys to Dockerfiles.

* Use new dsn

* Revert format

* Remove sentry dsn from compose temporarily

* Log sentry event id

* Move sentry to java base

* Remove sentry code from bigquery

* Update dockerfiles

* Fix build

* Update release tag format

* Bump version

* Add env to dockerfiles

* Fix e2e test connector dockerfil

* Fix snowflake bigquery dockerfile

* Mark new versions as unpublished

Co-authored-by: LiRen Tu <tuliren@gmail.com>
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
2022-01-29 16:58:35 -08:00
LiRen Tu
e4661fb92a Remove regex check from Java source acceptance test (#9829)
* Move getRegexTests to python source acceptance test

* Remove unused imports

* Update test template
2022-01-26 17:51:37 -08:00
LiRen Tu
d0b226bb1b Address review comments for mock source (#9773)
* Address review comments

* Add comments

* Format code

* Update user documentation
2022-01-25 13:39:28 -08:00
LiRen Tu
b269b9fb1e 🎉 Source e2e test: support custom catalog (#9720)
* Add continuous feed mode to source e2e-test

* Update connector catalog doc

* Fix sonar qube issues

* Add cloud variant

* Format code

* Add testing source connector to seed
2022-01-23 21:09:03 -08:00
Serhii Chvaliuk
844dd93122 Use multi-stage builds in dockerfiles to reduce java images (#9077)
* use multi-stage to reduce image size

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2021-12-23 22:27:05 +02:00
Oleksandr Sheheda
f0a9945f80 Revert "Change copy to add in dockerfiles to reduce container size (#8516)" (#8997)
This reverts commit 8ac2c6f4
2021-12-21 16:01:17 +02:00
Haoran Yu
8ac2c6f4a7 Change copy to add in dockerfiles to reduce container size (#8516)
Co-authored-by: Oleksandr Sheheda <alexandr-shegeda@users.noreply.github.com>
2021-12-16 23:52:59 -03:00
Lake Mossman
464c485b94 Add acceptance tests for source close timeout (#8217)
* add test connectors and bump versions

* add failure timeout acceptance test

* run gw format

* include exception in runtime exception

* mark as disabled and add comment
2021-12-16 15:21:26 -08:00
Lake Mossman
51866fd37a Stop using gentle close with heartbeat (#8036)
* use gentle close instead of gentle close with heartbeat when closing source

* also lower destination process gentle close duration

* add test connectors
2021-11-17 15:06:14 -08:00
Harsha Teja Kanna
3e7f95c25a 🎉 Support build on MacOS M1 (Apple Silicon) (#7104)
- See this doc for details: https://github.com/airbytehq/airbyte/blob/master/docs/contributing-to-airbyte/developing-locally.md
- Unit test does not work yet.
2021-10-19 11:20:21 -07:00
Charles
ba44f700b9 add final for params, local variables, and fields (#7084) 2021-10-15 16:41:04 -07:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
LiRen Tu
b9e1997d2f Split airbyte-db and move db dev commands to gradle (#5616)
# Summary

- A follow-up PR for #5543.
- This PR separates the `airbyte-db` project to two modules:
  - `lib` is the original `airbyte-db`.
  - `jooq` is for jOOQ code generation.
- This is necessary because the jOOQ generator requires a custom database implementation that can run Flyway migration. So the code generator logic needs to depend on the compilation of the original `airbyte-db` project.

# Commits
* Separate db to lib and jooq modules
* Update dependencies
* Add jobs db migrator test
* Fix compose build
* Add migration dev center
* Add schema dump task
* Update airbyte-db/lib/README.md
  * Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update readme
* Remove bom dependency
* Update readme
* Use jooq code in db config persistence
* Remove AirbyteConfigsTable

Co-authored-by: Davin Chia <davinchia@gmail.com>
2021-08-26 10:44:09 -07:00
Charles
c966fd7b33 fix acceptance tests deps (#4939) 2021-07-23 10:41:12 -07:00
Andrii Leonets
213fae17a1 MySQL source: Add comprehensive data type test (#3810) 2021-06-07 14:01:02 +03:00
Charles
aa6afb7282 Checkpointing: Worker use destination (instead of source) for state (#3290)
* Migrate BufferedStreamConsumer users (e.g. all JDBC destinations, MeiliSearch) (#3473)

* Add checkpointing test cases in Acceptance Tests (#3473)

* Add testing for emitting state in Destination Standard Test (#3546)

* Migrate BQ to support checkpointing (#3546)

* Migrate copy destinations support checkpointing (#3547)

* Checkpointing: Migrate CSV and JSON destinations (#3551)
2021-05-25 16:47:40 -07:00