1
0
mirror of synced 2025-12-22 11:31:02 -05:00
Commit Graph

138 Commits

Author SHA1 Message Date
Marius Posta
696118de6a gradle: remove broken mypy task (#31468) 2023-10-16 12:05:56 -07:00
Marius Posta
ae2ca40113 gradle: search for python3.11 binary for AL2023 support (#31227) 2023-10-10 13:15:02 -07:00
Marius Posta
f8edc18039 airbyte-ci,gradle: replace airbyte-docker with airbyte-ci (#30743) 2023-10-04 08:38:17 -07:00
Marius Posta
7ae97175a6 gradle: fix repo wide behaviour (#30607) 2023-09-28 05:01:13 -07:00
Aaron ("AJ") Steers
da32fc86bf 🚀 Launch Java CDK v0.1.0 (PR #30383)
- Launch Java CDK `v0.1.0`. 🚀
- Remove `airbyte-docker` gradle plugin from java connectors.
- Add `airbyte-java-connector` gradle plugin.
- Remove `airbyte-commons` from global build.gradle.
- Add `/publish-java-cdk` slash command.
- Initial launch of `/airbyte-ci` slash command to test connectors.
2023-09-27 15:46:02 -07:00
Ben Church
5c56ac1d84 Airbyte-ci: Remove gradle task connectorAcceptanceTest (#30326) 2023-09-19 15:16:37 -05:00
Marius Posta
532ef097eb airbyte-ci: fix integration test log buffering (#30438) 2023-09-14 11:06:19 -05:00
Ben Church
59f803e323 gradle docker: skip if not dev image (#30371) 2023-09-12 16:02:44 -07:00
Marius Posta
51c67d7eaa gradle: remove airbyteDocker.outputs dependencies (#30314) 2023-09-11 17:16:27 -07:00
Marius Posta
e6e62c65d1 gradle: airbyte-docker improvements (#30262)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-09-08 10:25:17 -07:00
Marius Posta
ef2849e35e gradle: fix airbyteDocker task inputs (#30187) 2023-09-07 03:46:31 -07:00
Marius Posta
be1e1adabd gradle: cleanup (#30060) 2023-09-05 14:05:40 -05:00
Edward Gao
934acaa137 Destination bigquery: rerelease 1s1t behind gate (#27936)
* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)""

This reverts commit 348c577dbb.

* version bumps+changelog

* Speed up BQ by having 2 queries, and not an OR (#27981)

* 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (#27856)

* only run t+d code if it's enabled

* dockerfile+changelog

* remove changelog entry

* Destinations V2: handle optional fields for `object` and `array` types (#27898)

* catch null schema

* fix null properties

* clean up

* consolidate + add more tests

* try catch

* empty json test

* Automated Commit - Formatting Changes

* remove todo

* destination bigquery: misc updates to 1s1t code (#28057)

* switch to checkedconsumer

* add unit test for buildColumnId

* use flag

* restructure prefix check

* fix build

* more type-parsing fixes (#28100)

* more type-parsing fixes

* handle duplicates

* Automated Commit - Format and Process Resources Changes

* add tests for asColumns

* Automated Commit - Format and Process Resources Changes

* log warnings instead of throwing exception

* better log message

* error level

---------

Co-authored-by: edgao <edgao@users.noreply.github.com>

* Automated Commit - Formatting Changes

* Improve protocol type parsing (#28126)

* Automated Commit - Formatting Changes

* Change from T&D every 10k records to an increasing time based interval (#28130)

* fifteen minute t&d

* add typing and deduping operation valve for increased intervals of typing and deduping

* Automated Commit - Format and Process Resources Changes

* resolve bizarre merge conflict

* Automated Commit - Format and Process Resources Changes

---------

Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>

* Simplify and speed up CDC delete support [DestinationsV2] (#28029)

* Simplify and speed up CDC delete support [DestinationsV2]

* better QUOTE

* spotbugs?

* recompile dbt image for local arch and use that when building images

* things compile, but tests fail

* tests working-ish

* comment

* fix logic to re-insert deleted records for cursor comparison.

tests pass!

* remove comment

* Skip CDC re-include logic if there are no CDC columns

* stop hardcoding pk (#28092)

* wip

* remove TODOs

---------

Co-authored-by: Edward Gao <edward.gao@airbyte.io>

* update method name

* Automated Commit - Formatting Changes

* depend on pinned normalization version

* implement 1s1t DATs for destination-bigquery (#27852)

* intiial implementation

* Automated Commit - Formatting Changes

* add second sync to test

* do concurrent things

* Automated Commit - Formatting Changes

* clarify comment

* minor tweaks

* more stuff

* Automated Commit - Formatting Changes

* minor cleanup

* lots of fixes

* handle sql vs json null better
* verify extra columns
* only check deleted_at if in DEDUP mode and the column exists
* add full refresh append test case

* Automated Commit - Formatting Changes

* add tests for the remaining sync modes

* Automated Commit - Formatting Changes

* readability stuff

* Automated Commit - Formatting Changes

* add test for gcs mode

* remove static fields

* Automated Commit - Formatting Changes

* add more test cases, tweak test scaffold

* cleanup

* Automated Commit - Formatting Changes

* extract recorddiffer

* and use it in the sql generator test

* fix

* comment

* naming+comment

* one more comment

* better assert

* remove unnecessary thing

* one last thing

* Automated Commit - Formatting Changes

* enable concurrent execution on all java integration tests

* add test for default namespace

* Automated Commit - Formatting Changes

* implement a 2-stream test

* Automated Commit - Formatting Changes

* extract methods

* invert jsonNodesNotEquivalent

* Automated Commit - Formatting Changes

* fix conditional

* pull out diffSingleRecord

* Automated Commit - Formatting Changes

* handle nulls correctly

* remove raw-specific handling; break up methods

* Automated Commit - Formatting Changes

---------

Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>

* Destinations V2: move create raw tables earlier (#28255)

* move create raw tables

* better log message

* stop building normalization (#28256)

* fix ability to run tests

* disable incremental t+d for now

* Automated Commit - Formatting Changes

---------

Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Cynthia Yin <cynthia@airbyte.io>
Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>
2023-07-14 09:34:56 -05:00
Alexandre Girard
97a353d5c5 Run mypy on airbyte-cdk as part of the build pipeline and fix typing issues in the file-based module (#27790)
* Try running only on modified files

* make a change

* return something with the wrong type

* Revert "return something with the wrong type"

This reverts commit 23b828371e.

* fix typing in file-based

* format

* Mypy

* fix

* leave as Mapping

* Revert "leave as Mapping"

This reverts commit 908f063f70.

* Use Dict

* update

* move dict()

* Revert "move dict()"

This reverts commit fa347a8236.

* Revert "Revert "move dict()""

This reverts commit c9237df2e4.

* Revert "Revert "Revert "move dict()"""

This reverts commit 5ac1616414.

* use Mapping

* point to config file

* comment

* strict = False

* remove --

* Revert "comment"

This reverts commit 6000814a82.

* install types

* install types in same command as mypy runs

* non-interactive

* freeze version

* pydantic plugin

* plugins

* update

* ignore missing import

* Revert "ignore missing import"

This reverts commit 1da7930fb7.

* Install pydantic instead

* fix

* this passes locally

* strict = true

* format

* explicitly import models

* Update

* remove old mypy.ini config

* temporarily disable mypy

* format

* any

* format

* fix tests

* format

* Automated Commit - Formatting Changes

* Revert "temporarily disable mypy"

This reverts commit eb8470fa3f.

* implicit reexport

* update test

* fix mypy

* Automated Commit - Formatting Changes

* fix some errors in tests

* more type fixes

* more fixes

* more

* .

* done with tests

* fix last files

* format

* Update gradle

* change source-stripe

* only run mypy on cdk

* remove strict

* Add more rules

* update

* ignore missing imports

* cast to string

* Allow untyped decorator

* reset to master

* move to the cdk

* derp

* move explicit imports around

* Automated Commit - Formatting Changes

* Revert "move explicit imports around"

This reverts commit 56e306b72f.

* move explicit imports around

* Upgrade mypy version

* point to config file

* Update readme

* Ignore errors in the models module

* Automated Commit - Formatting Changes

* move check to gradle build

* Any

* try checking out master too

* Revert "try checking out master too"

This reverts commit 8a8f3e373c.

* fetch master

* install mypy

* try without origin

* fetch from the script

* checkout master

* ls the branches

* remotes/origin/master

* remove some cruft

* comment

* remove pydantic types

* unpin mypy

* fetch from the script

* Update connectors base too

* modify a non-cdk file to confirm it doesn't get checked by mypy

* run mypy after generateComponentManifestClassFiles

* run from the venv

* pass files as arguments

* update

* fix when running without args

* with subdir

* path

* try without /

* ./

* remove filter

* try resetting

* Revert "try resetting"

This reverts commit 3a54c424de.

* exclude autogen file

* do not use the github action

* works locally

* remove extra fetch

* run on connectors base

* try bad  typing

* Revert "try bad  typing"

This reverts commit 33b512a3e4.

* reset stripe

* Revert "reset stripe"

This reverts commit 28f23fc6dd.

* Revert "Revert "reset stripe""

This reverts commit 5bf5dee371.

* missing return type

* do not ignore the autogen file

* remove extra installs

* run from venv

* Only check files modified on current branch

* Revert "Only check files modified on current branch"

This reverts commit b4b728e654.

* use merge-base

* Revert "use merge-base"

This reverts commit 3136670cbf.

* try with updated mypy

* bump

* run other steps after mypy

* reset task ordering

* run mypy though

* looser config

* tests pass

* fix mypy issues

* type: ignore

* optional

* this is always a bool

* ignore

* fix typing issues

* remove ignore

* remove mapping

* Automated Commit - Formatting Changes

* Revert "remove ignore"

This reverts commit 9ffeeb6cb1.

* update config

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
2023-07-13 16:55:48 -07:00
Catherine Noll
fbe4ecc5be Fix: only build docker images with the local CDK for connectors (#24259) 2023-03-20 19:41:18 -04:00
Sergio Ropero
7fa8508999 fix: Cast the number of threads to integer in the Java connector tests 😵‍💫 (#23917)
* reduce threads

* Fix real problem between CHAR and INTEGER
2023-03-09 19:12:33 +00:00
Sergio Ropero
3db452dff9 Property to limit test parallelisation (#23870)
* Adds new functionality on integration test pipeline and test pipeline in order to reduce parallelization.
It's done by creating a `gradle.properties` file in the connector root folder.

Extra: removes test concatenation for performance tests.

* rename parameter

* fix names

* Reduce threads
2023-03-08 18:48:20 +01:00
Sergio Ropero
f0909f6fa2 feat: Set number of forks to execute tests in parallel. (#23506)
* Set number of forks to execute tests in parallel.

* Add back `upToDateWhen`
2023-03-01 15:27:31 +01:00
Catherine Noll
d4ec2d952e Update /test to optionally test against the local CDK (#23434) 2023-02-28 11:53:51 -05:00
Joe Reuter
053d352a1c Make connector acceptance test gradle plugin work for destinations (#22609)
Fix connector acceptance test gradle plugin
2023-02-24 18:10:44 +01:00
Augustin
727229cb5f connector testing: remove superfluous tasks (#23208) 2023-02-21 22:24:40 +01:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846)
Rename source acceptance test to connector acceptance test
2023-02-02 11:38:19 +01:00
Evan Tahler
087018799a Use pytest v6.2.5 (#19848)
* Use pytest v6.2.5

* bump test definitions
2022-11-28 17:57:06 -08:00
Lake Mossman
684fbff12f Enable the Connector Builder server in the platform (#19269)
* init

* bad copy/paste

* move to top level

* Revert "move to top level"

This reverts commit aca3534d38.

* attempt to wire up connector builder frontend to server

* copy from octaviacli

* fix connection to builder server

* update

* delete

* Update

* delete python-version

* Revert "delete python-version"

This reverts commit f9258a7755.

* setup python

* install python

* rename

* kube stuff

* Install python

* missing kube file

* rename

* Update files

* Update bumpversion

* install python

* try with different entrypoint

* rename container

* point to docker-compose.yaml file

* derp

* copy acceptance_test.sh

* copy from acceptance tests

* delete cruft

* update

* remove application env

* reset

* reset to master

* update

* skip comprehensive incremental tests

* Revert "skip comprehensive incremental tests"

This reverts commit 9cee657596.

* reset to master

* remove cruft

* readme

* readme

* save

* buncha stuff

* it works

* improvements

* comment

* codesnippet

* delete superfluous steps

* update port to 8003

* reset to master

* Update publish docker

* move openapi spec to airbyte-connector-builder

* point to openapi spec

* dont expose the connector builder to localhost

* reset FE components to master

* Don't deploy the connector-builder

* Revert "Don't deploy the connector-builder"

This reverts commit 3d157494cf.

* Revert "Revert "Don't deploy the connector-builder""

This reverts commit beac3d48f0.

* comment out more things related to connector builder server

* more attempts at removing the connector builder

* comment out more things

* Update airbyte-connector-builder/README.md

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* fixes

* rename gradle task

* save

* copyright header

* Apply suggestions from code review

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update airbyte-webapp/src/config/configProviders.ts

Co-authored-by: Lake Mossman <lake@airbyte.io>

* update

* rename

* indent

* Revert "move openapi spec to airbyte-connector-builder"

This reverts commit 57dda04723.

* Revert "rename"

This reverts commit b2d802b8fa.

* Revert "Revert "rename""

This reverts commit 91db24fd4a.

* point to wrong file in case it fixes the build

* point to right openapi file

* Revert "Revert "move openapi spec to airbyte-connector-builder""

This reverts commit e46a837454.

* point to moved file

* fix path

* Update from master

* newline

* Add failing test

* Revert "Add failing test"

This reverts commit ed9fea09b5.

* comment

* update commented requires

* Add a comment

* comments

* save

* formatting is for losers anyways

* pyproject

* toml

* toml

* saved

* nice

* anothaone

* fmt

* fixmypy

* fix autogen notice header location

* improvecomment

* save working branch

* fmt

* delete license and disable caching

* removeyaml

* fix merge and uncomment flakeCheck

* undo css change

* fix typo

* format

* enable CORS

* move template to backend server

* try adding connector builder server to 'generate-docker' task to see if it fixes CI

* try adding buildDockerImage task

* try adding gradle.properties

* Revert "try adding buildDockerImage task"

This reverts commit 2102009ddb.

* Revert "try adding connector builder server to 'generate-docker' task to see if it fixes CI"

This reverts commit 40b5ae0fcd.

* process detail in apiOverride

* fix gradle

* fix scrollbars

* Run connector builder server tests and static analysis from a docker container (#19495)

* Revert "try adding buildDockerImage task"

This reverts commit 2102009ddb.

* Revert "try adding connector builder server to 'generate-docker' task to see if it fixes CI"

This reverts commit 40b5ae0fcd.

* missing file

* build airbyte-connector-builder-server image

* Add .

* comment

* build connector builder server as part of octavia_cli

* build connector builder too

* patch

* missing file

* comment

* comment

* run unit tests in a container

* Run integration tests

* run

* move to platform build

* format

* rootdir

* get the path from the project

* Update

* pyproject

* delete a bunch of code

* parameterize

* run tests

* update

* rename

* comment

* reset

* uncomment

* dont format

* run everything except isort

* delete dead code

* Revert "delete dead code"

This reverts commit b11df716a0.

* comment

* Update module

* reset to master

* mount in home

* commit formatted files

* try running format

* Better comment

* reset

* Revert "reset"

This reverts commit 20808f8262.

* Update buildSrc/src/main/groovy/airbyte-python-docker.gradle

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* comment

* Add everything except build and hidden directories

* symlink

* Update airbyte-connector-builder-server/run_format.sh

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-connector-builder-server/run_tests.sh

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: sherifnada <snadalive@gmail.com>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
2022-11-17 19:44:45 -08:00
Davin Chia
64736f0ab3 More Readable Gradle test config. (#18890)
The integration testing logs are extremely noisy today. The logs contain the entire test STDOUT and are a pain to read.

This PR contains a better test setup config to reduce logging output.

The main change is to comment out ShowStandardStreams in the integration tests plugin.
Then we add the started event so Gradle shows us when a test is started. Some tests take a longer time. It is useful to know they have begun so it's clear if the job is stuck or not.
Standardise the above setting across all of our custom test tasks.
2022-11-03 14:39:35 -07:00
Volodymyr Pochtar
a72cb7d1f8 feat: replace openjdk with amazoncorretto:17.0.4 on platform for security compliance (#16919) 2022-09-20 19:01:48 +03:00
Serhii Chvaliuk
8b3c6861d1 pin flake8==4.0.1 (#15155)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-08-01 15:12:49 +03:00
Evan Tahler
e14c71473e Java integration tests also depend on spotbugsMain (#14755) 2022-07-15 15:21:30 -07:00
Brian Lai
b73713e610 run source acceptance tests against image built from branch (#13401)
* run source acceptance tests against image built from branch + auto fail tests for validation

* remove auto fail tests used for validation
2022-06-02 12:22:18 -04:00
Sherif A. Nada
cc46255190 cull base-python and base-python-test (#12596) 2022-05-05 13:48:15 -07:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Sherif A. Nada
b85c51304f 🎉 Update hardway tutorial & generic source template to use SAT (#11908) 2022-04-11 20:12:06 -07:00
Jared Rhizor
526c5a0c64 automatically figure out ARM-related variables (#11450)
* automatically figure out ARM-related variables

* clean up

* explicitly use amd64-prefixed versions of alpine and postgres
2022-03-31 13:58:46 -07:00
Serhii Chvaliuk
84b2d320e6 upgrade black 22.1.0 -> 22.3.0 (#11494)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-03-29 14:13:41 +03:00
Charles
c1c8675366 Add readmes to all modules (#8893) 2022-03-13 14:45:36 -07:00
Maksym Pavlenok
61b0e9e196 🎉 Single py checker (#10246) 2022-02-21 14:01:42 +02:00
Maksym Pavlenok
b5b0976355 Fix deps for python build order (#10057) 2022-02-04 12:56:14 +02:00
Maksym Pavlenok
9aade158fc skip errors of coverage reports (#10028) 2022-02-03 11:38:48 +02:00
Maksym Pavlenok
91eff1dffd 🐛 Source S3: Loading of files' metadata (#8252) 2022-02-02 00:49:18 +02:00
Edward Gao
dad52edcc4 make airbyteDocker build cache functional (#9362) 2022-01-11 11:33:54 -08:00
Edward Gao
5dad18b377 fix destination-snowflake build for jdk17 (#8854) 2021-12-16 12:57:31 -08:00
Eugene
70d2f46bcc Source MySQL\MsSql\Postgres: added RDS base performance tests (#8215)
* Added RDS base performance tests for source-postgres, source-mssql and source-mysql
* updated perfomance test with cpu and memory limit


Co-authored-by: andriikorotkov <andrii.h.korotkov@globallogic.com>
2021-12-12 19:29:11 +02:00
Charles
e9f414d0a7 do not run excluded tags in integration tests (#8715) 2021-12-10 16:20:57 -08:00
Eugene
3a0eaafb5e 🎉 Source Mysql - added basic MySql performance tests (#7820)
* [ticket 7489, PR 7820] Source Mysql - added skeleton for MySql performance tests
2021-11-15 12:16:36 +02:00
Eugene Kulak
319492791e CDK: fix logging configuration inside source and streams (#7808) 2021-11-11 16:07:38 +02:00
Sherif A. Nada
efb5151011 🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering (#7719) 2021-11-08 19:26:32 -08:00
Jared Rhizor
1b5922896b upgrade to Gradle 7.2 (#7070)
* upgrade to gradle 7.2

* get build working with gradle 7.2

* fix build

* fix connector build?

* EXCLUDE distTar and switch includes to excludes

* switch back to includes

* temporarily try to adjust properties

* don't know why this was modified

* revert spotless upgrade

* upgrade gradle node (better arm support it says)

* add back in Xss since we are getting StackOverflowError without it

* revert bad spotless fmt

* fix acceptance tests duplicates strategy

* add zip for jdbc includes

* check if this helps for java connectors

* remove accidental commit

* remove redundant

* fix warning explosion

* oops

* override gradle.properties

* remove set-env
2021-10-15 14:03:30 -07:00
Dmytro
d767003375 Trello OAuth backend. (#6944) 2021-10-11 17:07:16 +03:00
Dmytro
7a6da86baa add coverage report (#6045)
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-09-15 10:50:07 +03:00
Sherif A. Nada
45ee6234f2 Only run pytest if test files are found in the test directories (#5012) 2021-07-27 08:33:51 -07:00