jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-06 06:04:16 -05:00
Code Issues Packages Projects Releases Wiki Activity
8,635 Commits 5,163 Branches 518 Tags
b4dfbdb67006d028c8fdbe0dfd2305273fa6df0d
Commit Graph

721 Commits

Author SHA1 Message Date
Denys Davydov
d97e300286 SAT: do not treat spec fields with const values as the ones that can hold secrets (#19465) 
* SAT: do not treat spec fields with const values as the ones that can hold secrets

* sat: upd changelog
 2022-11-16 14:14:52 +02:00
Ella Rohm-Ensing
f59709ea66 Parse list of dicts in json_schema_helper.find_nodes() (#19386) 
* Get test on nested list/dict passing - use index to query next object for list

* Fix flakecheck

* Test that get_node provides correct value

* Improve test and test cases

* Rewrite method for better comprehension

* Add test for base-level key. Rewrite method for comprehension and handling this case
 2022-11-15 17:40:04 -05:00
Denys Davydov
459856b73c #2576 oncall. SAT: test spec against exposed secrets (#19124) 
* #2576 oncall. SAT: test spec against exposed secrets

* #2576 upd changelog

* #2576 review fixes

* #2576 more test fixes
 2022-11-10 23:31:04 +02:00
Sherif A. Nada
125f35f8df Generate spec-first FastAPI Server (#18886) 2022-11-09 18:14:38 -08:00
Ella Rohm-Ensing
e73f535261 Add SAT test for no duplicates in enum properties (#19243) 
* Make pytest aware of backward compatibility mark

* Unit tests for duplicate values in enum properties

* Add docs about airbyte enforcements to enum usage

* Add docs info to test
 2022-11-09 15:47:05 -05:00
Augustin
c67bfc50bb SAT: script to create PR to migrate GA connector to high test strictness level. (#19138) 2022-11-08 23:11:21 +01:00
Augustin
a196cf712c SAT: make future_state configuration mandatory in high test strictness level (#19085) 2022-11-08 23:10:22 +01:00
Benoit Moriceau
1cdf1ba9e8 Bmoric/extract workspace api (#18996) 
* Extract Operation API

* Extract scheduler API

* Format

* extract source api

* Extract source definition api

* Add path

* Extract State API

* extract webbackend api

* extract webbackend api

* extract workspace api
 2022-11-08 10:15:57 -08:00
Akash Kulkarni
4fe8468e02 Migrate ssh exception -> config error exception (#19094) 
* Update SshTunnel.java

Migrate ssh exception -> config error exception

* Unnest exception

* Address PR comments

* Add test case
 2022-11-08 07:12:29 -08:00
Augustin
d815fb929b SAT: test strictness level migration tooling (create issues script) (#19020) 2022-11-07 17:04:50 +01:00
Augustin
7d73b08606 SAT: basic read on full catalog when test_strictness_level == high (#18937) 2022-11-07 08:09:34 +01:00
Eugene
9c39c0e045 Fixed tests for destination connectors (#19007) 
* Fixed tests for destination connectors
 2022-11-05 17:55:06 +02:00
Akash Kulkarni
972a8c0419 Handling configuration exceptions in IntegrationRunner (#18989) 
* Hanlde error msgs in integration runner

* Add ConfigErrorException

* Some formatting
 2022-11-04 14:38:26 -07:00
Subodh Kant Chaturvedi
3468516a2b cleanup: delete debezium 1-4-2 module (#18733) 
* mssql-source:upgrade debezium version to 1.9.6

* delete debezium 1-4-2

* more improvements
 2022-11-05 00:24:26 +05:30
Subodh Kant Chaturvedi
fe3f459a8f mssql-source:upgrade debezium version to 1.9.6 (#18732) 
* mssql-source:upgrade debezium version to 1.9.6

* more improvements

* upgrade version

* auto-bump connector version

* fix test

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-04 23:02:51 +05:30
Cole Snodgrass
9231e3ea35 removed unused code (#18788) 
* removed unused code

* reformat with proper line length
 2022-11-03 10:32:08 -07:00
Subodh Kant Chaturvedi
0a37a8d352 mysql-source:fix tinyint unsigned handling (#18619) 
* mysql-source:fix tinyint unsigned handling

* update doc

* format

* upgrade version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-11-02 20:54:05 +05:30
Greg Solovyev
6af98045f8 Parameterize test_empty_streams and test_stream_with_1_airbyte_column by destination (#18197) 
* Remove lines that always add Postgres to list of destinations
* Parameterize all tests in test_ephemeral by destination
 2022-11-01 12:55:32 -07:00
Greg Solovyev
8cf546483d 🐛 Add a drop table hook to drop scd tables in case of overwrite sync (#18015) 
* Add a drop table hook to drop scd tables in case of overwrite sync

* Add an integration test for dropping SCD table on overwrite

* skip new test for Oracle and TiDB

* Add normalization run after initial reset

* Bump normalization version
 2022-11-01 08:52:02 -07:00
oneshcheret
2d4f443c75 Sources postgres, mysql, alloydb: encode database name (#18538) 
* Source postgres: encode database name

* Source postgres, mysql: move encoding in util class, apply for mysql

* Source postgres, mysql: make var final

* Source postgres, mysql: bump version

* Source postgres, mysql: format code

* auto-bump connector version

* Source mysql: bump version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-31 19:53:20 +02:00
Augustin
c25ac8a748 SAT: make expect_records mandatory in high test_strictness_level (#18497) 2022-10-28 11:45:29 +02:00
Augustin
e2fe272086 normalization: Fix failing mypy checks (#18534) 2022-10-27 20:38:47 +02:00
Liren Tu
61c67bada9 Databricks destination: introduce personal staging location (#18557) 
* Update s3 parquet writer

- Rename gcsFileLocation.
- Merge local variables.

* Implement staging location getter

* Add integration test and fix post request

* Add comment and format code
 2022-10-27 09:42:59 -07:00
Edward Gao
f90ffb0a12 Destination Redshift: handle empty s3 bucket_path (#18434) 
* make bucket_path required

* check for nonempty path

* not required

* handle null/empty path

* version bump + changelog

* fix comment

* auto-bump connector version

* format

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-26 11:04:10 -07:00
Edward Gao
ba21d9048c fix formatting + imports (#18457) 2022-10-25 15:41:19 -07:00
N-o-Z
9b57d972af 🐛 Fix S3 destination integration tests to use bucket path (#18031) 
* Fix S3 destination tests

* Add Changelog

* CR Fixes

* add unit test

* version bump

* fix dockerfile

* auto-bump connector version

Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-25 15:21:32 -07:00
Akash Kulkarni
b0a8e5473f Follow-up fix for SSHExceptions (#18383) 
* Follow-up fix for SSHExceptions

* Catch RuntimeException instead of augmenting connection error messaging for now

* Update tests

* Bump versions + docs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-25 15:06:03 -07:00
Augustin
726d3bde4d SAT: enforce bypass_reason declaration on empty_streams when test_strictness_level == high (#18425) 2022-10-25 22:31:16 +02:00
Augustin
0865c62ee6 SAT: make all tests mandatory when test_strictness_level == high (#18414) 2022-10-25 21:45:12 +02:00
Augustin
e933de02e8 SAT: declare bypass_reason in acceptance-test-config.yml (#18364) 2022-10-25 08:52:35 +02:00
Augustin
a7fd147018 SAT: new test_strictness_level field in config (#18218) 2022-10-24 12:25:23 +02:00
Augustin
710c432f00 fix Connector Base build (#18351) 2022-10-24 11:19:21 +02:00
Ryan Fu
cc41385f50 Updates getPrivateKeyPair to throw exception when private key pair does not exist (#18263) 
* Changes method from returning null to throwing Exception to match method structure

* Added javadoc explaining that KeyPair should not be returning null

* Moved ConnectionErrorException to commons package

* Adds error handling for SshWrappedDestinations and markdown to include changes

* Bumps version numbers for Postgres, MySQL, MSSQL

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-21 16:04:39 -07:00
Jared Rhizor
e3ff75f3d6 🎉 New Source: Whisky Hunter API [low-code CDK] (#17918) 
* initial commit for source-whisky-hunter

* better logs

* finish up connector

* fix catalog

* fix bug in connector_runner.py that doesn't allow falsey configs

* add bootstrap.md

* add docs

* fix typo

* fix source-accept test

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-21 09:25:40 -03:00
Brian Lai
3521802545 Update Python Source connectors to use the latest airbyte-cdk minor version (#18261) 
* update all connectors currently only adhering to patch versions to pull latest minor version

* also add greenhouse

* fix scaffolds
 2022-10-20 16:06:02 -04:00
Akash Kulkarni
55d030f3d7 🎉 Enhanced error messaging for DB source connectors (#18087) 
* Enhanced error messaging for invalid cursor state

* Handle SSH config fail

* Handle SSH config fail

* Update AbstractDbSource.java

* Bump up versions + doc for beta+ connectors

Bump up versions + doc for beta+ connectors

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-19 12:22:42 -07:00
Serhii Chvaliuk
c9f79e6b42 SAT: Bump airbyte-cdk~=0.2.0 (#18174) 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-10-19 21:05:51 +03:00
Augustin
b564f3eb78 Protocol: make supported_sync_modes a required not empty list on AirbyteStream (#15591) 2022-10-19 15:22:25 +02:00
Benoit Moriceau
adf2c5ff8e Bmoric/remove dep connector worker (#17977) 
* test [ci skip]

* Autogenerated files

* Add missing annotation

* Remove unused json2Schema block from worker

* Move tess

* Missing deps and format

* Fix test build

* TMP

* Add missing dependencies

* PR comments

* Tmp

* [ci skip] Tmp

* Fix acceptance test and add the seed dependency

* Fix build

* For diff

* tmp

* Build pass

* make the worker to be  on the platform only

* fix setting.yaml

* remove dependency

* Fix pmd

* Fix Cron

* Add chart

* Fix cron

* Fix server build.gradle

* Fix jar conflict

* PR comments

* Add cron micronaut environemnt

* Update required build path

* remove all dependencies on worker

* Typo

* Add gradle exclusions

* Add exclusion to avoid conflict

* more exclusion

* Move more class

* Exclude worker from the general build

* Fix connector build
 2022-10-18 11:51:25 -07:00
Denys Davydov
7a3815a7cd SATs: allow new records in a sequential read for full refresh test (#17660) 
* SATs: allow new records in a sequential read for full refresh test

* SATs: upd changelog

* SATs: change the output when failing full refresh test

* SATs: upd according to code review
 2022-10-18 11:34:50 +03:00
Yevhen Sukhomud
ee2e2b5b1f 16250 Destination Redis: Add SSH support (#17951) 
* 16250 Destination Redis: Add SSH support

* 16250 Resolve port issue

* 11679 Bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-18 02:00:51 +07:00
Ryan Fu
ef3e84ce3a Checks for iterator hasNext element (#18041) 
* Checks for iterator hasNext element

* Fix linter with newline
 2022-10-17 10:08:07 -07:00
Greg Solovyev
a73ae2b09d Add SSH tunneling to Elastic Search Destination (#17805) 
* Add support for using URL in tunneled config
* Add support for SSH tunneling to destination-elasticsearch and destination-elasticsearch-strict-encrypt
 2022-10-14 16:12:16 -07:00
Edward Gao
94038a1864 fix formatting (#17913) 2022-10-12 15:05:56 -07:00
Edward Gao
bd0c2388df Destination Clickhouse: Publish normalization after removing native port (#17896) 
* bump version for publish

* add link for publish pr
 2022-10-12 13:51:55 -07:00
Denys Davydov
eb6a2429ad Source acceptance test: bugfix (#17757) 
* #17506 SATs: bugfix

* #17506 SATs: upd changelog

* SATs: bump version
 2022-10-12 22:53:38 +03:00
Augustin
0644a844aa SAT: backward compatiblity hypothesis testing: Disable filter too much health check (#17871) 2022-10-12 20:21:29 +02:00
Augustin
0cbd4deeb4 SAT: unit tests test_state_with_abnormally_large_values + add missing future_state_path (#17791) 2022-10-12 19:32:07 +02:00
Benoit Moriceau
f650005f2a Remove the dependency between the orchestrator container and the worker app (#17570) 
* test [ci skip]

* Autogenerated files

* Add missing annotation

* Remove unused json2Schema block from worker

* Move tess

* Missing deps and format

* Fix test build

* Add missing dependencies

* PR comments

* Fix acceptance test and add the seed dependency

* Fix build

* Fix build again
 2022-10-11 17:12:00 -07:00
Volodymyr Pochtar
a142141c98 Revert "Revert "feat: replace openjdk with amazoncorretto:17.0.4 on connectors for seсurity compliance (#17511)" (#17820)" (#17833) 
This reverts commit e47c94ecff.
 2022-10-11 14:22:54 +03:00