1
0
mirror of synced 2025-12-21 11:01:41 -05:00
Commit Graph

64 Commits

Author SHA1 Message Date
Wenqi Hu
8a6f866c1c [Snowflake source] Change CDK version to 0.1.31 instead of latest (#66311) 2025-09-16 14:10:25 -07:00
Wenqi Hu
5c1a2f2f69 Fix sampling bug for DefaultJdbcCursorIncrementalPartition (#66200) 2025-09-15 16:59:59 -07:00
Wenqi Hu
3fb8dda3e5 [Snowflake] Fix schema filtering in connector config (#66226) 2025-09-12 16:40:14 -07:00
Ian Alton
c8cf650229 Added a single source of truth for keypair authentication in Snowflake (#63726)
Co-authored-by: Conall Ó Gribín <167999304+Conall-airbyte@users.noreply.github.com>
2025-08-20 08:06:49 -07:00
Wenqi Hu
422a9a527d Fix ts data type for snowflake (#63780)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2025-07-28 17:55:50 +00:00
devin-ai-integration[bot]
02150fba71 fix(source-snowflake): revert base image from 2.0.3 to 2.0.2 to fix certificate errors (#63713) 2025-07-22 03:36:10 +00:00
Wenqi Hu
bbc46f3a74 [Snowflake] Update base image to 2.0.3 (#62939) 2025-07-15 22:51:13 +00:00
Matt Bayley
a4d4746020 Move source test framework to cdk (#62929) 2025-07-11 16:51:52 -07:00
Wenqi Hu
9bb0f19b96 [Snowflake] Create certified source connector (#61535) 2025-07-09 12:23:49 -07:00
devin-ai-integration[bot]
d125cca558 docs(source-snowflake): Add comprehensive incremental sync documentation (#62514)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ian.alton@airbyte.io <ian.alton@airbyte.io>
2025-07-03 18:37:09 -07:00
Augustin
39d6364c29 source-snowflake: Use airbyte/java-connector-base:2.0.0 (#51504) 2025-01-13 17:36:53 +02:00
Augustin
84fa6cc38e source-snowflake: Use airbyte/java-connector-base:1.0.0 (#49911)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2025-01-06 18:04:06 +01:00
Xiaohan Song
4370bf3c01 [source-snowflake] Upgrade snowflake jdbc driver (#48073) 2024-10-31 14:56:39 -07:00
aharbunou-branch
3f278ea4cd Support Snowflake key pair auth for source connector (#40424) 2024-07-09 13:40:36 -03:00
Audrey Maldonado
b9690ba76e Wrapping changelog in details for a collapsible view (#38801) 2024-06-04 13:40:53 -04:00
Evan Tahler
010da0e6ff source-snowflake: hide oauth option (#38317) 2024-05-17 13:00:12 -05:00
Natik Gadzhi
cb80e6922a [tools] prettier rules for .md + formatting cleanup 2024-05-07 08:19:33 -07:00
Marius Posta
b59098fe15 source-snowflake: adopt CDK 0.20.4 (#35220) 2024-02-14 11:54:57 -08:00
Chandler Prall
9a67b7fa10 🐛 bump cdk versions for java sources (#34453) 2024-02-01 14:26:38 -07:00
Subodh Kant Chaturvedi
b2eda46c7e sources(bigquery, redshift, snowflake): migrate to PER-STREAM state (#33484) 2023-12-19 01:04:44 +05:30
Joe Reuter
e2789b4f03 Update docusaurus to 3 (#33041) 2023-12-11 17:03:18 +01:00
Stephane Geneix
df9da285bf 🐛 📝 Source Snowflake: fix coercion-to-UTC for timestamp datasource (#31631)
Co-authored-by: stephane-airbyte <stephane-airbyte@users.noreply.github.com>
2023-10-24 17:59:56 +00:00
Xiaohan Song
4aff41087c Source Snowflake: Upgrade snowflake JDBC version to 3.14.1 (#31252) 2023-10-11 14:50:51 -07:00
Evan Tahler
9ce3cc6e79 Java DB Source connector licenses to Elv2 (#27737)
* Java DB Source connector licenses to Elv2

* Bump PR Ids
2023-06-28 18:52:02 -07:00
Akash Kulkarni
2f807440da Fix issue in streaming JDBC database (#27212)
* Fix issue in streaming JDBC database

* Bump + publish

* 🤖 Auto format source-db2 code [skip ci]

* 🤖 Auto format source-mssql code [skip ci]

* 🤖 Auto format source-postgres-strict-encrypt code [skip ci]

* 🤖 Auto format source-postgres code [skip ci]

* 🤖 Auto format source-mssql-strict-encrypt code [skip ci]

* 🤖 Auto format source-oracle code [skip ci]

* 🤖 Auto format source-tidb code [skip ci]

* 🤖 Auto format source-snowflake code [skip ci]

* 🤖 Auto format source-redshift code [skip ci]

* 🤖 Auto format source-mysql code [skip ci]

* 🤖 Auto format source-mysql-strict-encrypt code [skip ci]

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
2023-06-20 12:08:47 -05:00
Akash Kulkarni
d30906d89d Remove noisy logging for LegacyStateManager (#27335)
* Remove noisy logging for legacy state manager

Context : https://airbytehq-team.slack.com/archives/C043JHEEYKG/p1686654588724759

* Bump version + dockerfile
2023-06-16 17:13:33 -05:00
Subodh Kant Chaturvedi
ac5b109e09 fix-db-sources: introduce method to set cursor value for TIMESTAMP_WITH_TIMEZONE and TIME_WITH_TIMEZONE (#24693)
* fix-db-sources: introduce method to set cursor value for TIMESTAMP_WITH_TIMEZONE and TIME_WITH_TIMEZONE

* override method for snowflake

* update method and add changelog

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-30 09:06:55 -05:00
Subodh Kant Chaturvedi
526a45c875 fix db-sources: allow TIMESTAMP_WITH_TIMEZONE and TIME_WITH_TIMEZONE as cursor value (#24667)
* fix db-sources: allow TIMESTAMP_WITH_TIMEZONE and TIME_WITH_TIMEZONE as cursor value

* update docker version

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-29 23:52:54 +05:30
VitaliiMaltsev
12eac421de JDBC Sources: remove redundant date-time datatypes formatting (#20760)
* JDBC sources: consolidate date/time values mapping

* fixed Snowflake acceptance tests

* fixed TestJdbcUtils

* refactoring

* update setDate for incremental sync

* fixed CockroachDbJdbcSourceAcceptanceTest

* fixed CockroachDbSourceDatatypeTest

* fixed CockroachDbSourceDatatypeTest

* fixed MsSQLSourceDatatypeTest

* fixed MsSQL cdc datatype test

* refactoring

* fixed mssql cdc test

* fixed Redshift tests

* fixed Redshift tests

* updated db2 datatype tests

* fixed DB2 tests

* fixed DB2 strict-encrypt tests

* fixed Oracle strict-encrypt tests

* refactoring

* updated changelog

* refactoring

* removed redundant overrides

* bump versions

* auto-bump connector version

* JDBC Sources: remove redundant date parsing

* updated Snowflake test

* updated cockroachdb test

* updated date-time datatypes parsing in Postgres Source

* updated formatting in JDBC sources

* updated test cases

* updated Snowflake test

* updated Snowflake, Redshift, Cockroach, DB2 tests

* updated Snowflake, Postgres tests

* updated Redshift tests

* updated MySQL, MsSQL, DB2 tests

* updated MsSQL tests

* updated Oracle tests

* updated Oracle tests

* updated Oracle tests

* fixed some tests

* refactoring

* fixed checkstyle

* fixed TestJdbcUtils

* randomize Reshift test username

* removed redundant test

* updated sanity test

* Automated Change

* format time and bump versions

* auto-bump connector version

* bump versions

* auto-bump connector version

* auto-bump connector version

* manually update definitions

* Automated Change

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: ievgeniit <etsybaev@gmail.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-23 14:31:50 +00:00
Prateek Mukhedkar
33ca7f3451 Adding allowHosts to all non-GA source database connectors (#23455)
Adding allowedHosts to clickhouse, cockroachdb, db2, oracle, snowflake, tidb source Connectors. 
---------

Co-authored-by: prateekmukhedkar <prateekmukhedkar@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-07 19:52:15 -08:00
Rodi Reich Zilberman
49f97f1142 Integration Branch for DB/DW Sources team for Feb '23 code freeze (#23185)
* source-snowflake: use a safer method for parsing a BigInteger cursor value (#22358)

* use a safer method for parsing a BigInteger cursor value

* Add testing

* fix format change

* Fix failing integration tests

* Try removing the failing incremental test

* Try removing the failing incremental test

* Fix failing test

* Add metadata to connector logs (log level, class name, method name and line number) (#23105)

* Issue #17861 Add labels, class, method name and line numbers to connector logs

* Refactored unit test

* fix for warning about UTF8 charset in test class

---------

Co-authored-by: prateekmukhedkar <prateek@airbyte.io>

* Update docker image and release notes

* auto-bump connector version

* manually bump version on spec

---------

Co-authored-by: Prateek Mukhedkar <123108018+prateekmukhedkar@users.noreply.github.com>
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
Co-authored-by: Sergio Ropero <sergio@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-02-22 11:59:23 -08:00
VitaliiMaltsev
f5f4de2e5f JDBC sources: consolidate date/time values mapping (#20346)
* JDBC sources: consolidate date/time values mapping

* fixed Snowflake acceptance tests

* fixed TestJdbcUtils

* refactoring

* update setDate for incremental sync

* fixed CockroachDbJdbcSourceAcceptanceTest

* fixed CockroachDbSourceDatatypeTest

* fixed CockroachDbSourceDatatypeTest

* fixed MsSQLSourceDatatypeTest

* fixed MsSQL cdc datatype test

* refactoring

* fixed mssql cdc test

* fixed Redshift tests

* fixed Redshift tests

* updated db2 datatype tests

* fixed DB2 tests

* fixed DB2 strict-encrypt tests

* fixed Oracle strict-encrypt tests

* refactoring

* updated changelog

* refactoring

* removed redundant overrides

* bump versions

* auto-bump connector version

* resolved merge conflicts

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* bump postgres version

* auto-bump connector version

* updated seed definitions

* revert generated definitions for Postgres

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-01-18 19:21:11 +02:00
Rodi Reich Zilberman
94c84f783a Make schema field in source-snowflake mean a subset of the specified o… (#20465)
* Make schema field in source-postgres mean a subset of the specified of schema when during discover(). update UI

* Add missing file

* Fix failing acceptance test

* sanity

* update doc

* typo

* version bump and release note

* Fix failing test

* fix format
2023-01-12 09:30:43 -08:00
Rodi Reich Zilberman
fae6eee242 Make sure decimal numbers are serialized to json as a plain string ra… (#20407)
* Make sure decimal numbers are serialized to json as a plain string rather than scientific notation, as it breaks deserialization of integers

* bump version and release note

* Correct PR number on release note

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-12-14 15:36:32 -08:00
Akash Kulkarni
cbb2e986fb Set app_id in snowflake source connector based on oss/cloud (#19314)
* Set app_id in snowflake source connector based on oss/cloud

Set app_id in snowflake source connector based on

* Bump version + docs

* auto-bump connector version

* Formatting

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-11-10 17:24:10 -08:00
Akash Kulkarni
d2d5419306 Bump up snowflake source version (#19319)
* Bump up snowflake source version

Changes were made in this PR to fix a bug when new data is inserted at the same time the sync starts. However, they were never published

https://github.com/airbytehq/airbyte/pull/15535

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-11-10 15:59:12 -08:00
Liren Tu
888347a0d0 🎉 JDBC sources: store cursor record count in db state (#15535)
* Add cursor_record_count to db stream state

* Add cursor record count to cursor info

* Emit max cursor record count

* Add original cursor record count

* Unify logging format

* Add backward compatible methods

* Update unit tests for state decorating iterator

* Update test (not done yet)

* Fix one more unit test

* Change where clause operator according to record count

* Add branch for null cursor

* Skip saving record count when it is 0

* Fix log wording

* Set mock record count in test

* Check cursor value instead of cursor info

* Fix source jdbc test

* Read record count from state

* Fix tests

* Add an acceptance test case

* Fix npe

* Change record count from int to long to avoid type conversion

* Fix references

* Fix oracle container

* Use uppercase for snowflake

* Use uppercase for db2

* Fix and use uppercase

* Update test case to include the edge case

* Format code

* Remove extra assertion in clickhouse

* Merge ms sql incremental query method

* Log query for debugging

* Clean up name_and_timestamp table

* Fix db2 tests

* Fix mssql tests

* Fix oracle tests

* Fix oracle tests

* Fix cockroachdb tests

* Fix snowflake tests

* Add changelog

* Fix mssql tests

* Fix db2-strict-encrypt tests

* Fix oracle-strict-encrypt tests

* Bump postgres version

* Fix oracle-strict-encrypt tests

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-14 01:09:12 -07:00
VitaliiMaltsev
b8dfed0405 Source Snowflake: Handle date-time data types (#17144)
* Source Snowflake: Handle date-time data types

* updated changelog

* replaced star import

* bump version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-27 19:12:26 +03:00
Eugene
4e381abbd2 🐛Source-snowflake: added connection string identifier (#17116)
* [16767] Source-snowflake: added connection string identifier
2022-09-26 21:19:56 +03:00
Akash Kulkarni
44ef1c4ad6 🎉 Source Snowflake : Update JDBC driver for Snowflake to 3.13.22 (#16766)
Update JDBC driver for Snowflake

Co-authored-by: Liren Tu <tuliren.git@outlook.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-21 16:07:34 -07:00
Yurii Bidiuk
10e4aae891 Use LOG4J2 to wrap connectors logs to JSON format (#15668)
* Use LOG4J2 to wrap connectors logs to JSON format

* log connector's stack trace directly as a message

* add stack_trace field to json template

* bump versions

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* revert versions for destinations: postgres, mssql, mysql

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-14 23:27:13 +03:00
Liren Tu
29cef6635f 🎉 Redshift & snowflake sources: emit state messages more frequently (#16258)
* Emit more state messages in redshift source

* Emit more state messages in snowflake source

* auto-bump connector version [ci skip]

* Use random schema name in integration test

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-01 22:57:27 -07:00
oneshcheret
e3bc9bb4fe Snowflake source: add scope for role for oauth (#15797)
* Snowflake source: add ability to support different roles during oauth

* Snowflake source: add ability to support different roles during oauth

* Snowflake source: handle case with empty role fields

* Snowflake source: added host to spec seeds

* Snowflake source: bump version

* Snowflake source: update change log

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-22 14:38:42 +03:00
mkhokh-33
5776dcc949 Mkhokh/8858 cursor fields for incremental update (#14356)
* add logic to skip incremental mode for tables which do not have a field types that could not be used as cursor

* added tests for cursor fields incremental updates

* remove not used methods

* fix tests

* fix formatting

* revert RedshiftInsertDestinationAcceptanceTest

* fix ssl clickhouse test

* fix codereview comments

* fix clickhouse strict encrypt test

* fix postgres source after merge from master

* optimize ssl clickhouse test logic

* fix db2 test

* added check for cursor field should be not nullable, fix formatting

* fixed tests, formatting, issue after merge from master

* fix merge issues

* fix test for source-jdbc and remove some code used for local testing

* added description for cockroach db tests

* test changes

* refactoring for cursor field tests after code review comments

* refactoring for cursor field tests after code review comments

* fix code review comments

* remove unused imports

* fix formatting

* fix compilation error

* fix postgres and postgress ssl test

* Fixed bucket naming for S3

* remove CHAR, NCHAR from possible cursor types

* remove SYMBOL from possible cursor types

* removed redundant configs

* fixed mysql-strict-encrypt tests

* fixed mongodb-v2 tests

* minor format changes

* bump version

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: vmaltsev <ihbectop@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-18 19:48:22 +03:00
Edward Gao
eab00130f1 🐛 Source snowflake: int columns should be discovered as ints (#15314)
* snowflake discovers ints as ints

* version bump+changelog

* bump version+changelog

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-09 16:13:09 -07:00
Alexander Tsukanov
91853635d3 🎉Source Snowflake: Source/Destination doesn't respect DATE data type (#14828)
airbyte-5577: Respect date/datetime types for snowflake source.
2022-07-25 11:44:09 +03:00
Edward Gao
24efcedade 📝 improve error message when using unsupported JDBC type as cursor (#14714)
* improve error message

* also in JdbcSourceOperations

* bump versions + changelog

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-22 18:29:03 -07:00
VitaliiMaltsev
df445713aa Remove additionalProperties from JDBC source connectors (#14574)
* Remove additional properties from JDBC specs

* fixed SnowflakeSourceAuthAcceptanceTest

* fixed OracleStrictEncryptJdbcSourceAcceptanceTest

* fixed Oracle Strict encrypt expected spec

* updated Changelog

* fixed CockroachDbSourceTest

* fixed CockroachDb Tests

* fixed CockroachDb Tests for Linux OS

* bump versions

* bump clickhouse-source-strict-encrypt

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* fixed dependencies for Clickhouse Source

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* fixed OracleSourceDatatypeTest

* auto-bump connector version [ci skip]

* rollback ClickHouse Source changelog

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-21 11:01:34 +03:00
Topher Lubaway
9c6c092a22 Revert "Improving docusaurus sidebar generation (#1927) (#14369)" (#14596)
This reverts commit a2c194a11f.
2022-07-11 15:27:14 -05:00
Mykyta Serbynevskiy
a2c194a11f Improving docusaurus sidebar generation (#1927) (#14369)
* Improving docusaurus sidebar generation (#1927)

* Added "Career & open positions" folder to sidebar, adjusted "Project overview" folder

* Deleted "career-and-open-positions" folder from sidebar
2022-07-08 14:18:27 -05:00