* Clean fieldPath before use in projection
* Update release notes
* Automated Commit - Format and Process Resources Changes
* Remove prefix $ from field name
---------
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
* Record statistics about source database/collection
* Formatting
* Remove unnecessary log message
* Only record statistics during sync
* Emit structured Google log data
* Revert use of System.out.println
* Try using Log4j ObjectMessage
* Revert "Try using Log4j ObjectMessage"
This reverts commit d518c87aa8.
* Prepare release
* Update changelog
* Correct release version
* Update dependencies
* Use RawBsonDocument
* Add tests
* Formatting
* Incremental refresh test
* Formatting
* Fix NPE
* Test replica set/oplog via testcontainers
* Formatting
* PR feedback
* Automated Commit - Formatting Changes
* Add logging to debug integration test failures
* Fix failing test
* Remove credentials conversion logic
* Update documentation to include correct config format
* Use database from credentials
* Make expected catalog match test data
* Fix formatting
* Cleanup
* Improve logging
* Revert test catalog change
* Use test database
---------
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
* Fix issue in streaming JDBC database
* Bump + publish
* 🤖 Auto format source-db2 code [skip ci]
* 🤖 Auto format source-mssql code [skip ci]
* 🤖 Auto format source-postgres-strict-encrypt code [skip ci]
* 🤖 Auto format source-postgres code [skip ci]
* 🤖 Auto format source-mssql-strict-encrypt code [skip ci]
* 🤖 Auto format source-oracle code [skip ci]
* 🤖 Auto format source-tidb code [skip ci]
* 🤖 Auto format source-snowflake code [skip ci]
* 🤖 Auto format source-redshift code [skip ci]
* 🤖 Auto format source-mysql code [skip ci]
* 🤖 Auto format source-mysql-strict-encrypt code [skip ci]
---------
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
* initial commit
* cleanup
* cleanup
* Throw a configuration exception in case of bad jdbc url param
* End-to-End test session variable jdbc param
* reafctoring sanity
* sanity
* bump version and update note
* cherry pick fix to unrelated build error
* fix failing test
---------
Co-authored-by: Ryan Fu <ryan.fu@airbyte.io>
* Convert source mysql decimal number with 0 decimal point to an int
* Put int representation in json for decimal with 0 decimal digits
* bump docker image ver and release note
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix-db-sources: introduce method to set cursor value for TIMESTAMP_WITH_TIMEZONE and TIME_WITH_TIMEZONE
* override method for snowflake
* update method and add changelog
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix db-sources: allow TIMESTAMP_WITH_TIMEZONE and TIME_WITH_TIMEZONE as cursor value
* update docker version
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Changes to max fetch row size
- Add frequent logging for fetch size
- Increase MAX_ROW_FETCH_SIZE to 1,000,000,000
* Bump version
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* source-snowflake: use a safer method for parsing a BigInteger cursor value (#22358)
* use a safer method for parsing a BigInteger cursor value
* Add testing
* fix format change
* Fix failing integration tests
* Try removing the failing incremental test
* Try removing the failing incremental test
* Fix failing test
* Add metadata to connector logs (log level, class name, method name and line number) (#23105)
* Issue #17861 Add labels, class, method name and line numbers to connector logs
* Refactored unit test
* fix for warning about UTF8 charset in test class
---------
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
* Update docker image and release notes
* auto-bump connector version
* manually bump version on spec
---------
Co-authored-by: Prateek Mukhedkar <123108018+prateekmukhedkar@users.noreply.github.com>
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
Co-authored-by: Sergio Ropero <sergio@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* initial Exasol connector stub
* CUrrent version
* Exasol connector current version
* Exasol connector current version
* Exasol connector current version
* Exasol connector current version
* Remove unwanted changes
* Upgrade dependencies
* Use Exasol container for integration tests
* Purge database before test
* Fix integration tests
* Adapt to new API
* Fix integration tests
* Use separate fields for host, port and fingerprint
* Deactivate action on forks
* Activate action on forks
* Update documentation
* Adapt type of ID column
* Fix Exasol integration tests
* Pass certificate fingerprint when creating a DB connection
* Use the build directory for temp Exasol credentials
* Add reason for overriding testcontainers dependency
* Replace deprecated method
* Code cleanup
* Enable SSH support
* Use JdbcDestinationAcceptanceTest instead of generic base class
* Remove SSH support (will be added later)
* Apply suggestions from code review
Co-authored-by: Christoph Kuhnke <github@kuhnke.net>
* Improve debugging
* Use default name transformer implementation where possible
* Add unit tests for name transformer
* Add unit tests for destination
* Implement review findings by @ckunki
* Implement review findings by @itaseskii
* Apply suggestions from code review
Co-authored-by: Christoph Kuhnke <github@kuhnke.net>
* Describe data type limitation
* Make schema name mandatory
* Remove workaround for bug in ExasolContainer
* Add test for executing statements in a transaction
Replace Exasol specific code with default method
* add destination-exasol to destination_definitions.yaml
* add destination-exasol to destination_specs.yaml
---------
Co-authored-by: ThomasBestfleisch <thomas.bestfleisch@exasol.com>
Co-authored-by: Ivica Taseski <ivica.taseski94@gmail.com>
Co-authored-by: Christoph Kuhnke <github@kuhnke.net>
Co-authored-by: Sunny <6833405+sh4sh@users.noreply.github.com>
* add airbyte-protocol to deps.toml
* use published protocol jar for platform
* use published protocol jar for connectors
* point at published jar
* fix dep
* bump gcs storage
* fix build failures in standard-source-test
* fix deps
* downgrade alloy db because it is missing strictness tests
* Revert "downgrade alloy db because it is missing strictness tests"
This reverts commit cc6089d053.
---------
Co-authored-by: cgardens <charles@airbyte.io>
* add AttemptSyncConfig, move info out of JobSyncConfig
* get build working
* add db migration
* load config when building attempts
* persist AttemptSyncConfig
* it compiles
* fix job persistence test
* implement submitSync with attempt config
* fix TemporalClientTest
* reorganizing some code
* add GenerateInputActivity test
* verify AttemptSyncConfig is persisted
* add test for persistence changes
* add test for getAttemptByNumber
* use apis rather than direct db access
* fix compatibility with master
* copy update
* fix tests for allowed hosts addition
* remove debug logging
* fix: handle when state is not set on the connection
* fix: handle unset state (on the server this time)
* set state type when converting to internal representation
* Suggested Streams in in the Actor Definition
* Fix steam addition
* fix tests
* enable faker streams
* test resilaince.
* lint
* fig configLookup
* fixup definition load in webBackend
* fix build/tests
* Include 'suggested' in discover API response
* fix test
* Update airbyte-api/src/main/openapi/config.yaml
Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
* update build with typo
* fix test
* remove comment
* add more context
---------
Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
* Bump total allocation of the source's available memory for the JDBC buffer from 60% to 70%
* Profiling setup
* Update Dockerfile
* Fixes
* Update FetchSizeConstants.java
* Update dockerfiles
* Update JdbcStreamingQueryConfig.java
* auto-bump connector version
* Manually bump source-mysql
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Implement the persistence layer changes following #19191.
This PR handles writing and reading stats to the new stream stat_table and columns in the existing sync_stats table.
At the same time we introduce upserts of stats records - i.e. merge updates into a single record - in preparation for real time stats updates vs the current approach where a new stat record is always written.
There will be two remaining PRs after this:
- First PR will be to fully wire up and test the API.
- Second PR will be to actually save stats while jobs are running.
* updated IntegrationLauncherConfig.yaml and added to this class suportDBT and normalizationImage fields. Added to the GenerateInputActivityImpl and TemporalClient classes code parts for read destination_definition.yaml and get suportDBT and normalizationImage fields. Added logging and comparing normalization images from NormalizationRunnerFactory and destination_definition.yaml
* updated minor remarks
* updated minor remarks
* fixed minor remarks
* added normalization data to the tests
* fixed minor remarks
* removed NormalizationRunnerFactory
* fixed remarks
* fixed remarks
* fixed remarks
* updated acceptance tests
* updated acceptance tests
* updated check_images_exist.sh script
* updated method for get normalization image name for destination acceptance test
* fixed code style
* fixed code style and removed tests data
* updated JobErrorReporterTest.java
* updated JobErrorReporterTest.java
* fixed remarks
* added integration type field to the dectination_definition file and actor_definition table
* fixed tests
* fixed tests
* fixed minor changes after pulling master changes
* fixed minor changes after pulling master changes
* renamed integrationType to normalizationIntegrationType/ fixed minor remarks
* renamed extra dependencies
* updated docs
* updated docs
* fixed minor remarks
* added NormalizationDestinationDefinitionConfig.yaml for StandardDestinationDefinition.yaml and updated configuration
* updated normalization tag
* updated DestinationAcceptanceTest.java
* updated DestinationAcceptanceTest.java
* updated imports and descriptions
* fix button loading state when creating connector
* use workspace scoped endpoints
* get and update methods
* backfill script
* commit new migration file
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
