* try to publish new normalization version
* default to using ssl in postgres destinatoin
* tidy up
* Run normalization tests using postgres DB with SSL support
* bump version
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
* ssh-test
* add authentification via ssh tunnel with bastion docker host and postgres testcontainer
* created SshBastion class in base-java module
* implement Postgres source basic ssh tunneling connection for integration tests
* implement Postgres source ssh tunneling connection and refactoring SshBastion
* generate keys inside a bastion container
* remove throwing Exception from startTestContainers method
* fix checkstyle
* add documentation and changelog for Posthres source and destination
* update documentation for ssh readme.md | update version fo Postgres source and destination to 0.3.12
* update version of Postgres source and destination to 0.3.12
* removed static variables, removed version bump, rename class to SshBastionContainer, removed ci credentials for ssh Postgres Source and Destination
Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
* move logic from static JdbcUtils to JdbcSourceOperations
* format
* Add methods for all types in order to have possibility rewrite them and use common type mapping.
* Make BigQuery in line impl with JDBC changes
* fix binary type
* add database creations methods with sourceOperations input
* Update airbyte-db/lib/src/main/java/io/airbyte/db/SourceOperations.java
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* fix \u0000(NULL) value processing for Postgres + move postgres impl of SqlOperations to PostgresSqlOperations.
* changelog + format
* incr release version
* Add generic solution to adopt messages for a destination + remove unnecessary serialization
* revert version for build
* minor review fixes
* format
* add comments
* format
* incr version
# Summary
- A follow-up PR for #5543.
- This PR separates the `airbyte-db` project to two modules:
- `lib` is the original `airbyte-db`.
- `jooq` is for jOOQ code generation.
- This is necessary because the jOOQ generator requires a custom database implementation that can run Flyway migration. So the code generator logic needs to depend on the compilation of the original `airbyte-db` project.
# Commits
* Separate db to lib and jooq modules
* Update dependencies
* Add jobs db migrator test
* Fix compose build
* Add migration dev center
* Add schema dump task
* Update airbyte-db/lib/README.md
* Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update readme
* Remove bom dependency
* Update readme
* Use jooq code in db config persistence
* Remove AirbyteConfigsTable
Co-authored-by: Davin Chia <davinchia@gmail.com>
* add AIRBYTE_ENTRYPOINT for kubernetes support
* bump versions
* bump version in seed
* Update generic template
* keep scaffold sources at 0.1.0
* add missing newline
* handle python base versions correctly
* re-bump mysql and postgres sources
* re-bump snowflake destination
* add skip tests option
* switch to running tests
* reverse conditional to make it safer
* fix publish to include the test running
* fix iterable version
* fix file generation
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Update README icon links
* Update airbyte-specification doc
* Extend base connector
* Remove redundant region
* Separate warning from info
* Implement s3 destination
* Run format
* Clarify logging message
* Rename variables and functions
* Update documentation
* Rename and annotate interface
* Inject formatter factory
* Remove part size
* Fix spec field names and add unit tests
* Add unit tests for csv output formatter
* Format code
* Complete acceptance test and fix bugs
* Fix uuid
* Remove generator template files
They belong to another PR.
* Add unhappy test case
* Checkin airbyte state message
* Adjust stream transfer manager parameters
* Use underscore in filename
* Create csv sheet generator to handle data processing
* Format code
* Add partition id to filename
* Rename date format variable
* destination-specification: add supportsNormalization and supportsDBT attributes
* address review comment
* missed this one
* output after gradle format
* Migrate BufferedStreamConsumer users (e.g. all JDBC destinations, MeiliSearch) (#3473)
* Add checkpointing test cases in Acceptance Tests (#3473)
* Add testing for emitting state in Destination Standard Test (#3546)
* Migrate BQ to support checkpointing (#3546)
* Migrate copy destinations support checkpointing (#3547)
* Checkpointing: Migrate CSV and JSON destinations (#3551)
* use strict JSONL definition of new lines in destinations
* failing test case
* use next instead of nextLine
* add \n in string for test
* bump destination versions
* bump to even newer version
* bump versions in dockerfiles as well
* force mysql test to pass
Release all connectors affected by namespace change. Includes all JDBC sources and destinations.
Also add documentation for normalisation. Prerequisite to actually releasing 0.21.0-alpha.
This PR introduces the following behavior for JDBC sources:
Instead of streamName = schema.tableName, this is now streamName = tableName and namespace = schema. This means that, when replicating from these sources, data will be replicated into a form matching the source. e.g. public.users (postgres source) -> public.users (postgres destination) instead of current behaviour of public.public_users. Since MySQL does not have schemas, the MySQL source uses the database as it's namespace.
To do so:
- Make namespace a field class concept in Airbyte Protocol. This allows the source to propagate namespace and destinations to write to a source-defined namespace. Also sets us up for future namespace related configurability.
- Add an optional namespace field to the AirbyteRecordMessage. This field will be set by sources that support namespace.
- Introduce AirbyteStreamNameNamespacePair as a type-safe manner of identifying streams throughout our code base.
- Modify base_normalisation to better support source defined namespace, specifically allowing normalisation of tables with the same name to different schemas.
* add ssl for source-postgres
* add config in utf8 test
* correct comments from @jrhizor and @sherifnada
* correct config get
* add ssl test postgres
* add sh generate ssl files
* change pg ssl test
* use custom image
* correct spec.json
* correc tests
* remove unecessary config
* add config and correct spec.json
* add ssl to postgres destination
* add tools to generate custom dockers images and correct spec.json
* change how additional parameter is append
* add logic ssl for postgres destination
* remove if for append add params
* gradlew format
Add namespace field to the Airbyte Stream in preparation to propagate a source defined namespace to the Destination.
This namespace field is then consumed as the destination schema the table is written to.
This only applies to JDBC destinations.
This is Steps 1 - 4 of the namespace tech spec, seen at https://docs.google.com/document/d/1qFk4YqnwxE4MCGeJ9M2scGOYej6JnDy9A0zbICP_zjI/edit.
Some minor refactoring and commenting as I go.
* Remove unnecessary test classes as they match Integration tests in terms of what is being tested. They have no real value since the corresponding integration test can be run locally without additional credentials. The main value the classes brings is letting us run tests without building the docker image (the integration tests require doing so). however I feel this benefit is not worth the additional maintenance cost.
* Centralise DataArgumentProvider into it's own class for easier maintenance and usability.
Clean up Destination interfaces, with the goal of less repeated code and hopefully better readability for the next person.
* Rename the write method in the Destination interface to getConsumer to better reflect that the method is not writing itself, but returning a consumer that will write upon accepting a message. This was consuming me when I was reading the code.
* Remove generics from the FailureTrackingConsumer and the DestinationConsumer. Besides tests, there are no generic uses of the FaliureTrackingConsumer. Replace this with the AirbyteMessage to make explicit this is what the FailureTrackingConsumer use cases. Although this does restrict our future use of the FailureTrackingConsumer class, I'd rather limit this now and re-inject generics once we have more use cases. This was also confusing me - I kept on wondering what other data type can be consuming this interface.
* Rename FailureTrackingConsumer to FailureTrackingAirbyteMessageConsumer to better reflect how the consumer is meant to be used strictly as a DestinationConsumer (it implements the interface).
* Rename DestinationConsumer to AirbyteMessageConsumer.
In a subsequent PR, I plan to consolidate logic to error if the received Airbyte message is not of Record type, into the
FailureTrackingAirbyteMessageConsumer class.
* Handle destination sync mode in destinations
* Source & Destination sync modes are required (#2500)
* Provide Migration script making sure it is always defined for previous sync configs
* Fix JdbcSource handling of tables with same names in different schemas
* Previously the JdbcSource was combining the columns of any tables with the same name across different schemas into a single stream in the catalog.
* This was caught because in those tables there were columns of the same name with different types which triggered a precondition to check for this.
* The fix makes sure we group by both schema name and table name.
* Adds test to the standard jdbc tests to catch this case.
* This test does NOT run for mysql as, mysql has no concept of schemas.
