* add AIRBYTE_ENTRYPOINT for kubernetes support
* bump versions
* bump version in seed
* Update generic template
* keep scaffold sources at 0.1.0
* add missing newline
* handle python base versions correctly
* re-bump mysql and postgres sources
* re-bump snowflake destination
* add skip tests option
* switch to running tests
* reverse conditional to make it safer
* fix publish to include the test running
* fix iterable version
* fix file generation
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* source: implementation for mysql cdc
* add target file and position
* dont want to add file in this PR
* refine tests + add comments
* fix typo
* address review comments
* fix formatting error
* resolve conflicts
* update docs + bump docker minor version
* remove un-necessary new lines + add multiple checks for cdc
* address review comments from Davin
* increase the version in source_definitions.yaml
* rebuild seed
Release all connectors affected by namespace change. Includes all JDBC sources and destinations.
Also add documentation for normalisation. Prerequisite to actually releasing 0.21.0-alpha.
This PR introduces the following behavior for JDBC sources:
Instead of streamName = schema.tableName, this is now streamName = tableName and namespace = schema. This means that, when replicating from these sources, data will be replicated into a form matching the source. e.g. public.users (postgres source) -> public.users (postgres destination) instead of current behaviour of public.public_users. Since MySQL does not have schemas, the MySQL source uses the database as it's namespace.
To do so:
- Make namespace a field class concept in Airbyte Protocol. This allows the source to propagate namespace and destinations to write to a source-defined namespace. Also sets us up for future namespace related configurability.
- Add an optional namespace field to the AirbyteRecordMessage. This field will be set by sources that support namespace.
- Introduce AirbyteStreamNameNamespacePair as a type-safe manner of identifying streams throughout our code base.
- Modify base_normalisation to better support source defined namespace, specifically allowing normalisation of tables with the same name to different schemas.
* add ssl for source-postgres
* add config in utf8 test
* correct comments from @jrhizor and @sherifnada
* correct config get
* add ssl test postgres
* add sh generate ssl files
* change pg ssl test
* use custom image
* correct spec.json
* correc tests
* remove unecessary config
* add config and correct spec.json
* add ssl to postgres destination
* add tools to generate custom dockers images and correct spec.json
* change how additional parameter is append
* add logic ssl for postgres destination
* remove if for append add params
* gradlew format
Last step (besides documentation) of namespace changes. This is a follow up to #2767 .
After this change, the following JDBC sources will change their behaviour to the behaviour described in the above document.
Namely, instead of streamName = schema.tableName, this will become streamName = tableName and namespace = schema. This means that, when replicating from these sources, data will be replicated into a form matching the source. e.g. public.users (postgres source) -> public.users (postgres destination) instead of current behaviour of public.public_users. Since MySQL does not have schemas, the MySQL source uses the database as it's namespace.
I cleaned up some bits of the CatalogHelpers. This affected the destinations, so I'm also running the destination tests.
* spike
* more
* debezium wip
* use oneof for configuration
* iterator wrapping structure
* push current
* working loop
* move capability into source
* hack it into a sharable state
* debezium test runner (#2617)
* CDC Wait for Values (#2618)
* output actual AirbyteMessages for cdc (#2631)
* message conversion
* fmt
* add lsn extraction and comparison (#2613)
* postgres cdc catalog (#2673)
* update cdc catalog
* A
* table selection for cdc (#2690)
* table selection for cdc
* fix broken merge
* also test double quote in name
* Add state management to CDC (#2718)
* CDC: Fix Producer/Consumer State Machine (#2721)
* CDC Postgres Tests (#2777)
* fix postgres cdc image name and run check before reading data (#2785)
* minor postgres cdc fixes
* add test and fix check behavior
* fix
* improve comment
* remove unused props, remove todos, add some more sanity tests (#2791)
* cdc: add offset store tests (#2793)
* clean (#2798)
* postgres cdc docs (#2784)
* cdc docs
* Update docs/integrations/sources/postgres.md
Co-authored-by: Charles <giardina.charles@gmail.com>
* address gcp
* learn too english
* add link
* add more disk space warnings
* add additional cdc use case
* add information on how to find postgresql.conf
* add how to find the file
Co-authored-by: Charles <giardina.charles@gmail.com>
* various merge conflict fixes (#2799)
* cdc standard tests (#2813)
* require cdc users to create publications & update docs (#2818)
* postgres cdc race condition
* working? but different process
* add additional logging to help debug in the future
* everything done except working config
* remove unintended change
* Use oneOf in PG CDC spec (#2827)
* add oneOf configuration for postgres cdc (#2831)
* add oneof configuration for cdc postgres
* fmt
Co-authored-by: Charles <giardina.charles@gmail.com>
* fix test (#2834)
* fix test
* bump version
* add docs on creating replica identities (#2838)
* add docs on creating replica identities
* emphasize danger
* grammar
* bump pg version in source catalog
* generate seed files
Co-authored-by: cgardens <giardina.charles@gmail.com>
This PR is step 5 of this tech spec - https://docs.google.com/document/d/1qFk4YqnwxE4MCGeJ9M2scGOYej6JnDy9A0zbICP_zjI/edit.
The first of (at least) 2 PRs to implement this on the source side. I made some headway before deciding to break the changes into one PR implementing this for discover schema job, and another PR implementing this for read. The combined PR would have been too big otherwise.
Also refactor MoreResources as the test method was attempting to write to the location classes where loaded out from - the issue is we cannot guarantee where the class is loaded from can be written to. Changing this to write to a random folder in the temp directory.
* Handle destination sync mode in destinations
* Source & Destination sync modes are required (#2500)
* Provide Migration script making sure it is always defined for previous sync configs
* Fix JdbcSource handling of tables with same names in different schemas
* Previously the JdbcSource was combining the columns of any tables with the same name across different schemas into a single stream in the catalog.
* This was caught because in those tables there were columns of the same name with different types which triggered a precondition to check for this.
* The fix makes sure we group by both schema name and table name.
* Adds test to the standard jdbc tests to catch this case.
* This test does NOT run for mysql as, mysql has no concept of schemas.
* support cached builds for base -> base-java -> snowflake
* use plugin for image building
* fix matching on COPY from
* remove docker.gradle
* tmp commit
* update connectors
* finish rest of build files
* fix ide errors
* more build fixes
* clean up
* clean up for new soruces
* fix spotless
* fix flake problems
* add recommended empty file
* python caching
* fixes upon review
* clean up docker and build test files
* clean up python
* clean up
* fix integration test dependencies
* fix standard tests
* fix
* remove symlink
* re-add requirements to fix normalizatioon build
* fix symlink
* fix dumbest build problem of all
* add missing integration test def
* fix missing dep
* remove class exclusion
* move trim so null source versions are allowed
* rename map
* fix hardcoded value
* remove unnecessary dep
* use dashes for salesforce package name
* fix typo
* DRY and fix test image name
* Fix edit
* assert string is not empty
* build integration test image only for integrationTest
* move code generator to tools and rename docker build tasks
* make source test depend on integration test build, not the other way
* remove guard because the docker build should exist before the integrationtest is applied
* remove comment
* DRY up airbyte-source-test
* fix plugin compilation
* add missing dependency
* rename getTaggedImage to getDevTaggedImage
* fix test vs main docker build bug
