* Add toy source for the PokeAPI
* Address review comments.
* Run gradlew format
* Move Pokemon list constant to its own file
* Fix gradle formatting issues
* Prepare connector for publishing.
* Fix some naming bugs
* Update config file path
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
* Abort sync if one of the part fails to copy to temp table
* Check for record size when copying data from s3 to redshift
* Handle big record in RedshiftInsertDestination too
The Redshift Copy strategy currently has it's part size set to 10 MB. Since S3 allows a file to be broken into max of 10k parts, this results in a 100GB table limit. A user is trying to sync a table of 115GB and running into this issue.
This makes the part size configurable so users can increase this size if needed.
Release all connectors affected by namespace change. Includes all JDBC sources and destinations.
Also add documentation for normalisation. Prerequisite to actually releasing 0.21.0-alpha.
* Zendesk Talk #2346: full refresh/incremental sync connector with adopt best practices
Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* spike
* more
* debezium wip
* use oneof for configuration
* iterator wrapping structure
* push current
* working loop
* move capability into source
* hack it into a sharable state
* debezium test runner (#2617)
* CDC Wait for Values (#2618)
* output actual AirbyteMessages for cdc (#2631)
* message conversion
* fmt
* add lsn extraction and comparison (#2613)
* postgres cdc catalog (#2673)
* update cdc catalog
* A
* table selection for cdc (#2690)
* table selection for cdc
* fix broken merge
* also test double quote in name
* Add state management to CDC (#2718)
* CDC: Fix Producer/Consumer State Machine (#2721)
* CDC Postgres Tests (#2777)
* fix postgres cdc image name and run check before reading data (#2785)
* minor postgres cdc fixes
* add test and fix check behavior
* fix
* improve comment
* remove unused props, remove todos, add some more sanity tests (#2791)
* cdc: add offset store tests (#2793)
* clean (#2798)
* postgres cdc docs (#2784)
* cdc docs
* Update docs/integrations/sources/postgres.md
Co-authored-by: Charles <giardina.charles@gmail.com>
* address gcp
* learn too english
* add link
* add more disk space warnings
* add additional cdc use case
* add information on how to find postgresql.conf
* add how to find the file
Co-authored-by: Charles <giardina.charles@gmail.com>
* various merge conflict fixes (#2799)
* cdc standard tests (#2813)
* require cdc users to create publications & update docs (#2818)
* postgres cdc race condition
* working? but different process
* add additional logging to help debug in the future
* everything done except working config
* remove unintended change
* Use oneOf in PG CDC spec (#2827)
* add oneOf configuration for postgres cdc (#2831)
* add oneof configuration for cdc postgres
* fmt
Co-authored-by: Charles <giardina.charles@gmail.com>
* fix test (#2834)
* fix test
* bump version
* add docs on creating replica identities (#2838)
* add docs on creating replica identities
* emphasize danger
* grammar
* bump pg version in source catalog
* generate seed files
Co-authored-by: cgardens <giardina.charles@gmail.com>
* #2166 Issue: create Instagram connector and implement all relevant streams as full refresh
* #2166 Issue: add Insights streams
* #2273 Issue: add Incremental for streams
* #2273 Issue: code clean up
* update code after review
* add check on error for Story Insight
* add comments to code
* Source Instagram: adopt best practices, add docs, pull data from all IG business accounts (#2373)
* #2276 Issue: adopt best practices, add separate integration test for Insight's streams, create docs, update version of SDK library
* #2304 Issue: pull data from all IG business accounts
* add BASE_DIRECTORY to integration_test.py
* format configured_catalog(s)
* add credentials variables
* implement separated incremental states for different account_id, update docs
* Update instagram.md
* simplify state format
Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* add sample_config.json file
Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Google directory source #2110 - creating new source
* Google Directory #2110 - implementing new source
* Google directory #2110 - handling rate limit
* Google Directory #2110 - handling errors and rate limits
* Google Directory #2110 - reformat
* Google Directory #2110 - adding CI credentials
* Google Directory #2110 - adding to the source definition registry
* Google Directory #2110 - adding to the source definition registry(fix)
* Google Directory #2110 - injecting the config into the build environment
* Update google-directory.md
* Update google-directory.md
* Google directory #2110 - rename max_results to results_per_page and increase it to 100, fixing setup.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
