This PR introduces the following behavior for JDBC sources:
Instead of streamName = schema.tableName, this is now streamName = tableName and namespace = schema. This means that, when replicating from these sources, data will be replicated into a form matching the source. e.g. public.users (postgres source) -> public.users (postgres destination) instead of current behaviour of public.public_users. Since MySQL does not have schemas, the MySQL source uses the database as it's namespace.
To do so:
- Make namespace a field class concept in Airbyte Protocol. This allows the source to propagate namespace and destinations to write to a source-defined namespace. Also sets us up for future namespace related configurability.
- Add an optional namespace field to the AirbyteRecordMessage. This field will be set by sources that support namespace.
- Introduce AirbyteStreamNameNamespacePair as a type-safe manner of identifying streams throughout our code base.
- Modify base_normalisation to better support source defined namespace, specifically allowing normalisation of tables with the same name to different schemas.
* Zendesk Talk #2346: full refresh/incremental sync connector with adopt best practices
Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Last step (besides documentation) of namespace changes. This is a follow up to #2767 .
After this change, the following JDBC sources will change their behaviour to the behaviour described in the above document.
Namely, instead of streamName = schema.tableName, this will become streamName = tableName and namespace = schema. This means that, when replicating from these sources, data will be replicated into a form matching the source. e.g. public.users (postgres source) -> public.users (postgres destination) instead of current behaviour of public.public_users. Since MySQL does not have schemas, the MySQL source uses the database as it's namespace.
I cleaned up some bits of the CatalogHelpers. This affected the destinations, so I'm also running the destination tests.
* #2166 Issue: create Instagram connector and implement all relevant streams as full refresh
* #2166 Issue: add Insights streams
* #2273 Issue: add Incremental for streams
* #2273 Issue: code clean up
* update code after review
* add check on error for Story Insight
* add comments to code
* Source Instagram: adopt best practices, add docs, pull data from all IG business accounts (#2373)
* #2276 Issue: adopt best practices, add separate integration test for Insight's streams, create docs, update version of SDK library
* #2304 Issue: pull data from all IG business accounts
* add BASE_DIRECTORY to integration_test.py
* format configured_catalog(s)
* add credentials variables
* implement separated incremental states for different account_id, update docs
* Update instagram.md
* simplify state format
Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* add sample_config.json file
Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Google directory source #2110 - creating new source
* Google Directory #2110 - implementing new source
* Google directory #2110 - handling rate limit
* Google Directory #2110 - handling errors and rate limits
* Google Directory #2110 - reformat
* Google Directory #2110 - adding CI credentials
* Google Directory #2110 - adding to the source definition registry
* Google Directory #2110 - adding to the source definition registry(fix)
* Google Directory #2110 - injecting the config into the build environment
* Update google-directory.md
* Update google-directory.md
* Google directory #2110 - rename max_results to results_per_page and increase it to 100, fixing setup.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* add initialization for status dashboard static api
* working status reporter for integration tests
* try different formatting for date
* actually run tests
* only report build status for master
* some level of functions + datetime switching by architecture
* mixpanel integration
* add gcc to build signer tap
* fix source
* update catalogs
* fix registration name
* use base classes for source and tests
+ fix issues with creds and incremental in configured_catalog.json
* restore json config file
* set default start_date to 1 year ago
* format
* fix logging of errors in BaseSingerSource
* move integration test launching after caching
* run in separate job so it's not blocking
* move if to job level
* edit comment
* fix yaml error
* switch to using workflow path
* dump github context
* add comment id propagation
* add comment updates when things finish
* only have reactions for slash command dispatch, not the actual worker
* fix syntax
* fix syntax better
* yaml is terrible
* remove todo comments
* add error message and use not success
* final tweaks
* oops
* remove context dump
* add shopify singer
* temporarily run exchangerate on push
* depend on base singer docker output, not just base
* add static arg ref
* add integration test runner
* fmt
* add echos to show what is happening in script
* remove integration test and standard test building as part of main workflow
* remove empty integration test launcher script and re-enable checking for image existing
* format
* automatically run everything on master
* test running all on branch
* use list
* add required regex flag
* move inputs up to root of action
* move back to workflow dispath
* handle default value in non-dispatch workflows
* fix typo
* add exit 1 on failure to find connector
* add launcher
* make launcher executable
* add workflow launcher script
