* Adding Google Cloud Storage as destination
* Removed few comments and amended the version
* Added documentation in docs/integrations/destinations/gcs.md
* Amended gcs.md with the right pull id
* Implemented all the fixes requested by tuliren as per https://github.com/airbytehq/airbyte/pull/4329
* Renaming all the files
* Branch alligned to S3 0.1.7 (with Avro and Jsonl). Removed redundant file by making S3 a dependency for GCS
* Removed some additional duplicates between GCS and S3
* Revert changes in the root files
* Revert jdbc files
* Fix package names
* Refactor gcs config
* Format code
* Fix gcs connection
* Format code
* Add acceptance tests
* Fix parquet acceptance test
* Add ci credentials
* Register the connector and update documentations
* Fix typo
* Format code
* Add unit test
* Add comments
* Update readme
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Marco Fontana <marco.fontana@sohohouse.com>
Co-authored-by: marcofontana.ing@gmail.com <marcofontana.ing@gmail.com>
Co-authored-by: Marco Fontana <MaxwellJK@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
#4669 Stripe Source: Fix subscriptions stream to return all kinds of subscriptions (including expired and canceled)
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
* pre-PR
* add git config
* format
* Update airbyte-integrations/connectors/source-zendesk-sunshine/requirements.txt
upd requirements.txt remove extra
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
* Update airbyte-integrations/connectors/source-zendesk-sunshine/source_zendesk_sunshine/streams.py
backoff time int to float (btw real return type in headers is integer)
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
* requested changes
* fix newline absence && rm unnecessary temp file
* url_base to property
* rm extra var coming property
* rm extra var coming property
* save
* finishing updating the documentation
* forgotten definition
* add nullable to pass the test
* fix date in the log
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
* Added spec.json
* Initialization
* added oauth2 autorization
* added spec, check, discover + catalogs/configurared_catalogs
* updated request_params
* added paging, slicing (1d)
* Use oath2 for paypal
* incremental sync, acceptance test
* incremental sync, acceptance test
* Added spec.json
* Initialization
* added oauth2 autorization
* added spec, check, discover + catalogs/configurared_catalogs
* updated request_params
* added paging, slicing (1d)
* Use oath2 for paypal
* incremental sync, acceptance test
* incremental sync, acceptance test
* Added spec.json
* Initialization
* added oauth2 autorization
* added spec, check, discover + catalogs/configurared_catalogs
* updated request_params
* added paging, slicing (1d)
* Use oath2 for paypal
* incremental sync, acceptance test
* updated slices and api limits, added validation for input dates
* added tests, fixed cursor related information in schemas and configured catalogs, removed old comments, re-arranged Base PaypalTransactionStream class
* added input param 'env' to support production and sandbox envs
* added support for sandbox option, updated pattern for optional end date option
* added github secrets
* added support for sandbox option, updated pattern for optional end date option
* fixed Copyright date, removed debug mesages
* added docs
* fix for test failure - The sync should produce at least one STATE message
* removed optional parameter 'end_date'
* removed detailed info about balances schema
* Delete employees.json
* Delete customers.json
* Added requests_per_minute rate limit
* added unit tests, added custom backoff
* added test for stream slices with stream state
* removed comments
* updated docs pages
* fixed format for json files
* fixed types in schemas and link to the schema. fixed primary key for Transactions stream
* updated stream slices
* Updated tests, unified stream_slices for both streams, all instance variables instantiated directly in __init__ method
* added CHANGELOG.md
* Added build seeds
* fixed closing double quotation mark
* added paypal entry in builds.md
* add fixture helper
* added paypal transaction generator script
* fixed styling
* maximum allowed start_date is extracted from API response now.
* fixed schemas
* fixed schemas - removed datetime
* now maximum_allowed_start_date is identified by last_refreshed_datetime attr in API response.
* added possibility to specify additional properties
Co-authored-by: Sherif Nada <snadalive@gmail.com>
* add base files
* upd base
* save
* save sample files
* save & todo resolve state
* save the stage
* save the stage
* pre-gradle save
* fix catalogs
* merge && format
* normal stream slices
* apply requested changes
* requested_changes
* postfix
* update comment
* expand question stream add page_id
* upd typing once + rm missed inactual todo
* upd: caching with temp file
* upd requirements (requested)
* latest requested fixes
* posttext fix retn tempfile
* apply changes && comment
* newly requested changes
* return back spec -> changes to be set in new issue
* merge && usage comment
* add unit_test for get_updated_state function
* add simple date test
* parametrized (?) unittest
* upd comment on record_mode usage
* replace config with custom var
* pytest mark parametrized use
* rm unneeded var
* upd tests (requested)
* merge && upd texts
* add env airbyte_entrypoint
* Update README icon links
* Update airbyte-specification doc
* Extend base connector
* Remove redundant region
* Separate warning from info
* Implement s3 destination
* Run format
* Clarify logging message
* Rename variables and functions
* Update documentation
* Rename and annotate interface
* Inject formatter factory
* Remove part size
* Fix spec field names and add unit tests
* Add unit tests for csv output formatter
* Format code
* Complete acceptance test and fix bugs
* Fix uuid
* Remove generator template files
They belong to another PR.
* Add unhappy test case
* Checkin airbyte state message
* Adjust stream transfer manager parameters
* Use underscore in filename
* Create csv sheet generator to handle data processing
* Format code
* Add partition id to filename
* Rename date format variable
* Asana source
* Fix creds for CI.
* Update connection status in acceptance test config
Change status from `exception` to `failed`.
* Implement change request.
Remove few files from /integration_tests folder.
Use `stream_slices` and/or `request_params` functions instear of `read_records` function
* Update sample_config.json file.
* Update airbyte-integrations/connectors/source-asana/CHANGELOG.md
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Update `stream_slices` using.
Create reneric `read_stream` function in AsanaStream class and move there logic from `stream_slices` function.
* Rename functions.
rename `read_stream` to `read_slices_from_records`.
* Changes about publishing.
Add asana source to `source_definitions.yaml`.
Add `asana.svg`.
Create connector related file in `STANDARD_SOURCE_DEFINITION` folder.
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Create new test_ephemeral and refactor with test_normalization
* Add notes in docs
* Refactor common normalization tests into DbtIntegrationTest
* Bumpversion of normalization image
This PR introduces the following behavior for JDBC sources:
Instead of streamName = schema.tableName, this is now streamName = tableName and namespace = schema. This means that, when replicating from these sources, data will be replicated into a form matching the source. e.g. public.users (postgres source) -> public.users (postgres destination) instead of current behaviour of public.public_users. Since MySQL does not have schemas, the MySQL source uses the database as it's namespace.
To do so:
- Make namespace a field class concept in Airbyte Protocol. This allows the source to propagate namespace and destinations to write to a source-defined namespace. Also sets us up for future namespace related configurability.
- Add an optional namespace field to the AirbyteRecordMessage. This field will be set by sources that support namespace.
- Introduce AirbyteStreamNameNamespacePair as a type-safe manner of identifying streams throughout our code base.
- Modify base_normalisation to better support source defined namespace, specifically allowing normalisation of tables with the same name to different schemas.
* Zendesk Talk #2346: full refresh/incremental sync connector with adopt best practices
Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Last step (besides documentation) of namespace changes. This is a follow up to #2767 .
After this change, the following JDBC sources will change their behaviour to the behaviour described in the above document.
Namely, instead of streamName = schema.tableName, this will become streamName = tableName and namespace = schema. This means that, when replicating from these sources, data will be replicated into a form matching the source. e.g. public.users (postgres source) -> public.users (postgres destination) instead of current behaviour of public.public_users. Since MySQL does not have schemas, the MySQL source uses the database as it's namespace.
I cleaned up some bits of the CatalogHelpers. This affected the destinations, so I'm also running the destination tests.
