* Add GET_FBA_INVENTORY_AGED_DATA data
* Add GET_MERCHANT_LISTINGS_ALL_DATA stream support
* Update schemas
* Update configured_catalog.json
* Update connector to airbyte-cdk
* Add amazon seller partner test creds
* Update state sample files
* Apply code format
* Update acceptance-test-config.yml
* Add dummy integration test
* Refactor auth signature.
Update streams.py
* Remove print_function import from auth.py
* Refactor source class.
Add pydantic spec.
PR fixes.
* Add dummy integration test
* Typing added.
Add _create_prepared_request docstring.
* Add extra streams and schemas
* Update docs and spec
* Post merge code fixes
* Fix test setup
* Fix test setup
* Add sample_state.json
* Update reports streams logics.
Update test and config files.
* Update tests config.
Small code style fixes.
* Add reports stream slices.
Update check_connection method.
* Post review fixes.
* Streams update
* Add reports document retrieval and decrypting.
Update schemas and configs.
* Add CVS parsing into result rows
* Update ReportsAmazonSPStream class to be the child of Stream class.
Update GET_FLAT_FILE_OPEN_LISTINGS_DATA and GET_MERCHANT_LISTINGS_ALL_DATA schemas.
* Schema updates
* Source check method updated
* Update ReportsAmazonSPStream retry report logics
* Update check_connection source method
* Update reports read_records method.
Update report schemas.
* Update streams.py
* Update acceptance tests config.
Add small code fixes.
* Update report read_records logics
* Add reports streams rate limit handling logics.
Add rate limit unit tests.
* Source Amazon SP: Update reports streams logics. (#5311)
* Update check connection method
* #5796 silence printing full config when config validation fails (#5879)
* - #5796 silence printing full config when config validation fails
* fix unit tests after config validation check changes
Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com>
* Format google-search-console schemas (#6047)
* Update ads_insights.json (#5946)
fix ads_insights schema according to [facebook docs](https://developers.facebook.com/docs/marketing-api/reference/adgroup/insights/) and my own data
* Bump connectors version + update docs (#6060)
* 🐛 Source Facebook Marketing: Convert values' types according to schema types (#4978)
* Convert values' types according to schema types
* Put streams back to `configured_catalog.json`
Put back `ads_insights` and `ads_insights_age_and_gender` streams.
* Pickup changes from #5946
* Implement change request + fix previous PR
* Update schema
* Remove items_type from convert_to_schema_types()
* Bump connectors version
* add oauth to connector_base dependencies (#6064)
* use spec when persisting source configs (#6036)
* switch most usages of writing sources to using specs
* fix other usages
* fix test
* only wait on the server in the scheduler, not the worker
* fix
* rephrase sanity check and remove stdout
* 🎉 Source Stripe: Add `PaymentIntents` stream (#6004)
* Add `PaymentIntents` stream
* Update docs
* Implement change request + few updates
Split `source.py` file into `source.py` and `streams.py` files.
Update `payment_intents.json` file.
* Bump connectors version + update docs
* Add skeleton for databricks destination (#5629)
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
Co-authored-by: LiRen Tu <tuliren@gmail.com>
* Revert "Add skeleton for databricks destination (#5629)" (#6066)
This reverts commit 79256c46b5.
* 🎉 New Destination: Databricks (#5998)
Implement new destination connector for databricks delta lake.
Resolves#2075.
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Source PostHog: add support for self-hosted instances (#6058)
* publish #6058 (#6059)
* Destination Kafka: correct spec json and data types in config (#6040)
* correct spec json and data types in config
* bump version
* correct tests
* correct config parser NPE
* format files
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>
* Fix or delete broken links (#6069)
* Fix more doc issues (#6072)
* 🎉 Added optional platform flag for build image script (#6000)
* Fix dependabot security alert. (#6073)
* Pin set value to greater than 4.0.1 to fix security warning.
* Format the rest of the connectors.
* add coverage report (#6045)
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
* Fix the format of the data returned by Google Ads oauth to match the config accepted by the connector (#6032)
* update salesforce docs (#6081)
* 🎉 Source Github: add caching for all streams (#5949)
* Source Github: add checking for all streams
* bump version, update changelogs
* Disable automatic migration acceptance test (#5988)
- The automatic migration acceptance test no longer works because of the new Flyway migration system.
- The file-based migration system is being deprecated.
* 🎉 CDK: Add requests native authenticator support (#5731)
* Add requests native auth class
* Update init file.
Update type annotations.
Bump version.
* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.
* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md
* Add requests native authenticator tests
* Add CDK requests native __call__ method tests.
Update CHANGELOG.md
* Add outdated auth deprication messages
* Update requests native auth __call__ method tests
* Bump CDK version to 0.1.20
* Interface changes to support separating secrets from the config (#6065)
* Interface changes to support separating secrets from the config
* Cleanup from PR comments and whitespace
* Update log message for empty env variable (#6115)
Co-authored-by: Jared Rhizor <jared@dataline.io>
* Bump Airbyte version from 0.29.17-alpha to 0.29.18-alpha (#6125)
Co-authored-by: davinchia <davinchia@users.noreply.github.com>
* return auth spec in the API when getting definition specification (#6121)
* Ignore python test coverage files (#6144)
* CDK: support nested refs resolving (#6044)
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
* feat: path for nested fields (#6130)
* feat: path for nested fields
* fix: clipRule error
* fix: remove field name
* Fix request middleware for ConnectionService (#6148)
* Jamakase/update onboarding flow (#5656)
* Doc explains normalization full-refresh implications (#6097)
* update docs
* add info in quickstart connection page
* update abhi comments
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>
* Fix migration validation issue (#6154)
Resolves#6151.
* Bump Airbyte version from 0.29.18-alpha to 0.29.19-alpha (#6156)
Co-authored-by: tuliren <tuliren@users.noreply.github.com>
* Add information on which destinations support Incremental - Deduped History in their docs (#6031)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
* Update Airbyte Spec acknowledgements. (#6155)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
* Update new integration request
* Add back the migration acceptance test (#6163)
* 🎉 Create a Helm Chart For Airbyte (#5891)
See number #1868. This creates an initial helm chart for installing Airbyte in Kubernetes to make it easier for users who are more familiar with helm. It also includes GitHub actions to help continually test that the chart works in the most basic case.
All of the templates are based off of the kustomize folder, but minio and postgres have been removed in favor of adding the bitnami helm charts as dependencies since they have an active community and allow easily tweaking their install.
* Fix OAuth Summary strings (#6143)
Co-authored-by: Marcos Eliziario Santos <eliziario@users.noreply.github.com>
Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com>
Co-authored-by: oleh.zorenko <19872253+Zirochkaa@users.noreply.github.com>
Co-authored-by: Mauro <35332423+m-ronchi@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Jared Rhizor <jared@dataline.io>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
Co-authored-by: LiRen Tu <tuliren@gmail.com>
Co-authored-by: coeurdestenebres <90490546+coeurdestenebres@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>
Co-authored-by: Harsha Teja Kanna <h7kanna@users.noreply.github.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Dmytro <avida.d3@gmail.com>
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
Co-authored-by: Yevhenii <34103125+yevhenii-ldv@users.noreply.github.com>
Co-authored-by: Jenny Brown <85510829+airbyte-jenny@users.noreply.github.com>
Co-authored-by: davinchia <davinchia@users.noreply.github.com>
Co-authored-by: Iakov Salikov <36078770+isalikov@users.noreply.github.com>
Co-authored-by: Artem Astapenko <3767150+Jamakase@users.noreply.github.com>
Co-authored-by: tuliren <tuliren@users.noreply.github.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
Co-authored-by: Jonathan Stacks <jonstacks@users.noreply.github.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
* Bump source version.
Update source docs.
* Mock time.sleep in test_reports_stream_send_request_backoff_exception test
* Acceptance test basic_read test disabled
Co-authored-by: Marcos Eliziario Santos <eliziario@users.noreply.github.com>
Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com>
Co-authored-by: oleh.zorenko <19872253+Zirochkaa@users.noreply.github.com>
Co-authored-by: Mauro <35332423+m-ronchi@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Jared Rhizor <jared@dataline.io>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
Co-authored-by: LiRen Tu <tuliren@gmail.com>
Co-authored-by: coeurdestenebres <90490546+coeurdestenebres@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>
Co-authored-by: Harsha Teja Kanna <h7kanna@users.noreply.github.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Dmytro <avida.d3@gmail.com>
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
Co-authored-by: Yevhenii <34103125+yevhenii-ldv@users.noreply.github.com>
Co-authored-by: Jenny Brown <85510829+airbyte-jenny@users.noreply.github.com>
Co-authored-by: davinchia <davinchia@users.noreply.github.com>
Co-authored-by: Iakov Salikov <36078770+isalikov@users.noreply.github.com>
Co-authored-by: Artem Astapenko <3767150+Jamakase@users.noreply.github.com>
Co-authored-by: tuliren <tuliren@users.noreply.github.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
Co-authored-by: Jonathan Stacks <jonstacks@users.noreply.github.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Implement new destination connector for databricks delta lake.
Resolves#2075.
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Convert values' types according to schema types
* Put streams back to `configured_catalog.json`
Put back `ads_insights` and `ads_insights_age_and_gender` streams.
* Pickup changes from #5946
* Implement change request + fix previous PR
* Update schema
* Remove items_type from convert_to_schema_types()
* Bump connectors version
* added supported_sync_modes for AirbyteStream initialization to fic full_refresh test
* fix unit test with supported_sync_modes argument in AirbyteStream initialization
* Added ability to run the incremental sync within slicing, by leveraging the cached_state
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
* fixed s3 destination field naming for Parquet and Avro formats
* pull request number update
* updated resources for parquet
* snowflake s3 destination COPY is writing records from different table in the same raw table fix
* fixed s3 destination name transformer
* updated s3 destination name transformer
* updated snowflake s3 file name
* updated snowflake documentation
* updated snowflake documentation
* updated snowflake documentation
* updated code style
* updated code style
* updated code style
* updated redshift destination
* added test data for test filed with bad first char
* updated s3 documentation
* fixed remarks
* fixed code style
* fixed s3 tests
* New abstraction for NoSql database sources
* New MongoDbSource: partial impl
* Added MongoDataType
* Improved MongoDatabase and fixed read method
* code review changes;
* merge clean up;
* Renamed NoSqlDatabase to AbstractDatabase
* formatter changes;
* code review changes: changed mongodb-new to mongodb-v2; left only new connector info in all docs
* code review changes: changed mongodb-new to mongodb-v2; left only new connector info in all docs
* updated spec.json and toDatabaseConfig() method
* updated doc accordingly to spec.json changes
Co-authored-by: Iryna Kruk <iryna.o.kruk@globallogic.com>
* fix negative backoff
* correct incremental logic for comments
* remove test logs
* bump version
* fix spec
* remove old non-relevant test
* correct backoff default value
* correction a test
* save addl export cursor field
* fix incorrect optinal date-time
* add format date-time again
* correction after review
* fix integration test
* fix unit test
Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
* oracle normalization
* correct dbt_project function for oracle
* unit tests
* run format
* correct ephemeral tests
* add gradle dependency for oracle destination
* run int tests
* add oracle in settings.gradle for normalization run[
* use default airbyte columns
* format
* test all destinatoin ephemeral
* correct unit test
* correct unit test
* destination docs update
* correct mypy
* integration test all dest
* refactor oracle function
* merge master
* run all destinations
* flake8 escape regex
* surrogate key function
* correct few minor comments
* refactor scd sql function
* refactor scd function
* revert test
* refactor minor details
* revert tests
* revert ephemeral test
* revert unit test table_registry
* revert airbyte_protocol format
* format
* bump normalization version in worker
* minor chnages
* minor chages
* correct json_column for other destinations
* gradlew format
* revert tests
* remove comments
* add Oracle destination explicit in safe_cast_str
* add quote_in_parenthesis inside if clause
* gradlew format
* Fixed (StackOverflow) destination bigquery denormalized destination to handle the case when received schema doesn't contain a data type for Array type
* new mongo db destination
* fix remarks
* updated documentation and added loggers
* updated documentation
* added hashCode field to mongodb document and fix minor remarks
* fix code style
* updated mongodb data hash from integer to UUID string
