jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

🎉 Support incremental sync in Sendgrid (#3512)

Browse Source 
* Add incremental streams support.
Updated config catalogs.

* Format streams.py

* Add incremental test config.
Update README.md.
Remove unused code.

* Add SendgridStreamIncrementalMixin init type definitions

* Acceptance tests config full_refresh configured_catalog_path fixed.

* Add super() request_params call to SendgridStreamOffsetPagination request_params method

* Fix start time overwrite in incremental streams.
Update config files for abnormal state and catalogs.

* Bump source-sendgrid docker version
This commit is contained in:
Vadym
2021-05-25 15:49:48 +03:00
committed by GitHub
parent f31d409f73
commit 70f5dba415
12 changed files with 121 additions and 507 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
airbyte-integrations/connectors/source-sendgrid/README.md
View File

@@ -37,7 +37,7 @@ From the Airbyte repository root, run:
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/sendgrid)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_sendgrid/spec.json` file.
Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
See `sample_files/sample_config.json` for a sample config file.
See `integration_tests/sample_config.json` for a sample config file.
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source sendgrid test creds`
and place them into `secrets/config.json`.
@@ -48,7 +48,7 @@ and place them into `secrets/config.json`.
python main.py spec
python main.py check --config secrets/config.json
python main.py discover --config secrets/config.json
python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
```
### Unit Tests
@@ -78,7 +78,7 @@ Then run any of the connector commands as follows:
docker run --rm airbyte/source-sendgrid:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-sendgrid:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-sendgrid:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/source-sendgrid:dev read --config /secrets/config.json --catalog /sample_files/configured_catalog.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-sendgrid:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```
### Integration Tests