* 5-step tutorial * move * tiny bit of editing * Update tutorial * update docs * reset * move files * record selector, request options, and more links * update * update * connector definition * link * links * update example * footnote * typo * document string interpolation * note on string interpolation * update * fix code sample * fix * update sample * fix * use the actual config * Update as per comments * write as yaml * typo * Clarify options overloading * clarify that docker must be running * remove extra footnote * use venv directly * Apply suggestions from code review Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * signup instructions * update * clarify that both dot and bracket notations are interchangeable * Clarify how check works * create spec and config before updating connector definition * clarify what now_local() is * rename to yaml structure * Go through tutorial and update end of section code samples * fix link * update * update code samples * Update code samples * Update to bracket notation * remove superfluous comments * Update docs/connector-development/config-based/tutorial/2-install-dependencies.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/4-reading-data.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * fix path * update * motivation blurp * warning * warning * fix code block * update code samples * update code sample * update code samples * small updates * update yaml structure * custom class example * language annotations * update warning * Update tutorial to use dpath extractor * Update record selector docs * unit test * link to contributing * tiny update * $ in front of commands * $ in front of commands * More readings * link to existing config-based connectors * index * update * delete broken link * supported features * update * Add some links * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/record-selector.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * mention the unit * headers * remove mentions of interpolating on stream slice, etc. * update * exclude config-based docs Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: Augustin <augustin.lafanechere@gmail.com> Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
70 lines
2.6 KiB
Markdown
70 lines
2.6 KiB
Markdown
# Step 4: Reading data
|
|
|
|
Now that we're able to authenticate to the source API, we'll want to select data from the HTTP responses.
|
|
Let's first add the stream to the configured catalog in `source-exchange_rates-tutorial/integration_tests/configured_catalog.json`
|
|
|
|
```json
|
|
{
|
|
"streams": [
|
|
{
|
|
"stream": {
|
|
"name": "rates",
|
|
"json_schema": {},
|
|
"supported_sync_modes": [
|
|
"full_refresh"
|
|
]
|
|
},
|
|
"sync_mode": "full_refresh",
|
|
"destination_sync_mode": "overwrite"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
The configured catalog declares the sync modes supported by the stream (full refresh or incremental).
|
|
See the [catalog guide](https://docs.airbyte.io/understanding-airbyte/beginners-guide-to-catalog) for more information.
|
|
|
|
Let's define the stream schema in `source-exchange-rates-tutorial/source_exchange_rates_tutorial/schemas/rates.json`
|
|
|
|
You can download the JSON file describing the output schema with all currencies [here](https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-cdk/python/docs/tutorials/http_api_source_assets/exchange_rates.json) for convenience and place it in `schemas/`.
|
|
|
|
```bash
|
|
$ curl https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-cdk/python/docs/tutorials/http_api_source_assets/exchange_rates.json > source_exchange_rates_tutorial/schemas/rates.json
|
|
```
|
|
|
|
We can also delete the boilerplate schema files
|
|
|
|
```bash
|
|
$ rm source_exchange_rates_tutorial/schemas/customers.json
|
|
$ rm source_exchange_rates_tutorial/schemas/employees.json
|
|
```
|
|
|
|
Reading from the source can be done by running the `read` operation
|
|
|
|
```bash
|
|
$ python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
|
|
```
|
|
|
|
The logs should show that 1 record was read from the stream.
|
|
|
|
```
|
|
{"type": "LOG", "log": {"level": "INFO", "message": "Read 1 records from rates stream"}}
|
|
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing rates"}}
|
|
```
|
|
|
|
The `--debug` flag can be set to print out debug information, including the outgoing request and its associated response
|
|
|
|
```bash
|
|
$ python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json --debug
|
|
```
|
|
|
|
## Next steps
|
|
|
|
We now have a working implementation of a connector reading the latest exchange rates for a given currency.
|
|
We're however limited to only reading the latest exchange rate value.
|
|
Next, we'll [enhance the connector to read data for a given date, which will enable us to backfill the stream with historical data](5-incremental-reads.md).
|
|
|
|
## More readings
|
|
|
|
- [Record selector](../record-selector.md)
|
|
- [Catalog guide](https://docs.airbyte.io/understanding-airbyte/beginners-guide-to-catalog) |