airbyte/docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md

Step 3: Connecting to the API

We're now ready to start implementing the connector.

Over the course of this tutorial, we'll be editing a few files that were generated by the code generator:

• source-exchange-rates-tutorial/source_exchange_rates_tutorial/manifest.yaml: This is the connector manifest. It describes how the data should be read from the API source, as well as what inputs can be used to configure the connector.
• source-exchange_rates-tutorial/integration_tests/configured_catalog.json: This is the connector's catalog. It describes what data is available in a source
• source-exchange-rates-tutorial/integration_tests/sample_state.json: This is a sample state object to be used to test incremental syncs.

We'll also be creating the following files:

• source-exchange-rates-tutorial/secrets/config.json: This is the configuration file we'll be using to test the connector. Its schema should match the schema defined in the spec file.
• source-exchange-rates-tutorial/secrets/invalid_config.json: This is an invalid configuration file we'll be using to test the connector. Its schema should match the schema defined in the spec file.
• source_exchange_rates_tutorial/schemas/rates.json: This is the schema definition for the stream we'll implement.

Updating the connector spec and config

Let's populate the specification (spec) and the configuration (secrets/config.json) so the connector can access the access key and base currency.

1. We'll add these properties to the spec block in the source-exchange-rates-tutorial/source_exchange_rates_tutorial/manifest.yaml

spec: 
  documentation_url: https://docs.airbyte.io/integrations/sources/exchangeratesapi
  connection_specification:
    $schema: http://json-schema.org/draft-07/schema#
    title: exchangeratesapi.io Source Spec
    type: object
    required:
      - access_key
      - base
    additionalProperties: true
    properties:
      access_key:
        type: string
        description: >-
          Your API Access Key. See <a
          href="https://exchangeratesapi.io/documentation/">here</a>. The key is
          case sensitive.
        airbyte_secret: true
      base:
        type: string
        description: >-
          ISO reference currency. See <a
          href="https://www.ecb.europa.eu/stats/policy_and_exchange_rates/euro_reference_exchange_rates/html/index.en.html">here</a>.
        examples:
          - EUR
          - USD

2. We also need to fill in the connection config in the secrets/config.json Because of the sensitive nature of the access key, we recommend storing this config in the secrets directory because it is ignored by git.

echo '{"access_key": "<your_access_key>", "base": "USD"}'  > secrets/config.json

Updating the connector definition

Next, we'll update the connector definition (source-exchange-rates-tutorial/source_exchange_rates_tutorial/manifest.yaml). It was generated by the code generation script. More details on the connector definition file can be found in the overview and connection definition sections.

Let's fill this out these TODOs with the information found in the Exchange Rates API docs.

1. We'll first set the API's base url. According to the API documentation, the base url is "https://api.apilayer.com".

definitions:
  <...>
  requester:
    url_base: "https://api.apilayer.com"

2. Then, let's rename the stream from customers to rates, update the primary key to date, and set the path to "/exchangerates_data/latest" as per the API's documentation. This path is specific to the stream, so we'll set it within the rates_stream definition

  rates_stream:
    $ref: "#/definitions/base_stream"
    $parameters:
      name: "rates"
      primary_key: "date"
      path: "/exchangerates_data/latest"

We'll also update the reference in the streams block

streams:
  - "#/definitions/rates_stream"

3. Update the references in the check block

check:
  stream_names:
    - "rates"

Adding the reference in the check tells the check operation to use that stream to test the connection.

4. Next, we'll set up the authentication. The Exchange Rates API requires an access key to be passed as header named "apikey". This can be done using an ApiKeyAuthenticator, which we'll configure to point to the config's access_key field.

definitions:
  <...>
  requester:
    url_base: "https://api.apilayer.com"
    http_method: "GET"
    authenticator:
      type: ApiKeyAuthenticator
      header: "apikey"
      api_token: "{{ config['access_key'] }}"

5. According to the ExchangeRatesApi documentation, we can specify the base currency of interest in a request parameter. Let's assume the user will configure this via the connector configuration in parameter called base; we'll pass the value input by the user as a request parameter:

definitions:
  <...>
  requester:
    <...>
    request_options_provider:
      request_parameters:
        base: "{{ config['base'] }}"

The full connector definition should now look like

version: "0.1.0"

definitions:
  selector:
    extractor:
      field_path: [ ]
  requester:
    url_base: "https://api.apilayer.com"
    http_method: "GET"
    authenticator:
      type: ApiKeyAuthenticator
      header: "apikey"
      api_token: "{{ config['access_key'] }}"
    request_options_provider:
      request_parameters:
        base: "{{ config['base'] }}"
  retriever:
    record_selector:
      $ref: "#/definitions/selector"
    paginator:
      type: NoPagination
    requester:
      $ref: "#/definitions/requester"
  base_stream:
    retriever:
      $ref: "#/definitions/retriever"
  rates_stream:
    $ref: "#/definitions/base_stream"
    $parameters:
      name: "rates"
      primary_key: "date"
      path: "/exchangerates_data/latest"

streams:
  - "#/definitions/rates_stream"
check:
  stream_names:
    - "rates"
spec: 
  documentation_url: https://docs.airbyte.io/integrations/sources/exchangeratesapi
  connection_specification:
    $schema: http://json-schema.org/draft-07/schema#
    title: exchangeratesapi.io Source Spec
    type: object
    required:
      - access_key
      - base
    additionalProperties: true
    properties:
      access_key:
        type: string
        description: >-
          Your API Access Key. See <a
          href="https://exchangeratesapi.io/documentation/">here</a>. The key is
          case sensitive.
        airbyte_secret: true
      base:
        type: string
        description: >-
          ISO reference currency. See <a
          href="https://www.ecb.europa.eu/stats/policy_and_exchange_rates/euro_reference_exchange_rates/html/index.en.html">here</a>.
        examples:
          - EUR
          - USD

We can now run the check operation, which verifies the connector can connect to the API source.

python main.py check --config secrets/config.json

which should now succeed with logs similar to:

{"type": "LOG", "log": {"level": "INFO", "message": "Check succeeded"}}
{"type": "CONNECTION_STATUS", "connectionStatus": {"status": "SUCCEEDED"}}

Next steps

Next, we'll extract the records from the response

More readings

• Config-based connectors overview
• Authentication
• Request options providers
• Schema definition
• Connector specification reference
• Beginner's guide to catalog