airbyte/docs/integrations/destinations/heap-analytics.md

Heap Analytics

Heap Analytics is the only digital insights platform that gives you complete understanding of your customers’ digital journeys, so you can quickly improve conversion, retention, and customer delight.

Every action a user takes is autocaptured. See them live, analyze later, or let our Illuminate functionality automatically generate insights.

The Destination Connector here helps you load data to Heap, so that you could leverage the powerful analytics tool.

Prerequisites

• Heap Analytics Account
• App Id, also called Environment Id

Step 1: Set up Heap Analytics

Heap Analytics Account

If you don't have an Account

Sign up and create your Heap Analytics Account.

Understand Projects and Environments in Heap

Your Heap account is structured into a series of projects and environments. Projects in Heap can be thought of as blank slates and are completely independent of another. Environments are subsets of projects that share definitions (defined events, segments, and reports), but do not share any data. More info can be found from this doc.

Step 2: Prepare the environment Id

You can get the environment ID from the Heap Analytics dashboard. Choose Account --> Manage --> Projects.

Step 3: Set up the destination connector in Airbyte

1. In the left navigation bar, click Destinations. In the top-right corner, click + new destination.
2. On the destination setup page, select Heap Analytics from the Destination type dropdown and enter a name for this connector.
3. Fill in the environment Id to the field app_id
4. Pick the right API Type, we will cover more details next.

API Type Overview

Airbyte will load data to Heap by the server-side APIs. There are 3 API types

• Track Events
• Add User Properties
• Add Account Properties

The destination connector supports all types of schemas of source data. However, each configured catalog, or the connector instance, can load one stream only. The API type and the configuration determine the output stream. The transformation is run in memory that parses the source data to the schema compatible to the Server-Side API. Since there are 3 APIs, there are 3 different output schemas.

Step 4: Configure the transformation for an API Type

Track Events

Use this API to send custom events to Heap server-side.

The following is the sample cURL command:

curl \
  -X POST https://heapanalytics.com/api/track\
  -H "Content-Type: application/json" \
  -d '{
    "app_id": "11",
    "identity": "alice@example.com",
    "event": "Send Transactional Email",
    "timestamp": "2017-03-10T22:21:56+00:00", 
    "properties": {
      "subject": "Welcome to My App!",
      "variation": "A"
    }
  }' 

There are 4 properties in the request body.

• identity: An identity, typically corresponding to an existing user.
• event: The name of the server-side event.
• properties: An object with key-value properties you want associated with the event.
• timestamp: (optional), the datetime in ISO8601. e.g. "2017-03-10T22:21:56+00:00". Defaults to the current time if not provided.

For Add User Properties, You need to configure the following 4 fields in airbyte.

• Identity Column: The attribute name from the source data populated to identity.
• event_column: The attribute name from the source data populated to event.
• Timestamp Column: The attribute name from the source data populated to timestamp. This field is optional. It will be the current time if not provided.
• Property Columns: The attribute names from the source data populated to object properties. If you want to pick multiple attributes, split the names by comma(,). If you want to pick ALL attributes, simply put asterisk(*).

Note that, if you want to reference an attribute name in an object or an embedded object. You can the daisy-chained (.) connections. Let's use an example to illustrate it.

The data source is a json.

{
  "blocked": false,
  "created_at": "2022-10-21T04:09:54.622Z",
  "email": "evalyn_shields@hotmail.com",
  "email_verified": false,
  "family_name": "Brakus",
  "given_name": "Camden",
  "identities": {
    "user_id": "0a12757f-4b19-4e93-969e-c3a2e98fe82b",
    "connection": "Username-Password-Authentication",
    "provider": "auth0",
    "isSocial": false
  },
  "name": "Jordan Yost",
  "nickname": "Elroy",
  "updated_at": "2022-10-21T04:09:54.622Z",
  "user_id": "auth0|0a12757f-4b19-4e93-969e-c3a2e98fe82b"
}

If you configure the connector like this:

{
  "property_columns": "blocked,created_at,name",
  "event_column": "identities.connection",
  "identity_column": "email",
  "timestamp_column": "updated_at"
}

The final data will be transformed to

{
  "identity": "evalyn_shields@hotmail.com",
  "event": "Username-Password-Authentication",
  "timestamp": "2022-10-21T04:09:54.622Z", 
  "properties": {
    "blocked": false,
    "created_at": "2022-10-21T04:09:54.622Z",
    "name": "Jordan Yost"
  }
}

Add User Properties

This API allows you to attach custom properties to any identified users from your servers.

The following is the sample cURL command:

curl \
  -X POST https://heapanalytics.com/api/add_user_properties\
  -H "Content-Type: application/json" \
  -d '{
    "app_id": "11",
    "identity": "bob@example.com",
    "properties": {
      "age": "25",
      "language": "English",
      "profession": "Scientist",
      "email": "bob2@example2.com"
    }
  }' 

There are 2 properties in the request body.

• identity: An identity, typically corresponding to an existing user.
• properties: An object with key-value properties you want associated with the event.

For Add User Properties, You need to configure the following 2 fields in airbyte.

• Identity Column: The attribute name from the source data populated to identity.
• property_columns: The attribute names from the source data populated to object properties. If you want to pick multiple attributes, split the names by comma(,). If you want to pick ALL attributes, simply put asterisk(*).

Note that, if you want to reference an attribute name in an object or an embedded object. You can the daisy-chained (.) connections. Let's use an example to illustrate it.

The source data is a json

{
  "blocked": false,
  "created_at": "2022-10-21T04:09:59.328Z",
  "email": "marielle.murazik8@hotmail.com",
  "email_verified": false,
  "family_name": "Gutkowski",
  "given_name": "Alysha",
  "identities": {
    "user_id": "26d8952b-2e1e-4b79-b2aa-e363f062701a",
    "connection": "Username-Password-Authentication",
    "provider": "auth0",
    "isSocial": false
  },
  "name": "Lynn Crooks",
  "nickname": "Noe",
  "updated_at": "2022-10-21T04:09:59.328Z",
  "user_id": "auth0|26d8952b-2e1e-4b79-b2aa-e363f062701a",
}

If you configure the connector like this:

{
  "property_columns": "identities_provider,created_at,nickname",
  "identity_column": "user_id"
}

The final data will be transformed to

{
  "identity": "auth0|26d8952b-2e1e-4b79-b2aa-e363f062701a",
  "properties": {
    "identities_provider": "auth0",
    "created_at": "2022-10-21T04:09:59.328Z",
    "nickname": "Neo"
  }
}

Add Account Properties

This API allows you to attach custom account properties to users.

The following is the sample cURL command:

curl \
  -X POST https://heapanalytics.com/api/add_account_properties\
  -H "Content-Type: application/json" \
  -d '{ 
    "app_id": "123456789", 
    "account_id": "Fake Company",
    "properties": {
      "is_in_good_standing": "true",
      "revenue_potential": "123456",
      "account_hq": "United Kingdom",
      "subscription": "Monthly"
    }
  }'

There are 2 properties in the request body.

• account_id: Used for single account updates only. An ID for this account.
• properties: Used for single account updates only. An object with key-value properties you want associated with the account.

For Add Account Properties, you need to configure the following 2 fields in airbyte.

• Account ID Column: The attribute name from the source data populated to identity.
• Property Columns: The attribute names from the source data populated to object properties. If you want to pick multiple attributes, split the names by comma(,). If you want to pick ALL attributes, simply put asterisk(*).

Note that, if you want to reference an attribute name in an object or an embedded object. You can the daisy-chained (.) connections. Let's use an example to illustrate it.

The source data is a json

{
  "blocked": false,
  "created_at": "2022-10-21T04:08:53.393Z",
  "email": "nedra14@hotmail.com",
  "email_verified": false,
  "family_name": "Tillman",
  "given_name": "Jacinto",
  "identities": {
    "user_id": "815ff3c3-84fa-4f63-b959-ac2d11efc63c",
    "connection": "Username-Password-Authentication",
    "provider": "auth0",
    "isSocial": false
  },
  "name": "Lola Conn",
  "nickname": "Kenyatta",
  "updated_at": "2022-10-21T04:08:53.393Z",
  "user_id": "auth0|815ff3c3-84fa-4f63-b959-ac2d11efc63c"
}

If you configure the connector like this:

{
  "property_columns": "family_name,email_verified,blocked",
  "account_id_column": "identities.user_id"
}

The final data will be transformed to

{
  "account_id": "815ff3c3-84fa-4f63-b959-ac2d11efc63c",
  "properties": {
    "family_name": "Tillman",
    "email_verified": false,
    "blocked": false
  }
}

Features & Supported sync modes

Feature	Supported?(Yes/No)
Ful-Refresh Overwrite	Yes
Ful-Refresh Append	Yes
Incremental Append	Yes
Incremental Append-Deduplicate	Yes

Rate Limiting & Performance Considerations

Currently, the API Client for Heap Analytics sends one http request per row of source data. It slows down the performance if you have massive data to load to Heap. There is a bulk API offered under Heap, but it's not implemented in the first version.

Please kindly provide your feedback, I am happy to cache the transformed data in memory and load them to the bulk API.

Future improvements:

• Implement the Bulk API that loads multiple rows of data to Heap Analytics.

Changelog

Version	Date	Pull Request	Subject
0.1.0	2022-10-26	18530	Initial Release