airbyte/docs/integrations/sources/hubspot.md

HubSpot

This page contains the setup guide and reference information for the HubSpot source connector.

Prerequisites

Chose start date which is any data before this date will not be replicated and should be UTC date and time in the format 2017-01-25T00:00:00Z.

Setup guide

Step 1: Set up HubSpot

If you are using OAuth, most of the streams require the appropriate scopes enabled for the API account.

Stream	Required Scope
campaigns	content
companies	contacts
contact_lists	contacts
contacts	contacts
contacts_list_memberships	contacts
deal_pipelines	either the contacts scope (to fetch deals pipelines) or the tickets scope.
deals	contacts
email_events	content
engagements	contacts
engagements_emails	sales-email-read
feedback_submissions	crm.objects.feedback_submissions.read
forms	forms
form_submissions	forms
line_items	e-commerce
owners	contacts
products	e-commerce
property_history	contacts
quotes	no scope required
subscription_changes	content
tickets	tickets
workflows	automation

Step 2: Set up the HubSpot connector in Airbyte

For Airbyte Cloud:

1. Log into your Airbyte Cloud account.
2. In the left navigation bar, click Sources. In the top-right corner, click +new source.
3. On the Set up the source page, enter the name for the HubSpot connector and select HubSpot from the Source type dropdown.
4. Click Authenticate your account to sign in with Google and authorize your account.
5. Fill out a start date.
6. You're done.

For Airbyte OSS:

1. Fill out a API Key.
2. Fill out a start date.
3. You're done.

To obtain the API Key for the account, go to settings -> integrations (under the account banner) -> API Key. If you already have an API Key you can use that. Otherwise, generate a new one. See docs for more details.

Supported sync modes

The HubSpot source connector supports the following sync modes:

Feature	Supported?
Full Refresh Sync	Yes
Incremental Sync	Yes
Replicate Incremental Deletes	No
SSL connection	Yes

Supported Streams

• Campaigns
• Companies (Incremental)
• Contact Lists (Incremental)
• Contacts (Incremental)
• Contacts List Memberships
• Deal Pipelines
• Deals (including Contact associations) (Incremental)
• Email Events (Incremental)
• Engagements (Incremental)
• Engagements Calls (Incremental)
• Engagements Emails (Incremental)
• Engagements Meetings (Incremental)
• Engagements Notes (Incremental)
• Engagements Tasks (Incremental)
• Feedback Submissions (Incremental)
• Forms
• Form Submissions
• Line Items (Incremental)
• Marketing Emails
• Owners
• Products (Incremental)
• Property History (Incremental)
• Quotes (Incremental)
• Subscription Changes (Incremental)
• Tickets (Incremental)
• Ticket Pipelines
• Workflows

A note on the engagements stream

Objects in the engagements stream can have one of the following types: note, email, task, meeting, call.

Depending on the type of engagement, different properties will be set for that object in the engagements_metadata table in the destination.

• A call engagement will have a corresponding engagements_metadata object with non-null values in the toNumber, fromNumber, status, externalId, durationMilliseconds, externalAccountId, recordingUrl, body, and disposition columns.
• An email engagement will have a corresponding engagements_metadata object with with non-null values in the subject, html, and text columns. In addition, there will be records in four related tables, engagements_metadata_from, engagements_metadata_to, engagements_metadata_cc, engagements_metadata_bcc.
• A meeting engagement will have a corresponding engagements_metadata object with non-null values in the body, startTime, endTime, and title columns.
• A note engagement will have a corresponding engagements_metadata object with non-null values in the body column.
• A task engagement will have a corresponding engagements_metadata object with non-null values in the body, status, and forObjectType columns.

Note: HubSpot API currently only supports quotes endpoint using API Key, using OAuth it is impossible to access this stream (as reported by community.hubspot.com).

Performance considerations

The connector is restricted by normal HubSpot rate limitations.

Some streams, such as workflows need to be enabled before they can be read using a connector authenticated using an API Key. If reading a stream that is not enabled, a log message returned to the output and the sync operation only sync the other streams available.

Example of the output message when trying to read workflows stream with missing permissions for the API Key:

{
    "type": "LOG",
    "log": {
        "level": "WARN",
        "message": 'Stream `workflows` cannot be proceed. This API Key (EXAMPLE_API_KEY) does not have proper permissions! (requires any of [automation-access])'
    }
}

HubSpot's API will rate limit the amount of records you can sync daily, so make sure that you are on the appropriate plan if you are planning on syncing more than 250,000 records per day.

Changelog

Version	Date	Pull Request	Subject
0.1.62	2022-06-01	00000	Add line items to deals stream
0.1.61	2022-05-25	13381	Requests scopes as optional instead of required
0.1.60	2022-05-25	13159	Use RFC3339 datetime
0.1.59	2022-05-10	12711	Ensure oauth2.0 token has all needed scopes in "check" command
0.1.58	2022-05-04	12482	Update input configuration copy
0.1.57	2022-05-04	12198	Add deals associations for quotes
0.1.56	2022-05-02	12515	Extra logs for troubleshooting 403 errors
0.1.55	2022-04-28	12424	Correct schema for ticket_pipeline stream
0.1.54	2022-04-28	12335	Mock time slep in unit test s
0.1.53	2022-04-20	12230	Change spec json to yaml format
0.1.52	2022-03-25	11423	Add tickets associations to engagements streams
0.1.51	2022-03-24	11321	Fix updated at field non exists issue
0.1.50	2022-03-22	11266	Fix Engagements Stream Pagination
0.1.49	2022-03-17	11218	Anchor hyperlink in input configuration
0.1.48	2022-03-16	11105	Fix float numbers, upd docs
0.1.47	2022-03-15	11121	Add partition keys where appropriate
0.1.46	2022-03-14	10700	Handle 10k+ records reading in Hubspot streams
0.1.45	2022-03-04	10707	Remove stage history from deals stream to increase efficiency
0.1.44	2022-02-24	9027	Add associations companies to deals, ticket and contact stream
0.1.43	2022-02-24	10576	Cast timestamp to date/datetime
0.1.42	2022-02-22	10492	Add date-time format to datetime fields
0.1.41	2022-02-21	10177	Migrate to CDK
0.1.40	2022-02-10	10142	Add associations to ticket stream
0.1.39	2022-02-10	10055	Bug fix: reading not initialized stream
0.1.38	2022-02-03	9786	Add new streams for engagements(calls, emails, meetings, notes and tasks)
0.1.37	2022-01-27	9555	Getting form_submission for all forms
0.1.36	2022-01-22	7784	Add Property History Stream
0.1.35	2021-12-24	9081	Add Feedback Submissions stream and update Ticket Pipelines stream
0.1.34	2022-01-20	9641	Add more fields for email_events stream
0.1.33	2022-01-14	8887	More efficient support for incremental updates on Companies, Contact, Deals and Engagement streams
0.1.32	2022-01-13	8011	Add new stream form_submissions
0.1.31	2022-01-11	9385	Remove auto-generated properties from Engagements stream
0.1.30	2021-01-10	9129	Created Contacts list memberships streams
0.1.29	2021-12-17	8699	Add incremental sync support for companies, contact_lists, contacts, deals, line_items, products, quotes, tickets streams
0.1.28	2021-12-15	8429	Update fields and descriptions
0.1.27	2021-12-09	8658	Fixed config backward compatibility issue by allowing additional properties in the spec
0.1.26	2021-11-30	8329	Removed 'skip_dynamic_fields' config param
0.1.25	2021-11-23	8216	Add skip dynamic fields for testing only
0.1.24	2021-11-09	7683	Fix name issue 'Hubspot' -> 'HubSpot'
0.1.23	2021-11-08	7730	Fix OAuth flow schema
0.1.22	2021-11-03	7562	Migrate Hubspot source to CDK structure
0.1.21	2021-10-27	7405	Change of package import from urllib to urllib.parse
0.1.20	2021-10-26	7393	Hotfix for split_properties function, add the length of separator symbol ,(%2C in HTTP format) to the checking of the summary URL length
0.1.19	2021-10-26	6954	Fix issue with getting 414 HTTP error for streams
0.1.18	2021-10-18	5840	Add new marketing emails (with statistics) stream
0.1.17	2021-10-14	6995	Update discover method: disable quotes stream when using OAuth config
0.1.16	2021-09-27	6465	Implement OAuth support. Use CDK authenticator instead of connector specific authenticator
0.1.15	2021-09-23	6374	Use correct schema for owners stream
0.1.14	2021-09-08	5693	Include deal_to_contact association when pulling deal stream and include contact ID in contact stream
0.1.13	2021-09-08	5834	Fixed array fields without items property in schema
0.1.12	2021-09-02	5798	Treat empty string values as None for field with format to fix normalization errors
0.1.11	2021-08-26	5685	Remove all date-time format from schemas
0.1.10	2021-08-17	5463	Fix fail on reading stream using API Key without required permissions
0.1.9	2021-08-11	5334	Fix empty strings inside float datatype
0.1.8	2021-08-06	5250	Fix issue with printing exceptions
0.1.7	2021-07-27	4913	Update fields schema