jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

feat(source-hubspot) migrate to manifest only (#60855)

Browse Source 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
This commit is contained in:
Daryna Ishchenko
2025-05-28 12:07:03 +03:00
committed by GitHub
parent d85a6530a0
commit 4ae6b9ca60
48 changed files with 763 additions and 7674 deletions
Download Patch File Download Diff File Expand all files Collapse all files

105
airbyte-integrations/connectors/source-hubspot/README.md
View File

@@ -1,104 +1,39 @@
# Hubspot source connector
# HubSpot
This directory contains the manifest-only connector for `source-hubspot`.
This is the repository for the Hubspot source connector, written in Python.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/hubspot).
## Local development
### Prerequisites
- Python (~=3.9)
- Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation)
### Installing the connector
From this connector directory, run:
```bash
poetry install --with dev
```
### Create credentials
## Documentation reference:
Visit `https://developers.hubspot.com/` for API documentation
## Authentication setup
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/hubspot)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_hubspot/spec.yaml` file.
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
See `sample_files/sample_config.json` for a sample config file.
### Locally running the connector
## Usage
There are multiple ways to use this connector:
- You can use this connector as any other connector in Airbyte Marketplace.
- You can load this connector in `pyairbyte` using `get_source`!
- You can open this connector in Connector Builder, edit it, and publish to your workspaces.
```
poetry run source-hubspot spec
poetry run source-hubspot check --config secrets/config.json
poetry run source-hubspot discover --config secrets/config.json
poetry run source-hubspot read --config secrets/config.json --catalog integration_tests/basic_read_catalog.json
```
Please refer to the manifest-only connector documentation for more details.
### Running unit tests
## Local Development
We recommend you use the Connector Builder to edit this connector.
To run unit tests locally, from the connector directory run:
But, if you want to develop this connector locally, you can use the following steps.
```
poetry run pytest unit_tests
```
### Building the docker image
1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
2. Run the following command to build the docker image:
### Environment Setup
You will need `airbyte-ci` installed. You can find the documentation [here](airbyte-ci).
### Build
This will create a dev image (`source-hubspot:dev`) that you can use to test the connector locally.
```bash
airbyte-ci connectors --name=source-hubspot build
```
An image will be available on your host with the tag `airbyte/source-hubspot:dev`.
### Running as a docker container
Then run any of the connector commands as follows:
```
docker run --rm airbyte/source-hubspot:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-hubspot:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-hubspot:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-hubspot:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```
### Running our CI test suite
You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
### Test
This will run the acceptance tests for the connector.
```bash
airbyte-ci connectors --name=source-hubspot test
```
### Customizing acceptance Tests
Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
### Dependency Management
All of your dependencies should be managed via Poetry.
To add a new dependency, run:
```bash
poetry add <package-name>
```
Please commit the changes to `pyproject.toml` and `poetry.lock` files.
## Publishing a new version of the connector
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-hubspot test`
2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
- bump the `dockerImageTag` value in in `metadata.yaml`
- bump the `version` value in `pyproject.toml`
3. Make sure the `metadata.yaml` content is up to date.
4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/hubspot.md`).
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
6. Pat yourself on the back for being an awesome contributor.
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.

26
airbyte-integrations/connectors/source-hubspot/acceptance-test-config.yml
View File

@@ -34,34 +34,8 @@ acceptance_tests:
bypass_reason: Unable to populate (cost $20/month) - covered by integration tests
- name: owners_archived
bypass_reason: Unable to populate - covered by integration tests
- name: tickets_web_analytics
bypass_reason: Unable to populate - covered by integration tests
- name: deals_web_analytics
bypass_reason: Unable to populate - covered by integration tests
- name: companies_web_analytics
bypass_reason: Unable to populate - covered by integration tests
- name: deals_archived
bypass_reason: No value in re-create every 90 days
- name: engagements_calls_web_analytics
bypass_reason: Unable to populate - covered by integration tests
- name: engagements_emails_web_analytics
bypass_reason: Unable to populate - covered by integration tests
- name: engagements_meetings_web_analytics
bypass_reason: Unable to populate - covered by integration tests
- name: engagements_notes_web_analytics
bypass_reason: Unable to populate - covered by integration tests
- name: engagements_tasks_web_analytics
bypass_reason: Unable to populate - covered by integration tests
- name: goals_web_analytics
bypass_reason: Unable to populate - covered by integration tests
- name: line_items_web_analytics
bypass_reason: Unable to populate - covered by integration tests
- name: products_web_analytics
bypass_reason: Unable to populate - covered by integration tests
- name: pets_web_analytics
bypass_reason: Unable to populate - covered by integration tests
- name: cars_web_analytics
bypass_reason: Unable to populate - covered by integration tests
full_refresh:
tests:
- config_path: secrets/config.json

0
airbyte-integrations/connectors/source-hubspot/source_hubspot/components.py → airbyte-integrations/connectors/source-hubspot/components.py
View File

9
airbyte-integrations/connectors/source-hubspot/integration_tests/basic_read_catalog.json
View File

@@ -359,15 +359,6 @@
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "contacts_web_analytics",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
}
]
}

3
airbyte-integrations/connectors/source-hubspot/integration_tests/expected_records.jsonl
View File

File diff suppressed because one or more lines are too long

9
airbyte-integrations/connectors/source-hubspot/main.py
View File

@@ -1,9 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from source_hubspot.run import run
if __name__ == "__main__":
run()

321
airbyte-integrations/connectors/source-hubspot/source_hubspot/manifest.yaml → airbyte-integrations/connectors/source-hubspot/manifest.yaml
View File

@@ -1,4 +1,4 @@
version: 6.44.0
version: 6.51.0
type: DeclarativeSource
@@ -72,7 +72,9 @@ definitions:
http_codes:
- 403
error_message: >-
Access denied (403). The authenticated user does not have permissions to access the resource. Verify your permissions to access this endpoint.
Access denied (403). The authenticated user does not have permissions to access the resource.
Verify your scopes: {{ parameters.required_scopes }} to access stream {{ parameters.name }}.
See details: https://docs.airbyte.com/integrations/sources/hubspot#step-2-configure-the-scopes-for-your-streams-private-app-only
- type: HttpResponseFilter
action: FAIL
http_codes:
@@ -173,11 +175,11 @@ definitions:
type: RecordSelector
extractor:
type: CustomRecordExtractor
class_name: source_hubspot.components.HubspotSchemaExtractor
class_name: source_declarative_manifest.components.HubspotSchemaExtractor
field_path: []
schema_transformations:
- type: CustomTransformation
class_name: source_hubspot.components.HubspotRenamePropertiesTransformation
class_name: source_declarative_manifest.components.HubspotRenamePropertiesTransformation
schema_type_identifier:
type: SchemaTypeIdentifier
key_pointer:
@@ -221,13 +223,13 @@ definitions:
path: /properties/v2/deal/properties
schema_transformations:
- type: CustomTransformation
class_name: source_hubspot.components.NewtoLegacyFieldTransformation
class_name: source_declarative_manifest.components.NewtoLegacyFieldTransformation
field_mapping:
hs_date_entered_: "hs_v2_date_entered_"
hs_date_exited_: "hs_v2_date_exited_"
hs_time_in_: "hs_v2_latest_time_in_"
- type: CustomTransformation
class_name: source_hubspot.components.HubspotRenamePropertiesTransformation
class_name: source_declarative_manifest.components.HubspotRenamePropertiesTransformation
forms_schema_loader:
$ref: "#/definitions/base_dynamic_schema_loader"
@@ -266,6 +268,8 @@ definitions:
http_method: GET
request_headers:
Content-Type: "application/json"
error_handler:
$ref: "#/definitions/base_error_handler"
record_selector:
type: RecordSelector
extractor:
@@ -279,7 +283,7 @@ definitions:
type: RecordSelector
extractor:
type: CustomRecordExtractor
class_name: source_hubspot.components.HubspotPropertyHistoryExtractor
class_name: source_declarative_manifest.components.HubspotPropertyHistoryExtractor
field_path: ["results"]
entity_primary_key: companyId
additional_keys: ["archived"]
@@ -314,12 +318,13 @@ definitions:
is_client_side_incremental: true
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: timestamp
$parameters:
name: companies_property_history
path: /crm/v3/objects/companies
extractor_field_path: results
required_scopes: crm.objects.companies.read
schema_loader:
type: InlineSchemaLoader
schema:
@@ -353,6 +358,8 @@ definitions:
http_method: GET
request_headers:
Content-Type: "application/json"
error_handler:
$ref: "#/definitions/base_error_handler"
record_selector:
type: RecordSelector
extractor:
@@ -366,7 +373,7 @@ definitions:
type: RecordSelector
extractor:
type: CustomRecordExtractor
class_name: source_hubspot.components.HubspotPropertyHistoryExtractor
class_name: source_declarative_manifest.components.HubspotPropertyHistoryExtractor
field_path: ["results"]
entity_primary_key: vid
additional_keys: ["archived"] # The old additional_keys aren't valid for the Hubspot V3 endpoint
@@ -401,12 +408,13 @@ definitions:
is_client_side_incremental: true
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: timestamp
$parameters:
name: contacts_property_history
path: /crm/v3/objects/contacts
extractor_field_path: results
required_scopes: crm.objects.contacts.read
# The Hubspot V3 API uses camel case. To retain backwards compatibility, we convert fields kebab case from V1
transformations:
- type: AddFields
@@ -472,6 +480,8 @@ definitions:
http_method: GET
request_headers:
Content-Type: "application/json"
error_handler:
$ref: "#/definitions/base_error_handler"
record_selector:
type: RecordSelector
extractor:
@@ -485,7 +495,7 @@ definitions:
type: RecordSelector
extractor:
type: CustomRecordExtractor
class_name: source_hubspot.components.HubspotPropertyHistoryExtractor
class_name: source_declarative_manifest.components.HubspotPropertyHistoryExtractor
field_path: ["results"]
entity_primary_key: dealId
additional_keys: ["archived"]
@@ -520,12 +530,13 @@ definitions:
is_client_side_incremental: true
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: timestamp
$parameters:
name: deals_property_history
path: /crm/v3/objects/deals
extractor_field_path: results
required_scopes: crm.objects.deals.read
schema_loader:
type: InlineSchemaLoader
schema:
@@ -536,6 +547,9 @@ definitions:
- id
$ref: "#/definitions/stream_base"
name: marketing_emails
$parameters:
name: marketing_emails
required_scopes: content
retriever:
$ref: "#/definitions/base_retriever"
requester:
@@ -573,6 +587,9 @@ definitions:
email_subscriptions_stream:
primary_key:
- id
$parameters:
name: email_subscriptions
required_scopes: content
$ref: "#/definitions/stream_without_pagination"
name: email_subscriptions
retriever:
@@ -600,9 +617,9 @@ definitions:
- path: ["updatedAt"]
value: "{{ record.get('updatedAt') or record['createdAt'] }}"
- type: CustomTransformation
class_name: source_hubspot.components.HubspotFlattenAssociationsTransformation
class_name: source_declarative_manifest.components.HubspotFlattenAssociationsTransformation
- type: CustomTransformation
class_name: source_hubspot.components.NewtoLegacyFieldTransformation
class_name: source_declarative_manifest.components.NewtoLegacyFieldTransformation
field_mapping:
hs_date_entered_: "hs_v2_date_entered_"
hs_date_exited_: "hs_v2_date_exited_"
@@ -617,7 +634,7 @@ definitions:
$ref: "#/definitions/base_crm_search_incremental_stream"
transformations:
- type: CustomTransformation
class_name: source_hubspot.components.NewtoLegacyFieldTransformation
class_name: source_declarative_manifest.components.NewtoLegacyFieldTransformation
field_mapping:
hs_date_entered_: "hs_v2_date_entered_"
hs_date_exited_: "hs_v2_date_exited_"
@@ -635,6 +652,7 @@ definitions:
- contacts
- line_items
cursor_filter_property_field: hs_lastmodifieddate
required_scopes: crm.objects.deals.read
schema_loader:
- type: InlineSchemaLoader
schema:
@@ -642,19 +660,22 @@ definitions:
- $ref: "#/definitions/base_dynamic_schema_loader"
schema_transformations:
- type: CustomTransformation
class_name: source_hubspot.components.NewtoLegacyFieldTransformation
class_name: source_declarative_manifest.components.NewtoLegacyFieldTransformation
field_mapping:
hs_date_entered_: "hs_v2_date_entered_"
hs_date_exited_: "hs_v2_date_exited_"
hs_time_in_: "hs_v2_latest_time_in_"
- type: CustomTransformation
class_name: source_hubspot.components.HubspotRenamePropertiesTransformation
class_name: source_declarative_manifest.components.HubspotRenamePropertiesTransformation
deals_archived_stream:
primary_key:
- id
$ref: "#/definitions/stream_base"
name: deals_archived
$parameters:
name: deals_archived
required_scopes: contacts, crm.objects.deals.read
retriever:
$ref: "#/definitions/base_retriever"
requester:
@@ -698,14 +719,14 @@ definitions:
- results
schema_normalization:
type: CustomSchemaNormalization
class_name: source_hubspot.components.EntitySchemaNormalization
class_name: source_declarative_manifest.components.EntitySchemaNormalization
transformations:
- type: AddFields
fields:
- path: ["archivedAt"]
value: "{{ record.get('archivedAt') or record['createdAt'] }}"
- type: CustomTransformation
class_name: source_hubspot.components.NewtoLegacyFieldTransformation
class_name: source_declarative_manifest.components.NewtoLegacyFieldTransformation
field_mapping:
hs_date_entered_: "hs_v2_date_entered_"
hs_date_exited_: "hs_v2_date_exited_"
@@ -721,7 +742,7 @@ definitions:
is_client_side_incremental: true
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: archivedAt
schema_loader:
- $ref: "#/definitions/deals_archived_schema_loader"
@@ -734,6 +755,9 @@ definitions:
- id
$ref: "#/definitions/stream_base"
name: forms
$parameters:
name: forms
required_scopes: forms
retriever:
$ref: "#/definitions/base_retriever"
requester:
@@ -795,7 +819,7 @@ definitions:
is_client_side_incremental: true
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: updatedAt
schema_loader:
- $ref: "#/definitions/forms_schema_loader"
@@ -806,6 +830,9 @@ definitions:
form_submissions_stream:
$ref: "#/definitions/stream_base"
name: form_submissions
$parameters:
name: form_submissions
required_scopes: forms
retriever:
$ref: "#/definitions/base_retriever"
requester:
@@ -854,7 +881,7 @@ definitions:
is_client_side_incremental: true
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: updatedAt
cursor_format: "%ms"
schema_loader:
@@ -867,6 +894,9 @@ definitions:
- id
$ref: "#/definitions/stream_base"
name: owners
$parameters:
name: owners
required_scopes: crm.objects.owners.read
retriever:
$ref: "#/definitions/base_retriever"
requester:
@@ -890,7 +920,7 @@ definitions:
is_client_side_incremental: true
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: updatedAt
schema_loader:
type: InlineSchemaLoader
@@ -902,6 +932,9 @@ definitions:
- id
$ref: "#/definitions/stream_base"
name: owners_archived
$parameters:
name: owners_archived
required_scopes: crm.objects.owners.read
retriever:
$ref: "#/definitions/base_retriever"
requester:
@@ -927,7 +960,7 @@ definitions:
is_client_side_incremental: true
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: updatedAt
schema_loader:
type: InlineSchemaLoader
@@ -939,6 +972,13 @@ definitions:
- id
$ref: "#/definitions/stream_base"
name: ticket_pipelines
$parameters:
name: ticket_pipelines
required_scopes: >-
media_bridge.read, tickets, crm.schemas.custom.read, e-commerce, timeline, contacts, crm.schemas.contacts.read,
crm.objects.contacts.read, crm.objects.contacts.write, crm.objects.deals.read, crm.schemas.quotes.read,
crm.objects.deals.write, crm.objects.companies.read, crm.schemas.companies.read, crm.schemas.deals.read,
crm.schemas.line_items.read, crm.objects.companies.write
retriever:
$ref: "#/definitions/base_retriever"
requester:
@@ -959,7 +999,7 @@ definitions:
$ref: "#/schemas/ticket_pipelines"
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: updatedAt
campaigns_stream:
@@ -967,6 +1007,9 @@ definitions:
name: campaigns
primary_key:
- id
$parameters:
name: campaigns
required_scopes: crm.lists.read
retriever:
$ref: "#/definitions/base_retriever"
requester:
@@ -1014,7 +1057,7 @@ definitions:
$ref: "#/schemas/campaigns"
transformations:
- type: CustomTransformation
class_name: source_hubspot.components.AddFieldsFromEndpointTransformation
class_name: source_declarative_manifest.components.AddFieldsFromEndpointTransformation
requester:
$ref: "#/definitions/base_requester"
path: "/email/public/v1/campaigns/{{ stream_slice['parent_id'] }}"
@@ -1034,7 +1077,7 @@ definitions:
prefix: counters_
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: lastUpdatedTime
cursor_format: "%ms"
@@ -1042,6 +1085,9 @@ definitions:
$ref: "#/definitions/stream_base"
name: contact_lists
primary_key: listId
$parameters:
name: contact_lists
required_scopes: crm.lists.read
retriever:
$ref: "#/definitions/base_retriever"
requester:
@@ -1102,6 +1148,9 @@ definitions:
name: deal_pipelines
primary_key:
- pipelineId
$parameters:
name: deal_pipelines
required_scopes: crm.objects.contacts.read
retriever:
type: SimpleRetriever
requester:
@@ -1130,7 +1179,7 @@ definitions:
is_client_side_incremental: true
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: updatedAt
cursor_format: "%ms"
schema_loader:
@@ -1148,6 +1197,9 @@ definitions:
name: workflows
primary_key:
- id
$parameters:
name: workflows
required_scopes: automation
retriever:
type: SimpleRetriever
requester:
@@ -1191,7 +1243,7 @@ definitions:
$ref: "#/schemas/workflows"
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: updatedAt
cursor_format: "%ms"
@@ -1199,6 +1251,9 @@ definitions:
$ref: "#/definitions/stream_base"
name: email_events
primary_key: id
$parameters:
name: email_events
required_scopes: content
retriever:
$ref: "#/definitions/base_retriever"
requester:
@@ -1254,7 +1309,7 @@ definitions:
$ref: "#/schemas/email_events"
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: created
cursor_format: "%ms"
@@ -1262,11 +1317,14 @@ definitions:
$ref: "#/definitions/stream_base"
name: engagements
primary_key: id
$parameters:
name: engagements
required_scopes: crm.objects.companies.read, crm.objects.contacts.read, crm.objects.deals.read, tickets, e-commerce
retriever:
$ref: "#/definitions/base_retriever"
requester:
type: CustomRequester
class_name: source_hubspot.components.EngagementsHttpRequester
class_name: source_declarative_manifest.components.EngagementsHttpRequester
url_base: https://api.hubapi.com
authenticator:
$ref: "#/definitions/authenticator"
@@ -1338,7 +1396,7 @@ definitions:
$ref: "#/schemas/engagements"
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: lastUpdated
cursor_format: "%ms"
@@ -1385,14 +1443,14 @@ definitions:
$ref: "#/definitions/base_selector"
schema_normalization:
type: CustomSchemaNormalization
class_name: source_hubspot.components.EntitySchemaNormalization
class_name: source_declarative_manifest.components.EntitySchemaNormalization
transformations:
- type: AddFields
fields:
- path: ["updatedAt"]
value: "{{ record.get('updatedAt') or record['createdAt'] }}"
- type: CustomTransformation
class_name: source_hubspot.components.HubspotFlattenAssociationsTransformation
class_name: source_declarative_manifest.components.HubspotFlattenAssociationsTransformation
- type: DpathFlattenFields
field_path:
- properties
@@ -1450,20 +1508,20 @@ definitions:
type: DefaultPaginator
pagination_strategy:
type: CustomPaginationStrategy
class_name: source_hubspot.components.HubspotCRMSearchPaginationStrategy
class_name: source_declarative_manifest.components.HubspotCRMSearchPaginationStrategy
page_size: 200
record_selector:
type: RecordSelector
extractor:
type: CustomRecordExtractor
class_name: source_hubspot.components.HubspotAssociationsExtractor
class_name: source_declarative_manifest.components.HubspotAssociationsExtractor
field_path:
- results
entity_primary_key: "{{ parameters['entity'] }}"
associations_list: "{{ parameters['associations'] }}"
schema_normalization:
type: CustomSchemaNormalization
class_name: source_hubspot.components.EntitySchemaNormalization
class_name: source_declarative_manifest.components.EntitySchemaNormalization
transformations:
# - modular to allow support for NewtoLegacyFieldTransformation
- type: DpathFlattenFields
@@ -1485,6 +1543,7 @@ definitions:
associations:
- contacts
cursor_filter_property_field: hs_lastmodifieddate
required_scopes: crm.objects.contacts.read, crm.objects.companies.read
schema_loader:
- type: InlineSchemaLoader
schema:
@@ -1504,6 +1563,7 @@ definitions:
- deals
- tickets
cursor_filter_property_field: hs_lastmodifieddate
required_scopes: crm.objects.contacts.read
schema_loader:
- type: InlineSchemaLoader
schema:
@@ -1523,6 +1583,7 @@ definitions:
- deals
- tickets
cursor_filter_property_field: hs_lastmodifieddate
required_scopes: crm.objects.contacts.read, sales-email-read
schema_loader:
- type: InlineSchemaLoader
schema:
@@ -1541,6 +1602,7 @@ definitions:
- contacts
- deals
- tickets
required_scopes: crm.objects.contacts.read
cursor_filter_property_field: hs_lastmodifieddate
schema_loader:
- type: InlineSchemaLoader
@@ -1561,6 +1623,7 @@ definitions:
- deals
- tickets
cursor_filter_property_field: hs_lastmodifieddate
required_scopes: crm.objects.contacts.read
schema_loader:
- type: InlineSchemaLoader
schema:
@@ -1580,6 +1643,7 @@ definitions:
- deals
- tickets
cursor_filter_property_field: hs_lastmodifieddate
required_scopes: crm.objects.contacts.read
schema_loader:
- type: InlineSchemaLoader
schema:
@@ -1590,6 +1654,9 @@ definitions:
$ref: "#/definitions/stream_base"
name: subscription_changes
primary_key: []
$parameters:
name: subscription_changes
required_scopes: content
retriever:
$ref: "#/definitions/base_retriever"
requester:
@@ -1643,7 +1710,7 @@ definitions:
$ref: "#/schemas/subscription_changes"
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: timestamp
cursor_format: "%ms"
@@ -1681,7 +1748,7 @@ definitions:
$ref: "#/definitions/base_selector"
schema_normalization:
type: CustomSchemaNormalization
class_name: source_hubspot.components.EntitySchemaNormalization
class_name: source_declarative_manifest.components.EntitySchemaNormalization
incremental_sync:
type: DatetimeBasedCursor
cursor_field: updatedAt
@@ -1715,11 +1782,11 @@ definitions:
type: RecordSelector
extractor:
type: CustomRecordExtractor
class_name: source_hubspot.components.HubspotSchemaExtractor
class_name: source_declarative_manifest.components.HubspotSchemaExtractor
field_path: []
schema_transformations:
- type: CustomTransformation
class_name: source_hubspot.components.HubspotRenamePropertiesTransformation
class_name: source_declarative_manifest.components.HubspotRenamePropertiesTransformation
schema_type_identifier:
type: SchemaTypeIdentifier
key_pointer: ["name"]
@@ -1749,7 +1816,7 @@ definitions:
current_type: phone_number
state_migrations:
- type: CustomStateMigration
class_name: source_hubspot.components.MigrateEmptyStringState
class_name: source_declarative_manifest.components.MigrateEmptyStringState
cursor_field: updatedAt
cursor_format: "%Y-%m-%dT%H:%M:%S.%fZ"
@@ -1758,6 +1825,8 @@ definitions:
name: goals
$parameters:
crm_object_name: goal_targets
name: goals
required_scopes: crm.objects.goals.read
schema_loader:
- $ref: "#/definitions/base_crm_object_stream/schema_loader"
- type: InlineSchemaLoader
@@ -1768,7 +1837,9 @@ definitions:
$ref: "#/definitions/base_crm_object_stream"
name: products
$parameters:
name: products
crm_object_name: product
required_scopes: e-commerce
schema_loader:
- $ref: "#/definitions/base_crm_object_stream/schema_loader"
- type: InlineSchemaLoader
@@ -1780,6 +1851,8 @@ definitions:
name: line_items
$parameters:
crm_object_name: line_item
name: line_items
required_scopes: e-commerce, crm.objects.line_items.read
schema_loader:
- $ref: "#/definitions/base_crm_object_stream/schema_loader"
- type: InlineSchemaLoader
@@ -1796,9 +1869,9 @@ definitions:
- path: ["updatedAt"]
value: "{{ record.get('updatedAt') or record['createdAt'] }}"
- type: CustomTransformation
class_name: source_hubspot.components.HubspotFlattenAssociationsTransformation
class_name: source_declarative_manifest.components.HubspotFlattenAssociationsTransformation
- type: CustomTransformation
class_name: source_hubspot.components.NewtoLegacyFieldTransformation
class_name: source_declarative_manifest.components.NewtoLegacyFieldTransformation
field_mapping:
hs_lifecyclestage_: "hs_v2_date_entered_"
hs_date_exited_: "hs_v2_date_exited_"
@@ -1813,7 +1886,7 @@ definitions:
$ref: "#/definitions/base_crm_search_incremental_stream"
transformations:
- type: CustomTransformation
class_name: source_hubspot.components.NewtoLegacyFieldTransformation
class_name: source_declarative_manifest.components.NewtoLegacyFieldTransformation
field_mapping:
hs_lifecyclestage_: "hs_v2_date_entered_"
hs_date_exited_: "hs_v2_date_exited_"
@@ -1830,6 +1903,7 @@ definitions:
- contacts
- companies
cursor_filter_property_field: lastmodifieddate
required_scopes: crm.objects.contacts.read
schema_loader:
- type: InlineSchemaLoader
schema:
@@ -1837,13 +1911,13 @@ definitions:
- $ref: "#/definitions/base_dynamic_schema_loader"
schema_transformations:
- type: CustomTransformation
class_name: source_hubspot.components.NewtoLegacyFieldTransformation
class_name: source_declarative_manifest.components.NewtoLegacyFieldTransformation
field_mapping:
hs_lifecyclestage_: "hs_v2_date_entered_"
hs_date_exited_: "hs_v2_date_exited_"
hs_time_in_: "hs_v2_latest_time_in_"
- type: CustomTransformation
class_name: source_hubspot.components.HubspotRenamePropertiesTransformation
class_name: source_declarative_manifest.components.HubspotRenamePropertiesTransformation
deal_splits_stream:
type: StateDelegatingStream
@@ -1853,6 +1927,7 @@ definitions:
name: deal_splits
entity: deal_split
cursor_filter_property_field: hs_lastmodifieddate
required_scopes: crm.objects.deals.read
schema_loader:
- type: InlineSchemaLoader
schema:
@@ -1870,6 +1945,7 @@ definitions:
- companies
- contacts
cursor_filter_property_field: hs_lastmodifieddate
required_scopes: crm.objects.contacts.read, crm.objects.companies.read, crm.objects.leads.read
schema_loader:
- type: InlineSchemaLoader
schema:
@@ -1888,6 +1964,7 @@ definitions:
- contacts
- deals
cursor_filter_property_field: hs_lastmodifieddate
required_scopes: tickets
schema_loader:
- type: InlineSchemaLoader
schema:
@@ -1938,7 +2015,7 @@ dynamic_streams:
schema_properties: "schema_properties_placeholder" # will be set by components_resolver
schema_loader:
type: CustomSchemaLoader
class_name: source_hubspot.components.HubspotCustomObjectsSchemaLoader
class_name: source_declarative_manifest.components.HubspotCustomObjectsSchemaLoader
full_refresh_stream:
name: custom_object_stream_name # will be set by components_resolver
primary_key:
@@ -1968,7 +2045,7 @@ dynamic_streams:
$ref: "#/definitions/base_selector"
schema_normalization:
type: CustomSchemaNormalization
class_name: source_hubspot.components.EntitySchemaNormalization
class_name: source_declarative_manifest.components.EntitySchemaNormalization
transformations:
- type: AddFields
fields:
@@ -2014,7 +2091,7 @@ dynamic_streams:
type: DefaultPaginator
pagination_strategy:
type: CustomPaginationStrategy
class_name: source_hubspot.components.HubspotCRMSearchPaginationStrategy
class_name: source_declarative_manifest.components.HubspotCRMSearchPaginationStrategy
page_size: 200
page_size_option:
type: RequestOption
@@ -2028,7 +2105,7 @@ dynamic_streams:
- results
schema_normalization:
type: CustomSchemaNormalization
class_name: source_hubspot.components.EntitySchemaNormalization
class_name: source_declarative_manifest.components.EntitySchemaNormalization
transformations:
- type: DpathFlattenFields
field_path: ["properties"]
@@ -2137,6 +2214,148 @@ dynamic_streams:
- schema_properties
value: "{{ components_values.properties }}"
spec:
type: Spec
documentationUrl: https://docs.airbyte.com/integrations/sources/hubspot
connection_specification:
$schema: http://json-schema.org/draft-07/schema#
title: HubSpot Source Spec
type: object
required:
- credentials
additionalProperties: true
properties:
start_date:
type: string
title: Start date
pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$
description: >-
UTC date and time in the format 2017-01-25T00:00:00Z. Any data before
this date will not be replicated. If not set, "2006-06-01T00:00:00Z" (Hubspot creation date) will be used as start date.
It's recommended to provide relevant to your data start date value to optimize synchronization.
examples:
- "2017-01-25T00:00:00Z"
format: date-time
credentials:
title: Authentication
description: Choose how to authenticate to HubSpot.
type: object
oneOf:
- type: object
title: OAuth
required:
- client_id
- client_secret
- refresh_token
- credentials_title
properties:
credentials_title:
type: string
title: Auth Type
description: Name of the credentials
const: OAuth Credentials
order: 0
client_id:
title: Client ID
description: >-
The Client ID of your HubSpot developer application. See the <a
href="https://legacydocs.hubspot.com/docs/methods/oauth2/oauth2-quickstart">Hubspot docs</a>
if you need help finding this ID.
type: string
examples:
- "123456789000"
client_secret:
title: Client Secret
description: >-
The client secret for your HubSpot developer application. See the <a
href="https://legacydocs.hubspot.com/docs/methods/oauth2/oauth2-quickstart">Hubspot docs</a>
if you need help finding this secret.
type: string
examples:
- secret
airbyte_secret: true
refresh_token:
title: Refresh Token
description: >-
Refresh token to renew an expired access token. See the <a
href="https://legacydocs.hubspot.com/docs/methods/oauth2/oauth2-quickstart">Hubspot docs</a>
if you need help finding this token.
type: string
examples:
- refresh_token
airbyte_secret: true
- type: object
title: Private App
required:
- access_token
- credentials_title
properties:
credentials_title:
type: string
title: Auth Type
description: Name of the credentials set
const: Private App Credentials
order: 0
access_token:
title: Access token
description: >-
HubSpot Access token. See the <a
href="https://developers.hubspot.com/docs/api/private-apps">Hubspot docs</a>
if you need help finding this token.
type: string
airbyte_secret: true
enable_experimental_streams:
title: Enable experimental streams
description: If enabled then experimental streams become available for sync.
type: boolean
default: false
num_worker:
type: integer
title: Number of concurrent workers
minimum: 1
maximum: 40
default: 3
examples: [ 1, 2, 3 ]
description: The number of worker threads to use for the sync.
advanced_auth:
auth_flow_type: oauth2.0
predicate_key:
- credentials
- credentials_title
predicate_value: OAuth Credentials
oauth_config_specification:
complete_oauth_output_specification:
type: object
additionalProperties: false
properties:
refresh_token:
type: string
path_in_connector_config:
- credentials
- refresh_token
complete_oauth_server_input_specification:
type: object
additionalProperties: false
properties:
client_id:
type: string
client_secret:
type: string
complete_oauth_server_output_specification:
type: object
additionalProperties: false
properties:
client_id:
type: string
path_in_connector_config:
- credentials
- client_id
client_secret:
type: string
path_in_connector_config:
- credentials
- client_secret
# HubSpot account is limited to 110 requests every 10 seconds https://developers.hubspot.com/docs/guides/apps/api-usage/usage-details#rate-limits
concurrency_level:
type: ConcurrencyLevel

8
airbyte-integrations/connectors/source-hubspot/metadata.yaml
View File

@@ -6,11 +6,11 @@ data:
hosts:
- api.hubapi.com
connectorBuildOptions:
baseImage: docker.io/airbyte/python-connector-base:4.0.0@sha256:d9894b6895923b379f3006fa251147806919c62b7d9021b5cd125bb67d7bbe22
baseImage: docker.io/airbyte/source-declarative-manifest:6.51.0@sha256:890b109f243b8b9406f23ea7522de41025f7b3e87f6fc9710bc1e521213a276f
connectorSubtype: api
connectorType: source
definitionId: 36c891d9-4bd9-43ac-bad2-10e12756272c
dockerImageTag: 5.7.0
dockerImageTag: 5.8.0
dockerRepository: airbyte/source-hubspot
documentationUrl: https://docs.airbyte.com/integrations/sources/hubspot
erdUrl: https://dbdocs.io/airbyteio/source-hubspot?view=relationships
@@ -21,7 +21,7 @@ data:
name: HubSpot
remoteRegistries:
pypi:
enabled: true
enabled: false
packageName: airbyte-source-hubspot
registryOverrides:
cloud:
@@ -86,7 +86,7 @@ data:
- deals
supportLevel: certified
tags:
- language:python
- language:manifest-only
- cdk:low-code
connectorTestSuitesOptions:
- suite: liveTests

40
airbyte-integrations/connectors/source-hubspot/pyproject.toml
View File

@@ -1,40 +0,0 @@
[build-system]
requires = [ "poetry-core>=1.0.0",]
build-backend = "poetry.core.masonry.api"
[tool.poetry]
version = "5.7.0"
name = "source-hubspot"
description = "Source implementation for HubSpot."
authors = [ "Airbyte <contact@airbyte.io>",]
license = "ELv2"
readme = "README.md"
documentation = "https://docs.airbyte.com/integrations/sources/hubspot"
homepage = "https://airbyte.com"
repository = "https://github.com/airbytehq/airbyte"
[[tool.poetry.packages]]
include = "source_hubspot"
[tool.poetry.dependencies]
python = "^3.10,<3.12"
airbyte-cdk = "^6"
pendulum = "<3.0.0"
[tool.poetry.scripts]
source-hubspot = "source_hubspot.run:run"
[tool.poetry.group.dev.dependencies]
requests-mock = "^1.9.3"
mock = "^5.1.0"
pytest-mock = "^3.6"
pytest = "^8.0.0"
pytz = "2024.2"
freezegun = "0.3.4"
[tool.poe]
include = [
# Shared tasks definition file(s) can be imported here.
# Run `poe` or `poe --help` to see the list of available tasks.
"${POE_GIT_DIR}/poe-tasks/poetry-connector-tasks.toml",
]

6
airbyte-integrations/connectors/source-hubspot/source_hubspot/__init__.py
View File

@@ -1,6 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from .source import SourceHubspot
__all__ = ["SourceHubspot"]

6
airbyte-integrations/connectors/source-hubspot/source_hubspot/constants.py
View File

@@ -1,6 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
OAUTH_CREDENTIALS = "OAuth Credentials"
PRIVATE_APP_CREDENTIALS = "Private App Credentials"

63
airbyte-integrations/connectors/source-hubspot/source_hubspot/errors.py
View File

@@ -1,63 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from typing import Any
import requests
from requests import HTTPError
from airbyte_cdk.models import FailureType
from airbyte_cdk.utils import AirbyteTracedException
class HubspotError(AirbyteTracedException):
"""
Base error class.
Subclassing HTTPError to avoid breaking existing code that expects only HTTPErrors.
"""
def __init__(
self,
internal_message: str = None,
message: str = None,
failure_type: FailureType = FailureType.system_error,
exception: BaseException = None,
response: requests.Response = None,
):
super().__init__(internal_message, message, failure_type, exception)
self.response = response
class HubspotTimeout(HTTPError):
"""502/504 HubSpot has processing limits in place to prevent a single client from causing degraded performance,
and these responses indicate that those limits have been hit. You'll normally only see these timeout responses
when making a large number of requests over a sustained period. If you get one of these responses,
you should pause your requests for a few seconds, then retry.
"""
class HubspotInvalidAuth(HubspotError):
"""401 Unauthorized"""
class HubspotAccessDenied(HubspotError):
"""403 Forbidden"""
class HubspotRateLimited(HTTPError):
"""429 Rate Limit Reached"""
class HubspotBadRequest(HubspotError):
"""400 Bad Request"""
class InvalidStartDateConfigError(Exception):
"""Raises when the User inputs wrong or invalid `start_date` in inout configuration"""
def __init__(self, actual_value: Any, message: str):
super().__init__(
f"The value for `start_date` entered `{actual_value}` is ivalid and could not be processed.\nPlease use the real date/time value.\nFull message: {message}"
)

120
airbyte-integrations/connectors/source-hubspot/source_hubspot/helpers.py
View File

@@ -1,120 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import abc
import urllib.parse
from typing import Iterator, List, MutableMapping
class IRecordPostProcessor(abc.ABC):
"""
The interface is designed to post process records (like group them by ID and update) after the API response is parsed and
before they are emitted up the stack.
"""
@abc.abstractmethod
def add_record(self, record: MutableMapping):
""""""
@property
@abc.abstractmethod
def flat(self):
""""""
class GroupByKey(IRecordPostProcessor):
def __init__(self, primary_key: str = None):
self._storage = {}
self._primary_key = primary_key
def add_record(self, record: MutableMapping):
record_pk = record[self._primary_key]
if record_pk not in self._storage:
self._storage[record_pk] = record
stored_props = self._storage[record_pk].get("properties")
if stored_props:
stored_props.update(record.get("properties", {}))
self._storage[record_pk]["properties"] = stored_props
@property
def flat(self):
return list(self._storage.values())
class StoreAsIs(IRecordPostProcessor):
def __init__(self):
self._storage = []
def add_record(self, record: MutableMapping):
self._storage.append(record)
@property
def flat(self):
return self._storage
class IURLPropertyRepresentation(abc.ABC):
# The value is obtained experimentally, HubSpot allows the URL length up to ~16300 symbols,
# so it was decided to limit the length of the `properties` parameter to 15000 characters.
PROPERTIES_PARAM_MAX_LENGTH = 15000
def __init__(self, properties: List[str]):
self.properties = properties
def __bool__(self):
return bool(self.properties)
@property
@abc.abstractmethod
def as_url_param(self):
""""""
@property
@abc.abstractmethod
def _term_representation(self):
""""""
def split(self) -> Iterator["IURLPropertyRepresentation"]:
summary_length = 0
local_properties = []
for property_ in self.properties:
current_property_length = len(urllib.parse.quote(self._term_representation.format(property=property_)))
if current_property_length + summary_length >= self.PROPERTIES_PARAM_MAX_LENGTH:
yield type(self)(local_properties)
local_properties = []
summary_length = 0
local_properties.append(property_)
summary_length += current_property_length
if local_properties:
yield type(self)(local_properties)
@property
def too_many_properties(self) -> bool:
# Do not iterate over the generator until the end. Here we need to know if it produces more than one record
generator = self.split()
_ = next(generator)
return next(generator, None) is not None
class APIv1Property(IURLPropertyRepresentation):
_term_representation = "property={property}&"
def as_url_param(self):
return {"property": self.properties}
class APIv2Property(IURLPropertyRepresentation):
_term_representation = "property={property}&"
def as_url_param(self):
return {"property": self.properties}
class APIv3Property(IURLPropertyRepresentation):
_term_representation = "{property},"
def as_url_param(self):
return {"properties": ",".join(self.properties)}

54
airbyte-integrations/connectors/source-hubspot/source_hubspot/run.py
View File

@@ -1,54 +0,0 @@
#
# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
#
import sys
import traceback
from datetime import datetime
from typing import List
from orjson import orjson
from airbyte_cdk.entrypoint import AirbyteEntrypoint, launch, logger
from airbyte_cdk.exception_handler import init_uncaught_exception_handler
from airbyte_cdk.models import AirbyteErrorTraceMessage, AirbyteMessage, AirbyteMessageSerializer, AirbyteTraceMessage, TraceType, Type
from source_hubspot import SourceHubspot
def _get_source(args: List[str]):
catalog_path = AirbyteEntrypoint.extract_catalog(args)
config_path = AirbyteEntrypoint.extract_config(args)
state_path = AirbyteEntrypoint.extract_state(args)
try:
return SourceHubspot(
SourceHubspot.read_catalog(catalog_path) if catalog_path else None,
SourceHubspot.read_config(config_path) if config_path else None,
SourceHubspot.read_state(state_path) if state_path else None,
)
except Exception as error:
print(
orjson.dumps(
AirbyteMessageSerializer.dump(
AirbyteMessage(
type=Type.TRACE,
trace=AirbyteTraceMessage(
type=TraceType.ERROR,
emitted_at=int(datetime.now().timestamp() * 1000),
error=AirbyteErrorTraceMessage(
message=f"Error starting the sync. This could be due to an invalid configuration or catalog. Please contact Support for assistance. Error: {error}",
stack_trace=traceback.format_exc(),
),
),
)
)
).decode()
)
return None
def run() -> None:
init_uncaught_exception_handler(logger)
_args = sys.argv[1:]
source = _get_source(_args)
if source:
launch(source, _args)

32
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/companies.json
View File

@@ -1,32 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "Unique identifier for the company",
"type": ["null", "string"]
},
"createdAt": {
"description": "Date and time when the company was created",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "Date and time when the company was last updated",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates whether the company is archived or active",
"type": ["null", "boolean"]
},
"contacts": {
"description": "List of contacts associated with the company",
"type": ["null", "array"],
"items": {
"description": "Details of individual contacts",
"type": "string"
}
}
}
}

32
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/contacts.json
View File

@@ -1,32 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "Unique identifier for the contact.",
"type": ["null", "string"]
},
"createdAt": {
"description": "Date and time when the contact was created.",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "Date and time when the contact was last updated.",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates if the contact is archived or not.",
"type": ["null", "boolean"]
},
"companies": {
"description": "List of companies associated with the contact.",
"type": ["null", "array"],
"items": {
"description": "Details of a company associated with the contact.",
"type": ["null", "string"]
}
}
}
}

90
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/deal_splits.json
View File

@@ -1,90 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "Unique identifier for the deal split",
"type": ["null", "string"]
},
"properties": {
"description": "Deal Split properties",
"type": ["null", "object"],
"properties": {
"hs_deal_split_amount": {
"description": "Total amount of the deal split",
"type": ["null", "string"]
},
"hs_deal_split_percentage": {
"description": "Total percentage of the deal split",
"type": ["null", "string"]
},
"hs_createdate": {
"description": "Creation date of the deal",
"type": ["null", "string"],
"format": "date-time"
},
"hs_deal_id": {
"description": "Object id of the deal",
"type": ["null", "string"]
},
"hs_object_id": {
"description": "Unique object ID for the deal split",
"type": ["null", "string"]
},
"hs_lastmodifieddate": {
"description": "Last modified date of the deal",
"type": ["null", "string"],
"format": "date-time"
},
"hubspot_owner_id": {
"description": "Owner ID of the deal split",
"type": ["null", "string"]
}
}
},
"properties_hs_deal_split_amount": {
"description": "Total amount of the deal split",
"type": ["null", "string"]
},
"properties_hs_deal_split_percentage": {
"description": "Total percentage of the deal split",
"type": ["null", "string"]
},
"properties_hs_createdate": {
"description": "Creation date of the deal",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_deal_id": {
"description": "Object id of the deal",
"type": ["null", "string"]
},
"properties_hs_object_id": {
"description": "Unique object ID for the deal split",
"type": ["null", "string"]
},
"properties_hs_lastmodifieddate": {
"description": "Last modified date of the deal",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hubspot_owner_id": {
"description": "Owner ID of the deal split",
"type": ["null", "string"]
},
"createdAt": {
"description": "The date and time when the deal split was created",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "The date and time when the deal split was last updated",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates if the deal is archived",
"type": ["null", "boolean"]
}
}
}

45
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/deals.json
View File

@@ -1,45 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "Unique identifier for the deal",
"type": ["null", "string"]
},
"createdAt": {
"description": "The date and time when the deal was created",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "The date and time when the deal was last updated",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates if the deal is archived",
"type": ["null", "boolean"]
},
"companies": {
"description": "Information about companies associated with the deal",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"contacts": {
"description": "Information about contacts associated with the deal",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"line_items": {
"description": "Details of line items associated with the deal",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
}
}
}

498
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/engagements_calls.json
View File

@@ -1,498 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "Unique identifier for the call engagement.",
"type": ["null", "string"]
},
"properties": {
"description": "Information related to the properties of the call engagement.",
"type": "object",
"properties": {
"hs_activity_type": {
"description": "Type of activity associated with the call engagement.",
"type": ["null", "string"]
},
"hs_all_assigned_business_unit_ids": {
"description": "IDs of all business units assigned to the call engagement.",
"type": ["null", "string"]
},
"hs_at_mentioned_owner_ids": {
"description": "IDs of owners mentioned in the call engagement.",
"type": ["null", "string"]
},
"hs_attachment_ids": {
"description": "IDs of attachments associated with the call engagement.",
"type": ["null", "string"]
},
"hs_body_preview": {
"description": "Preview of the body content of the call engagement.",
"type": ["null", "string"]
},
"hs_body_preview_html": {
"description": "HTML-formatted preview of the body content of the call engagement.",
"type": ["null", "string"]
},
"hs_body_preview_is_truncated": {
"description": "Indicates if the body preview is truncated or not.",
"type": ["null", "boolean"]
},
"hs_call_app_id": {
"description": "App ID associated with the call engagement.",
"type": ["null", "number"]
},
"hs_call_authed_url_provider": {
"description": "Provider of the authenticated call URL.",
"type": ["null", "string"]
},
"hs_call_body": {
"description": "Body content of the call engagement.",
"type": ["null", "string"]
},
"hs_call_callee_object_id": {
"description": "Object ID of the callee associated with the call engagement.",
"type": ["null", "number"]
},
"hs_call_callee_object_type": {
"description": "Type of object of the callee associated with the call engagement.",
"type": ["null", "string"]
},
"hs_call_disposition": {
"description": "Disposition of the call engagement.",
"type": ["null", "string"]
},
"hs_call_duration": {
"description": "Duration of the call engagement.",
"type": ["null", "number"]
},
"hs_call_external_account_id": {
"description": "External account ID associated with the call engagement.",
"type": ["null", "string"]
},
"hs_call_external_id": {
"description": "External ID associated with the call engagement.",
"type": ["null", "string"]
},
"hs_call_from_number": {
"description": "Phone number from which the call was made.",
"type": ["null", "string"]
},
"hs_call_has_transcript": {
"description": "Indicates if the call has a transcript or not.",
"type": ["null", "boolean"]
},
"hs_call_recording_url": {
"description": "URL of the call recording.",
"type": ["null", "string"]
},
"hs_call_source": {
"description": "Source of the call engagement.",
"type": ["null", "string"]
},
"hs_call_status": {
"description": "Status of the call.",
"type": ["null", "string"]
},
"hs_call_title": {
"description": "Title of the call engagement.",
"type": ["null", "string"]
},
"hs_call_to_number": {
"description": "Phone number to which the call was made.",
"type": ["null", "string"]
},
"hs_call_transcription_id": {
"description": "Transcription ID of the call engagement.",
"type": ["null", "number"]
},
"hs_call_video_recording_url": {
"description": "URL of the video call recording.",
"type": ["null", "string"]
},
"hs_call_zoom_meeting_uuid": {
"description": "UUID of the Zoom meeting associated with the call engagement.",
"type": ["null", "string"]
},
"hs_calls_service_call_id": {
"description": "Service call ID associated with the call engagement.",
"type": ["null", "number"]
},
"hs_created_by": {
"description": "User who created the call engagement.",
"type": ["null", "number"]
},
"hs_created_by_user_id": {
"description": "User ID of the creator of the call engagement.",
"type": ["null", "number"]
},
"hs_createdate": {
"description": "Date and time when the call engagement was created.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_engagement_source": {
"description": "Source of the engagement.",
"type": ["null", "string"]
},
"hs_engagement_source_id": {
"description": "ID of the source of the engagement.",
"type": ["null", "string"]
},
"hs_follow_up_action": {
"description": "Follow-up action required for the engagement.",
"type": ["null", "string"]
},
"hs_gdpr_deleted": {
"description": "Indicates if the engagement is deleted due to GDPR compliance.",
"type": ["null", "boolean"]
},
"hs_lastmodifieddate": {
"description": "Date and time when the call engagement was last modified.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_merged_object_ids": {
"description": "IDs of merged objects associated with the engagement.",
"type": ["null", "string"]
},
"hs_modified_by": {
"description": "User who last modified the engagement.",
"type": ["null", "number"]
},
"hs_object_id": {
"description": "Object ID of the engagement.",
"type": ["null", "number"]
},
"hs_product_name": {
"description": "Name of the product associated with the engagement.",
"type": ["null", "string"]
},
"hs_queue_membership_ids": {
"description": "IDs of queue memberships associated with the engagement.",
"type": ["null", "string"]
},
"hs_timestamp": {
"description": "Timestamp of the engagement.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_unique_creation_key": {
"description": "Unique key for creation of the engagement.",
"type": ["null", "string"]
},
"hs_unique_id": {
"description": "Unique ID associated with the engagement.",
"type": ["null", "string"]
},
"hs_unknown_visitor_conversation": {
"description": "Indicates if the conversation is with an unknown visitor.",
"type": ["null", "boolean"]
},
"hs_updated_by_user_id": {
"description": "User ID of the last user who updated the engagement.",
"type": ["null", "number"]
},
"hs_user_ids_of_all_notification_followers": {
"description": "User IDs of all notification followers.",
"type": ["null", "string"]
},
"hs_user_ids_of_all_notification_unfollowers": {
"description": "User IDs of all notification unfollowers.",
"type": ["null", "string"]
},
"hs_user_ids_of_all_owners": {
"description": "User IDs of all owners associated with the engagement.",
"type": ["null", "string"]
},
"hubspot_owner_assigneddate": {
"description": "Date and time when the owner was assigned.",
"type": ["null", "string"],
"format": "date-time"
},
"hubspot_owner_id": {
"description": "Owner ID associated with the engagement.",
"type": ["null", "string"]
},
"hubspot_team_id": {
"description": "Team ID associated with the engagement.",
"type": ["null", "string"]
},
"hs_all_owner_ids": {
"description": "IDs of all owners associated with the call engagement.",
"type": ["null", "string"]
},
"hs_all_team_ids": {
"description": "IDs of all teams associated with the call engagement.",
"type": ["null", "string"]
},
"hs_all_accessible_team_ids": {
"description": "IDs of all teams that have access to the call engagement.",
"type": ["null", "string"]
}
}
},
"properties_hs_activity_type": {
"description": "Type of activity associated with the call engagement.",
"type": ["null", "string"]
},
"properties_hs_all_assigned_business_unit_ids": {
"description": "IDs of all business units assigned to the call engagement.",
"type": ["null", "string"]
},
"properties_hs_at_mentioned_owner_ids": {
"description": "IDs of owners mentioned in the call engagement.",
"type": ["null", "string"]
},
"properties_hs_attachment_ids": {
"description": "IDs of attachments associated with the call engagement.",
"type": ["null", "string"]
},
"properties_hs_body_preview": {
"description": "Preview of the body content of the call engagement.",
"type": ["null", "string"]
},
"properties_hs_body_preview_html": {
"description": "HTML-formatted preview of the body content of the call engagement.",
"type": ["null", "string"]
},
"properties_hs_body_preview_is_truncated": {
"description": "Indicates if the body preview is truncated or not.",
"type": ["null", "boolean"]
},
"properties_hs_call_app_id": {
"description": "App ID associated with the call engagement.",
"type": ["null", "number"]
},
"properties_hs_call_authed_url_provider": {
"description": "Provider of the authenticated call URL.",
"type": ["null", "string"]
},
"properties_hs_call_body": {
"description": "Body content of the call engagement.",
"type": ["null", "string"]
},
"properties_hs_call_callee_object_id": {
"description": "Object ID of the callee associated with the call engagement.",
"type": ["null", "number"]
},
"properties_hs_call_callee_object_type": {
"description": "Type of object of the callee associated with the call engagement.",
"type": ["null", "string"]
},
"properties_hs_call_disposition": {
"description": "Disposition of the call engagement.",
"type": ["null", "string"]
},
"properties_hs_call_duration": {
"description": "Duration of the call engagement.",
"type": ["null", "number"]
},
"properties_hs_call_external_account_id": {
"description": "External account ID associated with the call engagement.",
"type": ["null", "string"]
},
"properties_hs_call_external_id": {
"description": "External ID associated with the call engagement.",
"type": ["null", "string"]
},
"properties_hs_call_from_number": {
"description": "Phone number from which the call was made.",
"type": ["null", "string"]
},
"properties_hs_call_has_transcript": {
"description": "Indicates if the call has a transcript or not.",
"type": ["null", "boolean"]
},
"properties_hs_call_recording_url": {
"description": "URL of the call recording.",
"type": ["null", "string"]
},
"properties_hs_call_source": {
"description": "Source of the call engagement.",
"type": ["null", "string"]
},
"properties_hs_call_status": {
"description": "Status of the call.",
"type": ["null", "string"]
},
"properties_hs_call_title": {
"description": "Title of the call engagement.",
"type": ["null", "string"]
},
"properties_hs_call_to_number": {
"description": "Phone number to which the call was made.",
"type": ["null", "string"]
},
"properties_hs_call_transcription_id": {
"description": "Transcription ID of the call engagement.",
"type": ["null", "number"]
},
"properties_hs_call_video_recording_url": {
"description": "URL of the video call recording.",
"type": ["null", "string"]
},
"properties_hs_call_zoom_meeting_uuid": {
"description": "UUID of the Zoom meeting associated with the call engagement.",
"type": ["null", "string"]
},
"properties_hs_calls_service_call_id": {
"description": "Service call ID associated with the call engagement.",
"type": ["null", "number"]
},
"properties_hs_created_by": {
"description": "User who created the call engagement.",
"type": ["null", "number"]
},
"properties_hs_created_by_user_id": {
"description": "User ID of the creator of the call engagement.",
"type": ["null", "number"]
},
"properties_hs_createdate": {
"description": "Date and time when the call engagement was created.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_engagement_source": {
"description": "Source of the engagement.",
"type": ["null", "string"]
},
"properties_hs_engagement_source_id": {
"description": "ID of the source of the engagement.",
"type": ["null", "string"]
},
"properties_hs_follow_up_action": {
"description": "Follow-up action required for the engagement.",
"type": ["null", "string"]
},
"properties_hs_gdpr_deleted": {
"description": "Indicates if the engagement is deleted due to GDPR compliance.",
"type": ["null", "boolean"]
},
"properties_hs_lastmodifieddate": {
"description": "Date and time when the call engagement was last modified.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_merged_object_ids": {
"description": "IDs of merged objects associated with the engagement.",
"type": ["null", "string"]
},
"properties_hs_modified_by": {
"description": "User who last modified the engagement.",
"type": ["null", "number"]
},
"properties_hs_object_id": {
"description": "Object ID of the engagement.",
"type": ["null", "number"]
},
"properties_hs_product_name": {
"description": "Name of the product associated with the engagement.",
"type": ["null", "string"]
},
"properties_hs_queue_membership_ids": {
"description": "IDs of queue memberships associated with the engagement.",
"type": ["null", "string"]
},
"properties_hs_timestamp": {
"description": "Timestamp of the engagement.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_unique_creation_key": {
"description": "Unique key for creation of the engagement.",
"type": ["null", "string"]
},
"properties_hs_unique_id": {
"description": "Unique ID associated with the engagement.",
"type": ["null", "string"]
},
"properties_hs_unknown_visitor_conversation": {
"description": "Indicates if the conversation is with an unknown visitor.",
"type": ["null", "boolean"]
},
"properties_hs_updated_by_user_id": {
"description": "User ID of the last user who updated the engagement.",
"type": ["null", "number"]
},
"properties_hs_user_ids_of_all_notification_followers": {
"description": "User IDs of all notification followers.",
"type": ["null", "string"]
},
"properties_hs_user_ids_of_all_notification_unfollowers": {
"description": "User IDs of all notification unfollowers.",
"type": ["null", "string"]
},
"properties_hs_user_ids_of_all_owners": {
"description": "User IDs of all owners associated with the engagement.",
"type": ["null", "string"]
},
"properties_hubspot_owner_assigneddate": {
"description": "Date and time when the owner was assigned.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hubspot_owner_id": {
"description": "Owner ID associated with the engagement.",
"type": ["null", "string"]
},
"properties_hubspot_team_id": {
"description": "Team ID associated with the engagement.",
"type": ["null", "string"]
},
"properties_hs_all_owner_ids": {
"description": "IDs of all owners associated with the call engagement.",
"type": ["null", "string"]
},
"properties_hs_all_team_ids": {
"description": "IDs of all teams associated with the call engagement.",
"type": ["null", "string"]
},
"properties_hs_all_accessible_team_ids": {
"description": "IDs of all teams that have access to the call engagement.",
"type": ["null", "string"]
},
"createdAt": {
"description": "Date and time when the call engagement was created.",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "Date and time when the call engagement was last updated.",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates if the call engagement is archived or not.",
"type": ["null", "boolean"]
},
"contacts": {
"description": "Contacts associated with the call engagement.",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"deals": {
"description": "Deals associated with the call engagement.",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"companies": {
"description": "Companies associated with the call engagement.",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"tickets": {
"description": "Tickets associated with the call engagement.",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
}
}
}

707
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/engagements_emails.json
View File

@@ -1,707 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "Unique identifier for the engagement email",
"type": ["null", "string"]
},
"properties": {
"type": "object",
"properties": {
"hs_all_assigned_business_unit_ids": {
"description": "The IDs of all business units assigned to this engagement email",
"type": ["null", "string"]
},
"hs_at_mentioned_owner_ids": {
"description": "The IDs of owners mentioned in this engagement email",
"type": ["null", "string"]
},
"hs_attachment_ids": {
"description": "The IDs of attachments included in this engagement email",
"type": ["null", "string"]
},
"hs_body_preview": {
"description": "Preview text of the email body",
"type": ["null", "string"]
},
"hs_body_preview_html": {
"description": "HTML version of the preview text of the email body",
"type": ["null", "string"]
},
"hs_body_preview_is_truncated": {
"description": "Indicates if the body preview is truncated",
"type": ["null", "boolean"]
},
"hs_created_by": {
"description": "User who created the engagement email",
"type": ["null", "string"]
},
"hs_created_by_user_id": {
"description": "User ID of the creator of the engagement email",
"type": ["null", "number"]
},
"hs_createdate": {
"description": "Date and time when the engagement email was created",
"type": ["null", "string"],
"format": "date-time"
},
"hs_direction_and_unique_id": {
"description": "Direction and unique ID of the email",
"type": ["null", "string"]
},
"hs_email_attached_video_id": {
"description": "ID of the attached video in the email",
"type": ["null", "string"]
},
"hs_email_attached_video_name": {
"description": "Name of the attached video in the email",
"type": ["null", "string"]
},
"hs_email_attached_video_opened": {
"description": "Indicates if the attached video was opened",
"type": ["null", "boolean"]
},
"hs_email_attached_video_watched": {
"description": "Indicates if the attached video was watched",
"type": ["null", "boolean"]
},
"hs_email_bcc_email": {
"description": "Email address in BCC field of the email",
"type": ["null", "string"]
},
"hs_email_bcc_firstname": {
"description": "First name in BCC field of the email",
"type": ["null", "string"]
},
"hs_email_bcc_lastname": {
"description": "Last name in BCC field of the email",
"type": ["null", "string"]
},
"hs_email_bcc_raw": {
"description": "Raw data of BCC field of the email",
"type": ["null", "string"]
},
"hs_email_cc_email": {
"description": "Email address in CC field of the email",
"type": ["null", "string"]
},
"hs_email_cc_firstname": {
"description": "First name in CC field of the email",
"type": ["null", "string"]
},
"hs_email_cc_lastname": {
"description": "Last name in CC field of the email",
"type": ["null", "string"]
},
"hs_email_cc_raw": {
"description": "Raw data of CC field of the email",
"type": ["null", "string"]
},
"hs_email_direction": {
"description": "Direction of the email",
"type": ["null", "string"]
},
"hs_email_encoded_email_associations_request": {
"description": "Encoded email associations request",
"type": ["null", "string"]
},
"hs_email_error_message": {
"description": "Error message associated with the email",
"type": ["null", "string"]
},
"hs_email_facsimile_send_id": {
"description": "ID associated with the facsimile send",
"type": ["null", "string"]
},
"hs_email_from_email": {
"description": "Email address of the sender",
"type": ["null", "string"]
},
"hs_email_from_firstname": {
"description": "First name of the sender",
"type": ["null", "string"]
},
"hs_email_from_lastname": {
"description": "Last name of the sender",
"type": ["null", "string"]
},
"hs_email_from_raw": {
"description": "Raw data of the sender's email",
"type": ["null", "string"]
},
"hs_email_headers": {
"description": "Headers of the email",
"type": ["null", "string"]
},
"hs_email_html": {
"description": "HTML content of the email",
"type": ["null", "string"]
},
"hs_email_logged_from": {
"description": "Origin of the logged email",
"type": ["null", "string"]
},
"hs_email_media_processing_status": {
"description": "Status of media processing in the email",
"type": ["null", "string"]
},
"hs_email_member_of_forwarded_subthread": {
"description": "Indicates if the email is a member of a forwarded subthread",
"type": ["null", "boolean"]
},
"hs_email_message_id": {
"description": "Message ID of the email",
"type": ["null", "string"]
},
"hs_email_migrated_via_portal_data_migration": {
"description": "Indicates if the email was migrated via portal data migration",
"type": ["null", "string"]
},
"hs_email_pending_inline_image_ids": {
"description": "IDs of pending inline images in the email",
"type": ["null", "string"]
},
"hs_email_post_send_status": {
"description": "Status after sending the email",
"type": ["null", "string"]
},
"hs_email_recipient_drop_reasons": {
"description": "Reasons for dropping email recipients",
"type": ["null", "string"]
},
"hs_email_send_event_id": {
"description": "ID of the email send event",
"type": ["null", "string"]
},
"hs_email_send_event_id_created": {
"description": "Date and time when the email send event was created",
"type": ["null", "string"],
"format": "date-time"
},
"hs_email_sender_email": {
"description": "Email address of the sender of the email",
"type": ["null", "string"]
},
"hs_email_sender_firstname": {
"description": "First name of the sender of the email",
"type": ["null", "string"]
},
"hs_email_sender_lastname": {
"description": "Last name of the sender of the email",
"type": ["null", "string"]
},
"hs_email_sender_raw": {
"description": "Raw data of the sender of the email",
"type": ["null", "string"]
},
"hs_email_sent_via": {
"description": "Method through which the email was sent",
"type": ["null", "string"]
},
"hs_email_status": {
"description": "Status of the email",
"type": ["null", "string"]
},
"hs_email_subject": {
"description": "Subject of the email",
"type": ["null", "string"]
},
"hs_email_text": {
"description": "Text content of the email",
"type": ["null", "string"]
},
"hs_email_thread_id": {
"description": "Thread ID of the email",
"type": ["null", "string"]
},
"hs_email_to_email": {
"description": "Email address in 'To' field of the email",
"type": ["null", "string"]
},
"hs_email_to_firstname": {
"description": "First name in 'To' field of the email",
"type": ["null", "string"]
},
"hs_email_to_lastname": {
"description": "Last name in 'To' field of the email",
"type": ["null", "string"]
},
"hs_email_to_raw": {
"description": "Raw data of 'To' field of the email",
"type": ["null", "string"]
},
"hs_email_tracker_key": {
"description": "Key associated with email tracking",
"type": ["null", "string"]
},
"hs_email_validation_skipped": {
"description": "Indicates if email validation was skipped",
"type": ["null", "string"]
},
"hs_engagement_source": {
"description": "Source of engagement",
"type": ["null", "string"]
},
"hs_engagement_source_id": {
"description": "ID of the engagement source",
"type": ["null", "string"]
},
"hs_follow_up_action": {
"description": "Follow-up action related to the engagement",
"type": ["null", "string"]
},
"hs_gdpr_deleted": {
"description": "Indicates if the email has been GDPR deleted",
"type": ["null", "boolean"]
},
"hs_lastmodifieddate": {
"description": "Date and time when the engagement email was last modified",
"type": ["null", "string"],
"format": "date-time"
},
"hs_merged_object_ids": {
"description": "IDs of merged objects related to the email",
"type": ["null", "string"]
},
"hs_modified_by": {
"description": "User who last modified the email",
"type": ["null", "string"]
},
"hs_object_id": {
"description": "ID of the engagement email object",
"type": ["null", "number"]
},
"hs_product_name": {
"description": "Name of the product associated with the engagement email",
"type": ["null", "string"]
},
"hs_queue_membership_ids": {
"description": "IDs of queue memberships associated with this engagement email",
"type": ["null", "string"]
},
"hs_timestamp": {
"description": "Date and time of the timestamp for the engagement email",
"type": ["null", "string"],
"format": "date-time"
},
"hs_unique_creation_key": {
"description": "Unique key for the creation of the email",
"type": ["null", "string"]
},
"hs_unique_id": {
"description": "Unique ID of the engagement email",
"type": ["null", "string"]
},
"hs_updated_by_user_id": {
"description": "User ID of the user who last updated the email",
"type": ["null", "number"]
},
"hs_user_ids_of_all_notification_followers": {
"description": "User IDs of all notification followers of the email",
"type": ["null", "string"]
},
"hs_user_ids_of_all_notification_unfollowers": {
"description": "User IDs of all notification unfollowers of the email",
"type": ["null", "string"]
},
"hs_user_ids_of_all_owners": {
"description": "User IDs of all owners of the email",
"type": ["null", "string"]
},
"hubspot_owner_assigneddate": {
"description": "Date and time when the owner was assigned to the engagement email",
"type": ["null", "string"],
"format": "date-time"
},
"hubspot_owner_id": {
"description": "ID of the owner associated with the email",
"type": ["null", "string"]
},
"hubspot_team_id": {
"description": "ID of the team associated with the email",
"type": ["null", "string"]
},
"hs_all_owner_ids": {
"description": "The IDs of all owners associated with this engagement email",
"type": ["null", "string"]
},
"hs_all_team_ids": {
"description": "The IDs of all teams associated with this engagement email",
"type": ["null", "string"]
},
"hs_all_accessible_team_ids": {
"description": "The IDs of all the teams that have access to this engagement email",
"type": ["null", "string"]
}
}
},
"properties_hs_all_assigned_business_unit_ids": {
"description": "The IDs of all business units assigned to this engagement email",
"type": ["null", "string"]
},
"properties_hs_at_mentioned_owner_ids": {
"description": "The IDs of owners mentioned in this engagement email",
"type": ["null", "string"]
},
"properties_hs_attachment_ids": {
"description": "The IDs of attachments included in this engagement email",
"type": ["null", "string"]
},
"properties_hs_body_preview": {
"description": "Preview text of the email body",
"type": ["null", "string"]
},
"properties_hs_body_preview_html": {
"description": "HTML version of the preview text of the email body",
"type": ["null", "string"]
},
"properties_hs_body_preview_is_truncated": {
"description": "Indicates if the body preview is truncated",
"type": ["null", "boolean"]
},
"properties_hs_created_by": {
"description": "User who created the engagement email",
"type": ["null", "string"]
},
"properties_hs_created_by_user_id": {
"description": "User ID of the creator of the engagement email",
"type": ["null", "number"]
},
"properties_hs_createdate": {
"description": "Date and time when the engagement email was created",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_direction_and_unique_id": {
"description": "Direction and unique ID of the email",
"type": ["null", "string"]
},
"properties_hs_email_attached_video_id": {
"description": "ID of the attached video in the email",
"type": ["null", "string"]
},
"properties_hs_email_attached_video_name": {
"description": "Name of the attached video in the email",
"type": ["null", "string"]
},
"properties_hs_email_attached_video_opened": {
"description": "Indicates if the attached video was opened",
"type": ["null", "boolean"]
},
"properties_hs_email_attached_video_watched": {
"description": "Indicates if the attached video was watched",
"type": ["null", "boolean"]
},
"properties_hs_email_bcc_email": {
"description": "Email address in BCC field of the email",
"type": ["null", "string"]
},
"properties_hs_email_bcc_firstname": {
"description": "First name in BCC field of the email",
"type": ["null", "string"]
},
"properties_hs_email_bcc_lastname": {
"description": "Last name in BCC field of the email",
"type": ["null", "string"]
},
"properties_hs_email_bcc_raw": {
"description": "Raw data of BCC field of the email",
"type": ["null", "string"]
},
"properties_hs_email_cc_email": {
"description": "Email address in CC field of the email",
"type": ["null", "string"]
},
"properties_hs_email_cc_firstname": {
"description": "First name in CC field of the email",
"type": ["null", "string"]
},
"properties_hs_email_cc_lastname": {
"description": "Last name in CC field of the email",
"type": ["null", "string"]
},
"properties_hs_email_cc_raw": {
"description": "Raw data of CC field of the email",
"type": ["null", "string"]
},
"properties_hs_email_direction": {
"description": "Direction of the email",
"type": ["null", "string"]
},
"properties_hs_email_encoded_email_associations_request": {
"description": "Encoded email associations request",
"type": ["null", "string"]
},
"properties_hs_email_error_message": {
"description": "Error message associated with the email",
"type": ["null", "string"]
},
"properties_hs_email_facsimile_send_id": {
"description": "ID associated with the facsimile send",
"type": ["null", "string"]
},
"properties_hs_email_from_email": {
"description": "Email address of the sender",
"type": ["null", "string"]
},
"properties_hs_email_from_firstname": {
"description": "First name of the sender",
"type": ["null", "string"]
},
"properties_hs_email_from_lastname": {
"description": "Last name of the sender",
"type": ["null", "string"]
},
"properties_hs_email_from_raw": {
"description": "Raw data of the sender's email",
"type": ["null", "string"]
},
"properties_hs_email_headers": {
"description": "Headers of the email",
"type": ["null", "string"]
},
"properties_hs_email_html": {
"description": "HTML content of the email",
"type": ["null", "string"]
},
"properties_hs_email_logged_from": {
"description": "Origin of the logged email",
"type": ["null", "string"]
},
"properties_hs_email_media_processing_status": {
"description": "Status of media processing in the email",
"type": ["null", "string"]
},
"properties_hs_email_member_of_forwarded_subthread": {
"description": "Indicates if the email is a member of a forwarded subthread",
"type": ["null", "boolean"]
},
"properties_hs_email_message_id": {
"description": "Message ID of the email",
"type": ["null", "string"]
},
"properties_hs_email_migrated_via_portal_data_migration": {
"description": "Indicates if the email was migrated via portal data migration",
"type": ["null", "string"]
},
"properties_hs_email_pending_inline_image_ids": {
"description": "IDs of pending inline images in the email",
"type": ["null", "string"]
},
"properties_hs_email_post_send_status": {
"description": "Status after sending the email",
"type": ["null", "string"]
},
"properties_hs_email_recipient_drop_reasons": {
"description": "Reasons for dropping email recipients",
"type": ["null", "string"]
},
"properties_hs_email_send_event_id": {
"description": "ID of the email send event",
"type": ["null", "string"]
},
"properties_hs_email_send_event_id_created": {
"description": "Date and time when the email send event was created",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_email_sender_email": {
"description": "Email address of the sender of the email",
"type": ["null", "string"]
},
"properties_hs_email_sender_firstname": {
"description": "First name of the sender of the email",
"type": ["null", "string"]
},
"properties_hs_email_sender_lastname": {
"description": "Last name of the sender of the email",
"type": ["null", "string"]
},
"properties_hs_email_sender_raw": {
"description": "Raw data of the sender of the email",
"type": ["null", "string"]
},
"properties_hs_email_sent_via": {
"description": "Method through which the email was sent",
"type": ["null", "string"]
},
"properties_hs_email_status": {
"description": "Status of the email",
"type": ["null", "string"]
},
"properties_hs_email_subject": {
"description": "Subject of the email",
"type": ["null", "string"]
},
"properties_hs_email_text": {
"description": "Text content of the email",
"type": ["null", "string"]
},
"properties_hs_email_thread_id": {
"description": "Thread ID of the email",
"type": ["null", "string"]
},
"properties_hs_email_to_email": {
"description": "Email address in 'To' field of the email",
"type": ["null", "string"]
},
"properties_hs_email_to_firstname": {
"description": "First name in 'To' field of the email",
"type": ["null", "string"]
},
"properties_hs_email_to_lastname": {
"description": "Last name in 'To' field of the email",
"type": ["null", "string"]
},
"properties_hs_email_to_raw": {
"description": "Raw data of 'To' field of the email",
"type": ["null", "string"]
},
"properties_hs_email_tracker_key": {
"description": "Key associated with email tracking",
"type": ["null", "string"]
},
"properties_hs_email_validation_skipped": {
"description": "Indicates if email validation was skipped",
"type": ["null", "string"]
},
"properties_hs_engagement_source": {
"description": "Source of engagement",
"type": ["null", "string"]
},
"properties_hs_engagement_source_id": {
"description": "ID of the engagement source",
"type": ["null", "string"]
},
"properties_hs_follow_up_action": {
"description": "Follow-up action related to the engagement",
"type": ["null", "string"]
},
"properties_hs_gdpr_deleted": {
"description": "Indicates if the email has been GDPR deleted",
"type": ["null", "boolean"]
},
"properties_hs_lastmodifieddate": {
"description": "Date and time when the engagement email was last modified",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_merged_object_ids": {
"description": "IDs of merged objects related to the email",
"type": ["null", "string"]
},
"properties_hs_modified_by": {
"description": "User who last modified the email",
"type": ["null", "string"]
},
"properties_hs_object_id": {
"description": "ID of the engagement email object",
"type": ["null", "number"]
},
"properties_hs_product_name": {
"description": "Name of the product associated with the engagement email",
"type": ["null", "string"]
},
"properties_hs_queue_membership_ids": {
"description": "IDs of queue memberships associated with this engagement email",
"type": ["null", "string"]
},
"properties_hs_timestamp": {
"description": "Date and time of the timestamp for the engagement email",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_unique_creation_key": {
"description": "Unique key for the creation of the email",
"type": ["null", "string"]
},
"properties_hs_unique_id": {
"description": "Unique ID of the engagement email",
"type": ["null", "string"]
},
"properties_hs_updated_by_user_id": {
"description": "User ID of the user who last updated the email",
"type": ["null", "number"]
},
"properties_hs_user_ids_of_all_notification_followers": {
"description": "User IDs of all notification followers of the email",
"type": ["null", "string"]
},
"properties_hs_user_ids_of_all_notification_unfollowers": {
"description": "User IDs of all notification unfollowers of the email",
"type": ["null", "string"]
},
"properties_hs_user_ids_of_all_owners": {
"description": "User IDs of all owners of the email",
"type": ["null", "string"]
},
"properties_hubspot_owner_assigneddate": {
"description": "Date and time when the owner was assigned to the engagement email",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hubspot_owner_id": {
"description": "ID of the owner associated with the email",
"type": ["null", "string"]
},
"properties_hubspot_team_id": {
"description": "ID of the team associated with the email",
"type": ["null", "string"]
},
"properties_hs_all_owner_ids": {
"description": "The IDs of all owners associated with this engagement email",
"type": ["null", "string"]
},
"properties_hs_all_team_ids": {
"description": "The IDs of all teams associated with this engagement email",
"type": ["null", "string"]
},
"properties_hs_all_accessible_team_ids": {
"description": "The IDs of all the teams that have access to this engagement email",
"type": ["null", "string"]
},
"createdAt": {
"description": "Date and time when the engagement email was created",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "Date and time when the engagement email was last updated",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates if the engagement email is archived or not",
"type": ["null", "boolean"]
},
"contacts": {
"description": "List of contacts associated with the engagement email",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"deals": {
"description": "List of deals associated with the engagement email",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"companies": {
"description": "List of companies associated with the engagement email",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"tickets": {
"description": "List of tickets associated with the engagement email",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
}
}
}

486
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/engagements_meetings.json
View File

@@ -1,486 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "Unique identifier for the meeting engagement.",
"type": ["null", "string"]
},
"properties": {
"description": "Additional properties related to the meeting engagement.",
"type": "object",
"properties": {
"hs_activity_type": {
"description": "Type of activity associated with the meeting.",
"type": ["null", "string"]
},
"hs_all_assigned_business_unit_ids": {
"description": "IDs of all business units assigned to the meeting.",
"type": ["null", "string"]
},
"hs_at_mentioned_owner_ids": {
"description": "IDs of owners mentioned in the meeting.",
"type": ["null", "string"]
},
"hs_attachment_ids": {
"description": "IDs of attachments associated with the meeting.",
"type": ["null", "string"]
},
"hs_attendee_owner_ids": {
"description": "IDs of owners who are attendees in the meeting.",
"type": ["null", "string"]
},
"hs_body_preview": {
"description": "Preview of the meeting body.",
"type": ["null", "string"]
},
"hs_body_preview_html": {
"description": "HTML version of the meeting body preview.",
"type": ["null", "string"]
},
"hs_body_preview_is_truncated": {
"description": "Flag indicating if the body preview is truncated.",
"type": ["null", "boolean"]
},
"hs_created_by": {
"description": "User who created the meeting.",
"type": ["null", "number"]
},
"hs_created_by_user_id": {
"description": "User ID of the creator of the meeting.",
"type": ["null", "number"]
},
"hs_createdate": {
"description": "Date and time when the meeting was created.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_engagement_source": {
"description": "Source of the meeting engagement.",
"type": ["null", "string"]
},
"hs_engagement_source_id": {
"description": "ID of the source of the meeting engagement.",
"type": ["null", "string"]
},
"hs_follow_up_action": {
"description": "Follow-up action related to the meeting.",
"type": ["null", "string"]
},
"hs_gdpr_deleted": {
"description": "Flag indicating if the meeting is deleted due to GDPR.",
"type": ["null", "boolean"]
},
"hs_i_cal_uid": {
"description": "Unique identifier for the meeting in iCalendar format.",
"type": ["null", "string"]
},
"hs_internal_meeting_notes": {
"description": "Internal notes related to the meeting.",
"type": ["null", "string"]
},
"hs_lastmodifieddate": {
"description": "Date and time when the meeting was last modified.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_meeting_body": {
"description": "Full body of the meeting.",
"type": ["null", "string"]
},
"hs_meeting_calendar_event_hash": {
"description": "Unique hash for the meeting in the calendar event.",
"type": ["null", "string"]
},
"hs_meeting_change_id": {
"description": "Change ID associated with the meeting.",
"type": ["null", "string"]
},
"hs_meeting_created_from_link_id": {
"description": "ID of the link from which the meeting was created.",
"type": ["null", "string"]
},
"hs_meeting_end_time": {
"description": "End time of the meeting.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_meeting_external_url": {
"description": "External URL associated with the meeting.",
"type": ["null", "string"]
},
"hs_meeting_location": {
"description": "Location where the meeting took place.",
"type": ["null", "string"]
},
"hs_meeting_location_type": {
"description": "Type of location where the meeting took place.",
"type": ["null", "string"]
},
"hs_meeting_outcome": {
"description": "Outcome of the meeting.",
"type": ["null", "string"]
},
"hs_meeting_pre_meeting_prospect_reminders": {
"description": "Prospect reminders before the meeting.",
"type": ["null", "string"]
},
"hs_meeting_source": {
"description": "Source of the meeting.",
"type": ["null", "string"]
},
"hs_meeting_source_id": {
"description": "ID of the source of the meeting.",
"type": ["null", "string"]
},
"hs_meeting_start_time": {
"description": "Start time of the meeting.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_meeting_title": {
"description": "Title of the meeting.",
"type": ["null", "string"]
},
"hs_meeting_web_conference_meeting_id": {
"description": "Meeting ID for web conference.",
"type": ["null", "string"]
},
"hs_merged_object_ids": {
"description": "IDs of merged objects related to the meeting.",
"type": ["null", "string"]
},
"hs_modified_by": {
"description": "User who last modified the meeting.",
"type": ["null", "number"]
},
"hs_object_id": {
"description": "Object ID associated with the meeting.",
"type": ["null", "number"]
},
"hs_product_name": {
"description": "Name of the product associated with the meeting.",
"type": ["null", "string"]
},
"hs_queue_membership_ids": {
"description": "IDs of queues the meeting is associated with.",
"type": ["null", "string"]
},
"hs_scheduled_tasks": {
"description": "Scheduled tasks related to the meeting.",
"type": ["null", "string"]
},
"hs_timestamp": {
"description": "Timestamp for the meeting engagement.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_unique_creation_key": {
"description": "Unique key associated with the creation of the meeting.",
"type": ["null", "string"]
},
"hs_unique_id": {
"description": "Unique ID associated with the meeting.",
"type": ["null", "string"]
},
"hs_updated_by_user_id": {
"description": "User ID of the user who last updated the meeting.",
"type": ["null", "number"]
},
"hs_user_ids_of_all_notification_followers": {
"description": "IDs of users following notifications for the meeting.",
"type": ["null", "string"]
},
"hs_user_ids_of_all_notification_unfollowers": {
"description": "IDs of users who have unfollowed notifications for the meeting.",
"type": ["null", "string"]
},
"hs_user_ids_of_all_owners": {
"description": "IDs of all owners associated with the meeting.",
"type": ["null", "string"]
},
"hubspot_owner_assigneddate": {
"description": "Date and time when the owner was assigned to the meeting.",
"type": ["null", "string"],
"format": "date-time"
},
"hubspot_owner_id": {
"description": "ID of the owner associated with the meeting.",
"type": ["null", "string"]
},
"hubspot_team_id": {
"description": "ID of the team associated with the meeting.",
"type": ["null", "string"]
},
"hs_all_owner_ids": {
"description": "IDs of all owners associated with the meeting.",
"type": ["null", "string"]
},
"hs_all_team_ids": {
"description": "IDs of all teams associated with the meeting.",
"type": ["null", "string"]
},
"hs_all_accessible_team_ids": {
"description": "IDs of all teams that have access to the meeting.",
"type": ["null", "string"]
}
}
},
"properties_hs_activity_type": {
"description": "Type of activity associated with the meeting.",
"type": ["null", "string"]
},
"properties_hs_all_assigned_business_unit_ids": {
"description": "IDs of all business units assigned to the meeting.",
"type": ["null", "string"]
},
"properties_hs_at_mentioned_owner_ids": {
"description": "IDs of owners mentioned in the meeting.",
"type": ["null", "string"]
},
"properties_hs_attachment_ids": {
"description": "IDs of attachments associated with the meeting.",
"type": ["null", "string"]
},
"properties_hs_attendee_owner_ids": {
"description": "IDs of owners who are attendees in the meeting.",
"type": ["null", "string"]
},
"properties_hs_body_preview": {
"description": "Preview of the meeting body.",
"type": ["null", "string"]
},
"properties_hs_body_preview_html": {
"description": "HTML version of the meeting body preview.",
"type": ["null", "string"]
},
"properties_hs_body_preview_is_truncated": {
"description": "Flag indicating if the body preview is truncated.",
"type": ["null", "boolean"]
},
"properties_hs_created_by": {
"description": "User who created the meeting.",
"type": ["null", "number"]
},
"properties_hs_created_by_user_id": {
"description": "User ID of the creator of the meeting.",
"type": ["null", "number"]
},
"properties_hs_createdate": {
"description": "Date and time when the meeting was created.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_engagement_source": {
"description": "Source of the meeting engagement.",
"type": ["null", "string"]
},
"properties_hs_engagement_source_id": {
"description": "ID of the source of the meeting engagement.",
"type": ["null", "string"]
},
"properties_hs_follow_up_action": {
"description": "Follow-up action related to the meeting.",
"type": ["null", "string"]
},
"properties_hs_gdpr_deleted": {
"description": "Flag indicating if the meeting is deleted due to GDPR.",
"type": ["null", "boolean"]
},
"properties_hs_i_cal_uid": {
"description": "Unique identifier for the meeting in iCalendar format.",
"type": ["null", "string"]
},
"properties_hs_internal_meeting_notes": {
"description": "Internal notes related to the meeting.",
"type": ["null", "string"]
},
"properties_hs_lastmodifieddate": {
"description": "Date and time when the meeting was last modified.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_meeting_body": {
"description": "Full body of the meeting.",
"type": ["null", "string"]
},
"properties_hs_meeting_calendar_event_hash": {
"description": "Unique hash for the meeting in the calendar event.",
"type": ["null", "string"]
},
"properties_hs_meeting_change_id": {
"description": "Change ID associated with the meeting.",
"type": ["null", "string"]
},
"properties_hs_meeting_created_from_link_id": {
"description": "ID of the link from which the meeting was created.",
"type": ["null", "string"]
},
"properties_hs_meeting_end_time": {
"description": "End time of the meeting.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_meeting_external_url": {
"description": "External URL associated with the meeting.",
"type": ["null", "string"]
},
"properties_hs_meeting_location": {
"description": "Location where the meeting took place.",
"type": ["null", "string"]
},
"properties_hs_meeting_location_type": {
"description": "Type of location where the meeting took place.",
"type": ["null", "string"]
},
"properties_hs_meeting_outcome": {
"description": "Outcome of the meeting.",
"type": ["null", "string"]
},
"properties_hs_meeting_pre_meeting_prospect_reminders": {
"description": "Prospect reminders before the meeting.",
"type": ["null", "string"]
},
"properties_hs_meeting_source": {
"description": "Source of the meeting.",
"type": ["null", "string"]
},
"properties_hs_meeting_source_id": {
"description": "ID of the source of the meeting.",
"type": ["null", "string"]
},
"properties_hs_meeting_start_time": {
"description": "Start time of the meeting.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_meeting_title": {
"description": "Title of the meeting.",
"type": ["null", "string"]
},
"properties_hs_meeting_web_conference_meeting_id": {
"description": "Meeting ID for web conference.",
"type": ["null", "string"]
},
"properties_hs_merged_object_ids": {
"description": "IDs of merged objects related to the meeting.",
"type": ["null", "string"]
},
"properties_hs_modified_by": {
"description": "User who last modified the meeting.",
"type": ["null", "number"]
},
"properties_hs_object_id": {
"description": "Object ID associated with the meeting.",
"type": ["null", "number"]
},
"properties_hs_product_name": {
"description": "Name of the product associated with the meeting.",
"type": ["null", "string"]
},
"properties_hs_queue_membership_ids": {
"description": "IDs of queues the meeting is associated with.",
"type": ["null", "string"]
},
"properties_hs_scheduled_tasks": {
"description": "Scheduled tasks related to the meeting.",
"type": ["null", "string"]
},
"properties_hs_timestamp": {
"description": "Timestamp for the meeting engagement.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_unique_creation_key": {
"description": "Unique key associated with the creation of the meeting.",
"type": ["null", "string"]
},
"properties_hs_unique_id": {
"description": "Unique ID associated with the meeting.",
"type": ["null", "string"]
},
"properties_hs_updated_by_user_id": {
"description": "User ID of the user who last updated the meeting.",
"type": ["null", "number"]
},
"properties_hs_user_ids_of_all_notification_followers": {
"description": "IDs of users following notifications for the meeting.",
"type": ["null", "string"]
},
"properties_hs_user_ids_of_all_notification_unfollowers": {
"description": "IDs of users who have unfollowed notifications for the meeting.",
"type": ["null", "string"]
},
"properties_hs_user_ids_of_all_owners": {
"description": "IDs of all owners associated with the meeting.",
"type": ["null", "string"]
},
"properties_hubspot_owner_assigneddate": {
"description": "Date and time when the owner was assigned to the meeting.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hubspot_owner_id": {
"description": "ID of the owner associated with the meeting.",
"type": ["null", "string"]
},
"properties_hubspot_team_id": {
"description": "ID of the team associated with the meeting.",
"type": ["null", "string"]
},
"properties_hs_all_owner_ids": {
"description": "IDs of all owners associated with the meeting.",
"type": ["null", "string"]
},
"properties_hs_all_team_ids": {
"description": "IDs of all teams associated with the meeting.",
"type": ["null", "string"]
},
"properties_hs_all_accessible_team_ids": {
"description": "IDs of all teams with access to the meeting.",
"type": ["null", "string"]
},
"createdAt": {
"description": "Timestamp indicating when the meeting engagement was created.",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "Timestamp indicating when the meeting engagement was last updated.",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates whether the meeting engagement is archived or not.",
"type": ["null", "boolean"]
},
"contacts": {
"description": "Information about the contacts associated with the meeting engagement.",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"deals": {
"description": "Information about the deals associated with the meeting engagement.",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"companies": {
"description": "Information about the companies associated with the meeting engagement.",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"tickets": {
"description": "Information about the tickets associated with the meeting engagement.",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
}
}
}

330
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/engagements_notes.json
View File

@@ -1,330 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "Unique identifier for the engagement note",
"type": ["null", "string"]
},
"properties": {
"description": "Represents the properties associated with the engagement note",
"type": "object",
"properties": {
"hs_all_assigned_business_unit_ids": {
"description": "Business unit ids assigned to the note",
"type": ["null", "string"]
},
"hs_at_mentioned_owner_ids": {
"description": "Owner ids mentioned in the note",
"type": ["null", "string"]
},
"hs_attachment_ids": {
"description": "Attachment ids linked to the note",
"type": ["null", "string"]
},
"hs_body_preview": {
"description": "Preview of the note body",
"type": ["null", "string"]
},
"hs_body_preview_html": {
"description": "HTML version of the note body preview",
"type": ["null", "string"]
},
"hs_body_preview_is_truncated": {
"description": "Indicates if the body preview is truncated",
"type": ["null", "boolean"]
},
"hs_created_by": {
"description": "User who created the note",
"type": ["null", "number"]
},
"hs_created_by_user_id": {
"description": "User id of the creator",
"type": ["null", "number"]
},
"hs_createdate": {
"description": "Date and time of note creation",
"type": ["null", "string"],
"format": "date-time"
},
"hs_engagement_source": {
"description": "Source of the engagement",
"type": ["null", "string"]
},
"hs_engagement_source_id": {
"description": "ID of the engagement source",
"type": ["null", "string"]
},
"hs_follow_up_action": {
"description": "Follow-up action specified in the note",
"type": ["null", "string"]
},
"hs_gdpr_deleted": {
"description": "Indicates if the note is GDPR deleted",
"type": ["null", "boolean"]
},
"hs_lastmodifieddate": {
"description": "Date and time of the last modification",
"type": ["null", "string"],
"format": "date-time"
},
"hs_merged_object_ids": {
"description": "IDs of objects merged in the note",
"type": ["null", "string"]
},
"hs_modified_by": {
"description": "User who last modified the note",
"type": ["null", "number"]
},
"hs_note_body": {
"description": "Body content of the note",
"type": ["null", "string"]
},
"hs_object_id": {
"description": "ID of the note object",
"type": ["null", "number"]
},
"hs_product_name": {
"description": "Product name associated with the note",
"type": ["null", "string"]
},
"hs_queue_membership_ids": {
"description": "Queue membership IDs related to the note",
"type": ["null", "string"]
},
"hs_timestamp": {
"description": "Timestamp of the note",
"type": ["null", "string"],
"format": "date-time"
},
"hs_unique_creation_key": {
"description": "Unique key for note creation",
"type": ["null", "string"]
},
"hs_unique_id": {
"description": "Unique ID of the note",
"type": ["null", "string"]
},
"hs_updated_by_user_id": {
"description": "User ID who last updated the note",
"type": ["null", "number"]
},
"hs_user_ids_of_all_notification_followers": {
"description": "User IDs of all notification followers",
"type": ["null", "string"]
},
"hs_user_ids_of_all_notification_unfollowers": {
"description": "User IDs of all notification unfollowers",
"type": ["null", "string"]
},
"hs_user_ids_of_all_owners": {
"description": "User IDs of all owners",
"type": ["null", "string"]
},
"hubspot_owner_assigneddate": {
"description": "Date when owner was assigned",
"type": ["null", "string"],
"format": "date-time"
},
"hubspot_owner_id": {
"description": "Owner ID of the note",
"type": ["null", "string"]
},
"hubspot_team_id": {
"description": "Team ID associated with the note",
"type": ["null", "string"]
},
"hs_all_owner_ids": {
"description": "All owner ids associated with the note",
"type": ["null", "string"]
},
"hs_all_team_ids": {
"description": "All team ids associated with the note",
"type": ["null", "string"]
},
"hs_all_accessible_team_ids": {
"description": "All team ids that have access to the note",
"type": ["null", "string"]
}
}
},
"properties_hs_all_assigned_business_unit_ids": {
"description": "Business unit ids assigned to the note",
"type": ["null", "string"]
},
"properties_hs_at_mentioned_owner_ids": {
"description": "Owner ids mentioned in the note",
"type": ["null", "string"]
},
"properties_hs_attachment_ids": {
"description": "Attachment ids linked to the note",
"type": ["null", "string"]
},
"properties_hs_body_preview": {
"description": "Preview of the note body",
"type": ["null", "string"]
},
"properties_hs_body_preview_html": {
"description": "HTML version of the note body preview",
"type": ["null", "string"]
},
"properties_hs_body_preview_is_truncated": {
"description": "Indicates if the body preview is truncated",
"type": ["null", "boolean"]
},
"properties_hs_created_by": {
"description": "User who created the note",
"type": ["null", "number"]
},
"properties_hs_created_by_user_id": {
"description": "User id of the creator",
"type": ["null", "number"]
},
"properties_hs_createdate": {
"description": "Date and time of note creation",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_engagement_source": {
"description": "Source of the engagement",
"type": ["null", "string"]
},
"properties_hs_engagement_source_id": {
"description": "ID of the engagement source",
"type": ["null", "string"]
},
"properties_hs_follow_up_action": {
"description": "Follow-up action specified in the note",
"type": ["null", "string"]
},
"properties_hs_gdpr_deleted": {
"description": "Indicates if the note is GDPR deleted",
"type": ["null", "boolean"]
},
"properties_hs_lastmodifieddate": {
"description": "Date and time of the last modification",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_merged_object_ids": {
"description": "IDs of objects merged in the note",
"type": ["null", "string"]
},
"properties_hs_modified_by": {
"description": "User who last modified the note",
"type": ["null", "number"]
},
"properties_hs_note_body": {
"description": "Body content of the note",
"type": ["null", "string"]
},
"properties_hs_object_id": {
"description": "ID of the note object",
"type": ["null", "number"]
},
"properties_hs_product_name": {
"description": "Product name associated with the note",
"type": ["null", "string"]
},
"properties_hs_queue_membership_ids": {
"description": "Queue membership IDs related to the note",
"type": ["null", "string"]
},
"properties_hs_timestamp": {
"description": "Timestamp of the note",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_unique_creation_key": {
"description": "Unique key for note creation",
"type": ["null", "string"]
},
"properties_hs_unique_id": {
"description": "Unique ID of the note",
"type": ["null", "string"]
},
"properties_hs_updated_by_user_id": {
"description": "User ID who last updated the note",
"type": ["null", "number"]
},
"properties_hs_user_ids_of_all_notification_followers": {
"description": "User IDs of all notification followers",
"type": ["null", "string"]
},
"properties_hs_user_ids_of_all_notification_unfollowers": {
"description": "User IDs of all notification unfollowers",
"type": ["null", "string"]
},
"properties_hs_user_ids_of_all_owners": {
"description": "User IDs of all owners",
"type": ["null", "string"]
},
"properties_hubspot_owner_assigneddate": {
"description": "Date when owner was assigned to the note",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hubspot_owner_id": {
"description": "Owner ID of the note",
"type": ["null", "string"]
},
"properties_hubspot_team_id": {
"description": "Team ID associated with the note",
"type": ["null", "string"]
},
"properties_hs_all_owner_ids": {
"description": "All owner ids associated with the note",
"type": ["null", "string"]
},
"properties_hs_all_team_ids": {
"description": "All team ids associated with the note",
"type": ["null", "string"]
},
"properties_hs_all_accessible_team_ids": {
"description": "All team ids that have access to the note",
"type": ["null", "string"]
},
"createdAt": {
"description": "The date and time when the note was created",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "Date and time when the note was last updated",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates if the note has been archived",
"type": ["null", "boolean"]
},
"contacts": {
"description": "Contacts associated with the engagement note",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"deals": {
"description": "Deals associated with the engagement note",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"companies": {
"description": "Companies associated with the engagement note",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"tickets": {
"description": "Tickets associated with the engagement note",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
}
}
}

552
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/engagements_tasks.json
View File

@@ -1,552 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "Unique identifier for the task",
"type": ["null", "string"]
},
"properties": {
"description": "Custom properties associated with the task.",
"type": "object",
"properties": {
"hs_all_assigned_business_unit_ids": {
"description": "Array of IDs of business units this task is assigned to",
"type": ["null", "string"]
},
"hs_at_mentioned_owner_ids": {
"description": "Array of IDs of owners mentioned in the task",
"type": ["null", "string"]
},
"hs_attachment_ids": {
"description": "Array of attachment IDs associated with this task",
"type": ["null", "string"]
},
"hs_body_preview": {
"description": "Preview of the body content of the task",
"type": ["null", "string"]
},
"hs_body_preview_html": {
"description": "HTML version of the body content preview",
"type": ["null", "string"]
},
"hs_body_preview_is_truncated": {
"description": "Indicates if the body preview is truncated",
"type": ["null", "boolean"]
},
"hs_calendar_event_id": {
"description": "ID of the associated calendar event, if any",
"type": ["null", "string"]
},
"hs_created_by": {
"description": "Creator of the task",
"type": ["null", "number"]
},
"hs_created_by_user_id": {
"description": "ID of the user who created the task",
"type": ["null", "number"]
},
"hs_createdate": {
"description": "The date and time when the task was created",
"type": ["null", "string"],
"format": "date-time"
},
"hs_engagement_source": {
"description": "Source of the engagement task",
"type": ["null", "string"]
},
"hs_engagement_source_id": {
"description": "ID of the source of the engagement task",
"type": ["null", "string"]
},
"hs_follow_up_action": {
"description": "Action to follow up on the task",
"type": ["null", "string"]
},
"hs_gdpr_deleted": {
"description": "Indicates if the task has been deleted due to GDPR compliance",
"type": ["null", "boolean"]
},
"hs_lastmodifieddate": {
"description": "The date and time when the task was last modified",
"type": ["null", "string"],
"format": "date-time"
},
"hs_merged_object_ids": {
"description": "Array of IDs of merged objects",
"type": ["null", "string"]
},
"hs_modified_by": {
"description": "Last user who modified the task",
"type": ["null", "number"]
},
"hs_msteams_message_id": {
"description": "ID of the Microsoft Teams message associated with the task",
"type": ["null", "string"]
},
"hs_num_associated_companies": {
"description": "Number of companies associated with the task",
"type": ["null", "number"]
},
"hs_num_associated_contacts": {
"description": "Number of contacts associated with the task",
"type": ["null", "number"]
},
"hs_num_associated_deals": {
"description": "Number of deals associated with the task",
"type": ["null", "number"]
},
"hs_num_associated_queue_objects": {
"description": "Number of queue objects associated with the task",
"type": ["null", "number"]
},
"hs_num_associated_tickets": {
"description": "Number of tickets associated with the task",
"type": ["null", "number"]
},
"hs_object_id": {
"description": "ID of the engagement task object",
"type": ["null", "number"]
},
"hs_product_name": {
"description": "Name of the product associated with the task",
"type": ["null", "string"]
},
"hs_queue_membership_ids": {
"description": "Array of IDs of queue members associated with the task",
"type": ["null", "string"]
},
"hs_scheduled_tasks": {
"description": "Array of scheduled tasks related to this task",
"type": ["null", "string"]
},
"hs_task_body": {
"description": "Full body content of the task",
"type": ["null", "string"]
},
"hs_task_completion_date": {
"description": "The date and time when the task was completed",
"type": ["null", "string"],
"format": "date-time"
},
"hs_task_contact_timezone": {
"description": "Timezone of the contact related to the task",
"type": ["null", "string"]
},
"hs_task_for_object_type": {
"description": "Type of object the task is related to",
"type": ["null", "string"]
},
"hs_task_is_all_day": {
"description": "Indicates if the task spans the whole day",
"type": ["null", "boolean"]
},
"hs_task_last_contact_outreach": {
"description": "The date and time of the last contact outreach related to the task",
"type": ["null", "string"],
"format": "date-time"
},
"hs_task_last_sales_activity_timestamp": {
"description": "The date and time of the last sales activity related to the task",
"type": ["null", "string"],
"format": "date-time"
},
"hs_task_priority": {
"description": "Priority level of the task",
"type": ["null", "string"]
},
"hs_task_probability_to_complete": {
"description": "Probability of completing the task",
"type": ["null", "number"]
},
"hs_task_relative_reminders": {
"description": "Relative reminders set for the task",
"type": ["null", "string"]
},
"hs_task_reminders": {
"description": "Specific reminders set for the task",
"type": ["null", "string"]
},
"hs_task_repeat_interval": {
"description": "Interval for repeating the task",
"type": ["null", "string"]
},
"hs_task_send_default_reminder": {
"description": "Indicates if default reminders should be sent for the task",
"type": ["null", "boolean"]
},
"hs_task_sequence_enrollment_active": {
"description": "Indicates if the task is part of an active sequence enrollment",
"type": ["null", "boolean"]
},
"hs_task_sequence_step_enrollment_id": {
"description": "ID of the sequence step enrollment related to the task",
"type": ["null", "string"]
},
"hs_task_sequence_step_order": {
"description": "Order of the task within the sequence step",
"type": ["null", "number"]
},
"hs_task_status": {
"description": "Status of the task",
"type": ["null", "string"]
},
"hs_task_subject": {
"description": "Subject of the task",
"type": ["null", "string"]
},
"hs_task_template_id": {
"description": "ID of the task template, if any",
"type": ["null", "number"]
},
"hs_task_type": {
"description": "Type of the task",
"type": ["null", "string"]
},
"hs_timestamp": {
"description": "The timestamp associated with the task",
"type": ["null", "string"],
"format": "date-time"
},
"hs_unique_creation_key": {
"description": "Unique key for task creation",
"type": ["null", "string"]
},
"hs_unique_id": {
"description": "Unique ID of the task",
"type": ["null", "string"]
},
"hs_updated_by_user_id": {
"description": "ID of the user who last updated the task",
"type": ["null", "number"]
},
"hs_user_ids_of_all_notification_followers": {
"description": "Array of user IDs who are followers and receive notifications",
"type": ["null", "string"]
},
"hs_user_ids_of_all_notification_unfollowers": {
"description": "Array of user IDs who have unfollowed notifications for the task",
"type": ["null", "string"]
},
"hs_user_ids_of_all_owners": {
"description": "Array of user IDs of all owners",
"type": ["null", "string"]
},
"hubspot_owner_assigneddate": {
"description": "The date and time when the task was assigned to an owner",
"type": ["null", "string"],
"format": "date-time"
},
"hubspot_owner_id": {
"description": "ID of the owner of the task in HubSpot",
"type": ["null", "string"]
},
"hubspot_team_id": {
"description": "ID of the team of the task in HubSpot",
"type": ["null", "string"]
},
"hs_all_owner_ids": {
"description": "Array of IDs of all owners associated with this task",
"type": ["null", "string"]
},
"hs_all_team_ids": {
"description": "Array of IDs of all teams associated with this task",
"type": ["null", "string"]
},
"hs_all_accessible_team_ids": {
"description": "Array of IDs of teams that have access to this engagement task",
"type": ["null", "string"]
}
}
},
"properties_hs_all_assigned_business_unit_ids": {
"description": "List of all business unit IDs assigned to this task.",
"type": ["null", "string"]
},
"properties_hs_at_mentioned_owner_ids": {
"description": "List of user IDs mentioned in the task.",
"type": ["null", "string"]
},
"properties_hs_attachment_ids": {
"description": "List of attachment IDs associated with the task.",
"type": ["null", "string"]
},
"properties_hs_body_preview": {
"description": "Preview of the task body content.",
"type": ["null", "string"]
},
"properties_hs_body_preview_html": {
"description": "HTML formatted preview of the task body content.",
"type": ["null", "string"]
},
"properties_hs_body_preview_is_truncated": {
"description": "Flag indicating if the body preview is truncated.",
"type": ["null", "boolean"]
},
"properties_hs_calendar_event_id": {
"description": "ID of the calendar event associated with the task.",
"type": ["null", "string"]
},
"properties_hs_created_by": {
"description": "User who created the task.",
"type": ["null", "number"]
},
"properties_hs_created_by_user_id": {
"description": "User ID of the task creator.",
"type": ["null", "number"]
},
"properties_hs_createdate": {
"description": "The date and time when the task was created",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_engagement_source": {
"description": "Source of the task engagement.",
"type": ["null", "string"]
},
"properties_hs_engagement_source_id": {
"description": "ID of the task engagement source.",
"type": ["null", "string"]
},
"properties_hs_follow_up_action": {
"description": "Follow-up action associated with the task.",
"type": ["null", "string"]
},
"properties_hs_gdpr_deleted": {
"description": "Flag indicating if the task is deleted due to GDPR compliance.",
"type": ["null", "boolean"]
},
"properties_hs_lastmodifieddate": {
"description": "The date and time when the task was last modified",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_merged_object_ids": {
"description": "List of object IDs merged with this task.",
"type": ["null", "string"]
},
"properties_hs_modified_by": {
"description": "User who last modified the task.",
"type": ["null", "number"]
},
"properties_hs_msteams_message_id": {
"description": "ID of the Microsoft Teams message associated with the task.",
"type": ["null", "string"]
},
"properties_hs_num_associated_companies": {
"description": "Number of companies associated with the task.",
"type": ["null", "number"]
},
"properties_hs_num_associated_contacts": {
"description": "Number of contacts associated with the task.",
"type": ["null", "number"]
},
"properties_hs_num_associated_deals": {
"description": "Number of deals associated with the task.",
"type": ["null", "number"]
},
"properties_hs_num_associated_queue_objects": {
"description": "Number of queue objects associated with the task.",
"type": ["null", "number"]
},
"properties_hs_num_associated_tickets": {
"description": "Number of tickets associated with the task.",
"type": ["null", "number"]
},
"properties_hs_object_id": {
"description": "ID of the task object.",
"type": ["null", "number"]
},
"properties_hs_product_name": {
"description": "Product name associated with the task.",
"type": ["null", "string"]
},
"properties_hs_queue_membership_ids": {
"description": "List of queue membership IDs associated with the task.",
"type": ["null", "string"]
},
"properties_hs_scheduled_tasks": {
"description": "Flag indicating if the task is scheduled.",
"type": ["null", "string"]
},
"properties_hs_task_body": {
"description": "Full body content of the task.",
"type": ["null", "string"]
},
"properties_hs_task_completion_date": {
"description": "The date and time when the task was completed",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_task_contact_timezone": {
"description": "Time zone of the contact associated with the task.",
"type": ["null", "string"]
},
"properties_hs_task_for_object_type": {
"description": "Type of object the task is for (e.g., contact, deal).",
"type": ["null", "string"]
},
"properties_hs_task_is_all_day": {
"description": "Flag indicating if the task is an all-day task.",
"type": ["null", "boolean"]
},
"properties_hs_task_last_contact_outreach": {
"description": "The date and time of the last contact outreach related to the task",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_task_last_sales_activity_timestamp": {
"description": "The date and time of the last sales activity related to the task",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_task_priority": {
"description": "Priority level of the task.",
"type": ["null", "string"]
},
"properties_hs_task_probability_to_complete": {
"description": "Probability of completing the task.",
"type": ["null", "number"]
},
"properties_hs_task_relative_reminders": {
"description": "List of relative reminders set for the task.",
"type": ["null", "string"]
},
"properties_hs_task_reminders": {
"description": "List of reminders set for the task.",
"type": ["null", "string"]
},
"properties_hs_task_repeat_interval": {
"description": "Repeat interval for recurring tasks.",
"type": ["null", "string"]
},
"properties_hs_task_send_default_reminder": {
"description": "Flag indicating if default reminders should be sent for the task.",
"type": ["null", "boolean"]
},
"properties_hs_task_sequence_enrollment_active": {
"description": "Flag indicating if the task sequence enrollment is active.",
"type": ["null", "boolean"]
},
"properties_hs_task_sequence_step_enrollment_id": {
"description": "ID of the task sequence step enrollment.",
"type": ["null", "string"]
},
"properties_hs_task_sequence_step_order": {
"description": "Order of the task within the sequence step.",
"type": ["null", "number"]
},
"properties_hs_task_status": {
"description": "Status of the task (e.g., open, closed).",
"type": ["null", "string"]
},
"properties_hs_task_subject": {
"description": "Subject of the task.",
"type": ["null", "string"]
},
"properties_hs_task_template_id": {
"description": "ID of the task template.",
"type": ["null", "number"]
},
"properties_hs_task_type": {
"description": "Type of the task (e.g., call, email).",
"type": ["null", "string"]
},
"properties_hs_timestamp": {
"description": "The timestamp associated with the task",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_unique_creation_key": {
"description": "Unique key for identifying task creation.",
"type": ["null", "string"]
},
"properties_hs_unique_id": {
"description": "Unique ID for the task.",
"type": ["null", "string"]
},
"properties_hs_updated_by_user_id": {
"description": "User ID of the user who last updated the task.",
"type": ["null", "number"]
},
"properties_hs_user_ids_of_all_notification_followers": {
"description": "List of user IDs following notifications for the task.",
"type": ["null", "string"]
},
"properties_hs_user_ids_of_all_notification_unfollowers": {
"description": "List of user IDs not following notifications for the task.",
"type": ["null", "string"]
},
"properties_hs_user_ids_of_all_owners": {
"description": "List of user IDs who are owners of the task.",
"type": ["null", "string"]
},
"properties_hubspot_owner_assigneddate": {
"description": "The date and time when the task was assigned to an owner",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hubspot_owner_id": {
"description": "ID of the HubSpot owner associated with the task.",
"type": ["null", "string"]
},
"properties_hubspot_team_id": {
"description": "ID of the HubSpot team associated with the task.",
"type": ["null", "string"]
},
"properties_hs_all_owner_ids": {
"description": "List of all user IDs who are owners of this task.",
"type": ["null", "string"]
},
"properties_hs_all_team_ids": {
"description": "List of all team IDs assigned to this task.",
"type": ["null", "string"]
},
"properties_hs_all_accessible_team_ids": {
"description": "List of all team IDs that have access to this task.",
"type": ["null", "string"]
},
"createdAt": {
"description": "The date and time when the task was created",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "The date and time when the task was last updated",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates if the task has been archived",
"type": ["null", "boolean"]
},
"contacts": {
"description": "List of contacts associated with the task",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"deals": {
"description": "List of deals associated with the task",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"companies": {
"description": "List of companies associated with the task",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"tickets": {
"description": "List of tickets associated with the task",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
}
}
}

159
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/feedback_submissions.json
View File

@@ -1,159 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "Unique identifier for the feedback submission.",
"type": ["null", "string"]
},
"properties": {
"description": "Additional properties related to the feedback submission.",
"type": ["null", "object"],
"properties": {
"hs_all_accessible_team_ids": {
"description": "IDs of teams that have access to the submission.",
"type": ["null", "string"]
},
"hs_all_assigned_business_unit_ids": {
"description": "IDs of business units assigned to the submission.",
"type": ["null", "string"]
},
"hs_contact_email_rollup": {
"description": "Rollup of contact email addresses associated.",
"type": ["null", "string"]
},
"hs_contact_id": {
"description": "ID of the contact related to the submission.",
"type": ["null", "string"]
},
"hs_content": {
"description": "Content of the feedback submission.",
"type": ["null", "string"]
},
"hs_created_by_user_id": {
"description": "ID of the user who created the submission.",
"type": ["null", "string"]
},
"hs_createdate": {
"description": "The date when the submission was created.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_engagement_id": {
"description": "ID of the engagement associated with the submission.",
"type": ["null", "string"]
},
"hs_form_guid": {
"description": "GUID of the form used for the submission.",
"type": ["null", "string"]
},
"hs_ingestion_id": {
"description": "ID of the ingestion associated with the submission.",
"type": ["null", "string"]
},
"hs_knowledge_article_id": {
"description": "ID of the knowledge article linked.",
"type": ["null", "string"]
},
"hs_lastmodifieddate": {
"description": "The date when the submission was last modified.",
"type": ["null", "string"]
},
"hs_merged_object_ids": {
"description": "IDs of merged objects related to the submission.",
"type": ["null", "string"]
},
"hs_object_id": {
"description": "ID of the object associated with the submission.",
"type": ["null", "string"]
},
"hs_response_group": {
"description": "The group associated with the response.",
"type": ["null", "string"]
},
"hs_submission_name": {
"description": "Name of the feedback submission.",
"type": ["null", "string"]
},
"hs_submission_timestamp": {
"description": "Timestamp of the submission.",
"type": ["null", "string"]
},
"hs_submission_url": {
"description": "URL of the feedback submission.",
"type": ["null", "string"]
},
"hs_survey_channel": {
"description": "Channel through which the survey was conducted.",
"type": ["null", "string"]
},
"hs_survey_id": {
"description": "ID of the survey associated with the submission.",
"type": ["null", "string"]
},
"hs_survey_name": {
"description": "Name of the survey linked to the submission.",
"type": ["null", "string"]
},
"hs_survey_type": {
"description": "Type of the survey conducted.",
"type": ["null", "string"]
},
"hs_unique_creation_key": {
"description": "Unique key identifying the creation.",
"type": ["null", "string"]
},
"hs_updated_by_user_id": {
"description": "ID of the user who last updated the submission.",
"type": ["null", "string"]
},
"hs_user_ids_of_all_notification_followers": {
"description": "User IDs of all followers receiving notifications.",
"type": ["null", "string"]
},
"hs_user_ids_of_all_notification_unfollowers": {
"description": "User IDs of all followers who stopped notifications.",
"type": ["null", "string"]
},
"hs_user_ids_of_all_owners": {
"description": "User IDs of all owners of the submission.",
"type": ["null", "string"]
},
"hs_value": {
"description": "Value provided in the feedback submission.",
"type": ["null", "string"]
},
"hs_visitor_id": {
"description": "ID of the visitor associated with the submission.",
"type": ["null", "string"]
}
}
},
"createdAt": {
"description": "The timestamp when the feedback submission was created.",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "The timestamp of the last update made to the feedback submission.",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates if the feedback submission is archived or not.",
"type": ["null", "boolean"]
},
"archivedAt": {
"description": "The timestamp when the feedback submission was archived.",
"type": ["null", "string"],
"format": "date-time"
},
"contacts": {
"description": "List of contacts associated with the feedback submission.",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
}
}
}

112
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/goals.json
View File

@@ -1,112 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "Unique identifier for the goal.",
"type": ["null", "string"]
},
"properties": {
"description": "Represents the properties associated with the goal",
"type": ["null", "object"],
"properties": {
"hs_created_by_user_id": {
"description": "ID of the user who created the goal.",
"type": ["null", "string"]
},
"hs_createdate": {
"description": "Date and time when the goal was created.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_start_datetime": {
"description": "Start date and time of the goal period.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_end_datetime": {
"description": "End date and time of the goal period.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_goal_name": {
"description": "Name of the goal.",
"type": ["null", "string"]
},
"hs_lastmodifieddate": {
"description": "Date and time when the goal was last modified.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_kpi_value_last_calculated_at": {
"description": "Date and time when the KPI value was last calculated.",
"type": ["null", "string"],
"format": "date-time"
},
"hs_object_id": {
"description": "ID of the object associated with the goal.",
"type": ["null", "string"]
},
"hs_target_amount": {
"description": "Target amount set for the goal.",
"type": ["null", "string"]
}
}
},
"properties_hs_created_by_user_id": {
"description": "ID of the user who created the goal.",
"type": ["null", "string"]
},
"properties_hs_createdate": {
"description": "Date and time when the goal was created.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_start_datetime": {
"description": "Start date and time of the goal period.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_end_datetime": {
"description": "End date and time of the goal period.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_goal_name": {
"description": "Name of the goal.",
"type": ["null", "string"]
},
"properties_hs_lastmodifieddate": {
"description": "Date and time when the goal was last modified.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_kpi_value_last_calculated_at": {
"description": "Date and time when the KPI value was last calculated.",
"type": ["null", "string"],
"format": "date-time"
},
"properties_hs_object_id": {
"description": "ID of the object associated with the goal.",
"type": ["null", "string"]
},
"properties_hs_target_amount": {
"description": "Target amount set for the goal.",
"type": ["null", "string"]
},
"createdAt": {
"description": "Date and time when the goal was created.",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "Date and time when the goal was last updated.",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates if the goal is archived or not.",
"type": ["null", "boolean"]
}
}
}

24
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/line_items.json
View File

@@ -1,24 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "The unique identifier for the line item.",
"type": ["null", "string"]
},
"createdAt": {
"description": "The date and time when the line item was created.",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "The date and time when the line item was last updated.",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates whether the line item is archived or not.",
"type": ["null", "boolean"]
}
}
}

24
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/products.json
View File

@@ -1,24 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "The unique identifier of the product.",
"type": ["null", "string"]
},
"createdAt": {
"description": "The datetime when the product was created.",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "The datetime when the product was last updated.",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates whether the product is archived or active.",
"type": ["null", "boolean"]
}
}
}

155
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/shared/default_event_properties.json
View File

@@ -1,155 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"additionalProperties": true,
"properties": {
"properties_hs_asset_description": {
"type": ["null", "string"]
},
"properties_hs_asset_type": {
"type": ["null", "string"]
},
"properties_hs_browser": {
"type": ["null", "string"]
},
"properties_hs_campaign_id": {
"type": ["null", "string"]
},
"properties_hs_city": {
"type": ["null", "string"]
},
"properties_hs_country": {
"type": ["null", "string"]
},
"properties_hs_device_name": {
"type": ["null", "string"]
},
"properties_hs_device_type": {
"type": ["null", "string"]
},
"properties_hs_title": {
"type": ["null", "string"]
},
"properties_hs_form_correlation_id": {
"type": ["null", "string"]
},
"properties_hs_element_class": {
"type": ["null", "string"]
},
"properties_hs_element_id": {
"type": ["null", "string"]
},
"properties_hs_element_text": {
"type": ["null", "string"]
},
"properties_hs_language": {
"type": ["null", "string"]
},
"properties_hs_document_id": {
"type": ["null", "string"]
},
"properties_hs_presentation_id": {
"type": ["null", "string"]
},
"properties_hs_user_id": {
"type": ["null", "string"]
},
"properties_hs_link_href": {
"type": ["null", "string"]
},
"properties_hs_operating_system": {
"type": ["null", "string"]
},
"properties_hs_operating_version": {
"type": ["null", "string"]
},
"properties_hs_page_content_type": {
"type": ["null", "string"]
},
"properties_hs_page_id": {
"type": ["null", "string"]
},
"properties_hs_page_title": {
"type": ["null", "string"]
},
"properties_hs_page_url": {
"type": ["null", "string"]
},
"properties_hs_parent_module_id": {
"type": ["null", "string"]
},
"properties_hs_referrer": {
"type": ["null", "string"]
},
"properties_hs_region": {
"type": ["null", "string"]
},
"properties_hs_url": {
"type": ["null", "string"]
},
"properties_hs_screen_height": {
"type": ["null", "string"]
},
"properties_hs_screen_width": {
"type": ["null", "string"]
},
"properties_hs_touchpoint_source": {
"type": ["null", "string"]
},
"properties_hs_tracking_name": {
"type": ["null", "string"]
},
"properties_hs_user_agent": {
"type": ["null", "string"]
},
"properties_hs_utm_campaign": {
"type": ["null", "string"]
},
"properties_hs_utm_content": {
"type": ["null", "string"]
},
"properties_hs_utm_medium": {
"type": ["null", "string"]
},
"properties_hs_utm_source": {
"type": ["null", "string"]
},
"properties_hs_utm_term": {
"type": ["null", "string"]
},
"id": {
"type": ["null", "string"]
},
"properties_hs_base_url": {
"type": ["null", "string"]
},
"properties_hs_form_id": {
"type": ["null", "string"]
},
"properties_hs_form_type": {
"type": ["null", "string"]
},
"properties_hs_url_domain": {
"type": ["null", "string"]
},
"properties_hs_url_path": {
"type": ["null", "string"]
},
"properties_hs_visitor_type": {
"type": ["null", "string"]
},
"objectId": {
"type": ["null", "string"]
},
"objectType": {
"type": ["null", "string"]
},
"eventType": {
"type": ["null", "string"]
},
"occurredAt": {
"type": ["null", "string"],
"format": "date-time"
}
}
}

48
airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/tickets.json
View File

@@ -1,48 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"description": "Unique identifier for the ticket",
"type": ["null", "string"]
},
"createdAt": {
"description": "Date and time when the ticket was created",
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"description": "Date and time when the ticket was last updated",
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"description": "Indicates if the ticket is archived or not",
"type": ["null", "boolean"]
},
"contacts": {
"description": "Contacts associated with the ticket",
"type": ["null", "array"],
"items": {
"description": "Contact data",
"type": ["null", "integer"]
}
},
"companies": {
"description": "Companies associated with the ticket",
"type": ["null", "array"],
"items": {
"description": "Company data",
"type": ["null", "string"]
}
},
"deals": {
"description": "Deals associated with the ticket",
"type": ["null", "array"],
"items": {
"description": "Deal data",
"type": ["null", "string"]
}
}
}
}

282
airbyte-integrations/connectors/source-hubspot/source_hubspot/source.py
View File

@@ -1,282 +0,0 @@
#
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
#
import logging
from http import HTTPStatus
from itertools import chain
from typing import Any, Generator, List, Mapping, Optional, Tuple
from requests import HTTPError
from airbyte_cdk.models import ConfiguredAirbyteCatalog, FailureType
from airbyte_cdk.sources.declarative.declarative_stream import DeclarativeStream
from airbyte_cdk.sources.declarative.yaml_declarative_source import YamlDeclarativeSource
from airbyte_cdk.sources.source import TState
from airbyte_cdk.sources.streams import Stream
from airbyte_cdk.sources.streams.http import HttpClient
from airbyte_cdk.sources.streams.http.error_handlers import ErrorResolution, HttpStatusErrorHandler, ResponseAction
from source_hubspot.errors import HubspotInvalidAuth
from source_hubspot.streams import (
API,
BaseStream,
CompaniesWebAnalytics,
Contacts,
ContactsWebAnalytics,
CustomObject,
DealsWebAnalytics,
EngagementsCallsWebAnalytics,
EngagementsEmailsWebAnalytics,
EngagementsMeetingsWebAnalytics,
EngagementsNotesWebAnalytics,
EngagementsTasksWebAnalytics,
GoalsWebAnalytics,
LineItemsWebAnalytics,
ProductsWebAnalytics,
TicketsWebAnalytics,
WebAnalyticsStream,
)
"""
https://github.com/airbytehq/oncall/issues/3800
we use start date 2006-01-01 as date of creation of Hubspot to retrieve all data if start date was not provided
"""
DEFAULT_START_DATE = "2006-06-01T00:00:00Z"
scopes = {
"campaigns": {"crm.lists.read"},
"companies": {"crm.objects.contacts.read", "crm.objects.companies.read"},
"companies_property_history": {"crm.objects.companies.read"},
"contact_lists": {"crm.lists.read"},
"contacts": {"crm.objects.contacts.read"},
"contacts_property_history": {"crm.objects.contacts.read"},
"deal_pipelines": {"crm.objects.contacts.read"},
"deal_splits": {"crm.objects.deals.read"},
"deals": {"contacts", "crm.objects.deals.read"},
"deals_property_history": {"crm.objects.deals.read"},
"email_events": {"content"},
"email_subscriptions": {"content"},
"engagements": {"crm.objects.companies.read", "crm.objects.contacts.read", "crm.objects.deals.read", "tickets", "e-commerce"},
"engagements_calls": {"crm.objects.contacts.read"},
"engagements_emails": {"crm.objects.contacts.read", "sales-email-read"},
"engagements_meetings": {"crm.objects.contacts.read"},
"engagements_notes": {"crm.objects.contacts.read"},
"engagements_tasks": {"crm.objects.contacts.read"},
"marketing_emails": {"content"},
"deals_archived": {"contacts", "crm.objects.deals.read"},
"forms": {"forms"},
"form_submissions": {"forms"},
"goals": {"crm.objects.goals.read"},
"leads": {"crm.objects.contacts.read", "crm.objects.companies.read", "crm.objects.leads.read"},
"line_items": {"e-commerce", "crm.objects.line_items.read"},
"owners": {"crm.objects.owners.read"},
"owners_archived": {"crm.objects.owners.read"},
"products": {"e-commerce"},
"subscription_changes": {"content"},
"ticket_pipelines": {
"media_bridge.read",
"tickets",
"crm.schemas.custom.read",
"e-commerce",
"timeline",
"contacts",
"crm.schemas.contacts.read",
"crm.objects.contacts.read",
"crm.objects.contacts.write",
"crm.objects.deals.read",
"crm.schemas.quotes.read",
"crm.objects.deals.write",
"crm.objects.companies.read",
"crm.schemas.companies.read",
"crm.schemas.deals.read",
"crm.schemas.line_items.read",
"crm.objects.companies.write",
},
"tickets": {"tickets"},
"workflows": {"automation"},
}
properties_scopes = {
"companies_property_history": {"crm.schemas.companies.read"},
"contacts_property_history": {"crm.schemas.contacts.read"},
"deals_property_history": {"crm.schemas.deals.read"},
}
def scope_is_granted(stream: Stream, granted_scopes: List[str]) -> bool:
"""
Set of required scopes. Users need to grant at least one of the scopes for the stream to be avaialble to them
"""
granted_scopes = set(granted_scopes)
if isinstance(stream, BaseStream):
return stream.scope_is_granted(granted_scopes)
else:
# The default value is scopes for custom objects streams
return len(scopes.get(stream.name, {"crm.schemas.custom.read", "crm.objects.custom.read"}).intersection(granted_scopes)) > 0
def properties_scope_is_granted(stream: Stream, granted_scopes: List[str]) -> bool:
"""
Set of required scopes. Users need to grant at least one of the scopes for the stream to be avaialble to them
"""
granted_scopes = set(granted_scopes)
if isinstance(stream, BaseStream):
return stream.properties_scope_is_granted()
else:
return not properties_scopes.get(stream.name, set()) - granted_scopes
class SourceHubspot(YamlDeclarativeSource):
logger = logging.getLogger("airbyte")
def __init__(self, catalog: Optional[ConfiguredAirbyteCatalog], config: Optional[Mapping[str, Any]], state: TState, **kwargs):
super().__init__(catalog=catalog, config=config, state=state, **{"path_to_yaml": "manifest.yaml"})
def check_connection(self, logger: logging.Logger, config: Mapping[str, Any]) -> Tuple[bool, Optional[Any]]:
"""Check connection"""
common_params = self.get_common_params(config=config)
alive = True
error_msg = None
try:
contacts = Contacts(**common_params)
_ = contacts.properties
except HTTPError as error:
alive = False
error_msg = repr(error)
if error.response.status_code == HTTPStatus.BAD_REQUEST:
response_json = error.response.json()
error_msg = f"400 Bad Request: {response_json['message']}, please check if provided credentials are valid."
except HubspotInvalidAuth as e:
alive = False
error_msg = repr(e)
return alive, error_msg
def get_granted_scopes(self, authenticator):
try:
access_token = authenticator.get_access_token()
url = f"https://api.hubapi.com/oauth/v1/access-tokens/{access_token}"
error_resolution = ErrorResolution(
ResponseAction.RETRY, FailureType.transient_error, "Internal error attempting to get scopes."
)
error_mapping = {500: error_resolution, 502: error_resolution, 504: error_resolution}
http_client = HttpClient(
name="get hubspot granted scopes client",
logger=self.logger,
error_handler=HttpStatusErrorHandler(logger=self.logger, error_mapping=error_mapping),
)
request, response = http_client.send_request("get", url, request_kwargs={})
response.raise_for_status()
response_json = response.json()
granted_scopes = response_json["scopes"]
return granted_scopes
except Exception as e:
return False, repr(e)
@staticmethod
def get_api(config: Mapping[str, Any]) -> API:
credentials = config.get("credentials", {})
return API(credentials=credentials)
def get_common_params(self, config) -> Mapping[str, Any]:
start_date = config.get("start_date", DEFAULT_START_DATE)
credentials = config["credentials"]
api = self.get_api(config=config)
# Additional configuration is necessary for testing certain streams due to their specific restrictions.
acceptance_test_config = config.get("acceptance_test_config", {})
return dict(api=api, start_date=start_date, credentials=credentials, acceptance_test_config=acceptance_test_config)
def streams(self, config: Mapping[str, Any]) -> List[Stream]:
credentials = config.get("credentials", {})
common_params = self.get_common_params(config=config)
# Temporarily using `ConcurrentDeclarativeSource.streams()` to validate granted scopes.
streams = super().streams(config=config)
enable_experimental_streams = "enable_experimental_streams" in config and config["enable_experimental_streams"]
if enable_experimental_streams:
streams.extend(
[
ContactsWebAnalytics(**common_params),
CompaniesWebAnalytics(**common_params),
DealsWebAnalytics(**common_params),
TicketsWebAnalytics(**common_params),
EngagementsCallsWebAnalytics(**common_params),
EngagementsEmailsWebAnalytics(**common_params),
EngagementsMeetingsWebAnalytics(**common_params),
EngagementsNotesWebAnalytics(**common_params),
EngagementsTasksWebAnalytics(**common_params),
GoalsWebAnalytics(**common_params),
LineItemsWebAnalytics(**common_params),
ProductsWebAnalytics(**common_params),
]
)
api = API(credentials=credentials)
if api.is_oauth2():
authenticator = api.get_authenticator()
granted_scopes = self.get_granted_scopes(authenticator)
self.logger.info(f"The following scopes were granted: {granted_scopes}")
available_streams = [stream for stream in streams if scope_is_granted(stream, granted_scopes)]
unavailable_streams = [stream for stream in streams if not scope_is_granted(stream, granted_scopes)]
self.logger.info(f"The following streams are unavailable: {[s.name for s in unavailable_streams]}")
partially_available_streams = [stream for stream in streams if not properties_scope_is_granted(stream, granted_scopes)]
required_scoped = set(
chain(
*[
properties_scopes.get(x.name, set()) if isinstance(x, DeclarativeStream) else x.properties_scopes
for x in partially_available_streams
]
)
)
self.logger.info(
f"The following streams are partially available: {[s.name for s in partially_available_streams]}, "
f"add the following scopes to download all available data: {required_scoped}"
)
else:
self.logger.info("No scopes to grant when authenticating with API key.")
available_streams = streams
custom_object_streams = list(self.get_custom_object_streams(api=api, common_params=common_params))
if enable_experimental_streams:
custom_objects_web_analytics_streams = self.get_web_analytics_custom_objects_stream(
custom_object_stream_instances=custom_object_streams,
common_params=common_params,
)
available_streams.extend(custom_objects_web_analytics_streams)
return available_streams
def get_custom_object_streams(self, api: API, common_params: Mapping[str, Any]):
for entity, fully_qualified_name, schema, custom_properties in api.get_custom_objects_metadata():
yield CustomObject(
entity=entity,
schema=schema,
fully_qualified_name=fully_qualified_name,
custom_properties=custom_properties,
**common_params,
)
def get_web_analytics_custom_objects_stream(
self, custom_object_stream_instances: List[CustomObject], common_params: Any
) -> Generator[WebAnalyticsStream, None, None]:
for custom_object_stream_instance in custom_object_stream_instances:
def __init__(self, **kwargs: Any):
parent = custom_object_stream_instance.__class__(
entity=custom_object_stream_instance.entity,
schema=custom_object_stream_instance.schema,
fully_qualified_name=custom_object_stream_instance.fully_qualified_name,
custom_properties=custom_object_stream_instance.custom_properties,
**common_params,
)
super(self.__class__, self).__init__(parent=parent, **kwargs)
custom_web_analytics_stream_class = type(
f"{custom_object_stream_instance.name.capitalize()}WebAnalytics", (WebAnalyticsStream,), {"__init__": __init__}
)
yield custom_web_analytics_stream_class(**common_params)

139
airbyte-integrations/connectors/source-hubspot/source_hubspot/spec.yaml
View File

@@ -1,139 +0,0 @@
documentationUrl: https://docs.airbyte.com/integrations/sources/hubspot
connectionSpecification:
$schema: http://json-schema.org/draft-07/schema#
title: HubSpot Source Spec
type: object
required:
- credentials
additionalProperties: true
properties:
start_date:
type: string
title: Start date
pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$
description: >-
UTC date and time in the format 2017-01-25T00:00:00Z. Any data before
this date will not be replicated. If not set, "2006-06-01T00:00:00Z" (Hubspot creation date) will be used as start date.
It's recommended to provide relevant to your data start date value to optimize synchronization.
examples:
- "2017-01-25T00:00:00Z"
format: date-time
credentials:
title: Authentication
description: Choose how to authenticate to HubSpot.
type: object
oneOf:
- type: object
title: OAuth
required:
- client_id
- client_secret
- refresh_token
- credentials_title
properties:
credentials_title:
type: string
title: Auth Type
description: Name of the credentials
const: OAuth Credentials
order: 0
client_id:
title: Client ID
description: >-
The Client ID of your HubSpot developer application. See the <a
href="https://legacydocs.hubspot.com/docs/methods/oauth2/oauth2-quickstart">Hubspot docs</a>
if you need help finding this ID.
type: string
examples:
- "123456789000"
client_secret:
title: Client Secret
description: >-
The client secret for your HubSpot developer application. See the <a
href="https://legacydocs.hubspot.com/docs/methods/oauth2/oauth2-quickstart">Hubspot docs</a>
if you need help finding this secret.
type: string
examples:
- secret
airbyte_secret: true
refresh_token:
title: Refresh Token
description: >-
Refresh token to renew an expired access token. See the <a
href="https://legacydocs.hubspot.com/docs/methods/oauth2/oauth2-quickstart">Hubspot docs</a>
if you need help finding this token.
type: string
examples:
- refresh_token
airbyte_secret: true
- type: object
title: Private App
required:
- access_token
- credentials_title
properties:
credentials_title:
type: string
title: Auth Type
description: Name of the credentials set
const: Private App Credentials
order: 0
access_token:
title: Access token
description: >-
HubSpot Access token. See the <a
href="https://developers.hubspot.com/docs/api/private-apps">Hubspot docs</a>
if you need help finding this token.
type: string
airbyte_secret: true
enable_experimental_streams:
title: Enable experimental streams
description: If enabled then experimental streams become available for sync.
type: boolean
default: false
num_worker:
type: integer
title: Number of concurrent workers
minimum: 1
maximum: 40
default: 3
examples: [1, 2, 3]
description: The number of worker threads to use for the sync.
advanced_auth:
auth_flow_type: oauth2.0
predicate_key:
- credentials
- credentials_title
predicate_value: OAuth Credentials
oauth_config_specification:
complete_oauth_output_specification:
type: object
additionalProperties: false
properties:
refresh_token:
type: string
path_in_connector_config:
- credentials
- refresh_token
complete_oauth_server_input_specification:
type: object
additionalProperties: false
properties:
client_id:
type: string
client_secret:
type: string
complete_oauth_server_output_specification:
type: object
additionalProperties: false
properties:
client_id:
type: string
path_in_connector_config:
- credentials
- client_id
client_secret:
type: string
path_in_connector_config:
- credentials
- client_secret

1944
airbyte-integrations/connectors/source-hubspot/source_hubspot/streams.py
View File

File diff suppressed because it is too large Load Diff

127
airbyte-integrations/connectors/source-hubspot/unit_tests/conftest.py
View File

@@ -2,14 +2,20 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import pytest
from source_hubspot.source import SourceHubspot
from source_hubspot.streams import API
import sys
from pathlib import Path
import pytest
from airbyte_cdk.models import ConfiguredAirbyteCatalog
from airbyte_cdk.sources.declarative.yaml_declarative_source import YamlDeclarativeSource
from airbyte_cdk.test.catalog_builder import CatalogBuilder
from airbyte_cdk.test.entrypoint_wrapper import EntrypointOutput, read
from airbyte_cdk.test.state_builder import StateBuilder
pytest_plugins = ["airbyte_cdk.test.utils.manifest_only_fixtures"]
NUMBER_OF_PROPERTIES = 2000
OBJECTS_WITH_DYNAMIC_SCHEMA = [
"calls",
@@ -46,13 +52,6 @@ def oauth_config_fixture():
}
@pytest.fixture(name="common_params")
def common_params_fixture(config):
source = SourceHubspot(config, None, None)
common_params = source.get_common_params(config=config)
return common_params
@pytest.fixture(name="config_invalid_client_id")
def config_invalid_client_id_fixture():
return {
@@ -113,18 +112,34 @@ def migrated_properties_list():
]
@pytest.fixture(name="api")
def api(some_credentials):
return API(some_credentials)
@pytest.fixture
def http_mocker():
return None
def _get_manifest_path() -> Path:
source_declarative_manifest_path = Path("/airbyte/integration_code/source_declarative_manifest")
if source_declarative_manifest_path.exists():
return source_declarative_manifest_path
return Path(__file__).parent.parent
_SOURCE_FOLDER_PATH = _get_manifest_path()
_YAML_FILE_PATH = _SOURCE_FOLDER_PATH / "manifest.yaml"
sys.path.append(str(_SOURCE_FOLDER_PATH)) # to allow loading custom components
def get_source(config, state=None) -> YamlDeclarativeSource:
catalog = CatalogBuilder().build()
state = StateBuilder().build() if not state else state
return YamlDeclarativeSource(path_to_yaml=str(_YAML_FILE_PATH), catalog=catalog, config=config, state=state)
def find_stream(stream_name, config, state=None):
for stream in SourceHubspot(config=config, catalog=None, state=state).streams(config=config):
state = StateBuilder().build() if not state else state
streams = get_source(config, state).streams(config=config)
for stream in streams:
if stream.name == stream_name:
return stream
raise ValueError(f"Stream {stream_name} not found")
@@ -136,7 +151,7 @@ def patch_time(mocker):
def read_from_stream(cfg, stream: str, sync_mode, state=None, expecting_exception: bool = False) -> EntrypointOutput:
return read(SourceHubspot(cfg, None, None), cfg, CatalogBuilder().with_stream(stream, sync_mode).build(), state, expecting_exception)
return read(get_source(cfg, state), cfg, CatalogBuilder().with_stream(stream, sync_mode).build(), state, expecting_exception)
@pytest.fixture()
@@ -173,3 +188,81 @@ def mock_dynamic_schema_requests_with_skip(requests_mock, object_to_skip: list):
json=[{"name": "hs__test_field", "type": "enumeration"}],
status_code=200,
)
@pytest.fixture(name="custom_object_schema")
def custom_object_schema_fixture():
return {
"labels": {"this": "that"},
"requiredProperties": ["name"],
"searchableProperties": ["name"],
"primaryDisplayProperty": "name",
"secondaryDisplayProperties": [],
"archived": False,
"restorable": True,
"metaType": "PORTAL_SPECIFIC",
"id": "7232155",
"fullyQualifiedName": "p19936848_Animal",
"createdAt": "2022-06-17T18:40:27.019Z",
"updatedAt": "2022-06-17T18:40:27.019Z",
"objectTypeId": "2-7232155",
"properties": [
{
"name": "name",
"label": "Animal name",
"type": "string",
"fieldType": "text",
"description": "The animal name.",
"groupName": "animal_information",
"options": [],
"displayOrder": -1,
"calculated": False,
"externalOptions": False,
"hasUniqueValue": False,
"hidden": False,
"hubspotDefined": False,
"modificationMetadata": {"archivable": True, "readOnlyDefinition": True, "readOnlyValue": False},
"formField": True,
}
],
"associations": [],
"name": "animals",
}
@pytest.fixture(name="configured_catalog")
def configured_catalog_fixture():
configured_catalog = {
"streams": [
{
"stream": {
"name": "quotes",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": True,
"default_cursor_field": ["updatedAt"],
},
"sync_mode": "incremental",
"cursor_field": ["updatedAt"],
"destination_sync_mode": "append",
}
]
}
return ConfiguredAirbyteCatalog.parse_obj(configured_catalog)
@pytest.fixture(name="expected_custom_object_json_schema")
def expected_custom_object_json_schema():
return {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"additionalProperties": True,
"properties": {
"id": {"type": ["null", "string"]},
"createdAt": {"type": ["null", "string"], "format": "date-time"},
"updatedAt": {"type": ["null", "string"], "format": "date-time"},
"archived": {"type": ["null", "boolean"]},
"properties": {"type": ["null", "object"], "properties": {"name": {"type": ["null", "string"]}}},
"properties_name": {"type": ["null", "string"]},
},
}

8
airbyte-integrations/connectors/source-hubspot/unit_tests/integrations/__init__.py
View File

@@ -11,7 +11,6 @@ from airbyte_cdk.test.entrypoint_wrapper import EntrypointOutput, read
from airbyte_cdk.test.mock_http import HttpResponse, HttpRequest, HttpMocker
from airbyte_cdk.test.mock_http.response_builder import FieldPath, HttpResponseBuilder, RecordBuilder, create_record_builder, find_template
from airbyte_cdk.models import AirbyteStateMessage, SyncMode
from source_hubspot import SourceHubspot
from .config_builder import ConfigBuilder
from .request_builders.api import CustomObjectsRequestBuilder, OAuthRequestBuilder, PropertiesRequestBuilder, ScopesRequestBuilder
@@ -19,6 +18,7 @@ from .request_builders.streams import CRMStreamRequestBuilder, IncrementalCRMStr
from .response_builder.helpers import RootHttpResponseBuilder
from .response_builder.api import ScopesResponseBuilder
from .response_builder.streams import GenericResponseBuilder, HubspotStreamResponseBuilder
from ..conftest import get_source
OBJECTS_WITH_DYNAMIC_SCHEMA = [
"calls",
@@ -118,10 +118,6 @@ class HubspotTestCase:
response = GenericResponseBuilder().with_value("access_token", token).with_value("expires_in", 7200).build()
http_mocker.post(req, response)
@classmethod
def mock_scopes(cls, http_mocker: HttpMocker, token: str, scopes: List[str]):
http_mocker.get(ScopesRequestBuilder().with_access_token(token).build(), ScopesResponseBuilder(scopes).build())
@classmethod
def mock_custom_objects(cls, http_mocker: HttpMocker):
http_mocker.get(
@@ -190,4 +186,4 @@ class HubspotTestCase:
def read_from_stream(
cls, cfg, stream: str, sync_mode: SyncMode, state: Optional[List[AirbyteStateMessage]] = None, expecting_exception: bool = False
) -> EntrypointOutput:
return read(SourceHubspot(cfg, None, None), cfg, cls.catalog(stream, sync_mode), state, expecting_exception)
return read(get_source(cfg, state), cfg, cls.catalog(stream, sync_mode), state, expecting_exception)

9
airbyte-integrations/connectors/source-hubspot/unit_tests/integrations/test_engagements_calls.py
View File

@@ -10,7 +10,7 @@ from airbyte_cdk.models import SyncMode
from airbyte_cdk.test.mock_http import HttpMocker, HttpResponse
from airbyte_cdk.test.mock_http.response_builder import FieldPath
from . import HubspotTestCase
from . import OBJECTS_WITH_DYNAMIC_SCHEMA, HubspotTestCase
from .request_builders.streams import CRMStreamRequestBuilder
from .response_builder.streams import HubspotStreamResponseBuilder
@@ -51,7 +51,6 @@ class TestEngagementCallsStream(HubspotTestCase):
def _set_up_oauth(self, http_mocker: HttpMocker):
self.mock_oauth(http_mocker, self.ACCESS_TOKEN)
self.mock_scopes(http_mocker, self.ACCESS_TOKEN, self.SCOPES)
def _set_up_requests(
self, http_mocker: HttpMocker, with_oauth: bool = False, with_dynamic_schemas: bool = True, entities: Optional[List[str]] = None
@@ -69,8 +68,9 @@ class TestEngagementCallsStream(HubspotTestCase):
http_mocker,
with_oauth=True,
with_dynamic_schemas=True,
entities=["calls", "company", "contact", "emails", "meetings", "notes", "tasks"],
entities=OBJECTS_WITH_DYNAMIC_SCHEMA,
)
self.mock_response(http_mocker, self.request(), self.response())
self.read_from_stream(self.oauth_config(), self.STREAM_NAME, SyncMode.full_refresh)
@HttpMocker()
@@ -79,7 +79,7 @@ class TestEngagementCallsStream(HubspotTestCase):
http_mocker,
with_oauth=True,
with_dynamic_schemas=True,
entities=["calls", "company", "contact", "emails", "leads", "meetings", "notes", "tasks"],
entities=OBJECTS_WITH_DYNAMIC_SCHEMA,
)
self.mock_response(http_mocker, self.request(), self.response())
output = self.read_from_stream(self.oauth_config(), self.STREAM_NAME, SyncMode.full_refresh)
@@ -123,7 +123,6 @@ class TestEngagementCallsStream(HubspotTestCase):
@HttpMocker()
def test_given_missing_scopes_error_when_read_then_stop_sync(self, http_mocker: HttpMocker):
self.mock_oauth(http_mocker, self.ACCESS_TOKEN)
self.mock_scopes(http_mocker, self.ACCESS_TOKEN, [])
self.mock_custom_objects_streams(http_mocker)
self.read_from_stream(self.oauth_config(), self.STREAM_NAME, SyncMode.full_refresh, expecting_exception=True)

6
airbyte-integrations/connectors/source-hubspot/unit_tests/integrations/test_leads.py
View File

@@ -51,7 +51,6 @@ class TestLeadsStream(HubspotTestCase):
def _set_up_oauth(self, http_mocker: HttpMocker):
self.mock_oauth(http_mocker, self.ACCESS_TOKEN)
self.mock_scopes(http_mocker, self.ACCESS_TOKEN, self.SCOPES)
def _set_up_requests(self, http_mocker: HttpMocker, with_oauth: bool = False, with_dynamic_schema: bool = True):
if with_oauth:
@@ -63,12 +62,12 @@ class TestLeadsStream(HubspotTestCase):
@HttpMocker()
def test_given_oauth_authentication_when_read_then_perform_authenticated_queries(self, http_mocker: HttpMocker):
self._set_up_requests(http_mocker, with_oauth=True, with_dynamic_schema=False)
self._set_up_requests(http_mocker, with_oauth=True, with_dynamic_schema=True)
self.read_from_stream(self.oauth_config(), self.STREAM_NAME, SyncMode.full_refresh)
@HttpMocker()
def test_given_records_when_read_extract_desired_records(self, http_mocker: HttpMocker):
self._set_up_requests(http_mocker, with_oauth=True, with_dynamic_schema=False)
self._set_up_requests(http_mocker, with_oauth=True, with_dynamic_schema=True)
self.mock_response(http_mocker, self.request(), self.response())
output = self.read_from_stream(self.oauth_config(), self.STREAM_NAME, SyncMode.full_refresh)
assert len(output.records) == 1
@@ -111,7 +110,6 @@ class TestLeadsStream(HubspotTestCase):
@HttpMocker()
def test_given_missing_scopes_error_when_read_then_stop_sync(self, http_mocker: HttpMocker):
self.mock_oauth(http_mocker, self.ACCESS_TOKEN)
self.mock_scopes(http_mocker, self.ACCESS_TOKEN, [])
self.mock_custom_objects_streams(http_mocker)
self.read_from_stream(self.oauth_config(), self.STREAM_NAME, SyncMode.full_refresh, expecting_exception=True)

2
airbyte-integrations/connectors/source-hubspot/unit_tests/integrations/test_owners_archived.py
View File

@@ -43,8 +43,8 @@ class TestOwnersArchivedStream(HubspotTestCase):
@HttpMocker()
def test_given_one_page_when_read_stream_oauth_then_return_records(self, http_mocker: HttpMocker):
self.mock_oauth(http_mocker, self.ACCESS_TOKEN)
self.mock_scopes(http_mocker, self.ACCESS_TOKEN, self.SCOPES)
self.mock_custom_objects(http_mocker)
self.mock_dynamic_schema_requests(http_mocker)
self.mock_response(http_mocker, self.request().build(), self.response().build())
output = self.read_from_stream(self.oauth_config(), self.STREAM_NAME, SyncMode.full_refresh)
assert len(output.records) == 1

620
airbyte-integrations/connectors/source-hubspot/unit_tests/integrations/test_web_analytics_streams.py
View File

@@ -1,620 +0,0 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
import http
from datetime import datetime, timedelta
from typing import List, Optional, Tuple
import freezegun
import mock
import pytest
import pytz
from airbyte_cdk.models import AirbyteStateBlob, AirbyteStateMessage, AirbyteStateType, AirbyteStreamState, StreamDescriptor, SyncMode
from airbyte_cdk.test.mock_http import HttpMocker, HttpResponse
from airbyte_cdk.test.mock_http.response_builder import FieldPath
from . import HubspotTestCase
from .request_builders.streams import CRMStreamRequestBuilder, IncrementalCRMStreamRequestBuilder, WebAnalyticsRequestBuilder
from .response_builder.streams import HubspotStreamResponseBuilder
CRM_STREAMS = (
("tickets_web_analytics", "tickets", "ticket", ["contacts", "deals", "companies"]),
("deals_web_analytics", "deals", "deal", ["contacts", "companies", "line_items"]),
("companies_web_analytics", "companies", "company", ["contacts"]),
("contacts_web_analytics", "contacts", "contact", ["contacts", "companies"]),
("engagements_calls_web_analytics", "engagements_calls", "calls", ["contacts", "deals", "companies", "tickets"]),
("engagements_emails_web_analytics", "engagements_emails", "emails", ["contacts", "deals", "companies", "tickets"]),
("engagements_meetings_web_analytics", "engagements_meetings", "meetings", ["contacts", "deals", "companies", "tickets"]),
("engagements_notes_web_analytics", "engagements_notes", "notes", ["contacts", "deals", "companies", "tickets"]),
("engagements_tasks_web_analytics", "engagements_tasks", "tasks", ["contacts", "deals", "companies", "tickets"]),
)
CRM_INCREMENTAL_STREAMS = (
("goals_web_analytics", "goals", "goal_targets", []),
("line_items_web_analytics", "line_items", "line_item", []),
("products_web_analytics", "products", "product", []),
)
class WebAnalyticsTestCase(HubspotTestCase):
PARENT_CURSOR_FIELD = "updatedAt"
@classmethod
def response_builder(cls, stream):
return HubspotStreamResponseBuilder.for_stream(stream)
@classmethod
def web_analytics_request(
cls,
stream: str,
token: str,
object_id: str,
object_type: str,
start_date: Optional[str] = None,
end_date: Optional[str] = None,
first_page: bool = True,
):
start_date = start_date or cls.dt_str(cls.start_date())
end_date = end_date or cls.dt_str(cls.now())
query = {"limit": 100, "occurredAfter": start_date, "occurredBefore": end_date, "objectId": object_id, "objectType": object_type}
if not first_page:
query.update(cls.response_builder(stream).pagination_strategy.NEXT_PAGE_TOKEN)
return WebAnalyticsRequestBuilder().with_token(token).with_query(query).build()
@classmethod
def web_analytics_response(
cls, stream: str, with_pagination: bool = False, updated_on: Optional[str] = None, id: Optional[str] = None
) -> HttpResponse:
updated_on = updated_on or cls.dt_str(cls.updated_at())
record = cls.record_builder(stream, FieldPath(cls.CURSOR_FIELD)).with_field(FieldPath(cls.CURSOR_FIELD), updated_on)
if id:
record = record.with_field(FieldPath("objectId"), id)
response = cls.response_builder(stream).with_record(record)
if with_pagination:
response = response.with_pagination()
return response.build()
@classmethod
def mock_parent_object(
cls,
http_mocker: HttpMocker,
object_ids: List[str],
object_type: str,
stream_name: str,
associations: List[str],
properties: List[str],
first_page: bool = True,
with_pagination: bool = False,
date_range: Optional[Tuple[str, ...]] = None,
):
response_builder = cls.response_builder(stream_name)
for object_id in object_ids:
record = (
cls.record_builder(stream_name, FieldPath(cls.PARENT_CURSOR_FIELD))
.with_field(FieldPath(cls.PARENT_CURSOR_FIELD), cls.dt_str(cls.updated_at()))
.with_field(FieldPath("id"), object_id)
)
response_builder = response_builder.with_record(record)
if with_pagination:
response_builder = response_builder.with_pagination()
request_builder = CRMStreamRequestBuilder().for_entity(object_type).with_associations(associations).with_properties(properties)
if not first_page:
request_builder = request_builder.with_page_token(response_builder.pagination_strategy.NEXT_PAGE_TOKEN)
http_mocker.get(request_builder.build(), response_builder.build())
@freezegun.freeze_time("2024-03-03T14:42:00Z")
class TestCRMWebAnalyticsStream(WebAnalyticsTestCase):
SCOPES = ["tickets", "crm.objects.contacts.read", "crm.objects.companies.read", "contacts", "crm.objects.deals.read", "oauth"]
@classmethod
def extended_dt_ranges(cls) -> Tuple[Tuple[str, ...], ...]:
return (
(cls.dt_str(cls.now() - timedelta(days=60)), cls.dt_str(cls.now() - timedelta(days=30))),
(cls.dt_str(cls.now() - timedelta(days=30)), cls.dt_str(cls.now())),
)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_one_page_when_read_stream_oauth_then_return_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
self.mock_oauth(http_mocker, self.ACCESS_TOKEN)
self.mock_scopes(http_mocker, self.ACCESS_TOKEN, self.SCOPES)
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.MOCK_PROPERTIES_FOR_SCHEMA_LOADER)
self.mock_dynamic_schema_requests(http_mocker)
self.mock_parent_object(
http_mocker, [self.OBJECT_ID], object_type, parent_stream_name, parent_stream_associations, list(self.PROPERTIES.keys())
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type),
self.web_analytics_response(stream_name),
)
output = self.read_from_stream(self.oauth_config(), stream_name, SyncMode.full_refresh)
assert len(output.records) == 1
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_one_page_when_read_stream_private_token_then_return_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.PROPERTIES)
self.mock_dynamic_schema_requests(http_mocker)
self.mock_parent_object(
http_mocker, [self.OBJECT_ID], object_type, parent_stream_name, parent_stream_associations, list(self.PROPERTIES.keys())
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type),
self.web_analytics_response(stream_name),
)
output = self.read_from_stream(self.private_token_config(self.ACCESS_TOKEN), stream_name, SyncMode.full_refresh)
assert len(output.records) == 1
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_two_pages_when_read_then_return_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.PROPERTIES)
self.mock_dynamic_schema_requests(http_mocker)
self.mock_parent_object(
http_mocker, [self.OBJECT_ID], object_type, parent_stream_name, parent_stream_associations, list(self.PROPERTIES.keys())
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type),
self.web_analytics_response(stream_name, with_pagination=True),
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type, first_page=False),
self.web_analytics_response(stream_name),
)
output = self.read_from_stream(self.private_token_config(self.ACCESS_TOKEN), stream_name, SyncMode.full_refresh)
assert len(output.records) == 2
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_two_parent_pages_when_read_then_return_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.PROPERTIES)
self.mock_dynamic_schema_requests(http_mocker)
self.mock_parent_object(
http_mocker,
[self.OBJECT_ID],
object_type,
parent_stream_name,
parent_stream_associations,
with_pagination=True,
properties=list(self.PROPERTIES.keys()),
)
self.mock_parent_object(
http_mocker,
["another_object_id"],
object_type,
parent_stream_name,
parent_stream_associations,
first_page=False,
properties=list(self.PROPERTIES.keys()),
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type),
self.web_analytics_response(stream_name),
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, "another_object_id", object_type),
self.web_analytics_response(stream_name),
)
output = self.read_from_stream(self.private_token_config(self.ACCESS_TOKEN), stream_name, SyncMode.full_refresh)
assert len(output.records) == 2
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_wide_date_range_and_multiple_parent_records_when_read_then_return_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
date_ranges = self.extended_dt_ranges()
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.PROPERTIES)
self.mock_dynamic_schema_requests(http_mocker)
start_to_end = (date_ranges[0][0], date_ranges[-1][-1])
self.mock_parent_object(
http_mocker,
[self.OBJECT_ID, "another_object_id"],
object_type,
parent_stream_name,
parent_stream_associations,
list(self.PROPERTIES.keys()),
date_range=start_to_end,
)
for dt_range in date_ranges:
for _id in (self.OBJECT_ID, "another_object_id"):
start, end = dt_range
web_analytics_response = self.web_analytics_response(stream_name)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, _id, object_type, start, end),
web_analytics_response,
)
config_start_dt = date_ranges[0][0]
output = self.read_from_stream(self.private_token_config(self.ACCESS_TOKEN, config_start_dt), stream_name, SyncMode.full_refresh)
assert len(output.records) == 4 # 2 parent objects * 2 datetime slices
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_error_response_when_read_analytics_then_get_trace_message(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.PROPERTIES)
self.mock_parent_object(
http_mocker, [self.OBJECT_ID], object_type, parent_stream_name, parent_stream_associations, list(self.PROPERTIES.keys())
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type),
HttpResponse(status_code=500, body="{}"),
)
with mock.patch("time.sleep"):
output = self.read_from_stream(self.private_token_config(self.ACCESS_TOKEN), stream_name, SyncMode.full_refresh)
assert len(output.records) == 0
assert len(output.trace_messages) > 0
assert len(output.errors) > 0
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_500_then_200_when_read_then_return_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.PROPERTIES)
self.mock_dynamic_schema_requests(http_mocker)
self.mock_parent_object(
http_mocker, [self.OBJECT_ID], object_type, parent_stream_name, parent_stream_associations, list(self.PROPERTIES.keys())
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type),
[HttpResponse(status_code=500, body="{}"), self.web_analytics_response(stream_name)],
)
with mock.patch("time.sleep"):
output = self.read_from_stream(self.private_token_config(self.ACCESS_TOKEN), stream_name, SyncMode.full_refresh)
assert len(output.records) == 1
assert len(output.trace_messages) > 0
assert len(output.errors) == 0
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_missing_scopes_error_when_read_then_hault(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
self.mock_oauth(http_mocker, self.ACCESS_TOKEN)
self.mock_scopes(http_mocker, self.ACCESS_TOKEN, [])
self.read_from_stream(self.oauth_config(), stream_name, SyncMode.full_refresh, expecting_exception=True)
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_unauthorized_error_when_read_then_hault(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.PROPERTIES)
self.mock_parent_object(
http_mocker, [self.OBJECT_ID], object_type, parent_stream_name, parent_stream_associations, list(self.PROPERTIES.keys())
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type),
HttpResponse(status_code=http.HTTPStatus.UNAUTHORIZED, body="{}"),
)
with mock.patch("time.sleep"):
output = self.read_from_stream(self.private_token_config(self.ACCESS_TOKEN), stream_name, SyncMode.full_refresh)
assert len(output.records) == 0
assert len(output.trace_messages) > 0
assert len(output.errors) > 0
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_one_page_when_read_then_get_transformed_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.PROPERTIES)
self.mock_dynamic_schema_requests(http_mocker)
self.mock_parent_object(
http_mocker, [self.OBJECT_ID], object_type, parent_stream_name, parent_stream_associations, list(self.PROPERTIES.keys())
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type),
self.web_analytics_response(stream_name),
)
output = self.read_from_stream(self.private_token_config(self.ACCESS_TOKEN), stream_name, SyncMode.full_refresh)
record = output.records[0].record.data
assert "properties" not in record
prop_fields = len([f for f in record if f.startswith("properties_")])
assert prop_fields > 0
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_one_page_when_read_then_get_no_records_filtered(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
# validate that no filter is applied on the record set received from the API response
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.PROPERTIES)
self.mock_dynamic_schema_requests(http_mocker)
self.mock_parent_object(
http_mocker, [self.OBJECT_ID], object_type, parent_stream_name, parent_stream_associations, list(self.PROPERTIES.keys())
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type),
self.web_analytics_response(stream_name, updated_on=self.dt_str(self.now() - timedelta(days=365))),
)
output = self.read_from_stream(self.private_token_config(self.ACCESS_TOKEN), stream_name, SyncMode.full_refresh)
assert len(output.records) == 1
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_incremental_sync_when_read_then_state_message_produced_and_state_match_latest_record(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.PROPERTIES)
self.mock_dynamic_schema_requests(http_mocker)
self.mock_parent_object(
http_mocker, [self.OBJECT_ID], object_type, parent_stream_name, parent_stream_associations, list(self.PROPERTIES.keys())
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type),
self.web_analytics_response(stream_name, id=self.OBJECT_ID),
)
output = self.read_from_stream(self.private_token_config(self.ACCESS_TOKEN), stream_name, SyncMode.incremental)
assert len(output.state_messages) == 1
cursor_value_from_latest_record = output.records[-1].record.data.get(self.CURSOR_FIELD)
object_id_dict = getattr(output.most_recent_state.stream_state, self.OBJECT_ID)
cursor_value_from_most_recent_state = object_id_dict.get(self.CURSOR_FIELD)
assert cursor_value_from_most_recent_state == cursor_value_from_latest_record
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_state_with_no_current_slice_when_read_then_current_slice_in_state(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.PROPERTIES)
self.mock_dynamic_schema_requests(http_mocker)
self.mock_parent_object(
http_mocker, [self.OBJECT_ID], object_type, parent_stream_name, parent_stream_associations, list(self.PROPERTIES.keys())
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type),
self.web_analytics_response(stream_name, id=self.OBJECT_ID),
)
another_object_id = "another_object_id"
current_state = AirbyteStateMessage(
type=AirbyteStateType.STREAM,
stream=AirbyteStreamState(
stream_descriptor=StreamDescriptor(name=stream_name),
stream_state=AirbyteStateBlob(**{another_object_id: {self.CURSOR_FIELD: self.dt_str(self.now())}}),
),
)
output = self.read_from_stream(
self.private_token_config(self.ACCESS_TOKEN), stream_name, SyncMode.incremental, state=[current_state]
)
assert len(output.state_messages) == 1
first_object_id_dict = getattr(output.most_recent_state.stream_state, self.OBJECT_ID)
another_object_id_dict = getattr(output.most_recent_state.stream_state, another_object_id)
assert first_object_id_dict.get(self.CURSOR_FIELD)
assert another_object_id_dict.get(self.CURSOR_FIELD)
http_mocker.clear_all_matchers()
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_STREAMS)
@HttpMocker()
def test_given_state_with_current_slice_when_read_then_state_is_updated(
self, stream_name, parent_stream_name, object_type, parent_stream_associations, http_mocker: HttpMocker
):
self.mock_custom_objects(http_mocker)
self.mock_properties(http_mocker, object_type, self.PROPERTIES)
self.mock_dynamic_schema_requests(http_mocker)
self.mock_parent_object(
http_mocker, [self.OBJECT_ID], object_type, parent_stream_name, parent_stream_associations, list(self.PROPERTIES.keys())
)
self.mock_response(
http_mocker,
self.web_analytics_request(stream_name, self.ACCESS_TOKEN, self.OBJECT_ID, object_type),
self.web_analytics_response(stream_name, id=self.OBJECT_ID),
)
current_state = AirbyteStateMessage(
type=AirbyteStateType.STREAM,
stream=AirbyteStreamState(
stream_descriptor=StreamDescriptor(name=stream_name),
stream_state=AirbyteStateBlob(**{self.OBJECT_ID: {self.CURSOR_FIELD: self.dt_str(self.start_date() - timedelta(days=30))}}),
),
)
output = self.read_from_stream(
self.private_token_config(self.ACCESS_TOKEN), stream_name, SyncMode.incremental, state=[current_state]
)
assert len(output.state_messages) == 1
object_id_dict = getattr(output.most_recent_state.stream_state, self.OBJECT_ID)
assert object_id_dict.get(self.CURSOR_FIELD) == self.dt_str(self.updated_at())
http_mocker.clear_all_matchers()
@freezegun.freeze_time("2024-03-03T14:42:00Z")
class TestIncrementalCRMWebAnalyticsStreamFullRefresh(TestCRMWebAnalyticsStream):
SCOPES = ["e-commerce", "oauth", "crm.objects.feedback_submissions.read", "crm.objects.goals.read"]
@classmethod
def dt_conversion(cls, dt: str) -> str:
return str(int(datetime.strptime(dt, cls.DT_FORMAT).replace(tzinfo=pytz.utc).timestamp()) * 1000)
@classmethod
def mock_parent_object(
cls,
http_mocker: HttpMocker,
object_ids: List[str],
object_type: str,
stream_name: str,
associations: List[str],
properties: List[str],
first_page: bool = True,
with_pagination: bool = False,
date_range: Optional[Tuple[str]] = None,
):
date_range = date_range or (cls.dt_str(cls.start_date()), cls.dt_str(cls.now()))
response_builder = cls.response_builder(stream_name)
for object_id in object_ids:
record = (
cls.record_builder(stream_name, FieldPath(cls.PARENT_CURSOR_FIELD))
.with_field(FieldPath(cls.PARENT_CURSOR_FIELD), cls.dt_str(cls.updated_at()))
.with_field(FieldPath("id"), object_id)
)
response_builder = response_builder.with_record(record)
if with_pagination:
response_builder = response_builder.with_pagination()
start, end = date_range
request_builder = (
IncrementalCRMStreamRequestBuilder()
.for_entity(object_type)
.with_associations(associations)
.with_dt_range(("startTimestamp", cls.dt_conversion(start)), ("endTimestamp", cls.dt_conversion(end)))
.with_properties(properties)
)
if not first_page:
request_builder = request_builder.with_page_token(response_builder.pagination_strategy.NEXT_PAGE_TOKEN)
http_mocker.get(request_builder.build(), response_builder.build())
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_one_page_when_read_stream_oauth_then_return_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations
):
super().test_given_one_page_when_read_stream_oauth_then_return_records(
stream_name, parent_stream_name, object_type, parent_stream_associations
)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_one_page_when_read_stream_private_token_then_return_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations
):
super().test_given_one_page_when_read_stream_private_token_then_return_records(
stream_name, parent_stream_name, object_type, parent_stream_associations
)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_two_pages_when_read_then_return_records(self, stream_name, parent_stream_name, object_type, parent_stream_associations):
super().test_given_two_pages_when_read_then_return_records(stream_name, parent_stream_name, object_type, parent_stream_associations)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_wide_date_range_and_multiple_parent_records_when_read_then_return_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations
):
super().test_given_wide_date_range_and_multiple_parent_records_when_read_then_return_records(
stream_name, parent_stream_name, object_type, parent_stream_associations
)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_error_response_when_read_analytics_then_get_trace_message(
self, stream_name, parent_stream_name, object_type, parent_stream_associations
):
super().test_given_error_response_when_read_analytics_then_get_trace_message(
stream_name, parent_stream_name, object_type, parent_stream_associations
)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_500_then_200_when_read_then_return_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations
):
super().test_given_500_then_200_when_read_then_return_records(
stream_name, parent_stream_name, object_type, parent_stream_associations
)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_missing_scopes_error_when_read_then_hault(
self, stream_name, parent_stream_name, object_type, parent_stream_associations
):
super().test_given_missing_scopes_error_when_read_then_hault(
stream_name, parent_stream_name, object_type, parent_stream_associations
)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_unauthorized_error_when_read_then_hault(self, stream_name, parent_stream_name, object_type, parent_stream_associations):
super().test_given_unauthorized_error_when_read_then_hault(stream_name, parent_stream_name, object_type, parent_stream_associations)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_one_page_when_read_then_get_transformed_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations
):
super().test_given_one_page_when_read_then_get_transformed_records(
stream_name, parent_stream_name, object_type, parent_stream_associations
)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_one_page_when_read_then_get_no_records_filtered(
self, stream_name, parent_stream_name, object_type, parent_stream_associations
):
super().test_given_one_page_when_read_then_get_no_records_filtered(
stream_name, parent_stream_name, object_type, parent_stream_associations
)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_incremental_sync_when_read_then_state_message_produced_and_state_match_latest_record(
self, stream_name, parent_stream_name, object_type, parent_stream_associations
):
super().test_given_incremental_sync_when_read_then_state_message_produced_and_state_match_latest_record(
stream_name, parent_stream_name, object_type, parent_stream_associations
)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_state_with_no_current_slice_when_read_then_current_slice_in_state(
self, stream_name, parent_stream_name, object_type, parent_stream_associations
):
super().test_given_state_with_no_current_slice_when_read_then_current_slice_in_state(
stream_name, parent_stream_name, object_type, parent_stream_associations
)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_state_with_current_slice_when_read_then_state_is_updated(
self, stream_name, parent_stream_name, object_type, parent_stream_associations
):
super().test_given_state_with_current_slice_when_read_then_state_is_updated(
stream_name, parent_stream_name, object_type, parent_stream_associations
)
@pytest.mark.parametrize(("stream_name", "parent_stream_name", "object_type", "parent_stream_associations"), CRM_INCREMENTAL_STREAMS)
def test_given_two_parent_pages_when_read_then_return_records(
self, stream_name, parent_stream_name, object_type, parent_stream_associations
):
super().test_given_two_parent_pages_when_read_then_return_records(
stream_name, parent_stream_name, object_type, parent_stream_associations
)

86
airbyte-integrations/connectors/source-hubspot/poetry.lock → airbyte-integrations/connectors/source-hubspot/unit_tests/poetry.lock generated
View File

@@ -1,14 +1,14 @@
# This file is automatically @generated by Poetry 1.8.3 and should not be changed by hand.
# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand.
[[package]]
name = "airbyte-cdk"
version = "6.50.0"
version = "6.51.0"
description = "A framework for writing Airbyte Connectors."
optional = false
python-versions = "<3.13,>=3.10"
files = [
{file = "airbyte_cdk-6.50.0-py3-none-any.whl", hash = "sha256:b67a0d74abff23df3ad641dae46ac09e5a49a6ff1240a23be5926bd660b7979e"},
{file = "airbyte_cdk-6.50.0.tar.gz", hash = "sha256:cf20cae758444d4213a8e579f31eecd02cfb640ab74ef9365c833f1722e1c7e8"},
{file = "airbyte_cdk-6.51.0-py3-none-any.whl", hash = "sha256:a776494905370626b42564eca6390e0c364ec5f0881f791e2700f4be44b4e8ce"},
{file = "airbyte_cdk-6.51.0.tar.gz", hash = "sha256:52f8a9b557ccd8d0de73dbfbac86d8d6b59f5718825dcbade81ddf2edb963eb9"},
]
[package.dependencies]
@@ -532,18 +532,17 @@ test = ["pytest (>=6)"]
[[package]]
name = "freezegun"
version = "0.3.4"
version = "1.5.1"
description = "Let your Python tests travel through time"
optional = false
python-versions = "*"
python-versions = ">=3.7"
files = [
{file = "freezegun-0.3.4-py2.py3-none-any.whl", hash = "sha256:d15d5daa22260891955d436899f94c8b80525daa895aec74c0afa5a25ac0230e"},
{file = "freezegun-0.3.4.tar.gz", hash = "sha256:8d5eb5656c324125cce80e2e9ae572af6da997b7065b3bb6599c20f1b28dcf46"},
{file = "freezegun-1.5.1-py3-none-any.whl", hash = "sha256:bf111d7138a8abe55ab48a71755673dbaa4ab87f4cff5634a4442dfec34c15f1"},
{file = "freezegun-1.5.1.tar.gz", hash = "sha256:b29dedfcda6d5e8e083ce71b2b542753ad48cfec44037b3fc79702e2980a89e9"},
]
[package.dependencies]
python-dateutil = ">=1.0,<2.0 || >2.0"
six = "*"
python-dateutil = ">=2.7"
[[package]]
name = "genson"
@@ -859,13 +858,13 @@ i18n = ["Babel (>=2.7)"]
[[package]]
name = "joblib"
version = "1.5.0"
version = "1.5.1"
description = "Lightweight pipelining with Python functions"
optional = false
python-versions = ">=3.9"
files = [
{file = "joblib-1.5.0-py3-none-any.whl", hash = "sha256:206144b320246485b712fc8cc51f017de58225fa8b414a1fe1764a7231aca491"},
{file = "joblib-1.5.0.tar.gz", hash = "sha256:d8757f955389a3dd7a23152e43bc297c2e0c2d3060056dad0feefc88a06939b5"},
{file = "joblib-1.5.1-py3-none-any.whl", hash = "sha256:4719a31f054c7d766948dcd83e9613686b27114f190f717cec7eaa2084f8a74a"},
{file = "joblib-1.5.1.tar.gz", hash = "sha256:f4f86e351f39fe3d0d32a9f2c3d8af1ee4cec285aafcb27003dda5205576b444"},
]
[[package]]
@@ -959,7 +958,10 @@ files = [
[package.dependencies]
httpx = ">=0.23.0,<1"
orjson = {version = ">=3.9.14,<4.0.0", markers = "platform_python_implementation != \"PyPy\""}
pydantic = {version = ">=1,<3", markers = "python_full_version < \"3.12.4\""}
pydantic = [
{version = ">=1,<3", markers = "python_full_version < \"3.12.4\""},
{version = ">=2.7.4,<3.0.0", markers = "python_full_version >= \"3.12.4\""},
]
requests = ">=2,<3"
requests-toolbelt = ">=1.0.0,<2.0.0"
@@ -1291,6 +1293,7 @@ files = [
numpy = [
{version = ">=1.22.4", markers = "python_version < \"3.11\""},
{version = ">=1.23.2", markers = "python_version == \"3.11\""},
{version = ">=1.26.0", markers = "python_version >= \"3.12\""},
]
python-dateutil = ">=2.8.2"
pytz = ">=2020.1"
@@ -1321,40 +1324,6 @@ sql-other = ["SQLAlchemy (>=2.0.0)", "adbc-driver-postgresql (>=0.8.0)", "adbc-d
test = ["hypothesis (>=6.46.1)", "pytest (>=7.3.2)", "pytest-xdist (>=2.2.0)"]
xml = ["lxml (>=4.9.2)"]
[[package]]
name = "pendulum"
version = "2.1.2"
description = "Python datetimes made easy"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
files = [
{file = "pendulum-2.1.2-cp27-cp27m-macosx_10_15_x86_64.whl", hash = "sha256:b6c352f4bd32dff1ea7066bd31ad0f71f8d8100b9ff709fb343f3b86cee43efe"},
{file = "pendulum-2.1.2-cp27-cp27m-win_amd64.whl", hash = "sha256:318f72f62e8e23cd6660dbafe1e346950281a9aed144b5c596b2ddabc1d19739"},
{file = "pendulum-2.1.2-cp35-cp35m-macosx_10_15_x86_64.whl", hash = "sha256:0731f0c661a3cb779d398803655494893c9f581f6488048b3fb629c2342b5394"},
{file = "pendulum-2.1.2-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:3481fad1dc3f6f6738bd575a951d3c15d4b4ce7c82dce37cf8ac1483fde6e8b0"},
{file = "pendulum-2.1.2-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:9702069c694306297ed362ce7e3c1ef8404ac8ede39f9b28b7c1a7ad8c3959e3"},
{file = "pendulum-2.1.2-cp35-cp35m-win_amd64.whl", hash = "sha256:fb53ffa0085002ddd43b6ca61a7b34f2d4d7c3ed66f931fe599e1a531b42af9b"},
{file = "pendulum-2.1.2-cp36-cp36m-macosx_10_15_x86_64.whl", hash = "sha256:c501749fdd3d6f9e726086bf0cd4437281ed47e7bca132ddb522f86a1645d360"},
{file = "pendulum-2.1.2-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:c807a578a532eeb226150d5006f156632df2cc8c5693d778324b43ff8c515dd0"},
{file = "pendulum-2.1.2-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:2d1619a721df661e506eff8db8614016f0720ac171fe80dda1333ee44e684087"},
{file = "pendulum-2.1.2-cp36-cp36m-win_amd64.whl", hash = "sha256:f888f2d2909a414680a29ae74d0592758f2b9fcdee3549887779cd4055e975db"},
{file = "pendulum-2.1.2-cp37-cp37m-macosx_10_15_x86_64.whl", hash = "sha256:e95d329384717c7bf627bf27e204bc3b15c8238fa8d9d9781d93712776c14002"},
{file = "pendulum-2.1.2-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:4c9c689747f39d0d02a9f94fcee737b34a5773803a64a5fdb046ee9cac7442c5"},
{file = "pendulum-2.1.2-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:1245cd0075a3c6d889f581f6325dd8404aca5884dea7223a5566c38aab94642b"},
{file = "pendulum-2.1.2-cp37-cp37m-win_amd64.whl", hash = "sha256:db0a40d8bcd27b4fb46676e8eb3c732c67a5a5e6bfab8927028224fbced0b40b"},
{file = "pendulum-2.1.2-cp38-cp38-macosx_10_15_x86_64.whl", hash = "sha256:f5e236e7730cab1644e1b87aca3d2ff3e375a608542e90fe25685dae46310116"},
{file = "pendulum-2.1.2-cp38-cp38-manylinux1_i686.whl", hash = "sha256:de42ea3e2943171a9e95141f2eecf972480636e8e484ccffaf1e833929e9e052"},
{file = "pendulum-2.1.2-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:7c5ec650cb4bec4c63a89a0242cc8c3cebcec92fcfe937c417ba18277d8560be"},
{file = "pendulum-2.1.2-cp38-cp38-win_amd64.whl", hash = "sha256:33fb61601083f3eb1d15edeb45274f73c63b3c44a8524703dc143f4212bf3269"},
{file = "pendulum-2.1.2-cp39-cp39-manylinux1_i686.whl", hash = "sha256:29c40a6f2942376185728c9a0347d7c0f07905638c83007e1d262781f1e6953a"},
{file = "pendulum-2.1.2-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:94b1fc947bfe38579b28e1cccb36f7e28a15e841f30384b5ad6c5e31055c85d7"},
{file = "pendulum-2.1.2.tar.gz", hash = "sha256:b06a0ca1bfe41c990bbf0c029f0b6501a7f2ec4e38bfec730712015e8860f207"},
]
[package.dependencies]
python-dateutil = ">=2.6,<3.0"
pytzdata = ">=2020.1"
[[package]]
name = "platformdirs"
version = "4.3.8"
@@ -1489,13 +1458,13 @@ files = [
[[package]]
name = "pydantic"
version = "2.11.4"
version = "2.11.5"
description = "Data validation using Python type hints"
optional = false
python-versions = ">=3.9"
files = [
{file = "pydantic-2.11.4-py3-none-any.whl", hash = "sha256:d9615eaa9ac5a063471da949c8fc16376a84afb5024688b3ff885693506764eb"},
{file = "pydantic-2.11.4.tar.gz", hash = "sha256:32738d19d63a226a52eed76645a98ee07c1f410ee41d93b4afbfa85ed8111c2d"},
{file = "pydantic-2.11.5-py3-none-any.whl", hash = "sha256:f9c26ba06f9747749ca1e5c94d6a85cb84254577553c8785576fd38fa64dc0f7"},
{file = "pydantic-2.11.5.tar.gz", hash = "sha256:7f853db3d0ce78ce8bbb148c401c2cdd6431b3473c0cdff2755c7690952a7b7a"},
]
[package.dependencies]
@@ -1784,17 +1753,6 @@ files = [
{file = "pytz-2024.2.tar.gz", hash = "sha256:2aa355083c50a0f93fa581709deac0c9ad65cca8a9e9beac660adcbd493c798a"},
]
[[package]]
name = "pytzdata"
version = "2020.1"
description = "The Olson timezone database for Python."
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
files = [
{file = "pytzdata-2020.1-py2.py3-none-any.whl", hash = "sha256:e1e14750bcf95016381e4d472bad004eef710f2d6417240904070b3d6654485f"},
{file = "pytzdata-2020.1.tar.gz", hash = "sha256:3efa13b335a00a8de1d345ae41ec78dd11c9f8807f522d39850f2dd828681540"},
]
[[package]]
name = "pyyaml"
version = "6.0.2"
@@ -2534,5 +2492,5 @@ files = [
[metadata]
lock-version = "2.0"
python-versions = "^3.10,<3.12"
content-hash = "7999a4270d0d25a0724d2ec7830144746e524754ce9cb93a6a79180113de515a"
python-versions = "^3.10,<3.13"
content-hash = "6c643c93c4f1a38e8f188bf170f7902ac49a4099169b2623e5d16f4487fb5945"

20
airbyte-integrations/connectors/source-hubspot/unit_tests/pyproject.toml Normal file
View File

@@ -0,0 +1,20 @@
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.poetry]
name = "source-hubspot"
version = "0.0.0"
description = "Unit tests for source-hubspot"
authors = ["Airbyte <contact@airbyte.io>"]
[tool.poetry.dependencies]
python = "^3.10,<3.13"
airbyte-cdk = "^6"
pytest = "^8"
freezegun = "^1.4.0"
pytest-mock = "^3.6.1"
requests-mock = "^1.12.1"
mock = "^5.1.0"
[tool.pytest.ini_options]
filterwarnings = [
"ignore:This class is experimental*"
]

50
airbyte-integrations/connectors/source-hubspot/unit_tests/test_components.py
View File

@@ -7,15 +7,6 @@ from unittest.mock import Mock, patch
import pytest
import requests
from requests import Response
from source_hubspot.components import (
HubspotAssociationsExtractor,
HubspotFlattenAssociationsTransformation,
HubspotPropertyHistoryExtractor,
HubspotRenamePropertiesTransformation,
MigrateEmptyStringState,
NewtoLegacyFieldTransformation,
)
from source_hubspot.streams import DEALS_NEW_TO_LEGACY_FIELDS_MAPPING
from airbyte_cdk.sources.declarative.retrievers import SimpleRetriever
@@ -92,8 +83,13 @@ from airbyte_cdk.sources.declarative.retrievers import SimpleRetriever
"Does not overwrite value for legacy field if legacy field exists",
],
)
def test_new_to_legacy_field_transformation(input, expected):
transformer = NewtoLegacyFieldTransformation(DEALS_NEW_TO_LEGACY_FIELDS_MAPPING)
def test_new_to_legacy_field_transformation(input, expected, components_module):
deals_new_to_legacy_mapping = {
"hs_date_entered_": "hs_v2_date_entered_",
"hs_date_exited_": "hs_v2_date_exited_",
"hs_time_in_": "hs_v2_latest_time_in_",
}
transformer = components_module.NewtoLegacyFieldTransformation(deals_new_to_legacy_mapping)
transformer.transform(input)
assert input == expected
@@ -109,8 +105,8 @@ def test_new_to_legacy_field_transformation(input, expected):
"Valid state: date string, no need to migrate",
],
)
def test_migrate_empty_string_state(config, state, expected_should_migrate, expected_state):
state_migration = MigrateEmptyStringState("updatedAt", config)
def test_migrate_empty_string_state(config, state, expected_should_migrate, expected_state, components_module):
state_migration = components_module.MigrateEmptyStringState("updatedAt", config)
actual_should_migrate = state_migration.should_migrate(stream_state=state)
assert actual_should_migrate is expected_should_migrate
@@ -119,7 +115,7 @@ def test_migrate_empty_string_state(config, state, expected_should_migrate, expe
assert state_migration.migrate(stream_state=state) == expected_state
def test_hubspot_rename_properties_transformation():
def test_hubspot_rename_properties_transformation(components_module):
expected_properties = {
"properties_amount": {"type": ["null", "number"]},
"properties_hs_v2_date_entered_closedwon": {"format": "date-time", "type": ["null", "string"]},
@@ -142,7 +138,7 @@ def test_hubspot_rename_properties_transformation():
"hs_v2_date_exited_closedlost": {"format": "date-time", "type": ["null", "string"]},
"hs_v2_latest_time_in_contractsent": {"format": "date-time", "type": ["null", "string"]},
}
transformation = HubspotRenamePropertiesTransformation()
transformation = components_module.HubspotRenamePropertiesTransformation()
transformation.transform(record=dynamic_properties_record)
@@ -162,7 +158,7 @@ def test_hubspot_rename_properties_transformation():
assert dynamic_properties_record["properties"] == expected_properties["properties"]
def test_property_history_extractor():
def test_property_history_extractor(components_module):
expected_records = [
{
"dealId": "1234",
@@ -297,7 +293,7 @@ def test_property_history_extractor():
decoder = Mock()
decoder.decode.return_value = response
extractor = HubspotPropertyHistoryExtractor(
extractor = components_module.HubspotPropertyHistoryExtractor(
field_path=["results"], entity_primary_key="dealId", additional_keys=["archived"], decoder=decoder, config={}, parameters={}
)
@@ -306,7 +302,7 @@ def test_property_history_extractor():
assert actual_records == expected_records
def test_property_history_extractor_ignore_hs_lastmodifieddate():
def test_property_history_extractor_ignore_hs_lastmodifieddate(components_module):
expected_records = [
{
"dealId": "1234",
@@ -361,7 +357,7 @@ def test_property_history_extractor_ignore_hs_lastmodifieddate():
decoder = Mock()
decoder.decode.return_value = response
extractor = HubspotPropertyHistoryExtractor(
extractor = components_module.HubspotPropertyHistoryExtractor(
field_path=["results"], entity_primary_key="dealId", additional_keys=[], decoder=decoder, config={}, parameters={}
)
@@ -370,10 +366,10 @@ def test_property_history_extractor_ignore_hs_lastmodifieddate():
assert actual_records == expected_records
def test_flatten_associations_transformation():
def test_flatten_associations_transformation(components_module):
expected_record = {"id": "a2b", "Contacts": [101, 102], "Companies": [202, 209]}
transformation = HubspotFlattenAssociationsTransformation()
transformation = components_module.HubspotFlattenAssociationsTransformation()
current_record = {
"id": "a2b",
@@ -388,7 +384,7 @@ def test_flatten_associations_transformation():
assert current_record == expected_record
def test_associations_extractor(config):
def test_associations_extractor(config, components_module):
expected_records = [
{"id": "123", "companies": ["909", "424"], "contacts": ["408"]},
{
@@ -436,7 +432,7 @@ def test_associations_extractor(config):
},
]
extractor = HubspotAssociationsExtractor(
extractor = components_module.HubspotAssociationsExtractor(
field_path=["results"],
entity="deals",
associations_list=["companies", "contacts"],
@@ -460,10 +456,10 @@ def test_associations_extractor(config):
assert records[1]["contacts"] == expected_records[1]["contacts"]
def test_extractor_supports_entity_interpolation(config):
def test_extractor_supports_entity_interpolation(config, components_module):
parameters = {"entity": "engagements_emails"}
extractor = HubspotAssociationsExtractor(
extractor = components_module.HubspotAssociationsExtractor(
field_path=["results"],
entity="{{ parameters['entity'] }}",
associations_list=["companies", "contacts"],
@@ -483,8 +479,8 @@ def test_extractor_supports_entity_interpolation(config):
pytest.param("{{ parameters['associations'] }}", id="test_interpolated_associations"),
],
)
def test_extractor_supports_associations_list_interpolation(config, associations_list_value):
extractor = HubspotAssociationsExtractor(
def test_extractor_supports_associations_list_interpolation(config, associations_list_value, components_module):
extractor = components_module.HubspotAssociationsExtractor(
field_path=["results"],
entity="emails",
associations_list=associations_list_value,

44
airbyte-integrations/connectors/source-hubspot/unit_tests/test_entity_schema_normalization.py
View File

@@ -1,44 +0,0 @@
#
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
#
import pytest
from source_hubspot.components import EntitySchemaNormalization
@pytest.mark.parametrize(
"original_value,field_schema,expected_value",
[
pytest.param("sample_string", {"type": ["null", "string"]}, "sample_string", id="test_string_transform"),
pytest.param("", {"type": ["null", "number"]}, None, id="test_empty_string_returns_none"),
pytest.param("700.0", {"type": ["null", "number"]}, 700.0, id="test_transform_float"),
pytest.param("10293848576", {"type": ["null", "number"]}, 10293848576, id="test_do_not_cast_numeric_id_to_float"),
pytest.param("true", {"type": ["null", "boolean"]}, True, id="test_transform_boolean_true"),
pytest.param("false", {"type": ["null", "boolean"]}, False, id="test_transform_boolean_false"),
pytest.param("Not real", {"type": ["null", "boolean"]}, "Not real", id="test_do_not_transform_non_boolean"),
pytest.param("TrUe", {"type": ["null", "boolean"]}, True, id="test_transform_boolean_case_insensitive"),
pytest.param(
"2025-02-19T11:49:03.544Z",
{"type": "string", "format": "date-time"},
"2025-02-19T11:49:03.544000+00:00",
id="test_transform_datetime_string",
),
pytest.param("1746082800", {"type": "string", "format": "date"}, "2025-05-01", id="test_timestamp_seconds_to_date"),
pytest.param(
"1746082800", {"type": "string", "format": "date-time"}, "2025-05-01T07:00:00+00:00", id="test_timestamp_seconds_to_datetime"
),
pytest.param(
"1746082800",
{"type": "string", "format": "date-time", "__ab_apply_cast_datetime": False},
"1746082800",
id="test_do_not_apply_cast_datetime",
),
pytest.param("not_parsable_date", {"type": "string", "format": "date"}, "not_parsable_date", id="test_do_not_apply_cast_datetime"),
],
)
def test_entity_schema_normalization(original_value, field_schema, expected_value):
transform_function = EntitySchemaNormalization().get_transform_function()
transformed_value = transform_function(original_value=original_value, field_schema=field_schema)
assert type(transformed_value) == type(expected_value)
assert transformed_value == expected_value

106
airbyte-integrations/connectors/source-hubspot/unit_tests/test_field_type_converting.py
View File

@@ -1,106 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import pytest
from source_hubspot.streams import BaseStream
@pytest.mark.parametrize(
"field_type,expected",
[
("string", {"type": ["null", "string"]}),
("integer", {"type": ["null", "integer"]}),
("number", {"type": ["null", "number"]}),
("bool", {"type": ["null", "boolean"]}),
("boolean", {"type": ["null", "boolean"]}),
("enumeration", {"type": ["null", "string"]}),
("object", {"type": ["null", "object"]}),
("array", {"type": ["null", "array"]}),
("date", {"type": ["null", "string"], "format": "date"}),
("date-time", {"type": ["null", "string"], "format": "date-time"}),
("datetime", {"type": ["null", "string"], "format": "date-time"}),
("json", {"type": ["null", "string"]}),
("phone_number", {"type": ["null", "string"]}),
],
)
def test_field_type_format_converting(field_type, expected):
assert BaseStream._get_field_props(field_type=field_type) == expected
@pytest.mark.parametrize(
"field_type,expected",
[
("_unsupported_field_type_", {"type": ["null", "string"]}),
(None, {"type": ["null", "string"]}),
(1, {"type": ["null", "string"]}),
],
)
def test_bad_field_type_converting(field_type, expected, caplog, capsys):
assert BaseStream._get_field_props(field_type=field_type) == expected
logs = caplog.records
assert logs
assert logs[0].levelname == "WARNING"
assert logs[0].msg == f"Unsupported type {field_type} found"
@pytest.mark.parametrize(
"declared_field_types,field_name,field_value,format,casted_value",
[
# test for None in field_values
(["null", "string"], "some_field", None, None, None),
(["null", "number"], "some_field", None, None, None),
(["null", "integer"], "some_field", None, None, None),
(["null", "object"], "some_field", None, None, None),
(["null", "boolean"], "some_field", None, None, None),
# specific cases
("string", "some_field", "test", None, "test"),
(["null", "number"], "some_field", "123.456", None, 123.456),
(["null", "number"], "some_field", "123,123.456", None, 123123.456),
(["null", "number"], "user_id", "123", None, 123),
(["null", "string"], "some_field", "123", None, "123"),
# when string has empty field_value (empty string)
(["null", "string"], "some_field", "", None, ""),
# when NOT string type but has empty sting in field_value, instead of double or null,
# we should use None instead, to have it properly casted to the correct type
(["null", "number"], "some_field", "", None, None),
(["null", "integer"], "some_field", "", None, None),
(["null", "object"], "some_field", "", None, None),
(["null", "boolean"], "some_field", "", None, None),
# when string needs to be cast as booleans
(["null", "boolean"], "some_field", "false", None, False),
(["null", "boolean"], "some_field", "true", None, True),
# Test casting fields with format specified
(["null", "string"], "some_field", "", "date-time", None),
(["string"], "some_field", "", "date-time", ""),
(["null", "string"], "some_field", "2020", "date-time", "2020-01-01T00:00:00+00:00"),
],
)
def test_cast_type_if_needed(declared_field_types, field_name, field_value, format, casted_value):
assert (
BaseStream._cast_value(
declared_field_types=declared_field_types, field_name=field_name, field_value=field_value, declared_format=format
)
== casted_value
)
@pytest.mark.parametrize(
"field_value, declared_format, expected_casted_value",
[
("1653696000000", "date", "2022-05-28"),
("1645608465000", "date-time", "2022-02-23T09:27:45+00:00"),
(1645608465000, "date-time", "2022-02-23T09:27:45+00:00"),
("2022-05-28", "date", "2022-05-28"),
("2022-02-23 09:27:45", "date-time", "2022-02-23T09:27:45+00:00"),
("", "date", ""),
(None, "date", None),
("2022-02-23 09:27:45", "date", "2022-02-23"),
("2022-05-28", "date-time", "2022-05-28T00:00:00+00:00"),
],
)
def test_cast_timestamp_to_date(field_value, declared_format, expected_casted_value):
casted_value = BaseStream._cast_datetime("hs_recurring_billing_end_date", field_value, declared_format=declared_format)
assert casted_value == expected_casted_value

5
airbyte-integrations/connectors/source-hubspot/unit_tests/test_handle_retry_token_expired.py
View File

@@ -5,7 +5,8 @@ import pytest
from airbyte_cdk.models import SyncMode
from airbyte_cdk.sources.streams.http.exceptions import UserDefinedBackoffException
from unit_tests.conftest import find_stream
from .conftest import find_stream
def test_handle_request_with_retry(config, requests_mock):
@@ -23,7 +24,7 @@ def test_handle_request_with_retry(config, requests_mock):
assert len(stream_slices) == 1
list(stream_instance.read_records(sync_mode=SyncMode.full_refresh, stream_slice=stream_slices[0]))
# one request per each mock
assert requests_mock.call_count == 4
assert requests_mock.call_count == 3
def test_handle_request_with_retry_token_expired(config, requests_mock):

424
airbyte-integrations/connectors/source-hubspot/unit_tests/test_source.py
View File

@@ -4,25 +4,17 @@
import logging
import random
from datetime import timedelta
from http import HTTPStatus
from unittest.mock import MagicMock
from urllib.parse import urlencode
import mock
import pendulum
import pytest
from source_hubspot.errors import HubspotRateLimited, InvalidStartDateConfigError
from source_hubspot.helpers import APIv3Property
from source_hubspot.source import SourceHubspot
from source_hubspot.streams import API, BaseStream, Deals
from airbyte_cdk.models import ConfiguredAirbyteCatalog, ConfiguredAirbyteCatalogSerializer, SyncMode
from airbyte_cdk.test.entrypoint_wrapper import read
from airbyte_cdk.models import SyncMode
from airbyte_cdk.test.entrypoint_wrapper import discover
from airbyte_cdk.test.state_builder import StateBuilder
from airbyte_cdk.utils.datetime_helpers import ab_datetime_now
from .conftest import find_stream, mock_dynamic_schema_requests_with_skip, read_from_stream
from .conftest import find_stream, get_source, mock_dynamic_schema_requests_with_skip, read_from_stream
from .utils import read_full_refresh, read_incremental
@@ -39,32 +31,35 @@ def time_sleep_mock(mocker):
def test_check_connection_ok(requests_mock, config):
responses = [
{"json": [], "status_code": 200},
{
"json": [
{
"name": "hs__migration_soft_delete",
"type": "enumeration",
}
],
"status_code": 200,
},
]
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
requests_mock.register_uri("GET", "/properties/v2/contact/properties", responses)
ok, error_msg = SourceHubspot(config, None, None).check_connection(logger, config=config)
requests_mock.register_uri("GET", "/crm/v3/objects/contact", {})
ok, error_msg = get_source(config).check_connection(logger, config=config)
assert ok
assert not error_msg
def test_check_connection_empty_config(config):
def test_check_connection_empty_config(caplog):
config = {}
with pytest.raises(KeyError):
SourceHubspot(config, None, None).check_connection(logger, config=config)
def test_check_connection_invalid_config(config):
config.pop("credentials")
with pytest.raises(KeyError):
SourceHubspot(config, None, None).check_connection(logger, config=config)
get_source(config).check_connection(logger, config=config)
assert "KeyError: ['credentials', 'credentials_title']" in caplog.records[0].message
assert caplog.records[0].levelname == "ERROR"
def test_check_connection_exception(config):
ok, error_msg = SourceHubspot(config, None, None).check_connection(logger, config=config)
ok, error_msg = get_source(config).check_connection(logger, config=config)
assert not ok
assert error_msg
@@ -75,91 +70,69 @@ def test_check_connection_bad_request_exception(requests_mock, config_invalid_cl
{"json": {"message": "invalid client_id"}, "status_code": 400},
]
requests_mock.register_uri("POST", "/oauth/v1/token", responses)
ok, error_msg = SourceHubspot(config_invalid_client_id, None, None).check_connection(logger, config=config_invalid_client_id)
ok, error_msg = get_source(config_invalid_client_id).check_connection(logger, config=config_invalid_client_id)
assert not ok
assert error_msg
def test_check_connection_invalid_start_date_exception(config_invalid_date):
with pytest.raises(InvalidStartDateConfigError):
ok, error_msg = SourceHubspot(config_invalid_date, None, None).check_connection(logger, config=config_invalid_date)
assert not ok
assert error_msg
def test_streams(requests_mock, config):
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
streams = get_source(config).streams(config)
assert len(streams) == 32
@mock.patch("source_hubspot.source.SourceHubspot.get_custom_object_streams")
def test_streams_forbidden_returns_default_streams(mock_get_custom_object_streams, requests_mock, config):
def test_streams_forbidden_returns_default_streams(requests_mock, config):
# 403 forbidden → no custom streams, should fall back to the 32 built-in ones
requests_mock.get(
"https://api.hubapi.com/crm/v3/schemas",
json={"status": "error", "message": "This access_token does not have proper permissions!"},
status_code=403,
)
streams = SourceHubspot(config, None, None).streams(config)
streams = get_source(config).streams(config)
assert len(streams) == 32
# ToDo: This test works locally but fails in CI
# @mock.patch("source_hubspot.source.SourceHubspot.get_custom_object_streams")
# def test_streams_ok_with_one_custom_stream(mock_get_custom_object_streams, requests_mock, config):
# # 200 OK → one custom “cars” stream added to the 32 built-ins, total = 33
# requests_mock.get(
# "https://api.hubapi.com/crm/v3/schemas",
# json={"results": [{"name": "cars", "fullyQualifiedName": "cars", "properties": {}}]},
# status_code=200,
# )
# streams = SourceHubspot(config, None, None).streams(config)
# assert len(streams) == 33
@mock.patch("source_hubspot.source.SourceHubspot.get_custom_object_streams")
def test_streams_incremental(mock_get_custom_object_streams, requests_mock, config_experimental):
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
streams = SourceHubspot(config_experimental, None, None).streams(config_experimental)
assert len(streams) == 44
def test_custom_streams(config_experimental):
custom_object_stream_instances = [MagicMock()]
streams = SourceHubspot(config_experimental, None, None).get_web_analytics_custom_objects_stream(
custom_object_stream_instances=custom_object_stream_instances,
common_params={"api": MagicMock(), "start_date": "2021-01-01T00:00:00Z", "credentials": config_experimental["credentials"]},
)
assert len(list(streams)) == 1
def test_check_credential_title_exception(config):
config["credentials"].pop("credentials_title")
with pytest.raises(Exception):
SourceHubspot(config, None, None).check_connection(logger, config=config)
ok, message = get_source(config).check_connection(logger, config=config)
assert ok == False
assert "`authenticator_selection_path` is not found in the config" in message
def test_parse_and_handle_errors(some_credentials):
response = MagicMock()
response.status_code = HTTPStatus.TOO_MANY_REQUESTS
with pytest.raises(HubspotRateLimited):
API(some_credentials)._parse_and_handle_errors(response)
def test_convert_datetime_to_string():
pendulum_time = pendulum.now()
assert BaseStream._convert_datetime_to_string(pendulum_time, declared_format="date")
assert BaseStream._convert_datetime_to_string(pendulum_time, declared_format="date-time")
def test_streams_ok_with_one_custom_stream(requests_mock, config, mock_dynamic_schema_requests):
# 200 OK → one custom “cars” stream added to the 32 built-ins, total = 33
adapter = requests_mock.get(
"https://api.hubapi.com/crm/v3/schemas",
json={"results": [{"name": "cars", "fullyQualifiedName": "cars", "properties": {}}]},
status_code=200,
)
streams = discover(get_source(config), config).catalog.catalog.streams
assert adapter.called
assert len(streams) == 33
def test_check_connection_backoff_on_limit_reached(requests_mock, config):
"""Error once, check that we retry and not fail"""
prop_response = [
{
"json": [
{
"name": "hs__migration_soft_delete",
"type": "enumeration",
}
],
"status_code": 200,
}
]
responses = [
{"json": {"error": "limit reached"}, "status_code": 429, "headers": {"Retry-After": "0"}},
{"json": [], "status_code": 200},
]
requests_mock.register_uri("GET", "/properties/v2/contact/properties", responses)
source = SourceHubspot(config, None, None)
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
requests_mock.register_uri("GET", "/properties/v2/contact/properties", prop_response)
requests_mock.register_uri("GET", "/crm/v3/objects/contact", responses)
source = get_source(config)
alive, error = source.check_connection(logger=logger, config=config)
assert alive
@@ -168,12 +141,25 @@ def test_check_connection_backoff_on_limit_reached(requests_mock, config):
def test_check_connection_backoff_on_server_error(requests_mock, config):
"""Error once, check that we retry and not fail"""
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
prop_response = [
{
"json": [
{
"name": "hs__migration_soft_delete",
"type": "enumeration",
}
],
"status_code": 200,
}
]
responses = [
{"json": {"error": "something bad"}, "status_code": 500},
{"json": [], "status_code": 200},
]
requests_mock.register_uri("GET", "/properties/v2/contact/properties", responses)
source = SourceHubspot(config, None, None)
requests_mock.register_uri("GET", "/properties/v2/contact/properties", prop_response)
requests_mock.register_uri("GET", "/crm/v3/objects/contact", responses)
source = get_source(config)
alive, error = source.check_connection(logger=logger, config=config)
assert alive
@@ -241,7 +227,7 @@ class TestSplittingPropertiesFunctionality:
response = api._session.get(api.BASE_URL + url, params=params)
return api._parse_and_handle_errors(response)
def test_stream_with_splitting_properties(self, requests_mock, api, fake_properties_list, config, mock_dynamic_schema_requests):
def test_stream_with_splitting_properties(self, requests_mock, fake_properties_list, config, mock_dynamic_schema_requests):
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
"""
Check working stream `companies` with large list of properties using new functionality with splitting properties
@@ -283,34 +269,8 @@ class TestSplittingPropertiesFunctionality:
record_responses,
)
after_id = id_list[-1]
catalog = ConfiguredAirbyteCatalogSerializer.load(
{
"streams": [
{
"stream": {
"name": "companies",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
},
"sync_mode": "full_refresh",
"destination_sync_mode": "append",
}
]
}
)
state = (
StateBuilder()
.with_stream_state(
"companies",
{},
)
.build()
)
stream_records = read(
SourceHubspot(config=config, catalog=catalog, state=state), config=config, catalog=catalog, state=state
).records
stream_records = read_from_stream(config, "companies", SyncMode.full_refresh).records
# check that we have records for all set ids, and that each record has 2000 properties (not more, and not less)
assert len(stream_records) == sum([len(ids) for ids in record_ids_paginated])
for record_ab_message in stream_records:
@@ -319,7 +279,7 @@ class TestSplittingPropertiesFunctionality:
properties = [field for field in record if field.startswith("properties_")]
assert len(properties) == NUMBER_OF_PROPERTIES
def test_stream_with_splitting_properties_with_pagination(self, requests_mock, config, common_params, api, fake_properties_list):
def test_stream_with_splitting_properties_with_pagination(self, requests_mock, config, fake_properties_list):
"""
Check working stream `products` with large list of properties using new functionality with splitting properties
"""
@@ -356,22 +316,6 @@ class TestSplittingPropertiesFunctionality:
f"{test_stream.retriever.requester.url_base}/{test_stream.retriever.requester.get_path()}?{urlencode(params)}",
record_responses,
)
catalog = ConfiguredAirbyteCatalogSerializer.load(
{
"streams": [
{
"stream": {
"name": "products",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
},
"sync_mode": "incremental",
"destination_sync_mode": "append",
}
]
}
)
state = (
StateBuilder()
.with_stream_state(
@@ -381,9 +325,7 @@ class TestSplittingPropertiesFunctionality:
.build()
)
stream_records = read(
SourceHubspot(config=config, catalog=catalog, state=state), config=config, catalog=catalog, state=state
).records
stream_records = read_from_stream(config, "products", SyncMode.incremental, state).records
assert len(stream_records) == 5
for record_ab_message in stream_records:
@@ -392,61 +334,6 @@ class TestSplittingPropertiesFunctionality:
properties = [field for field in record if field.startswith("properties_")]
assert len(properties) == NUMBER_OF_PROPERTIES
def test_stream_with_splitting_properties_with_new_record(self, requests_mock, common_params, api, fake_properties_list):
"""
Check working stream `workflows` with large list of properties using new functionality with splitting properties
"""
parsed_properties = list(APIv3Property(fake_properties_list).split())
self.set_mock_properties(requests_mock, "/properties/v2/deal/properties", fake_properties_list)
test_stream = Deals(**common_params)
ids_list = ["6043593519", "1092593519", "1092593518", "1092593517", "1092593516"]
for property_slice in parsed_properties:
record_responses = [
{
"json": {
"results": [
{**self.BASE_OBJECT_BODY, **{"id": id, "properties": {p: "fake_data" for p in property_slice.properties}}}
for id in ids_list
],
"paging": {},
},
"status_code": 200,
}
]
test_stream._sync_mode = SyncMode.full_refresh
prop_key, prop_val = next(iter(property_slice.as_url_param().items()))
requests_mock.register_uri("GET", f"{test_stream.url}?{prop_key}={prop_val}", record_responses)
test_stream._sync_mode = None
ids_list.append("1092593513")
stream_records = read_full_refresh(test_stream)
assert len(stream_records) == 6
@pytest.fixture(name="configured_catalog")
def configured_catalog_fixture():
configured_catalog = {
"streams": [
{
"stream": {
"name": "quotes",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": True,
"default_cursor_field": ["updatedAt"],
},
"sync_mode": "incremental",
"cursor_field": ["updatedAt"],
"destination_sync_mode": "append",
}
]
}
return ConfiguredAirbyteCatalog.parse_obj(configured_catalog)
def test_search_based_stream_should_not_attempt_to_get_more_than_10k_records(
requests_mock, config, fake_properties_list, mock_dynamic_schema_requests
@@ -508,21 +395,6 @@ def test_search_based_stream_should_not_attempt_to_get_more_than_10k_records(
# Create test_stream instance with some state
test_stream = find_stream("companies", config)
catalog = ConfiguredAirbyteCatalogSerializer.load(
{
"streams": [
{
"stream": {
"name": "companies",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
},
"sync_mode": "incremental",
"destination_sync_mode": "append",
}
]
}
)
state = (
StateBuilder()
.with_stream_state(
@@ -546,7 +418,7 @@ def test_search_based_stream_should_not_attempt_to_get_more_than_10k_records(
[{"status_code": 200, "json": {"results": [{"from": {"id": "1"}, "to": [{"toObjectId": "2"}]}]}}],
)
output = read(SourceHubspot(config=config, catalog=catalog, state=state), config=config, catalog=catalog, state=state)
output = read_from_stream(config, "companies", SyncMode.incremental, state)
# The stream should not attempt to get more than 10K records.
# Instead, it should use the new state to start a new search query.
assert len(output.records) == 11000
@@ -617,22 +489,6 @@ def test_search_based_incremental_stream_should_sort_by_id(requests_mock, config
"/crm/v4/associations/company/contacts/batch/read",
[{"status_code": 200, "json": {"results": [{"from": {"id": f"{x}"}, "to": [{"toObjectId": "2"}]}]}} for x in range(1, 11001, 200)],
)
catalog = ConfiguredAirbyteCatalogSerializer.load(
{
"streams": [
{
"stream": {
"name": "companies",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
},
"sync_mode": "incremental",
"destination_sync_mode": "append",
}
]
}
)
state = (
StateBuilder()
.with_stream_state(
@@ -641,7 +497,7 @@ def test_search_based_incremental_stream_should_sort_by_id(requests_mock, config
)
.build()
)
output = read(SourceHubspot(config=config, catalog=catalog, state=state), config=config, catalog=catalog, state=state)
output = read_from_stream(config, "companies", SyncMode.incremental, state)
records = output.records
# The stream should not attempt to get more than 10K records.
# Instead, it should use the new state to start a new search query.
@@ -654,7 +510,7 @@ def test_search_based_incremental_stream_should_sort_by_id(requests_mock, config
assert output.state_messages[1].state.stream.stream_state.updatedAt == "2022-02-25T16:43:11.000000Z"
def test_engagements_stream_pagination_works(requests_mock, common_params, config):
def test_engagements_stream_pagination_works(requests_mock, config):
"""
Tests the engagements stream handles pagination correctly, for both
full_refresh and incremental sync modes.
@@ -736,7 +592,7 @@ def test_engagements_stream_pagination_works(requests_mock, common_params, confi
assert len(records) == 100
def test_engagements_stream_since_old_date(mock_dynamic_schema_requests, requests_mock, common_params, fake_properties_list, config):
def test_engagements_stream_since_old_date(mock_dynamic_schema_requests, requests_mock, fake_properties_list, config):
"""
Connector should use 'All Engagements' API for old dates (more than 30 days)
"""
@@ -758,21 +614,6 @@ def test_engagements_stream_since_old_date(mock_dynamic_schema_requests, request
# Mocking Request
requests_mock.register_uri("GET", "/engagements/v1/engagements/paged?count=250", responses)
catalog = ConfiguredAirbyteCatalogSerializer.load(
{
"streams": [
{
"stream": {
"name": "engagements",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
},
"sync_mode": "incremental",
"destination_sync_mode": "append",
}
]
}
)
state = (
StateBuilder()
.with_stream_state(
@@ -781,19 +622,19 @@ def test_engagements_stream_since_old_date(mock_dynamic_schema_requests, request
)
.build()
)
output = read(SourceHubspot(config=config, catalog=catalog, state=state), config=config, catalog=catalog, state=state)
output = read_from_stream(config, "engagements", SyncMode.incremental, state)
assert len(output.records) == 100
assert int(output.state_messages[0].state.stream.stream_state.lastUpdated) == recent_date
def test_engagements_stream_since_recent_date(mock_dynamic_schema_requests, requests_mock, common_params, fake_properties_list, config):
def test_engagements_stream_since_recent_date(mock_dynamic_schema_requests, requests_mock, fake_properties_list, config):
"""
Connector should use 'Recent Engagements' API for recent dates (less than 30 days)
"""
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
recent_date = pendulum.now() - timedelta(days=10) # 10 days ago
recent_date = ab_datetime_now() - timedelta(days=10) # 10 days ago
recent_date = int(recent_date.timestamp() * 1000)
responses = [
{
@@ -806,37 +647,16 @@ def test_engagements_stream_since_recent_date(mock_dynamic_schema_requests, requ
"status_code": 200,
}
]
# Create test_stream instance with some state
catalog = ConfiguredAirbyteCatalogSerializer.load(
{
"streams": [
{
"stream": {
"name": "engagements",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
},
"sync_mode": "incremental",
"destination_sync_mode": "overwrite",
}
]
}
)
state = StateBuilder().with_stream_state("engagements", {"lastUpdated": recent_date}).build()
# Mocking Request
requests_mock.register_uri("GET", f"/engagements/v1/engagements/recent/modified?count=250&since={recent_date}", responses)
output = read(SourceHubspot(config=config, catalog=catalog, state=state), config=config, catalog=catalog, state=state)
output = read_from_stream(config, "engagements", SyncMode.incremental, state)
# The stream should not attempt to get more than 10K records.
assert len(output.records) == 100
assert int(output.state_messages[0].state.stream.stream_state.lastUpdated) == recent_date
def test_engagements_stream_since_recent_date_more_than_10k(
mock_dynamic_schema_requests, requests_mock, common_params, fake_properties_list, config
):
def test_engagements_stream_since_recent_date_more_than_10k(mock_dynamic_schema_requests, requests_mock, fake_properties_list, config):
"""
Connector should use 'Recent Engagements' API for recent dates (less than 30 days).
If response from 'Recent Engagements' API returns 10k records, it means that there more records,
@@ -844,7 +664,7 @@ def test_engagements_stream_since_recent_date_more_than_10k(
"""
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
recent_date = pendulum.now() - timedelta(days=10) # 10 days ago
recent_date = ab_datetime_now() - timedelta(days=10) # 10 days ago
recent_date = int(recent_date.timestamp() * 1000)
responses = [
{
@@ -857,36 +677,17 @@ def test_engagements_stream_since_recent_date_more_than_10k(
"status_code": 200,
}
]
# Create test_stream instance with some state
catalog = ConfiguredAirbyteCatalogSerializer.load(
{
"streams": [
{
"stream": {
"name": "engagements",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
},
"sync_mode": "incremental",
"destination_sync_mode": "overwrite",
}
]
}
)
state = StateBuilder().with_stream_state("engagements", {"lastUpdated": recent_date}).build()
# Mocking Request
requests_mock.register_uri("GET", f"/engagements/v1/engagements/recent/modified?count=250&since={recent_date}", responses)
requests_mock.register_uri("GET", "/engagements/v1/engagements/paged?count=250", responses)
output = read(SourceHubspot(config=config, catalog=catalog, state=state), config=config, catalog=catalog, state=state)
output = read_from_stream(config, "engagements", SyncMode.incremental, state)
assert len(output.records) == 100
assert int(output.state_messages[0].state.stream.stream_state.lastUpdated) == recent_date
@mock.patch("source_hubspot.source.SourceHubspot.get_custom_object_streams")
def test_pagination_marketing_emails_stream(mock_get_custom_objects_stream, requests_mock, common_params, config):
def test_pagination_marketing_emails_stream(requests_mock, config):
"""
Test pagination for Marketing Emails stream
"""
@@ -930,52 +731,3 @@ def test_pagination_marketing_emails_stream(mock_get_custom_objects_stream, requ
records = read_full_refresh(test_stream)
# The stream should handle pagination correctly and output 600 records.
assert len(records) == 600
def test_get_granted_scopes(requests_mock, mocker):
authenticator = mocker.Mock()
authenticator.get_access_token.return_value = "the-token"
expected_scopes = ["a", "b", "c"]
response = [
{"json": {"scopes": expected_scopes}, "status_code": 200},
]
requests_mock.register_uri("GET", "https://api.hubapi.com/oauth/v1/access-tokens/the-token", response)
actual_scopes = SourceHubspot({}, None, None).get_granted_scopes(authenticator)
assert expected_scopes == actual_scopes
def test_get_granted_scopes_retry(requests_mock, mocker):
authenticator = mocker.Mock()
expected_token = "the-token"
authenticator.get_access_token.return_value = expected_token
mock_url = f"https://api.hubapi.com/oauth/v1/access-tokens/{expected_token}"
response = [
{"json": {}, "status_code": 500},
]
requests_mock.register_uri("GET", mock_url, response)
actual_scopes = SourceHubspot({}, None, None).get_granted_scopes(authenticator)
assert len(requests_mock.request_history) > 1
def test_streams_oauth_2_auth_no_suitable_scopes(requests_mock, mocker, config):
authenticator = mocker.Mock()
authenticator.get_access_token.return_value = "the-token"
mocker.patch("source_hubspot.streams.API.is_oauth2", return_value=True)
mocker.patch("source_hubspot.streams.API.get_authenticator", return_value=authenticator)
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
expected_scopes = ["no.scopes.granted"]
response = [
{"json": {"scopes": expected_scopes}, "status_code": 200},
]
requests_mock.register_uri("GET", "https://api.hubapi.com/oauth/v1/access-tokens/the-token", response)
streams = SourceHubspot(config, None, None).streams(config)
assert len(streams) == 0

34
airbyte-integrations/connectors/source-hubspot/unit_tests/test_split_properties.py
View File

@@ -1,34 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import pytest
from source_hubspot.helpers import APIv1Property, APIv3Property
lorem_ipsum = """Lorem ipsum dolor sit amet, consectetur adipiscing elit"""
lorem_ipsum = lorem_ipsum.lower().replace(",", "")
many_properties = lorem_ipsum.split(" ") * 1000
few_properties = ["firstname", "lastname", "age", "dob", "id"]
@pytest.mark.parametrize(
("cls", "properties", "chunks_expected"),
(
(APIv1Property, few_properties, 1),
(APIv3Property, few_properties, 1),
(APIv1Property, many_properties, 11),
(APIv3Property, many_properties, 5),
),
)
def test_split_properties(cls, properties, chunks_expected):
chunked_properties = set()
index = 0
for index, chunk in enumerate(cls(properties).split()):
chunked_properties |= set(chunk.properties)
as_string = next(iter(chunk.as_url_param().values()))
assert len(as_string) <= cls.PROPERTIES_PARAM_MAX_LENGTH
chunks = index + 1
assert chunked_properties == set(properties)
assert chunks == chunks_expected

466
airbyte-integrations/connectors/source-hubspot/unit_tests/test_streams.py
View File

@@ -3,32 +3,27 @@
#
import json
from unittest.mock import patch
import mock
import pendulum
import pytest
from source_hubspot.streams import (
ContactsWebAnalytics,
CustomObject,
Deals,
RecordUnnester,
from airbyte_cdk.models import (
AirbyteStateBlob,
AirbyteStateMessage,
AirbyteStateType,
AirbyteStreamState,
ConfiguredAirbyteCatalogSerializer,
StreamDescriptor,
SyncMode,
)
from airbyte_cdk.models import AirbyteStateBlob, AirbyteStateMessage, AirbyteStateType, AirbyteStreamState, StreamDescriptor, SyncMode
from airbyte_cdk.sources.types import Record
from airbyte_cdk.test.entrypoint_wrapper import discover, read
from airbyte_cdk.test.state_builder import StateBuilder
from .conftest import find_stream, mock_dynamic_schema_requests_with_skip, read_from_stream
from .conftest import find_stream, get_source, mock_dynamic_schema_requests_with_skip, read_from_stream
from .utils import read_full_refresh, read_incremental
@pytest.fixture(autouse=True)
def time_sleep_mock(mocker):
time_mock = mocker.patch("time.sleep", lambda x: None)
yield time_mock
def test_updated_at_field_non_exist_handler(requests_mock, config, common_params, fake_properties_list, custom_object_schema):
def test_updated_at_field_non_exist_handler(requests_mock, config, fake_properties_list, custom_object_schema):
requests_mock.register_uri("GET", "/crm/v3/schemas", json={"results": [custom_object_schema]})
stream = find_stream("contact_lists", config)
created_at = "2022-03-25T16:43:11Z"
@@ -87,16 +82,12 @@ def test_updated_at_field_non_exist_handler(requests_mock, config, common_params
("workflows", "", {"updatedAt": 1675121674226}),
],
)
@mock.patch("source_hubspot.source.SourceHubspot.get_custom_object_streams")
def test_streams_read(
mock_get_custom_object_streams, stream_class, endpoint, cursor_value, requests_mock, common_params, fake_properties_list, config
):
def test_streams_read(stream_class, endpoint, cursor_value, requests_mock, fake_properties_list, config):
mock_dynamic_schema_requests_with_skip(requests_mock, [])
stream = find_stream(stream_class, config)
data_field = (
stream.retriever.record_selector.extractor.field_path[0] if len(stream.retriever.record_selector.extractor.field_path) > 0 else None
)
list_entities = [
{
"id": "test_id",
@@ -192,25 +183,18 @@ def test_streams_read(
"DealsArchived stream with v2 field transformations",
],
)
@mock.patch("source_hubspot.source.SourceHubspot.get_custom_object_streams")
def test_stream_read_with_legacy_field_transformation(
mock_get_custom_object_streams,
stream_class,
endpoint,
cursor_value,
requests_mock,
common_params,
fake_properties_list,
migrated_properties_list,
config,
):
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
if isinstance(stream_class, str):
stream = find_stream(stream_class, config)
data_field = stream.retriever.record_selector.extractor.field_path[0]
else:
stream = stream_class(**common_params)
data_field = stream.data_field
stream = find_stream(stream_class, config)
data_field = stream.retriever.record_selector.extractor.field_path[0]
responses = [
{
"json": {
@@ -283,12 +267,8 @@ def test_stream_read_with_legacy_field_transformation(
@pytest.mark.parametrize("sync_mode", [SyncMode.full_refresh, SyncMode.incremental])
@mock.patch("source_hubspot.source.SourceHubspot.get_custom_object_streams")
def test_crm_search_streams_with_no_associations(
mock_get_custom_object_streams, sync_mode, common_params, requests_mock, fake_properties_list, config
):
def test_crm_search_streams_with_no_associations(sync_mode, requests_mock, fake_properties_list, config):
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
stream_state = AirbyteStateMessage(
type=AirbyteStateType.STREAM,
stream=AirbyteStreamState(
@@ -351,8 +331,7 @@ def test_crm_search_streams_with_no_associations(
{"json": {}, "status_code": 504},
],
)
@mock.patch("source_hubspot.source.SourceHubspot.get_custom_object_streams")
def test_common_error_retry(mock_get_custom_object_streams, error_response, requests_mock, common_params, fake_properties_list, config):
def test_common_error_retry(error_response, requests_mock, config, fake_properties_list, mock_dynamic_schema_requests):
"""Error once, check that we retry and not fail"""
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
@@ -394,10 +373,8 @@ def test_common_error_retry(mock_get_custom_object_streams, error_response, requ
assert len(requests_mock.request_history) > 1
def test_contact_lists_transform(requests_mock, common_params, config, custom_object_schema):
def test_contact_lists_transform(requests_mock, config, custom_object_schema, mock_dynamic_schema_requests):
requests_mock.register_uri("GET", "/crm/v3/schemas", json={"results": [custom_object_schema]})
stream = find_stream("contact_lists", config)
responses = [
{
"json": {
@@ -420,21 +397,16 @@ def test_contact_lists_transform(requests_mock, common_params, config, custom_ob
]
requests_mock.register_uri("POST", "https://api.hubapi.com/crm/v3/lists/search", responses)
records = read_full_refresh(stream)
records = read_from_stream(config, "contact_lists", SyncMode.full_refresh).records
assert len(records) > 0
for record in records:
assert isinstance(record["updatedAt"], str)
assert isinstance(record.record.data["updatedAt"], str)
@mock.patch("source_hubspot.source.SourceHubspot.get_custom_object_streams")
def test_client_side_incremental_stream(
mock_get_custom_object_streams, mock_dynamic_schema_requests, requests_mock, common_params, fake_properties_list, config
):
def test_client_side_incremental_stream(mock_dynamic_schema_requests, requests_mock, fake_properties_list, config):
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
stream = find_stream("forms", config)
data_field = stream.retriever.record_selector.extractor.field_path[0]
data_field = "results"
latest_cursor_value = "2024-01-30T23:46:36.287000Z"
responses = [
{
@@ -457,216 +429,36 @@ def test_client_side_incremental_stream(
}
]
stream_url = stream.retriever.requester.url_base + stream.retriever.requester.path
requests_mock.register_uri("GET", stream_url, responses)
requests_mock.register_uri("GET", "https://api.hubapi.com/marketing/v3/forms", responses)
requests_mock.register_uri("GET", "/properties/v2/form/properties", properties_response)
output = read_from_stream(config, "forms", SyncMode.incremental)
assert output.state_messages[-1].state.stream.stream_state.__dict__[stream.cursor_field] == latest_cursor_value
@pytest.fixture(name="custom_object_schema")
def custom_object_schema_fixture():
return {
"labels": {"this": "that"},
"requiredProperties": ["name"],
"searchableProperties": ["name"],
"primaryDisplayProperty": "name",
"secondaryDisplayProperties": [],
"archived": False,
"restorable": True,
"metaType": "PORTAL_SPECIFIC",
"id": "7232155",
"fullyQualifiedName": "p19936848_Animal",
"createdAt": "2022-06-17T18:40:27.019Z",
"updatedAt": "2022-06-17T18:40:27.019Z",
"objectTypeId": "2-7232155",
"properties": [
{
"name": "name",
"label": "Animal name",
"type": "string",
"fieldType": "text",
"description": "The animal name.",
"groupName": "animal_information",
"options": [],
"displayOrder": -1,
"calculated": False,
"externalOptions": False,
"hasUniqueValue": False,
"hidden": False,
"hubspotDefined": False,
"modificationMetadata": {"archivable": True, "readOnlyDefinition": True, "readOnlyValue": False},
"formField": True,
}
],
"associations": [],
"name": "animals",
}
@pytest.fixture(name="expected_custom_object_json_schema")
def expected_custom_object_json_schema():
return {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"additionalProperties": True,
"properties": {
"id": {"type": ["null", "string"]},
"createdAt": {"type": ["null", "string"], "format": "date-time"},
"updatedAt": {"type": ["null", "string"], "format": "date-time"},
"archived": {"type": ["null", "boolean"]},
"properties": {"type": ["null", "object"], "properties": {"name": {"type": ["null", "string"]}}},
"properties_name": {"type": ["null", "string"]},
},
}
assert output.state_messages[-1].state.stream.stream_state.__dict__["updatedAt"] == latest_cursor_value
def test_custom_object_stream_doesnt_call_hubspot_to_get_json_schema_if_available(
requests_mock, custom_object_schema, expected_custom_object_json_schema, common_params
requests_mock, custom_object_schema, config, expected_custom_object_json_schema, mock_dynamic_schema_requests
):
stream = CustomObject(
entity="animals",
schema=expected_custom_object_json_schema,
fully_qualified_name="p123_animals",
custom_properties={"name": {"type": ["null", "string"]}},
**common_params,
)
adapter = requests_mock.register_uri("GET", "/crm/v3/schemas", [{"json": {"results": [custom_object_schema]}}])
json_schema = stream.get_json_schema()
adapter = requests_mock.register_uri("GET", "/crm/v3/schemas", json={"results": [custom_object_schema]})
streams = discover(get_source(config), config)
json_schema = [s.json_schema for s in streams.catalog.catalog.streams if s.name == "animals"][0]
assert json_schema == expected_custom_object_json_schema
assert not adapter.called
# called only once when creating dynamic streams
assert adapter.call_count == 1
def test_get_custom_objects_metadata_success(requests_mock, custom_object_schema, expected_custom_object_json_schema, api):
def test_get_custom_objects_metadata_success(
requests_mock, custom_object_schema, expected_custom_object_json_schema, config, mock_dynamic_schema_requests
):
requests_mock.register_uri("GET", "/crm/v3/schemas", json={"results": [custom_object_schema]})
for entity, fully_qualified_name, schema, custom_properties in api.get_custom_objects_metadata():
assert entity == "animals"
assert fully_qualified_name == "p19936848_Animal"
assert schema == expected_custom_object_json_schema
source_hubspot = get_source(config)
streams = discover(source_hubspot, config)
custom_stream = [s for s in source_hubspot.streams(config) if s.name == "animals"][0]
custom_stream_json_schema = [s.json_schema for s in streams.catalog.catalog.streams if s.name == "animals"][0]
@pytest.mark.parametrize(
"input_data, unnest_fields, expected_output",
(
(
[{"id": 1, "createdAt": "2020-01-01", "email": {"from": "integration-test@airbyte.io", "to": "michael_scott@gmail.com"}}],
[],
[{"id": 1, "createdAt": "2020-01-01", "email": {"from": "integration-test@airbyte.io", "to": "michael_scott@gmail.com"}}],
),
(
[
{
"id": 1,
"createdAt": "2020-01-01",
"email": {"from": "integration-test@airbyte.io", "to": "michael_scott@gmail.com"},
"properties": {"phone": "+38044-111-111", "address": "31, Cleveland str, Washington DC"},
}
],
[],
[
{
"id": 1,
"createdAt": "2020-01-01",
"email": {"from": "integration-test@airbyte.io", "to": "michael_scott@gmail.com"},
"properties": {"phone": "+38044-111-111", "address": "31, Cleveland str, Washington DC"},
"properties_phone": "+38044-111-111",
"properties_address": "31, Cleveland str, Washington DC",
}
],
),
(
[
{
"id": 1,
"createdAt": "2020-01-01",
"email": {"from": "integration-test@airbyte.io", "to": "michael_scott@gmail.com"},
}
],
["email"],
[
{
"id": 1,
"createdAt": "2020-01-01",
"email": {"from": "integration-test@airbyte.io", "to": "michael_scott@gmail.com"},
"email_from": "integration-test@airbyte.io",
"email_to": "michael_scott@gmail.com",
}
],
),
(
[
{
"id": 1,
"createdAt": "2020-01-01",
"email": {"from": "integration-test@airbyte.io", "to": "michael_scott@gmail.com"},
"properties": {"phone": "+38044-111-111", "address": "31, Cleveland str, Washington DC"},
}
],
["email"],
[
{
"id": 1,
"createdAt": "2020-01-01",
"email": {"from": "integration-test@airbyte.io", "to": "michael_scott@gmail.com"},
"email_from": "integration-test@airbyte.io",
"email_to": "michael_scott@gmail.com",
"properties": {"phone": "+38044-111-111", "address": "31, Cleveland str, Washington DC"},
"properties_phone": "+38044-111-111",
"properties_address": "31, Cleveland str, Washington DC",
}
],
),
),
)
def test_records_unnester(input_data, unnest_fields, expected_output):
unnester = RecordUnnester(fields=unnest_fields)
assert list(unnester.unnest(input_data)) == expected_output
def test_web_analytics_stream_slices(common_params, mocker):
parent_slicer_mock = mocker.patch("airbyte_cdk.sources.streams.http.HttpSubStream.stream_slices")
parent_slicer_mock.return_value = (_ for _ in [{"parent": {"id": 1}}])
pendulum_now_mock = mocker.patch("pendulum.now")
pendulum_now_mock.return_value = pendulum.parse(common_params["start_date"]).add(days=50)
stream = ContactsWebAnalytics(**common_params)
slices = list(stream.stream_slices(SyncMode.incremental, cursor_field="occurredAt"))
assert len(slices) == 2
assert all(map(lambda slice: slice["objectId"] == 1, slices))
assert [("2021-01-10T00:00:00Z", "2021-02-09T00:00:00Z"), ("2021-02-09T00:00:00Z", "2021-03-01T00:00:00Z")] == [
(s["occurredAfter"], s["occurredBefore"]) for s in slices
]
def test_web_analytics_latest_state(common_params, mocker):
parent_slicer_mock = mocker.patch("airbyte_cdk.sources.streams.http.HttpSubStream.stream_slices")
parent_slicer_mock.return_value = (_ for _ in [{"parent": {"id": "1"}}])
pendulum_now_mock = mocker.patch("pendulum.now")
pendulum_now_mock.return_value = pendulum.parse(common_params["start_date"]).add(days=10)
parent_slicer_mock = mocker.patch("source_hubspot.streams.BaseStream.read_records")
parent_slicer_mock.return_value = (_ for _ in [{"objectId": "1", "occurredAt": "2021-01-02T00:00:00Z"}])
stream = ContactsWebAnalytics(**common_params)
stream.state = {"1": {"occurredAt": "2021-01-01T00:00:00Z"}}
slices = list(stream.stream_slices(SyncMode.incremental, cursor_field="occurredAt"))
records = [
list(stream.read_records(SyncMode.incremental, cursor_field="occurredAt", stream_slice=stream_slice)) for stream_slice in slices
]
assert len(slices) == 1
assert len(records) == 1
assert len(records[0]) == 1
assert records[0][0]["objectId"] == "1"
assert stream.state["1"]["occurredAt"] == "2021-01-02T00:00:00Z"
assert custom_stream_json_schema == expected_custom_object_json_schema
assert custom_stream.retriever._parameters["entity"] == "p19936848_Animal"
@pytest.mark.parametrize(
@@ -678,21 +470,13 @@ def test_web_analytics_latest_state(common_params, mocker):
("marketing_emails", {"updated": 1634050455543}, {"rootMicId": 1234.56}, {"rootMicId": "1234.56"}),
],
)
@mock.patch("source_hubspot.source.SourceHubspot.get_custom_object_streams")
def test_cast_record_fields_with_schema_if_needed(
mock_get_custom_object_stream, stream_class, cursor_value, requests_mock, common_params, data_to_cast, expected_casted_data, config
):
def test_cast_record_fields_with_schema_if_needed(stream_class, cursor_value, requests_mock, data_to_cast, expected_casted_data, config):
"""
Test that the stream cast record fields with stream json schema if needed
"""
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
if isinstance(stream_class, str):
stream = find_stream(stream_class, config)
data_field = stream.retriever.record_selector.extractor.field_path[0]
else:
stream = stream_class(**common_params)
data_field = stream.data_field
stream = find_stream(stream_class, config)
data_field = stream.retriever.record_selector.extractor.field_path[0]
responses = [
{
@@ -737,7 +521,7 @@ def test_cast_record_fields_with_schema_if_needed(
{"hs_closed_amount": "123456"},
),
(
Deals,
"deals",
"deal",
{"updatedAt": "2022-02-25T16:43:11Z"},
[("hs_closed_amount", "integer")],
@@ -745,7 +529,7 @@ def test_cast_record_fields_with_schema_if_needed(
{"hs_closed_amount": 123456},
),
(
Deals,
"deals",
"deal",
{"updatedAt": "2022-02-25T16:43:11Z"},
[("hs_closed_amount", "number")],
@@ -753,7 +537,7 @@ def test_cast_record_fields_with_schema_if_needed(
{"hs_closed_amount": 123456.10},
),
(
Deals,
"deals",
"deal",
{"updatedAt": "2022-02-25T16:43:11Z"},
[("hs_closed_amount", "boolean")],
@@ -762,31 +546,25 @@ def test_cast_record_fields_with_schema_if_needed(
),
],
)
@mock.patch("source_hubspot.source.SourceHubspot.get_custom_object_streams")
def test_cast_record_fields_if_needed(
mock_get_custom_object_streams,
stream_class,
endpoint,
cursor_value,
fake_properties_list_response,
requests_mock,
common_params,
data_to_cast,
expected_casted_data,
custom_object_schema,
config,
mock_dynamic_schema_requests,
):
"""
Test that the stream cast record fields in properties key with properties endpoint response if needed
"""
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
if isinstance(stream_class, str):
stream = find_stream(stream_class, config)
data_field = stream.retriever.record_selector.extractor.field_path[0]
else:
stream = stream_class(**common_params)
data_field = stream.data_field
stream = find_stream(stream_class, config)
data_field = stream.retriever.record_selector.extractor.field_path[0]
responses = [
{
@@ -815,8 +593,150 @@ def test_cast_record_fields_if_needed(
requests_mock.register_uri("GET", stream_url, responses)
requests_mock.register_uri("GET", f"/properties/v2/{endpoint}/properties", properties_response)
records = read_full_refresh(stream)
records = read_from_stream(config, stream_class, SyncMode.full_refresh).records
assert records
record = records[0]
for casted_key, casted_value in expected_casted_data.items():
assert record["properties"][casted_key] == casted_value
assert record.record.data["properties"][casted_key] == casted_value
@pytest.mark.parametrize(
"stream, scopes, url, method",
[
("campaigns", "crm.lists.read", "https://api.hubapi.com/email/public/v1/campaigns", "GET"),
("companies", "crm.objects.contacts.read, crm.objects.companies.read", "https://api.hubapi.com/crm/v3/objects/company", "GET"),
("companies_property_history", "crm.objects.companies.read", "https://api.hubapi.com/properties/v2/companies/properties", "GET"),
("contact_lists", "crm.lists.read", "https://api.hubapi.com/crm/v3/lists/search", "POST"),
("contacts_property_history", "crm.objects.contacts.read", "https://api.hubapi.com/properties/v2/contacts/properties", "GET"),
("deal_pipelines", "crm.objects.contacts.read", "https://api.hubapi.com/crm-pipelines/v1/pipelines/deals", "GET"),
("deals", "crm.objects.deals.read", "https://api.hubapi.com/crm/v3/objects/deal", "GET"),
("deals_property_history", "crm.objects.deals.read", "https://api.hubapi.com/properties/v2/deals/properties", "GET"),
("email_events", "content", "https://api.hubapi.com/email/public/v1/events", "GET"),
("email_subscriptions", "content", "https://api.hubapi.com/email/public/v1/subscriptions", "GET"),
(
"engagements",
"crm.objects.companies.read, crm.objects.contacts.read, crm.objects.deals.read, tickets, e-commerce",
"https://api.hubapi.com/engagements/v1/engagements/paged",
"GET",
),
("engagements_calls", "crm.objects.contacts.read", "https://api.hubapi.com/crm/v3/objects/calls", "GET"),
("engagements_emails", "crm.objects.contacts.read, sales-email-read", "https://api.hubapi.com/crm/v3/objects/emails", "GET"),
("engagements_meetings", "crm.objects.contacts.read", "https://api.hubapi.com/crm/v3/objects/meetings", "GET"),
("engagements_notes", "crm.objects.contacts.read", "https://api.hubapi.com/crm/v3/objects/notes", "GET"),
("engagements_tasks", "crm.objects.contacts.read", "https://api.hubapi.com/crm/v3/objects/tasks", "GET"),
("marketing_emails", "content", "https://api.hubapi.com/marketing-emails/v1/emails/with-statistics", "GET"),
("deals_archived", "contacts, crm.objects.deals.read", "https://api.hubapi.com/crm/v3/objects/deals", "GET"),
("forms", "forms", "https://api.hubapi.com/marketing/v3/forms", "GET"),
# form_submissions have parent stream forms
# ("form_submissions", "forms", "https://api.hubapi.com/marketing/v3/forms", "GET"),
("goals", "crm.objects.goals.read", "https://api.hubapi.com/crm/v3/objects/goal_targets", "GET"),
("line_items", "e-commerce, crm.objects.line_items.read", "https://api.hubapi.com/crm/v3/objects/line_item", "GET"),
("owners", "crm.objects.owners.read", "https://api.hubapi.com/crm/v3/owners", "GET"),
("owners_archived", "crm.objects.owners.read", "https://api.hubapi.com/crm/v3/owners", "GET"),
("products", "e-commerce", "https://api.hubapi.com/crm/v3/objects/product", "GET"),
("subscription_changes", "content", "https://api.hubapi.com/email/public/v1/subscriptions/timeline", "GET"),
(
"ticket_pipelines",
"media_bridge.read, tickets, crm.schemas.custom.read, e-commerce, timeline, contacts, crm.schemas.contacts.read, crm.objects.contacts.read, crm.objects.contacts.write, crm.objects.deals.read, crm.schemas.quotes.read, crm.objects.deals.write, crm.objects.companies.read, crm.schemas.companies.read, crm.schemas.deals.read, crm.schemas.line_items.read, crm.objects.companies.write",
"https://api.hubapi.com/crm/v3/pipelines/tickets",
"GET",
),
("workflows", "automation", "https://api.hubapi.com/automation/v3/workflows", "GET"),
("contacts", "crm.objects.contacts.read", "https://api.hubapi.com/crm/v3/objects/contact", "GET"),
("deal_splits", "crm.objects.deals.read", "https://api.hubapi.com/crm/v3/objects/deal_split", "GET"),
(
"leads",
"crm.objects.contacts.read, crm.objects.companies.read, crm.objects.leads.read",
"https://api.hubapi.com/crm/v3/objects/leads",
"GET",
),
("tickets", "tickets", "https://api.hubapi.com/crm/v3/objects/ticket", "GET"),
],
)
def test_streams_raise_error_message_if_scopes_missing(stream, scopes, url, method, requests_mock, config, mock_dynamic_schema_requests):
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
requests_mock.register_uri(method, url, [{"status_code": 403}])
catalog = ConfiguredAirbyteCatalogSerializer.load(
{
"streams": [
{
"stream": {"name": stream, "json_schema": {}, "supported_sync_modes": ["full_refresh"]},
"sync_mode": "full_refresh",
"destination_sync_mode": "append",
}
]
}
)
state = (
StateBuilder()
.with_stream_state(
stream,
{},
)
.build()
)
source_hubspot = get_source(config)
output = read(source_hubspot, config=config, catalog=catalog, state=state)
assert output.errors[0].trace.error.message == (
"Access denied (403). The authenticated user does not have permissions to access the resource. "
f"Verify your scopes: {scopes} to access stream {stream}. "
f"See details: https://docs.airbyte.com/integrations/sources/hubspot#step-2-configure-the-scopes-for-your-streams-private-app-only"
)
def test_discover_if_scopes_missing(config, requests_mock, mock_dynamic_schema_requests):
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
requests_mock.register_uri("GET", "https://api.hubapi.com/properties/v2/companies/properties", [{"status_code": 403}])
source_hubspot = get_source(config)
output = discover(source_hubspot, config=config)
assert output.catalog
def test_read_catalog_with_missing_scopes(config, requests_mock, mock_dynamic_schema_requests):
requests_mock.get("https://api.hubapi.com/crm/v3/schemas", json={}, status_code=200)
requests_mock.get("https://api.hubapi.com/marketing-emails/v1/emails/with-statistics", json={}, status_code=403)
requests_mock.get("https://api.hubapi.com/email/public/v1/subscriptions", json={}, status_code=403)
catalog = ConfiguredAirbyteCatalogSerializer.load(
{
"streams": [
{
"stream": {"name": "marketing_emails", "json_schema": {}, "supported_sync_modes": ["full_refresh"]},
"sync_mode": "full_refresh",
"destination_sync_mode": "append",
},
{
"stream": {"name": "email_subscriptions", "json_schema": {}, "supported_sync_modes": ["full_refresh"]},
"sync_mode": "full_refresh",
"destination_sync_mode": "append",
},
]
}
)
state = (
StateBuilder()
.with_stream_state(
"marketing_emails",
{},
)
.with_stream_state(
"email_subscriptions",
{},
)
.build()
)
source_hubspot = get_source(config)
output = read(source_hubspot, config=config, catalog=catalog, state=state)
assert output.errors
assert not output.records
error_messages = [error.trace.error.message for error in output.errors]
assert error_messages
assert (
"Access denied (403). The authenticated user does not have permissions to access the resource. "
"Verify your scopes: content to access stream email_subscriptions. See details: "
"https://docs.airbyte.com/integrations/sources/hubspot#step-2-configure-the-scopes-for-your-streams-private-app-only"
) in error_messages
assert (
"Access denied (403). The authenticated user does not have permissions to access the resource. "
"Verify your scopes: content to access stream marketing_emails. See details: "
"https://docs.airbyte.com/integrations/sources/hubspot#step-2-configure-the-scopes-for-your-streams-private-app-only"
) in error_messages

1
docs/integrations/sources/hubspot.md
View File

@@ -333,6 +333,7 @@ The connector is restricted by normal HubSpot [rate limitations](https://legacyd
| Version | Date | Pull Request | Subject |
|:-----------|:-----------|:---------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 5.8.0 | 2025-05-28 | [60855](https://github.com/airbytehq/airbyte/pull/60855) | Migrate to manifest-only |
| 5.7.0 | 2025-05-27 | [60919](https://github.com/airbytehq/airbyte/pull/60919) | Promoting release candidate 5.7.0-rc.2 to a main version. |
| 5.7.0-rc.2 | 2025-05-23 | [60881](https://github.com/airbytehq/airbyte/pull/60881) | Ignore 403 errors for dynamic streams to prevent sync failures |
| 5.7.0-rc.1 | 2025-05-22 | [60830](https://github.com/airbytehq/airbyte/pull/60830) | Migrate CustomObjects streams |