jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

feat(source-bing-ads) migrate to manifest only (#62520)

Browse Source 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: ChristoGrab <christo.grab@gmail.com>
This commit is contained in:
Daryna Ishchenko
2025-07-08 11:50:47 +03:00
committed by GitHub
parent 21c9c3bdc9
commit 2413b49c38
57 changed files with 1361 additions and 5444 deletions
Download Patch File Download Diff File Expand all files Collapse all files

102
airbyte-integrations/connectors/source-bing-ads/README.md
View File

@@ -1,57 +1,42 @@
# Bing-Ads source connector
# Bing Ads
This directory contains the manifest-only connector for `source-bing-ads`.
This is the repository for the Bing-Ads source connector, written in Python.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/bing-ads).
## Local development
### Prerequisites
- Python (~=3.9)
- Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation)
### Installing the connector
From this connector directory, run:
```bash
poetry install --with dev
```
### Create credentials
## Documentation reference:
Visit `https://learn.microsoft.com/en-us/advertising/guides/?view=bingads-13` for API documentation
## Authentication setup
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/bing-ads)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_bing_ads/spec.yaml` file.
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the spec defined in `manifest.yaml` file.
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
See `sample_files/sample_config.json` for a sample config file.
### Locally running the connector
## Usage
There are multiple ways to use this connector:
- You can use this connector as any other connector in Airbyte Marketplace.
- You can load this connector in `pyairbyte` using `get_source`!
- You can open this connector in Connector Builder, edit it, and publish to your workspaces.
```
poetry run source-bing-ads spec
poetry run source-bing-ads check --config secrets/config.json
poetry run source-bing-ads discover --config secrets/config.json
poetry run source-bing-ads read --config secrets/config.json --catalog sample_files/configured_catalog.json
```
Please refer to the manifest-only connector documentation for more details.
### Running unit tests
## Local Development
We recommend you use the Connector Builder to edit this connector.
To run unit tests locally, from the connector directory run:
But, if you want to develop this connector locally, you can use the following steps.
```
poetry run pytest unit_tests
```
### Building the docker image
1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
2. Run the following command to build the docker image:
### Environment Setup
You will need `airbyte-ci` installed. You can find the documentation [here](airbyte-ci).
### Build
This will create a dev image (`source-bing-ads:dev`) that you can use to test the connector locally.
```bash
airbyte-ci connectors --name=source-bing-ads build
```
An image will be available on your host with the tag `airbyte/source-bing-ads:dev`.
### Test
This will run the acceptance tests for the connector.
```bash
airbyte-ci connectors --name=source-bing-ads test
```
### Running as a docker container
@@ -63,42 +48,3 @@ docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-bing-ads:dev check --c
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-bing-ads:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-bing-ads:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```
### Running our CI test suite
You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
```bash
airbyte-ci connectors --name=source-bing-ads test
```
### Customizing acceptance Tests
Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
### Dependency Management
All of your dependencies should be managed via Poetry.
To add a new dependency, run:
```bash
poetry add <package-name>
```
Please commit the changes to `pyproject.toml` and `poetry.lock` files.
## Publishing a new version of the connector
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-bing-ads test`
2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
- bump the `dockerImageTag` value in in `metadata.yaml`
- bump the `version` value in `pyproject.toml`
3. Make sure the `metadata.yaml` content is up to date.
4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/bing-ads.md`).
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
6. Pat yourself on the back for being an awesome contributor.
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.

0
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/components.py → airbyte-integrations/connectors/source-bing-ads/components.py
View File

9
airbyte-integrations/connectors/source-bing-ads/main.py
View File

@@ -1,9 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from source_bing_ads.run import run
if __name__ == "__main__":
run()

254
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/manifest.yaml → airbyte-integrations/connectors/source-bing-ads/manifest.yaml
View File

@@ -117,7 +117,7 @@ definitions:
type: RecordSelector
record_filter:
type: CustomRecordFilter
class_name: source_bing_ads.components.DuplicatedRecordsFilter
class_name: source_declarative_manifest.components.DuplicatedRecordsFilter
extractor:
type: DpathExtractor
field_path: ["Accounts"]
@@ -148,6 +148,14 @@ definitions:
- parsed_account_id
value: "{{ record.Id | string }}"
value_type: string
- type: AddFields
condition: "{{ record.get('TaxCertificate') is not none and record.get('TaxCertificate').get('TaxCertificates') is not none and 'KeyValuePairOfstringbase64Binary' in record.get('TaxCertificate').get('TaxCertificates') }}"
fields:
- type: AddedFieldDefinition
path:
- TaxCertificate
- TaxCertificates
value: "{{ record.get('TaxCertificate').get('TaxCertificates').get('KeyValuePairOfstringbase64Binary') if record.get('TaxCertificate') is not none and record.get('TaxCertificate').get('TaxCertificates') is not none and 'KeyValuePairOfstringbase64Binary' in record.get('TaxCertificate').get('TaxCertificates') else none }}"
campaigns_stream:
type: DeclarativeStream
name: campaigns
@@ -233,7 +241,7 @@ definitions:
- UrlCustomParameters
value: "{{ {'Parameters': {'CustomParameter': record.UrlCustomParameters.Parameters}} if record.get('UrlCustomParameters') and record.get('UrlCustomParameters').get('Parameters') else record.get('UrlCustomParameters') }}"
- type: CustomTransformation
class_name: source_bing_ads.components.BingAdsCampaignsRecordTransformer
class_name: source_declarative_manifest.components.BingAdsCampaignsRecordTransformer
- type: RemoveFields
field_pointers: []
ad_groups_stream:
@@ -2474,11 +2482,11 @@ definitions:
- "Account Id"
value: "{{ stream_partition['account_id'] }}"
- type: CustomTransformation
class_name: source_bing_ads.components.BulkDatetimeToRFC3339
class_name: source_declarative_manifest.components.BulkDatetimeToRFC3339
incremental_sync: "#/definitions/incremental_bulk_datetime_cursor"
state_migrations:
- type: CustomStateMigration
class_name: source_bing_ads.components.BulkStreamsStateMigration
class_name: source_declarative_manifest.components.BulkStreamsStateMigration
- type: LegacyToPerPartitionStateMigration
# base retrievers
@@ -2616,7 +2624,7 @@ definitions:
# substream partition routers
account_substream_partition_router:
type: CustomPartitionRouter
class_name: source_bing_ads.components.LightSubstreamPartitionRouter
class_name: source_declarative_manifest.components.LightSubstreamPartitionRouter
parent_stream_configs:
- type: ParentStreamConfig
# Id is the primary key of the accounts stream but an integer
@@ -5219,7 +5227,7 @@ dynamic_streams:
error_message_contains: Please check your request parameters.
schema_loader:
type: CustomSchemaLoader
class_name: source_bing_ads.components.CustomReportSchemaLoader
class_name: source_declarative_manifest.components.CustomReportSchemaLoader
incremental_sync:
$ref: "#/definitions/incremental_sync_report_datetime_cursor"
end_datetime:
@@ -5227,7 +5235,7 @@ dynamic_streams:
datetime: "{{ format_datetime(now_utc(), parameters['end_date_time_format']) }}"
transformations:
- type: CustomTransformation
class_name: source_bing_ads.components.CustomReportTransformation
class_name: source_declarative_manifest.components.CustomReportTransformation
components_resolver:
type: ConfigComponentsResolver
stream_config:
@@ -17471,3 +17479,235 @@ concurrency_level:
type: ConcurrencyLevel
default_concurrency: 2
max_concurrency: 10
spec:
type: Spec
documentationUrl: https://docs.airbyte.com/integrations/sources/bing-ads
connection_specification:
"$schema": http://json-schema.org/draft-07/schema#
title: Bing Ads Spec
type: object
required:
- developer_token
- client_id
- refresh_token
additionalProperties: true
properties:
auth_method:
type: string
const: oauth2.0
tenant_id:
type: string
title: Tenant ID
description: The Tenant ID of your Microsoft Advertising developer application.
Set this to "common" unless you know you need a different value.
airbyte_secret: true
default: common
order: 0
client_id:
type: string
title: Client ID
description: The Client ID of your Microsoft Advertising developer application.
airbyte_secret: true
order: 1
client_secret:
type: string
title: Client Secret
description: The Client Secret of your Microsoft Advertising developer application.
default: ''
airbyte_secret: true
order: 2
refresh_token:
type: string
title: Refresh Token
description: Refresh Token to renew the expired Access Token.
airbyte_secret: true
order: 3
developer_token:
type: string
title: Developer Token
description: Developer token associated with user. See more info <a href="https://docs.microsoft.com/en-us/advertising/guides/get-started?view=bingads-13#get-developer-token">
in the docs</a>.
airbyte_secret: true
order: 4
account_names:
title: Account Names Predicates
description: Predicates that will be used to sync data by specific accounts.
type: array
order: 5
items:
description: Account Names Predicates Config.
type: object
properties:
operator:
title: Operator
description: An Operator that will be used to filter accounts. The Contains
predicate has features for matching words, matching inflectional forms
of words, searching using wildcard characters, and searching using proximity.
The Equals is used to return all rows where account name is equal(=)
to the string that you provided
type: string
enum:
- Contains
- Equals
name:
title: Account Name
description: Account Name is a string value for comparing with the specified
predicate.
type: string
required:
- operator
- name
reports_start_date:
type: string
title: Reports replication start date
format: date
description: The start date from which to begin replicating report data. Any
data generated before this date will not be replicated in reports. This is
a UTC date in YYYY-MM-DD format. If not set, data from previous and current
calendar year will be replicated.
order: 6
lookback_window:
title: Lookback window
description: Also known as attribution or conversion window. How far into the
past to look for records (in days). If your conversion window has an hours/minutes
granularity, round it up to the number of days exceeding. Used only for performance
report streams in incremental mode without specified Reports Start Date.
type: integer
default: 0
minimum: 0
maximum: 90
order: 7
custom_reports:
title: Custom Reports
description: You can add your Custom Bing Ads report by creating one.
order: 8
type: array
items:
title: Custom Report Config
type: object
properties:
name:
title: Report Name
description: The name of the custom report, this name would be used as
stream name
type: string
examples:
- Account Performance
- AdDynamicTextPerformanceReport
- custom report
reporting_object:
title: Reporting Data Object
description: The name of the the object derives from the ReportRequest
object. You can find it in Bing Ads Api docs - Reporting API - Reporting
Data Objects.
type: string
enum:
- AccountPerformanceReportRequest
- AdDynamicTextPerformanceReportRequest
- AdExtensionByAdReportRequest
- AdExtensionByKeywordReportRequest
- AdExtensionDetailReportRequest
- AdGroupPerformanceReportRequest
- AdPerformanceReportRequest
- AgeGenderAudienceReportRequest
- AudiencePerformanceReportRequest
- CallDetailReportRequest
- CampaignPerformanceReportRequest
- ConversionPerformanceReportRequest
- DestinationUrlPerformanceReportRequest
- DSAAutoTargetPerformanceReportRequest
- DSACategoryPerformanceReportRequest
- DSASearchQueryPerformanceReportRequest
- GeographicPerformanceReportRequest
- GoalsAndFunnelsReportRequest
- HotelDimensionPerformanceReportRequest
- HotelGroupPerformanceReportRequest
- KeywordPerformanceReportRequest
- NegativeKeywordConflictReportRequest
- ProductDimensionPerformanceReportRequest
- ProductMatchCountReportRequest
- ProductNegativeKeywordConflictReportRequest
- ProductPartitionPerformanceReportRequest
- ProductPartitionUnitPerformanceReportRequest
- ProductSearchQueryPerformanceReportRequest
- ProfessionalDemographicsAudienceReportRequest
- PublisherUsagePerformanceReportRequest
- SearchCampaignChangeHistoryReportRequest
- SearchQueryPerformanceReportRequest
- ShareOfVoiceReportRequest
- UserLocationPerformanceReportRequest
report_columns:
title: Columns
description: A list of available report object columns. You can find it
in description of reporting object that you want to add to custom report.
type: array
items:
description: Name of report column.
type: string
minItems: 1
report_aggregation:
title: Aggregation
description: A list of available aggregations.
type: string
items:
title: ValidEnums
description: An enumeration of aggregations.
enum:
- Hourly
- Daily
- Weekly
- Monthly
- DayOfWeek
- HourOfDay
- WeeklyStartingMonday
- Summary
default:
- Hourly
required:
- name
- reporting_object
- report_columns
- report_aggregation
advanced_auth:
auth_flow_type: oauth2.0
predicate_key:
- auth_method
predicate_value: oauth2.0
oauth_config_specification:
complete_oauth_output_specification:
type: object
additionalProperties: false
properties:
refresh_token:
type: string
path_in_connector_config:
- refresh_token
complete_oauth_server_input_specification:
type: object
additionalProperties: false
properties:
client_id:
type: string
client_secret:
type: string
complete_oauth_server_output_specification:
type: object
additionalProperties: false
properties:
client_id:
type: string
path_in_connector_config:
- client_id
client_secret:
type: string
path_in_connector_config:
- client_secret
oauth_user_input_from_connector_config_specification:
type: object
additionalProperties: false
properties:
tenant_id:
type: string
path_in_connector_config:
- tenant_id

10
airbyte-integrations/connectors/source-bing-ads/metadata.yaml
View File

@@ -12,11 +12,11 @@ data:
- api.ads.microsoft.com
- clientcenter.api.bingads.microsoft.com
connectorBuildOptions:
baseImage: docker.io/airbyte/python-connector-base:4.0.0@sha256:d9894b6895923b379f3006fa251147806919c62b7d9021b5cd125bb67d7bbe22
baseImage: docker.io/airbyte/source-declarative-manifest:6.57.1@sha256:0ed3b25bf9c1a91f980b38d1a60ca3921dedb584373d71d6e6a4e5625ecfbb12
connectorSubtype: api
connectorType: source
definitionId: 47f25999-dd5e-4636-8c39-e7cea2453331
dockerImageTag: 2.22.0
dockerImageTag: 2.23.0-rc.1
dockerRepository: airbyte/source-bing-ads
documentationUrl: https://docs.airbyte.com/integrations/sources/bing-ads
erdUrl: https://dbdocs.io/airbyteio/source-bing-ads?view=relationships
@@ -27,7 +27,7 @@ data:
name: Bing Ads
remoteRegistries:
pypi:
enabled: true
enabled: false
packageName: airbyte-source-bing-ads
registryOverrides:
cloud:
@@ -37,7 +37,7 @@ data:
releaseStage: generally_available
releases:
rolloutConfiguration:
enableProgressiveRollout: false
enableProgressiveRollout: true
breakingChanges:
1.0.0:
message: Version 1.0.0 removes the primary keys from the geographic performance report streams. This will prevent the connector from losing data in the incremental append+dedup sync mode because of deduplication and incorrect primary keys. A data reset and schema refresh of all the affected streams is required for the changes to take effect.
@@ -62,7 +62,7 @@ data:
- ad_groups
supportLevel: certified
tags:
- language:python
- language:manifest-only
- cdk:low-code
connectorTestSuitesOptions:
- suite: liveTests

41
airbyte-integrations/connectors/source-bing-ads/pyproject.toml
View File

@@ -1,41 +0,0 @@
[build-system]
requires = [ "poetry-core>=1.0.0",]
build-backend = "poetry.core.masonry.api"
[tool.poetry]
version = "2.22.0"
name = "source-bing-ads"
description = "Source implementation for Bing Ads."
authors = [ "Airbyte <contact@airbyte.io>",]
license = "MIT"
readme = "README.md"
documentation = "https://docs.airbyte.com/integrations/sources/bing-ads"
homepage = "https://airbyte.com"
repository = "https://github.com/airbytehq/airbyte"
[[tool.poetry.packages]]
include = "source_bing_ads"
[tool.poetry.dependencies]
python = "^3.10,<3.12"
bingads = "==13.0.18.1"
urllib3 = "==1.26.18"
airbyte-cdk = "^6"
cached-property = "==1.5.2"
pendulum = "<3.0.0"
[tool.poetry.scripts]
source-bing-ads = "source_bing_ads.run:run"
[tool.poetry.group.dev.dependencies]
freezegun = "^1.4.0"
pytest-mock = "^3.6.1"
pytest = "^8.0.0"
requests-mock = "^1.12.1"
[tool.poe]
include = [
# Shared tasks definition file(s) can be imported here.
# Run `poe` or `poe --help` to see the list of available tasks.
"${POE_GIT_DIR}/poe-tasks/poetry-connector-tasks.toml",
]

27
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/__init__.py
View File

@@ -1,27 +0,0 @@
"""
MIT License
Copyright (c) 2020 Airbyte
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .source import SourceBingAds
__all__ = ["SourceBingAds"]

277
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/base_streams.py
View File

@@ -1,277 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import ssl
import time
from abc import ABC, abstractmethod
from typing import Any, Iterable, List, Mapping, MutableMapping, Optional, Union
from urllib.error import URLError
from bingads.service_client import ServiceClient
from bingads.v13.reporting.reporting_service_manager import ReportingServiceManager
from suds import sudsobject
from airbyte_cdk.models import SyncMode
from airbyte_cdk.sources.streams import Stream
from source_bing_ads.client import Client
class BingAdsBaseStream(Stream, ABC):
primary_key: Optional[Union[str, List[str], List[List[str]]]] = None
def __init__(self, client: Client, config: Mapping[str, Any]) -> None:
super().__init__()
self.client = client
self.config = config
class BingAdsStream(BingAdsBaseStream, ABC):
@property
@abstractmethod
def operation_name(self) -> str:
"""
Specifies operation name to use for a current stream
"""
@property
@abstractmethod
def service_name(self) -> str:
"""
Specifies bing ads service name for a current stream
"""
@property
def parent_key_to_foreign_key_map(self) -> MutableMapping[str, str]:
"""
Specifies dict with field in record as kay and slice key as value to be inserted in record in transform method.
"""
return {}
def transform(self, record: MutableMapping[str, Any], stream_slice: Mapping[str, Any], **kwargs) -> MutableMapping[str, Any]:
foreign_keys = {key: stream_slice.get(value) for key, value in self.parent_key_to_foreign_key_map.items()}
return record | foreign_keys
@property
def _service(self) -> Union[ServiceClient, ReportingServiceManager]:
return self.client.get_service(service_name=self.service_name)
@property
def _user_id(self) -> int:
return self._get_user_id()
# TODO remove once Microsoft support confirm their SSL certificates are always valid...
def _get_user_id(self, number_of_retries=10):
""""""
try:
return self._service.GetUser().User.Id
except URLError as error:
if isinstance(error.reason, ssl.SSLError):
self.logger.warning("SSL certificate error, retrying...")
if number_of_retries > 0:
time.sleep(1)
return self._get_user_id(number_of_retries - 1)
else:
raise error
def next_page_token(self, response: sudsobject.Object, **kwargs: Mapping[str, Any]) -> Optional[Mapping[str, Any]]:
"""
Default method for streams that don't support pagination
"""
return None
def send_request(self, params: Mapping[str, Any], customer_id: str, account_id: str = None) -> Mapping[str, Any]:
request_kwargs = {
"service_name": self.service_name,
"customer_id": customer_id,
"account_id": account_id,
"operation_name": self.operation_name,
"params": params,
}
request = self.client.request(**request_kwargs)
return request
def read_records(
self,
sync_mode: SyncMode,
stream_slice: Mapping[str, Any] = None,
stream_state: Mapping[str, Any] = None,
**kwargs: Mapping[str, Any],
) -> Iterable[Mapping[str, Any]]:
stream_state = stream_state or {}
next_page_token = None
account_id = str(stream_slice.get("account_id")) if stream_slice else None
customer_id = str(stream_slice.get("customer_id")) if stream_slice else None
while True:
params = self.request_params(
stream_state=stream_state,
stream_slice=stream_slice,
next_page_token=next_page_token,
account_id=account_id,
)
response = self.send_request(params, customer_id=customer_id, account_id=account_id)
for record in self.parse_response(response):
yield self.transform(record, stream_slice)
next_page_token = self.next_page_token(response, current_page_token=next_page_token)
if not next_page_token:
break
def parse_response(self, response: sudsobject.Object, **kwargs) -> Iterable[Mapping]:
if response is not None and hasattr(response, self.data_field):
yield from self.client.asdict(response)[self.data_field]
class BingAdsCampaignManagementStream(BingAdsStream, ABC):
service_name: str = "CampaignManagement"
@property
@abstractmethod
def data_field(self) -> str:
"""
Specifies root object name in a stream response
"""
@property
@abstractmethod
def additional_fields(self) -> Optional[str]:
"""
Specifies which additional fields to fetch for a current stream.
Expected format: field names separated by space
"""
def parse_response(self, response: sudsobject.Object, **kwargs) -> Iterable[Mapping]:
if response is not None and hasattr(response, self.data_field):
yield from self.client.asdict(response)[self.data_field]
class Accounts(BingAdsStream):
"""
Searches for accounts that the current authenticated user can access.
API doc: https://docs.microsoft.com/en-us/advertising/customer-management-service/searchaccounts?view=bingads-13
Account schema: https://docs.microsoft.com/en-us/advertising/customer-management-service/advertiseraccount?view=bingads-13
Stream caches incoming responses to be able to reuse this data in Campaigns stream
"""
primary_key = "Id"
# Stream caches incoming responses to avoid duplicated http requests
use_cache: bool = True
data_field: str = "AdvertiserAccount"
service_name: str = "CustomerManagementService"
operation_name: str = "SearchAccounts"
additional_fields: str = "TaxCertificate AccountMode"
# maximum page size
page_size_limit: int = 1000
def __init__(self, client: Client, config: Mapping[str, Any]) -> None:
super().__init__(client, config)
self._account_names = config.get("account_names", [])
self._unique_account_ids = set()
def next_page_token(self, response: sudsobject.Object, current_page_token: Optional[int]) -> Optional[Mapping[str, Any]]:
current_page_token = current_page_token or 0
if response is not None and hasattr(response, self.data_field):
return None if self.page_size_limit > len(response[self.data_field]) else current_page_token + 1
else:
return None
def stream_slices(
self,
**kwargs: Mapping[str, Any],
) -> Iterable[Optional[Mapping[str, Any]]]:
user_id_predicate = {
"Field": "UserId",
"Operator": "Equals",
"Value": self._user_id,
}
if self._account_names:
for account_config in self._account_names:
account_name_predicate = {"Field": "AccountName", "Operator": account_config["operator"], "Value": account_config["name"]}
yield {"predicates": {"Predicate": [user_id_predicate, account_name_predicate]}}
else:
yield {"predicates": {"Predicate": [user_id_predicate]}}
def request_params(
self,
next_page_token: Mapping[str, Any] = None,
stream_slice: Mapping[str, Any] = None,
**kwargs: Mapping[str, Any],
) -> MutableMapping[str, Any]:
paging = self._service.factory.create("ns5:Paging")
paging.Index = next_page_token or 0
paging.Size = self.page_size_limit
return {
"PageInfo": paging,
"Predicates": stream_slice["predicates"],
"ReturnAdditionalFields": self.additional_fields,
}
def _transform_tax_fields(self, record: Mapping[str, Any]) -> Mapping[str, Any]:
tax_certificates = record["TaxCertificate"].get("TaxCertificates", {}) if record.get("TaxCertificate") is not None else {}
if tax_certificates and not isinstance(tax_certificates, list):
tax_certificate_pairs = tax_certificates.get("KeyValuePairOfstringbase64Binary")
if tax_certificate_pairs:
record["TaxCertificate"]["TaxCertificates"] = tax_certificate_pairs
return record
def parse_response(self, response: sudsobject.Object, **kwargs) -> Iterable[Mapping]:
if response is not None and hasattr(response, self.data_field):
records = self.client.asdict(response)[self.data_field]
for record in records:
if record["Id"] not in self._unique_account_ids:
self._unique_account_ids.add(record["Id"])
yield self._transform_tax_fields(record)
class Campaigns(BingAdsCampaignManagementStream):
"""
Gets the campaigns for all provided accounts.
API doc: https://docs.microsoft.com/en-us/advertising/campaign-management-service/getcampaignsbyaccountid?view=bingads-13
Campaign schema: https://docs.microsoft.com/en-us/advertising/campaign-management-service/campaign?view=bingads-13
Stream caches incoming responses to be able to reuse this data in AdGroups stream
"""
primary_key = "Id"
# Stream caches incoming responses to avoid duplicated http requests
use_cache: bool = True
data_field: str = "Campaign"
operation_name: str = "GetCampaignsByAccountId"
additional_fields: Iterable[str] = [
"AdScheduleUseSearcherTimeZone",
"BidStrategyId",
"CpvCpmBiddingScheme",
"DynamicDescriptionSetting",
"DynamicFeedSetting",
"MaxConversionValueBiddingScheme",
"MultimediaAdsBidAdjustment",
"TargetImpressionShareBiddingScheme",
"TargetSetting",
"VerifiedTrackingSetting",
]
campaign_types: Iterable[str] = ["Audience", "DynamicSearchAds", "Search", "Shopping", "PerformanceMax"]
parent_key_to_foreign_key_map = {
"AccountId": "account_id",
"CustomerId": "customer_id",
}
def request_params(
self,
stream_slice: Mapping[str, Any] = None,
**kwargs: Mapping[str, Any],
) -> MutableMapping[str, Any]:
return {
"AccountId": stream_slice["account_id"],
"CampaignType": " ".join(self.campaign_types),
"ReturnAdditionalFields": " ".join(self.additional_fields),
}
def stream_slices(
self,
**kwargs: Mapping[str, Any],
) -> Iterable[Optional[Mapping[str, Any]]]:
accounts = Accounts(self.client, self.config)
for _slice in accounts.stream_slices():
for account in accounts.read_records(SyncMode.full_refresh, _slice):
yield {"account_id": account["Id"], "customer_id": account["ParentCustomerId"]}

279
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/client.py
View File

@@ -1,279 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import logging
import os
import socket
import ssl
import sys
import uuid
from datetime import datetime, timedelta, timezone
from functools import lru_cache
from typing import Any, Iterator, List, Mapping, Optional, Union
from urllib.error import URLError
import backoff
import pendulum
from bingads.authorization import AuthorizationData, OAuthTokens, OAuthWebAuthCodeGrant
from bingads.exceptions import OAuthTokenRequestException
from bingads.service_client import ServiceClient
from bingads.util import errorcode_of_exception
from bingads.v13.bulk import BulkServiceManager, DownloadParameters
from bingads.v13.reporting.exceptions import ReportingDownloadException
from bingads.v13.reporting.reporting_service_manager import ReportingServiceManager
from suds import WebFault, sudsobject
from airbyte_cdk.models import FailureType
from airbyte_cdk.utils import AirbyteTracedException
FILE_TYPE = "Csv"
TIMEOUT_IN_MILLISECONDS = 3_600_000
class Client:
api_version: int = 13
refresh_token_safe_delta: int = 10 # in seconds
logger: logging.Logger = logging.getLogger("airbyte")
# retry on: rate limit errors, auth token expiration, internal errors
# https://docs.microsoft.com/en-us/advertising/guides/services-protocol?view=bingads-13#throttling
# https://docs.microsoft.com/en-us/advertising/guides/operation-error-codes?view=bingads-13
retry_on_codes: Iterator[str] = ["117", "207", "4204", "109", "0"]
max_retries: int = 5
# A backoff factor to apply between attempts after the second try
# {retry_factor} * (2 ** ({number of total retries} - 1))
retry_factor: int = 15
# environments supported by Microsoft Advertising: sandbox, production
environment: str = "production"
# The time interval in milliseconds between two status polling attempts.
report_poll_interval: int = 15000
# Timeout of downloading report
_download_timeout = 300000
_max_download_timeout = 600000
reports_start_date = None
def __init__(
self,
tenant_id: str,
reports_start_date: str = None,
developer_token: str = None,
client_id: str = None,
client_secret: str = None,
refresh_token: str = None,
**kwargs: Mapping[str, Any],
) -> None:
self.refresh_token = refresh_token
self.developer_token = developer_token
self.client_id = client_id
self.client_secret = client_secret
self.authentication = self._get_auth_client(client_id, tenant_id, client_secret)
self.oauth: OAuthTokens = self._get_access_token()
if reports_start_date:
self.reports_start_date = pendulum.parse(reports_start_date).astimezone(tz=timezone.utc)
def _get_auth_client(self, client_id: str, tenant_id: str, client_secret: str = None) -> OAuthWebAuthCodeGrant:
# https://github.com/BingAds/BingAds-Python-SDK/blob/e7b5a618e87a43d0a5e2c79d9aa4626e208797bd/bingads/authorization.py#L390
auth_creds = {
"client_id": client_id,
"redirection_uri": "", # should be empty string
"client_secret": None,
"tenant": tenant_id,
}
# the `client_secret` should be provided for `non-public clients` only
# https://docs.microsoft.com/en-us/advertising/guides/authentication-oauth-get-tokens?view=bingads-13#request-accesstoken
if client_secret and client_secret != "":
auth_creds["client_secret"] = client_secret
return OAuthWebAuthCodeGrant(**auth_creds)
@lru_cache(maxsize=4)
def _get_auth_data(self, customer_id: str = None, account_id: Optional[str] = None) -> AuthorizationData:
return AuthorizationData(
account_id=account_id,
customer_id=customer_id,
developer_token=self.developer_token,
authentication=self.authentication,
)
def _get_access_token(self) -> OAuthTokens:
self.logger.info("Fetching access token ...")
# clear caches to be able to use new access token
self.get_service.cache_clear()
self._get_auth_data.cache_clear()
try:
tokens = self.authentication.request_oauth_tokens_by_refresh_token(self.refresh_token)
except OAuthTokenRequestException as e:
raise AirbyteTracedException(
message=str(e),
internal_message="Failed to get OAuth access token by refresh token. "
"The user could not be authenticated as the grant is expired. The user must sign in again.",
failure_type=FailureType.config_error,
)
return tokens
def is_token_expiring(self) -> bool:
"""
Performs check if access token expiring in less than refresh_token_safe_delta seconds
"""
token_total_lifetime: timedelta = datetime.utcnow() - self.oauth.access_token_received_datetime
token_updated_expires_in: int = self.oauth.access_token_expires_in_seconds - token_total_lifetime.seconds
return False if token_updated_expires_in > self.refresh_token_safe_delta else True
def should_give_up(self, error: Union[WebFault, URLError, ReportingDownloadException]) -> bool:
if isinstance(error, URLError):
if (
isinstance(error.reason, socket.timeout)
or isinstance(error.reason, ssl.SSLError)
or isinstance(error.reason, socket.gaierror) # temporary failure in name resolution
):
return False
if isinstance(error, ReportingDownloadException):
self.logger.info("Reporting file download tracking status timeout.")
if self._download_timeout < self._max_download_timeout:
self._download_timeout = self._download_timeout + 10000
self.logger.info(f"Increasing time of timeout to {self._download_timeout}")
return False
error_code = str(errorcode_of_exception(error))
give_up = error_code not in self.retry_on_codes
if give_up:
self.logger.error(f"Giving up for returned error code: {error_code}. Error details: {self._get_error_message(error)}")
return give_up
def _get_error_message(self, error: WebFault) -> str:
return str(self.asdict(error.fault)) if hasattr(error, "fault") else str(error)
def log_retry_attempt(self, details: Mapping[str, Any]) -> None:
_, exc, _ = sys.exc_info()
self.logger.info(
f"Caught retryable error: {self._get_error_message(exc)} after {details['tries']} tries. Waiting {details['wait']} seconds then retrying..."
)
def request(self, **kwargs: Mapping[str, Any]) -> Mapping[str, Any]:
return backoff.on_exception(
backoff.expo,
(WebFault, URLError, ReportingDownloadException),
max_tries=self.max_retries,
factor=self.retry_factor,
jitter=None,
on_backoff=self.log_retry_attempt,
giveup=self.should_give_up,
)(self._request)(**kwargs)
def _request(
self,
service_name: Optional[str],
operation_name: str,
customer_id: Optional[str],
account_id: Optional[str],
params: Mapping[str, Any],
is_report_service: bool = False,
) -> Mapping[str, Any]:
"""
Executes appropriate Service Operation on Bing Ads API
"""
if self.is_token_expiring():
self.oauth = self._get_access_token()
if is_report_service:
service = self._get_reporting_service(customer_id=customer_id, account_id=account_id)
else:
service = self.get_service(service_name=service_name, customer_id=customer_id, account_id=account_id)
if operation_name == "download_report":
params["download_parameters"].timeout_in_milliseconds = self._download_timeout
return getattr(service, operation_name)(**params)
@lru_cache(maxsize=4)
def get_service(
self,
service_name: str,
customer_id: str = None,
account_id: Optional[str] = None,
) -> ServiceClient:
return ServiceClient(
service=service_name,
version=self.api_version,
authorization_data=self._get_auth_data(customer_id, account_id),
environment=self.environment,
)
@lru_cache(maxsize=4)
def _get_reporting_service(
self,
customer_id: Optional[str] = None,
account_id: Optional[str] = None,
) -> ServiceClient:
return ReportingServiceManager(
authorization_data=self._get_auth_data(customer_id, account_id),
poll_interval_in_milliseconds=self.report_poll_interval,
environment=self.environment,
)
@classmethod
def asdict(cls, suds_object: sudsobject.Object) -> Mapping[str, Any]:
"""
Converts nested Suds Object into serializable format.
Input sample:
{
obj[] =
{
value = 1
},
{
value = "str"
},
}
Output sample: =>
{'obj': [{'value': 1}, {'value': 'str'}]}
"""
result: Mapping[str, Any] = {}
for field, val in sudsobject.asdict(suds_object).items():
if hasattr(val, "__keylist__"):
result[field] = cls.asdict(val)
elif isinstance(val, list):
result[field] = []
for item in val:
if hasattr(item, "__keylist__"):
result[field].append(cls.asdict(item))
else:
result[field].append(item)
elif isinstance(val, datetime):
result[field] = val.isoformat()
else:
result[field] = val
return result
def _bulk_service_manager(self, customer_id: Optional[str] = None, account_id: Optional[str] = None):
return BulkServiceManager(
authorization_data=self._get_auth_data(customer_id, account_id),
poll_interval_in_milliseconds=5000,
environment=self.environment,
)
def get_bulk_entity(
self,
download_entities: List[str],
data_scope: List[str],
customer_id: Optional[str] = None,
account_id: Optional[str] = None,
start_date: Optional[str] = None,
) -> str:
"""
Return path with zipped csv archive
"""
download_parameters = DownloadParameters(
# campaign_ids=None,
data_scope=data_scope,
download_entities=download_entities,
file_type=FILE_TYPE,
last_sync_time_in_utc=start_date,
result_file_directory=os.getcwd(),
result_file_name=str(uuid.uuid4()),
overwrite_result_file=True, # Set this value true if you want to overwrite the same file.
timeout_in_milliseconds=TIMEOUT_IN_MILLISECONDS, # You may optionally cancel the download after a specified time interval.
)
bulk_service_manager = self._bulk_service_manager(customer_id=customer_id, account_id=account_id)
return bulk_service_manager.download_file(download_parameters)

50
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/report_streams.py
View File

@@ -1,50 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import re
import xml.etree.ElementTree as ET
from abc import ABC
from typing import Any, List, Mapping, Tuple, Union
from urllib.parse import urlparse
from bingads.v13.internal.reporting.row_report import _RowReport
from suds import WebFault
from airbyte_cdk.sources.utils.transform import TransformConfig, TypeTransformer
from source_bing_ads.reports import BingAdsReportingServicePerformanceStream, BingAdsReportingServiceStream, HourlyReportTransformerMixin
from source_bing_ads.utils import transform_date_format_to_rfc_3339, transform_report_hourly_datetime_format_to_rfc_3339
class AccountPerformanceReport(BingAdsReportingServicePerformanceStream, ABC):
report_name: str = "AccountPerformanceReport"
report_schema_name = "account_performance_report"
primary_key = [
"AccountId",
"TimePeriod",
"CurrencyCode",
"AdDistribution",
"DeviceType",
"Network",
"DeliveredMatchType",
"DeviceOS",
"TopVsOther",
"BidMatchType",
]
class AccountPerformanceReportHourly(HourlyReportTransformerMixin, AccountPerformanceReport):
report_aggregation = "Hourly"
report_schema_name = "account_performance_report_hourly"
class AccountPerformanceReportDaily(AccountPerformanceReport):
report_aggregation = "Daily"
class AccountPerformanceReportWeekly(AccountPerformanceReport):
report_aggregation = "Weekly"
class AccountPerformanceReportMonthly(AccountPerformanceReport):
report_aggregation = "Monthly"

19
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/reports/__init__.py
View File

@@ -1,19 +0,0 @@
from .bing_ads_reporting_service_stream import BingAdsReportingServiceStream
from .bing_ads_reporting_service_performance_stream import BingAdsReportingServicePerformanceStream
from .ad_performance_report import (
AdPerformanceReportWeekly,
AdPerformanceReportMonthly,
AdPerformanceReportDaily,
AdPerformanceReportHourly,
)
from .hourly_report_transformer_mixin import HourlyReportTransformerMixin
__all__ = [
"BingAdsReportingServiceStream",
"BingAdsReportingServicePerformanceStream",
"AdPerformanceReportWeekly",
"AdPerformanceReportMonthly",
"AdPerformanceReportDaily",
"AdPerformanceReportHourly",
"HourlyReportTransformerMixin",
]

47
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/reports/ad_performance_report.py
View File

@@ -1,47 +0,0 @@
#
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
#
from abc import ABC
from .bing_ads_reporting_service_performance_stream import BingAdsReportingServicePerformanceStream
from .hourly_report_transformer_mixin import HourlyReportTransformerMixin
class AdPerformanceReport(BingAdsReportingServicePerformanceStream, ABC):
report_name: str = "AdPerformanceReport"
report_schema_name = "ad_performance_report"
primary_key = [
"AccountId",
"CampaignId",
"AdGroupId",
"AdId",
"TimePeriod",
"CurrencyCode",
"AdDistribution",
"DeviceType",
"Language",
"Network",
"DeviceOS",
"TopVsOther",
"BidMatchType",
"DeliveredMatchType",
]
class AdPerformanceReportHourly(HourlyReportTransformerMixin, AdPerformanceReport):
report_aggregation = "Hourly"
report_schema_name = "ad_performance_report_hourly"
class AdPerformanceReportDaily(AdPerformanceReport):
report_aggregation = "Daily"
class AdPerformanceReportWeekly(AdPerformanceReport):
report_aggregation = "Weekly"
class AdPerformanceReportMonthly(AdPerformanceReport):
report_aggregation = "Monthly"

20
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/reports/bing_ads_reporting_service_performance_stream.py
View File

@@ -1,20 +0,0 @@
#
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
#
from abc import ABC
from typing import Any, Mapping
from .bing_ads_reporting_service_stream import BingAdsReportingServiceStream
class BingAdsReportingServicePerformanceStream(BingAdsReportingServiceStream, ABC):
def get_start_date(self, stream_state: Mapping[str, Any] = None, account_id: str = None):
start_date = super().get_start_date(stream_state, account_id)
if self.config.get("lookback_window") and start_date:
# Datetime subtract won't work with days = 0
# it'll output an AirbyteError
return start_date.subtract(days=self.config["lookback_window"])
else:
return start_date

230
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/reports/bing_ads_reporting_service_stream.py
View File

@@ -1,230 +0,0 @@
#
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
#
import _csv
from abc import ABC, abstractmethod
from datetime import datetime
from typing import Any, Iterable, List, Mapping, MutableMapping, Optional, Set, Tuple, Union
import pendulum
from bingads import ServiceClient
from bingads.v13.internal.reporting.row_report import _RowReport
from bingads.v13.internal.reporting.row_report_iterator import _RowReportRecord
from bingads.v13.reporting import ReportingDownloadParameters
from cached_property import cached_property
from suds import sudsobject
from airbyte_cdk.models import SyncMode
from airbyte_cdk.sources.streams.core import package_name_from_class
from airbyte_cdk.sources.utils.schema_helpers import ResourceSchemaLoader
from airbyte_cdk.sources.utils.transform import TransformConfig, TypeTransformer
from source_bing_ads.base_streams import Accounts, BingAdsStream
class BingAdsReportingServiceStream(BingAdsStream, ABC):
# The directory where the file with report will be downloaded.
file_directory: str = "/tmp"
# timeout for reporting download operations in milliseconds
timeout: int = 300000
report_file_format: str = "Csv"
transformer: TypeTransformer = TypeTransformer(TransformConfig.DefaultSchemaNormalization)
primary_key: List[str] = ["TimePeriod", "Network", "DeviceType"]
cursor_field = "TimePeriod"
service_name: str = "ReportingService"
operation_name: str = "download_report"
def get_json_schema(self) -> Mapping[str, Any]:
return ResourceSchemaLoader(package_name_from_class(self.__class__)).get_schema(self.report_schema_name)
@property
@abstractmethod
def report_name(self) -> str:
"""
Specifies bing ads report naming
"""
@property
@abstractmethod
def report_aggregation(self) -> Optional[str]:
"""
Specifies bing ads report aggregation type
Supported types: Hourly, Daily, Weekly, Monthly
"""
@property
@abstractmethod
def report_schema_name(self) -> str:
"""
Specifies file name with schema
"""
@property
def default_time_periods(self):
# used when reports start date is not provided
return ["LastYear", "ThisYear"] if self.report_aggregation not in ("DayOfWeek", "HourOfDay") else ["ThisYear"]
@property
def report_columns(self) -> Iterable[str]:
return list(self.get_json_schema().get("properties", {}).keys())
def parse_response(self, response: sudsobject.Object, **kwargs: Mapping[str, Any]) -> Iterable[Mapping]:
if response is not None:
try:
for row in response.report_records:
yield {column: self.get_column_value(row, column) for column in self.report_columns}
except _csv.Error as e:
self.logger.warning(f"CSV report file for stream `{self.name}` is broken or cannot be read correctly: {e}, skipping ...")
def get_column_value(self, row: _RowReportRecord, column: str) -> Union[str, None, int, float]:
"""
Reads field value from row and transforms:
1. empty values to logical None
2. Percent values to numeric string e.g. "12.25%" -> "12.25"
"""
value = row.value(column)
if not value or value == "--":
return None
if "%" in value:
value = value.replace("%", "")
if value and column in self._get_schema_numeric_properties:
value = value.replace(",", "")
return value
@cached_property
def _get_schema_numeric_properties(self) -> Set[str]:
return set(k for k, v in self.get_json_schema()["properties"].items() if set(v.get("type")) & {"integer", "number"})
def get_request_date(self, reporting_service: ServiceClient, date: datetime) -> sudsobject.Object:
"""
Creates XML Date object based on datetime.
https://docs.microsoft.com/en-us/advertising/reporting-service/date?view=bingads-13
The [suds.client.Factory-class.html factory] namespace provides a factory that may be used
to create instances of objects and types defined in the WSDL.
"""
request_date = reporting_service.factory.create("Date")
request_date.Day = date.day
request_date.Month = date.month
request_date.Year = date.year
return request_date
def request_params(
self, stream_state: Mapping[str, Any] = None, account_id: str = None, **kwargs: Mapping[str, Any]
) -> Mapping[str, Any]:
stream_slice = kwargs["stream_slice"]
start_date = self.get_start_date(stream_state, account_id)
reporting_service = self.client.get_service("ReportingService")
request_time_zone = reporting_service.factory.create("ReportTimeZone")
report_time = reporting_service.factory.create("ReportTime")
report_time.ReportTimeZone = request_time_zone.GreenwichMeanTimeDublinEdinburghLisbonLondon
if start_date:
report_time.CustomDateRangeStart = self.get_request_date(reporting_service, start_date)
report_time.CustomDateRangeEnd = self.get_request_date(reporting_service, datetime.utcnow())
report_time.PredefinedTime = None
else:
report_time.CustomDateRangeStart = None
report_time.CustomDateRangeEnd = None
report_time.PredefinedTime = stream_slice["time_period"]
report_request = self.get_report_request(account_id, False, False, False, self.report_file_format, False, report_time)
return {
"report_request": report_request,
"result_file_directory": self.file_directory,
"result_file_name": self.report_name,
"overwrite_result_file": True,
"timeout_in_milliseconds": self.timeout,
}
def get_start_date(self, stream_state: Mapping[str, Any] = None, account_id: str = None):
if stream_state and account_id:
# we've observed that account_id is being passed as an integer upstream, so we convert it to string
parsed_account_id = str(account_id)
if stream_state.get(parsed_account_id, {}).get(self.cursor_field):
return pendulum.parse(stream_state[parsed_account_id][self.cursor_field])
return self.client.reports_start_date
def get_updated_state(
self,
current_stream_state: MutableMapping[str, Any],
latest_record: Mapping[str, Any],
) -> Mapping[str, Any]:
account_id = str(latest_record["AccountId"])
current_stream_state[account_id] = current_stream_state.get(account_id, {})
current_stream_state[account_id][self.cursor_field] = max(
self.get_report_record_timestamp(latest_record[self.cursor_field]),
current_stream_state.get(account_id, {}).get(self.cursor_field, ""),
)
return current_stream_state
def send_request(self, params: Mapping[str, Any], customer_id: str, account_id: str) -> _RowReport:
request_kwargs = {
"service_name": None,
"customer_id": customer_id,
"account_id": account_id,
"operation_name": self.operation_name,
"is_report_service": True,
"params": {"download_parameters": ReportingDownloadParameters(**params)},
}
return self.client.request(**request_kwargs)
def get_report_request(
self,
account_id: str,
exclude_column_headers: bool,
exclude_report_footer: bool,
exclude_report_header: bool,
report_file_format: str,
return_only_complete_data: bool,
time: sudsobject.Object,
) -> sudsobject.Object:
reporting_service = self.client.get_service(self.service_name)
report_request = reporting_service.factory.create(f"{self.report_name}Request")
if self.report_aggregation:
report_request.Aggregation = self.report_aggregation
report_request.ExcludeColumnHeaders = exclude_column_headers
report_request.ExcludeReportFooter = exclude_report_footer
report_request.ExcludeReportHeader = exclude_report_header
report_request.Format = report_file_format
report_request.FormatVersion = "2.0"
report_request.ReturnOnlyCompleteData = return_only_complete_data
report_request.Time = time
report_request.ReportName = self.report_name
# Defines the set of accounts and campaigns to include in the report.
scope = reporting_service.factory.create("AccountThroughCampaignReportScope")
scope.AccountIds = {"long": [account_id]}
scope.Campaigns = None
report_request.Scope = scope
columns = reporting_service.factory.create(f"ArrayOf{self.report_name}Column")
getattr(columns, f"{self.report_name}Column").append(self.report_columns)
report_request.Columns = columns
return report_request
def get_report_record_timestamp(self, datestring: str) -> str:
"""
Parse report date field based on aggregation type
"""
return (
self.transformer._custom_normalizer(datestring, self.get_json_schema()["properties"][self.cursor_field])
if self.transformer._custom_normalizer
else datestring
)
def stream_slices(
self, *, sync_mode: SyncMode, cursor_field: Optional[List[str]] = None, stream_state: Optional[Mapping[str, Any]] = None
) -> Iterable[Optional[Mapping[str, Any]]]:
accounts = Accounts(self.client, self.config)
for _slice in accounts.stream_slices():
for account in accounts.read_records(SyncMode.full_refresh, _slice):
if self.get_start_date(stream_state, account["Id"]): # if start date is not provided default time periods will be used
yield {"account_id": account["Id"], "customer_id": account["ParentCustomerId"]}
else:
for period in self.default_time_periods:
yield {"account_id": account["Id"], "customer_id": account["ParentCustomerId"], "time_period": period}

18
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/reports/hourly_report_transformer_mixin.py
View File

@@ -1,18 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from airbyte_cdk.sources.utils.transform import TransformConfig, TypeTransformer
from source_bing_ads.utils import transform_report_hourly_datetime_format_to_rfc_3339
class HourlyReportTransformerMixin:
transformer: TypeTransformer = TypeTransformer(TransformConfig.DefaultSchemaNormalization | TransformConfig.CustomSchemaNormalization)
@staticmethod
@transformer.registerCustomTransform
def custom_transform_datetime_rfc3339(original_value, field_schema):
if original_value and "format" in field_schema and field_schema["format"] == "date-time":
transformed_value = transform_report_hourly_datetime_format_to_rfc_3339(original_value)
return transformed_value
return original_value

55
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/run.py
View File

@@ -1,55 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import sys
import traceback
from datetime import datetime
from typing import List
from orjson import orjson
from airbyte_cdk.entrypoint import AirbyteEntrypoint, launch, logger
from airbyte_cdk.exception_handler import init_uncaught_exception_handler
from airbyte_cdk.models import AirbyteErrorTraceMessage, AirbyteMessage, AirbyteMessageSerializer, AirbyteTraceMessage, TraceType, Type
from source_bing_ads import SourceBingAds
def _get_source(args: List[str]):
catalog_path = AirbyteEntrypoint.extract_catalog(args)
config_path = AirbyteEntrypoint.extract_config(args)
state_path = AirbyteEntrypoint.extract_state(args)
try:
return SourceBingAds(
SourceBingAds.read_catalog(catalog_path) if catalog_path else None,
SourceBingAds.read_config(config_path) if config_path else None,
SourceBingAds.read_state(state_path) if state_path else None,
)
except Exception as error:
print(
orjson.dumps(
AirbyteMessageSerializer.dump(
AirbyteMessage(
type=Type.TRACE,
trace=AirbyteTraceMessage(
type=TraceType.ERROR,
emitted_at=int(datetime.now().timestamp() * 1000),
error=AirbyteErrorTraceMessage(
message=f"Error starting the sync. This could be due to an invalid configuration or catalog. Please contact Support for assistance. Error: {error}",
stack_trace=traceback.format_exc(),
),
),
)
)
).decode()
)
return None
def run() -> None:
init_uncaught_exception_handler(logger)
_args = sys.argv[1:]
source = _get_source(_args)
if source:
launch(source, _args)

159
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/schemas/account_performance_report.json
View File

@@ -1,159 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"AccountId": {
"description": "Unique identifier for the Bing Ads account",
"type": ["null", "integer"]
},
"TimePeriod": {
"description": "Time period for the report",
"type": ["null", "string"],
"format": "date"
},
"CurrencyCode": {
"description": "Currency code used for reporting",
"type": ["null", "string"]
},
"AdDistribution": {
"description": "Type of ad distribution (search, content, both)",
"type": ["null", "string"]
},
"DeviceType": {
"description": "Type of device used",
"type": ["null", "string"]
},
"Network": {
"description": "Type of network (search, audience)",
"type": ["null", "string"]
},
"DeliveredMatchType": {
"description": "Type of match in ad delivery",
"type": ["null", "string"]
},
"DeviceOS": {
"description": "Operating system of the device",
"type": ["null", "string"]
},
"TopVsOther": {
"description": "Performance comparison between top and other ad positions",
"type": ["null", "string"]
},
"BidMatchType": {
"description": "Type of bidding match (exact, phrase, broad)",
"type": ["null", "string"]
},
"AccountName": {
"description": "Name of the Bing Ads account",
"type": ["null", "string"]
},
"AccountNumber": {
"description": "Numeric account number",
"type": ["null", "string"]
},
"PhoneImpressions": {
"description": "Number of ad impressions on phone devices",
"type": ["null", "integer"]
},
"PhoneCalls": {
"description": "Number of phone calls generated",
"type": ["null", "integer"]
},
"Clicks": {
"description": "Total number of clicks",
"type": ["null", "integer"]
},
"Ctr": {
"description": "Click-through rate",
"type": ["null", "number"]
},
"Spend": {
"description": "Total spend on ad campaigns",
"type": ["null", "number"]
},
"Impressions": {
"description": "Total number of ad impressions",
"type": ["null", "integer"]
},
"CostPerConversion": {
"description": "Cost per conversion",
"type": ["null", "number"]
},
"Ptr": {
"description": "Phone-through rate",
"type": ["null", "number"]
},
"Assists": {
"description": "Number of assist conversions",
"type": ["null", "integer"]
},
"ReturnOnAdSpend": {
"description": "Return on ad spend",
"type": ["null", "number"]
},
"CostPerAssist": {
"description": "Cost per assist conversion",
"type": ["null", "number"]
},
"AverageCpc": {
"description": "Average cost per click",
"type": ["null", "number"]
},
"AveragePosition": {
"description": "Average ad position",
"type": ["null", "number"]
},
"AverageCpm": {
"description": "Average cost per thousand impressions",
"type": ["null", "number"]
},
"Conversions": {
"description": "Total number of conversions",
"type": ["null", "number"]
},
"ConversionsQualified": {
"description": "Number of qualified conversions",
"type": ["null", "number"]
},
"ConversionRate": {
"description": "Percentage of conversions from clicks",
"type": ["null", "number"]
},
"LowQualityClicks": {
"description": "Number of low-quality clicks",
"type": ["null", "integer"]
},
"LowQualityClicksPercent": {
"description": "Percentage of low-quality clicks",
"type": ["null", "number"]
},
"LowQualityImpressions": {
"description": "Number of low-quality impressions",
"type": ["null", "integer"]
},
"LowQualitySophisticatedClicks": {
"description": "Number of sophisticated low-quality clicks",
"type": ["null", "integer"]
},
"LowQualityConversions": {
"description": "Total number of low-quality conversions",
"type": ["null", "integer"]
},
"LowQualityConversionRate": {
"description": "Conversion rate for low-quality clicks",
"type": ["null", "number"]
},
"Revenue": {
"description": "Total revenue generated",
"type": ["null", "number"]
},
"RevenuePerConversion": {
"description": "Revenue per conversion",
"type": ["null", "number"]
},
"RevenuePerAssist": {
"description": "Revenue per assist conversion",
"type": ["null", "number"]
}
}
}

160
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/schemas/account_performance_report_hourly.json
View File

@@ -1,160 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"AccountId": {
"description": "The unique identifier for the Bing Ads account",
"type": ["null", "integer"]
},
"TimePeriod": {
"description": "The time period for the report data",
"type": ["null", "string"],
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
},
"CurrencyCode": {
"description": "The currency code used for monetary values",
"type": ["null", "string"]
},
"AdDistribution": {
"description": "The distribution network for the ad (search partners, audience network, etc.)",
"type": ["null", "string"]
},
"DeviceType": {
"description": "The type of device on which the ad was displayed",
"type": ["null", "string"]
},
"Network": {
"description": "The network on which the ad appeared (e.g., Bing, AOL)",
"type": ["null", "string"]
},
"DeliveredMatchType": {
"description": "The match type for the delivered ad",
"type": ["null", "string"]
},
"DeviceOS": {
"description": "The operating system of the device on which the ad was displayed",
"type": ["null", "string"]
},
"TopVsOther": {
"description": "Indicates whether the ad appeared at the top or other positions",
"type": ["null", "string"]
},
"BidMatchType": {
"description": "The match type for which the bid was set",
"type": ["null", "string"]
},
"AccountName": {
"description": "The name of the Bing Ads account",
"type": ["null", "string"]
},
"AccountNumber": {
"description": "The account number associated with the Bing Ads account",
"type": ["null", "string"]
},
"PhoneImpressions": {
"description": "The number of impressions that included a phone number",
"type": ["null", "integer"]
},
"PhoneCalls": {
"description": "The number of phone calls generated by the ad",
"type": ["null", "integer"]
},
"Clicks": {
"description": "The total number of clicks on the ad",
"type": ["null", "integer"]
},
"Ctr": {
"description": "The click-through rate for the ad",
"type": ["null", "number"]
},
"Spend": {
"description": "The total spend on the ad campaign",
"type": ["null", "number"]
},
"Impressions": {
"description": "The total number of impressions generated by the ad",
"type": ["null", "integer"]
},
"CostPerConversion": {
"description": "The cost per conversion generated by the ad",
"type": ["null", "number"]
},
"Ptr": {
"description": "The phone-through rate for the ad",
"type": ["null", "number"]
},
"Assists": {
"description": "The number of assists generated by the ad",
"type": ["null", "integer"]
},
"ReturnOnAdSpend": {
"description": "The return on ad spend (ROAS) for the ad campaign",
"type": ["null", "number"]
},
"CostPerAssist": {
"description": "The cost per assist generated by the ad",
"type": ["null", "number"]
},
"AverageCpc": {
"description": "The average cost per click for the ad",
"type": ["null", "number"]
},
"AveragePosition": {
"description": "The average position where the ad appeared on the search results page",
"type": ["null", "number"]
},
"AverageCpm": {
"description": "The average cost per thousand impressions for the ad",
"type": ["null", "number"]
},
"Conversions": {
"description": "The total number of conversions generated by the ad",
"type": ["null", "number"]
},
"ConversionsQualified": {
"description": "The number of conversions that met certain criteria",
"type": ["null", "number"]
},
"ConversionRate": {
"description": "The rate at which clicks on the ad led to conversions",
"type": ["null", "number"]
},
"LowQualityClicks": {
"description": "The number of low-quality clicks on the ad",
"type": ["null", "integer"]
},
"LowQualityClicksPercent": {
"description": "The percentage of low-quality clicks out of total clicks",
"type": ["null", "number"]
},
"LowQualityImpressions": {
"description": "The number of low-quality impressions generated by the ad",
"type": ["null", "integer"]
},
"LowQualitySophisticatedClicks": {
"description": "The number of sophisticated clicks recognized as low-quality",
"type": ["null", "integer"]
},
"LowQualityConversions": {
"description": "The number of conversions from low-quality clicks",
"type": ["null", "integer"]
},
"LowQualityConversionRate": {
"description": "The conversion rate for low-quality clicks",
"type": ["null", "number"]
},
"Revenue": {
"description": "The total revenue generated by the ad",
"type": ["null", "number"]
},
"RevenuePerConversion": {
"description": "The revenue per conversion generated by the ad",
"type": ["null", "number"]
},
"RevenuePerAssist": {
"description": "The revenue per assist generated by the ad",
"type": ["null", "number"]
}
}
}

197
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/schemas/accounts.json
View File

@@ -1,197 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"Id": {
"description": "ID of the account",
"type": ["null", "integer"]
},
"AccountFinancialStatus": {
"description": "The financial status of the account",
"type": ["null", "string"]
},
"AccountLifeCycleStatus": {
"description": "The life cycle status of the account",
"type": ["null", "string"]
},
"AutoTagType": {
"description": "The type of auto-tagging",
"type": ["null", "string"]
},
"AccountMode": {
"description": "The mode of the account",
"type": ["null", "string"]
},
"ForwardCompatibilityMap": {
"description": "Map for forward compatibility",
"type": ["null", "string"]
},
"PaymentMethodType": {
"description": "Type of the payment method",
"type": ["null", "string"]
},
"Language": {
"description": "The language used in the account",
"type": ["null", "string"]
},
"LinkedAgencies": {
"type": ["null", "object"],
"properties": {
"Id": {
"description": "ID of the linked agency",
"type": ["null", "integer"]
},
"Name": {
"description": "Name of the linked agency",
"type": ["null", "string"]
}
}
},
"TaxInformation": {
"description": "Tax information of the account",
"type": ["null", "string"]
},
"CurrencyCode": {
"description": "The currency code used by the account",
"type": ["null", "string"]
},
"TimeZone": {
"description": "The time zone of the account",
"type": ["null", "string"]
},
"BusinessAddress": {
"description": "The business address associated with the account.",
"type": ["null", "object"],
"properties": {
"City": {
"description": "The city of the business address",
"type": ["null", "string"]
},
"CountryCode": {
"description": "The country code of the business address",
"type": ["null", "string"]
},
"Id": {
"description": "ID of the business address",
"type": ["null", "integer"]
},
"Line1": {
"description": "Address line 1",
"type": ["null", "string"]
},
"Line2": {
"description": "Address line 2",
"type": ["null", "string"]
},
"Line3": {
"description": "Address line 3",
"type": ["null", "string"]
},
"Line4": {
"description": "Address line 4",
"type": ["null", "string"]
},
"PostalCode": {
"description": "The postal code of the business address",
"type": ["null", "string"]
},
"StateOrProvince": {
"description": "The state or province of the business address",
"type": ["null", "string"]
},
"TimeStamp": {
"description": "Timestamp of the business address",
"type": ["null", "string"]
},
"BusinessName": {
"description": "The business name",
"type": ["null", "string"]
}
}
},
"BackUpPaymentInstrumentId": {
"description": "ID of the backup payment instrument",
"type": ["null", "integer"]
},
"BillingThresholdAmount": {
"description": "The threshold amount for billing",
"type": ["null", "number"]
},
"BillToCustomerId": {
"description": "Customer ID for billing",
"type": ["null", "integer"]
},
"LastModifiedByUserId": {
"description": "ID of the user who last modified the account",
"type": ["null", "integer"]
},
"LastModifiedTime": {
"description": "The date and time of the last modification",
"type": ["null", "string"],
"format": "date-time",
"airbyte_type": "timestamp_without_timezone"
},
"Name": {
"description": "The name of the account",
"type": ["null", "string"]
},
"Number": {
"description": "The account number",
"type": ["null", "string"]
},
"ParentCustomerId": {
"description": "ID of the parent customer",
"type": ["null", "integer"]
},
"PauseReason": {
"description": "Reason for pausing the account",
"type": ["null", "integer"]
},
"PaymentMethodId": {
"description": "ID of the payment method",
"type": ["null", "integer"]
},
"PrimaryUserId": {
"description": "ID of the primary user",
"type": ["null", "integer"]
},
"SalesHouseCustomerId": {
"description": "Customer ID for sales house",
"type": ["null", "integer"]
},
"SoldToPaymentInstrumentId": {
"description": "ID of the payment instrument for sales",
"type": ["null", "integer"]
},
"TimeStamp": {
"description": "Timestamp of the account",
"type": ["null", "string"]
},
"TaxCertificate": {
"type": ["null", "object"],
"properties": {
"TaxCertificateBlobContainerName": {
"type": ["null", "string"]
},
"Status": {
"type": ["null", "string"],
"enum": ["Invalid", "Pending", "Valid"]
},
"TaxCertificates": {
"type": ["null", "array"],
"items": {
"type": ["null", "object"],
"properties": {
"key": {
"type": ["null", "string"]
},
"value": {
"type": ["null", "string"]
}
}
}
}
}
}
}
}

207
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/schemas/ad_performance_report.json
View File

@@ -1,207 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"AccountId": {
"description": "The unique ID of the account to which the ad belongs",
"type": ["null", "integer"]
},
"CampaignId": {
"description": "The unique ID of the campaign to which the ad belongs",
"type": ["null", "integer"]
},
"AdGroupId": {
"description": "The ID of the ad group to which the ad belongs",
"type": ["null", "integer"]
},
"AdId": {
"description": "The unique ID of the ad",
"type": ["null", "integer"]
},
"TimePeriod": {
"description": "The time period for the report data",
"type": ["null", "string"],
"format": "date"
},
"AbsoluteTopImpressionRatePercent": {
"description": "The percentage of times your ad is shown in the absolute top location",
"type": ["null", "number"]
},
"TopImpressionRatePercent": {
"description": "The percentage of times your ad is shown either at the top or absolute top location",
"type": ["null", "number"]
},
"CurrencyCode": {
"description": "The currency code used for monetary values",
"type": ["null", "string"]
},
"AdDistribution": {
"description": "The distribution network where the ad was shown",
"type": ["null", "string"]
},
"DeviceType": {
"description": "The type of device where the ad was displayed (Desktop, Mobile, Tablet)",
"type": ["null", "string"]
},
"Language": {
"description": "The language targeting of the ad",
"type": ["null", "string"]
},
"Network": {
"description": "The network where the ad was displayed (Bing, Syndicated search partners)",
"type": ["null", "string"]
},
"DeviceOS": {
"description": "The operating system of the device where the ad was displayed",
"type": ["null", "string"]
},
"TopVsOther": {
"description": "The comparison between showing at the top or other positions",
"type": ["null", "string"]
},
"BidMatchType": {
"description": "The match type of the keyword that triggered the ad",
"type": ["null", "string"]
},
"DeliveredMatchType": {
"description": "The match type of the keyword that was matched to deliver the ad",
"type": ["null", "string"]
},
"AccountName": {
"description": "The name of the account to which the ad belongs",
"type": ["null", "string"]
},
"CampaignName": {
"description": "The name of the campaign to which the ad belongs",
"type": ["null", "string"]
},
"CampaignType": {
"description": "The type of campaign (Search, Display, etc.)",
"type": ["null", "string"]
},
"AdGroupName": {
"description": "The name of the ad group to which the ad belongs",
"type": ["null", "string"]
},
"Impressions": {
"description": "The total number of times the ad was shown",
"type": ["null", "integer"]
},
"Clicks": {
"description": "The total number of clicks on the ad",
"type": ["null", "integer"]
},
"Ctr": {
"description": "The click-through rate",
"type": ["null", "number"]
},
"Spend": {
"description": "The total cost spent on the ad",
"type": ["null", "number"]
},
"CostPerConversion": {
"description": "The cost per conversion",
"type": ["null", "number"]
},
"DestinationUrl": {
"description": "The URL where the user is directed when clicking the ad",
"type": ["null", "string"]
},
"Assists": {
"description": "The number of assist conversions generated",
"type": ["null", "integer"]
},
"ReturnOnAdSpend": {
"description": "The return on ad spend",
"type": ["null", "number"]
},
"CostPerAssist": {
"description": "The cost per assist conversion",
"type": ["null", "number"]
},
"CustomParameters": {
"description": "Custom parameters passed in the ad URL",
"type": ["null", "string"]
},
"FinalAppUrl": {
"description": "The final URL for specific apps in the ad",
"type": ["null", "string"]
},
"AdDescription": {
"description": "The description text of the ad",
"type": ["null", "string"]
},
"AdDescription2": {
"description": "The second description line of the ad",
"type": ["null", "string"]
},
"ViewThroughConversions": {
"description": "The total number of view-through conversions",
"type": ["null", "integer"]
},
"ViewThroughConversionsQualified": {
"description": "The total number of qualified view-through conversions",
"type": ["null", "number"]
},
"AllCostPerConversion": {
"description": "The cost per conversion for all conversion actions",
"type": ["null", "number"]
},
"AllReturnOnAdSpend": {
"description": "The return on ad spend for all conversion actions",
"type": ["null", "number"]
},
"Conversions": {
"description": "The total number of conversions",
"type": ["null", "number"]
},
"ConversionRate": {
"description": "The conversion rate",
"type": ["null", "number"]
},
"ConversionsQualified": {
"description": "The total number of qualified conversions",
"type": ["null", "number"]
},
"AverageCpc": {
"description": "The average cost per click",
"type": ["null", "number"]
},
"AveragePosition": {
"description": "The average position in which the ad appeared",
"type": ["null", "number"]
},
"AverageCpm": {
"description": "The average cost per thousand impressions",
"type": ["null", "number"]
},
"AllConversions": {
"description": "The total number of all conversion actions",
"type": ["null", "integer"]
},
"AllConversionRate": {
"description": "The conversion rate for all conversion actions",
"type": ["null", "number"]
},
"AllRevenue": {
"description": "The total revenue generated from all conversion actions",
"type": ["null", "number"]
},
"AllRevenuePerConversion": {
"description": "The average revenue per conversion for all conversion actions",
"type": ["null", "number"]
},
"Revenue": {
"description": "The total revenue generated by the ad",
"type": ["null", "number"]
},
"RevenuePerConversion": {
"description": "The revenue per conversion",
"type": ["null", "number"]
},
"RevenuePerAssist": {
"description": "The revenue per assist conversion",
"type": ["null", "number"]
}
}
}

200
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/schemas/ad_performance_report_hourly.json
View File

@@ -1,200 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"AccountId": {
"description": "The unique identifier for the account to which the ad belongs",
"type": ["null", "integer"]
},
"CampaignId": {
"description": "The unique identifier for the campaign to which the ad belongs",
"type": ["null", "integer"]
},
"AdGroupId": {
"description": "The unique identifier for the ad group to which the ad belongs",
"type": ["null", "integer"]
},
"AdId": {
"description": "The unique identifier for the ad",
"type": ["null", "integer"]
},
"TimePeriod": {
"description": "The time period to which the data corresponds",
"type": ["null", "string"],
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
},
"CurrencyCode": {
"description": "The currency code used for monetary values",
"type": ["null", "string"]
},
"AdDistribution": {
"description": "The distribution channel for the ad (e.g., Search, Audience Network)",
"type": ["null", "string"]
},
"DeviceType": {
"description": "The type of device on which the ad was displayed (e.g., Desktop, Mobile)",
"type": ["null", "string"]
},
"Language": {
"description": "The language targeting of the ad",
"type": ["null", "string"]
},
"Network": {
"description": "The network where the ad was displayed (e.g., Bing, AOL)",
"type": ["null", "string"]
},
"DeviceOS": {
"description": "The operating system of the device on which the ad was displayed",
"type": ["null", "string"]
},
"TopVsOther": {
"description": "The performance comparison of top positions vs. other positions",
"type": ["null", "string"]
},
"BidMatchType": {
"description": "The type of keyword match (e.g., Broad, Phrase, Exact) for the bid",
"type": ["null", "string"]
},
"DeliveredMatchType": {
"description": "The type of keyword match for which the ad has been delivered",
"type": ["null", "string"]
},
"AccountName": {
"description": "The name of the account to which the ad belongs",
"type": ["null", "string"]
},
"CampaignName": {
"description": "The name of the campaign to which the ad belongs",
"type": ["null", "string"]
},
"CampaignType": {
"description": "The type of the campaign (e.g., Search, Audience Network)",
"type": ["null", "string"]
},
"AdGroupName": {
"description": "The name of the ad group to which the ad belongs",
"type": ["null", "string"]
},
"Impressions": {
"description": "The total number of times the ad was shown",
"type": ["null", "integer"]
},
"Clicks": {
"description": "The total number of clicks on the ad",
"type": ["null", "integer"]
},
"Ctr": {
"description": "The click-through rate",
"type": ["null", "number"]
},
"Spend": {
"description": "The total amount spent on the ad campaign",
"type": ["null", "number"]
},
"CostPerConversion": {
"description": "The cost per specific conversion",
"type": ["null", "number"]
},
"DestinationUrl": {
"description": "The destination URL of the ad",
"type": ["null", "string"]
},
"Assists": {
"description": "The number of assists (when an ad indirectly results in a conversion)",
"type": ["null", "integer"]
},
"ReturnOnAdSpend": {
"description": "The return on ad spend for specific conversions",
"type": ["null", "number"]
},
"CostPerAssist": {
"description": "The cost per assist (indirect conversion)",
"type": ["null", "number"]
},
"CustomParameters": {
"description": "Any custom parameters set for the ad",
"type": ["null", "string"]
},
"FinalAppUrl": {
"description": "The final URL shown in the ad for app installations",
"type": ["null", "string"]
},
"AdDescription": {
"description": "The text of the first description line in the ad",
"type": ["null", "string"]
},
"AdDescription2": {
"description": "The text of the second description line in the ad",
"type": ["null", "string"]
},
"ViewThroughConversions": {
"description": "The total number of view-through conversions",
"type": ["null", "integer"]
},
"ViewThroughConversionsQualified": {
"description": "The number of qualified view-through conversions",
"type": ["null", "number"]
},
"AllCostPerConversion": {
"description": "The cost per conversion for all conversions",
"type": ["null", "number"]
},
"AllReturnOnAdSpend": {
"description": "The return on ad spend for all conversions",
"type": ["null", "number"]
},
"Conversions": {
"description": "The total number of specific conversions",
"type": ["null", "number"]
},
"ConversionRate": {
"description": "The conversion rate for specific conversions",
"type": ["null", "number"]
},
"ConversionsQualified": {
"description": "The number of qualified conversions",
"type": ["null", "number"]
},
"AverageCpc": {
"description": "The average cost per click",
"type": ["null", "number"]
},
"AveragePosition": {
"description": "The average position of the ad on the search results page",
"type": ["null", "number"]
},
"AverageCpm": {
"description": "The average cost per 1,000 impressions",
"type": ["null", "number"]
},
"AllConversions": {
"description": "The total number of all conversions",
"type": ["null", "integer"]
},
"AllConversionRate": {
"description": "The conversion rate for all conversions",
"type": ["null", "number"]
},
"AllRevenue": {
"description": "The total revenue from all conversions",
"type": ["null", "number"]
},
"AllRevenuePerConversion": {
"description": "The revenue per conversion for all conversions",
"type": ["null", "number"]
},
"Revenue": {
"description": "The total revenue generated by the ad",
"type": ["null", "number"]
},
"RevenuePerConversion": {
"description": "The revenue per specific conversion",
"type": ["null", "number"]
},
"RevenuePerAssist": {
"description": "The revenue per assist (indirect conversion)",
"type": ["null", "number"]
}
}
}

189
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/schemas/campaigns.json
View File

@@ -1,189 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"AccountId": {
"description": "The unique identifier of the account associated with the campaign.",
"type": ["null", "integer"]
},
"CustomerId": {
"description": "The unique identifier of the customer associated with the campaign.",
"type": ["null", "integer"]
},
"AudienceAdsBidAdjustment": {
"description": "Bid adjustment value for audience targeting ads.",
"type": ["null", "number"]
},
"BiddingScheme": {
"description": "Details of the bidding scheme for the campaign",
"type": ["null", "object"],
"properties": {
"Type": {
"description": "The type of bidding strategy used for the campaign.",
"type": ["null", "string"]
},
"MaxCpc": {
"description": "Details of the maximum cost-per-click bid",
"type": ["null", "object"],
"properties": {
"Amount": {
"description": "The maximum cost-per-click bid for the campaign.",
"type": ["null", "number"]
}
}
}
}
},
"BudgetType": {
"description": "The type of budget (e.g., daily, monthly) for the campaign.",
"type": ["null", "string"]
},
"MultimediaAdsBidAdjustment": {
"description": "Bid adjustment value for multimedia ads.",
"type": ["null", "number"]
},
"DailyBudget": {
"description": "The daily budget amount set for the campaign.",
"type": ["null", "number"]
},
"ExperimentId": {
"description": "The identifier of the experiment linked to the campaign.",
"type": ["null", "number"]
},
"FinalUrlSuffix": {
"description": "The final URL suffix appended to campaign URLs.",
"type": ["null", "string"]
},
"ForwardCompatibilityMap": {
"description": "Forward compatibility map for potential future enhancements",
"type": ["null", "array"],
"items": {
"type": ["null", "object"],
"properties": {
"key": {
"description": "The key identifying a forward compatibility setting.",
"type": ["null", "string"]
},
"value": {
"description": "The value associated with the forward compatibility setting.",
"type": ["null", "string"]
}
}
}
},
"Id": {
"description": "The unique identifier of the campaign.",
"type": ["null", "number"]
},
"Name": {
"description": "The name of the campaign.",
"type": ["null", "string"]
},
"Status": {
"description": "The status of the campaign (e.g., Active, Paused).",
"type": ["null", "string"]
},
"SubType": {
"description": "The subtype of the campaign, providing additional context.",
"type": ["null", "string"]
},
"TimeZone": {
"description": "The time zone setting for the campaign.",
"type": ["null", "string"]
},
"TrackingUrlTemplate": {
"description": "The tracking URL template used for the campaign.",
"type": ["null", "string"]
},
"UrlCustomParameters": {
"description": "Custom parameters for campaign URLs",
"type": ["null", "object"],
"properties": {
"Parameters": {
"description": "Specific URL parameters",
"type": ["null", "array"],
"items": {
"type": ["null", "object"],
"properties": {
"Key": {
"description": "The key parameter for URL customization.",
"type": ["null", "string"]
},
"Value": {
"description": "The value parameter for URL customization.",
"type": ["null", "string"]
}
}
}
}
}
},
"CampaignType": {
"description": "The type of campaign (e.g., Search, Display, Video) being run.",
"type": ["null", "string"]
},
"Settings": {
"description": "Settings related to the campaign",
"type": ["null", "object"],
"properties": {
"Setting": {
"description": "Specific setting details",
"type": ["null", "array"],
"items": {
"type": ["null", "object"],
"properties": {
"Type": {
"description": "The type of setting applied to the campaign.",
"type": ["null", "string"]
},
"Details": {
"description": "Specific details of the setting",
"type": ["null", "object"],
"properties": {
"TargetSettingDetail": {
"description": "Specific target setting details",
"type": ["null", "array"],
"items": {
"type": ["null", "object"],
"properties": {
"CriterionTypeGroup": {
"description": "The group type for targeting.",
"type": ["null", "string"]
},
"TargetAndBid": {
"description": "Indicates whether targeting is set to 'Bid only' or 'Target and bid'.",
"type": ["null", "boolean"]
}
}
}
}
}
}
}
}
}
}
},
"BudgetId": {
"description": "The identifier of the budget associated with the campaign.",
"type": ["null", "number"]
},
"Languages": {
"description": "Languages targeted in the campaign",
"type": ["null", "object"],
"properties": {
"string": {
"description": "The languages targeted by the campaign.",
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
}
}
},
"AdScheduleUseSearcherTimeZone": {
"description": "Indicates whether ad schedules should be based on the searcher's time zone.",
"type": ["null", "boolean"]
}
}
}

231
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/schemas/product_search_query_performance_report.json
View File

@@ -1,231 +0,0 @@
{
"$schema": "https://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"TimePeriod": {
"description": "Time period for the data",
"type": ["null", "string"],
"format": "date"
},
"AccountId": {
"description": "Unique identifier for the account",
"type": ["null", "integer"]
},
"AccountNumber": {
"description": "Account number associated with the account",
"type": ["null", "string"]
},
"AccountName": {
"description": "Name of the account",
"type": ["null", "string"]
},
"AdId": {
"description": "ID of the ad",
"type": ["null", "integer"]
},
"AdGroupId": {
"description": "ID of the ad group",
"type": ["null", "integer"]
},
"AdGroupName": {
"description": "Name of the ad group",
"type": ["null", "string"]
},
"CampaignId": {
"description": "ID of the campaign",
"type": ["null", "integer"]
},
"CampaignName": {
"description": "Name of the campaign",
"type": ["null", "string"]
},
"DestinationUrl": {
"description": "URL of the landing page",
"type": ["null", "string"]
},
"DeviceType": {
"description": "Type of device",
"type": ["null", "string"]
},
"DeviceOS": {
"description": "Operating system of the device",
"type": ["null", "string"]
},
"Language": {
"description": "Language targeting of the ad",
"type": ["null", "string"]
},
"SearchQuery": {
"description": "Search query that triggered the ad",
"type": ["null", "string"]
},
"Network": {
"description": "Network where the ad was displayed",
"type": ["null", "string"]
},
"MerchantProductId": {
"description": "ID of the merchant product",
"type": ["null", "string"]
},
"Title": {
"description": "Title of the ad",
"type": ["null", "string"]
},
"ClickTypeId": {
"description": "ID of the click type",
"type": ["null", "string"]
},
"TotalClicksOnAdElements": {
"description": "Total clicks on ad elements (e.g., sitelinks)",
"type": ["null", "number"]
},
"ClickType": {
"description": "Type of click (e.g., headline, sitelink)",
"type": ["null", "string"]
},
"AdGroupCriterionId": {
"description": "ID of the ad group criterion",
"type": ["null", "string"]
},
"ProductGroup": {
"description": "Grouping of products",
"type": ["null", "string"]
},
"PartitionType": {
"description": "Type of the partition",
"type": ["null", "string"]
},
"Impressions": {
"description": "Total number of impressions",
"type": ["null", "integer"]
},
"Clicks": {
"description": "Total number of clicks",
"type": ["null", "integer"]
},
"Ctr": {
"description": "Click-through rate",
"type": ["null", "number"]
},
"AverageCpc": {
"description": "Average cost per click",
"type": ["null", "number"]
},
"Spend": {
"description": "Total spend on the ad",
"type": ["null", "number"]
},
"Conversions": {
"description": "Total number of conversions",
"type": ["null", "integer"]
},
"ConversionRate": {
"description": "Percentage of conversions over clicks",
"type": ["null", "number"]
},
"Assists": {
"description": "Total number of assists",
"type": ["null", "integer"]
},
"CostPerAssist": {
"description": "Average cost per assist",
"type": ["null", "number"]
},
"Revenue": {
"description": "Total revenue generated",
"type": ["null", "number"]
},
"CostPerConversion": {
"description": "Average cost per conversion",
"type": ["null", "number"]
},
"RevenuePerConversion": {
"description": "Average revenue generated per conversion",
"type": ["null", "number"]
},
"RevenuePerAssist": {
"description": "Average revenue generated per assist",
"type": ["null", "number"]
},
"CustomerId": {
"description": "Unique identifier for the customer",
"type": ["null", "integer"]
},
"CustomerName": {
"description": "Name of the customer",
"type": ["null", "string"]
},
"AssistedImpressions": {
"description": "Number of impressions in which the ad was assisted",
"type": ["null", "integer"]
},
"AssistedClicks": {
"description": "Number of assisted clicks",
"type": ["null", "integer"]
},
"AssistedConversions": {
"description": "Number of assisted conversions",
"type": ["null", "integer"]
},
"AllConversions": {
"description": "Total number of conversions",
"type": ["null", "integer"]
},
"AllRevenue": {
"description": "Total revenue generated",
"type": ["null", "number"]
},
"AllConversionRate": {
"description": "Percentage of conversions over all clicks",
"type": ["null", "number"]
},
"AllCostPerConversion": {
"description": "Average cost per conversion",
"type": ["null", "number"]
},
"AllRevenuePerConversion": {
"description": "Average revenue generated per conversion",
"type": ["null", "number"]
},
"Goal": {
"description": "Specific goal targeted by the ad",
"type": ["null", "string"]
},
"GoalType": {
"description": "Type of goal",
"type": ["null", "string"]
},
"AbsoluteTopImpressionRatePercent": {
"description": "The percentage of times the ad showed at the absolute top of the search results",
"type": ["null", "number"]
},
"AverageCpm": {
"description": "Average cost per thousand impressions",
"type": ["null", "number"]
},
"ConversionsQualified": {
"description": "Number of conversions that meet specific criteria",
"type": ["null", "number"]
},
"AssistedConversionsQualified": {
"description": "Number of assisted conversions that meet specific criteria",
"type": ["null", "number"]
},
"AllConversionsQualified": {
"description": "Number of conversions that meet specific criteria",
"type": ["null", "number"]
},
"CampaignType": {
"description": "Type of the campaign",
"type": ["null", "string"]
},
"AssetGroupId": {
"description": "ID of the asset group",
"type": ["null", "integer"]
},
"AssetGroupName": {
"description": "Name of the asset group",
"type": ["null", "string"]
}
}
}

232
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/schemas/product_search_query_performance_report_hourly.json
View File

@@ -1,232 +0,0 @@
{
"$schema": "https://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"TimePeriod": {
"description": "The time period of the report",
"type": ["null", "string"],
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
},
"AccountId": {
"description": "ID of the account associated with the data",
"type": ["null", "integer"]
},
"AccountNumber": {
"description": "Number of the account associated with the data",
"type": ["null", "string"]
},
"AccountName": {
"description": "Name of the account associated with the data",
"type": ["null", "string"]
},
"AdId": {
"description": "ID of the ad",
"type": ["null", "integer"]
},
"AdGroupId": {
"description": "ID of the ad group",
"type": ["null", "integer"]
},
"AdGroupName": {
"description": "Name of the ad group",
"type": ["null", "string"]
},
"CampaignId": {
"description": "ID of the campaign",
"type": ["null", "integer"]
},
"CampaignName": {
"description": "Name of the campaign",
"type": ["null", "string"]
},
"DestinationUrl": {
"description": "URL where the ad directs the user",
"type": ["null", "string"]
},
"DeviceType": {
"description": "Type of device used",
"type": ["null", "string"]
},
"DeviceOS": {
"description": "Operating system of the device",
"type": ["null", "string"]
},
"Language": {
"description": "Language targeting for the ad",
"type": ["null", "string"]
},
"SearchQuery": {
"description": "The search query entered by the user",
"type": ["null", "string"]
},
"Network": {
"description": "The network where the ad was displayed",
"type": ["null", "string"]
},
"MerchantProductId": {
"description": "ID of the merchant product",
"type": ["null", "string"]
},
"Title": {
"description": "Title of the ad",
"type": ["null", "string"]
},
"ClickTypeId": {
"description": "ID of the click type",
"type": ["null", "string"]
},
"TotalClicksOnAdElements": {
"description": "Total number of clicks on elements within the ad",
"type": ["null", "number"]
},
"ClickType": {
"description": "Type of click",
"type": ["null", "string"]
},
"AdGroupCriterionId": {
"description": "Unique ID for the ad group criterion",
"type": ["null", "string"]
},
"ProductGroup": {
"description": "Grouping of products",
"type": ["null", "string"]
},
"PartitionType": {
"description": "Type of partition",
"type": ["null", "string"]
},
"Impressions": {
"description": "Total number of times the ad was shown",
"type": ["null", "integer"]
},
"Clicks": {
"description": "Total number of clicks on the ad",
"type": ["null", "integer"]
},
"Ctr": {
"description": "Click-through rate of the ad",
"type": ["null", "number"]
},
"AverageCpc": {
"description": "Average cost per click for the ad",
"type": ["null", "number"]
},
"Spend": {
"description": "Total amount spent on the ad",
"type": ["null", "number"]
},
"Conversions": {
"description": "Total number of times the ad resulted in a conversion",
"type": ["null", "integer"]
},
"ConversionRate": {
"description": "The conversion rate for the ad",
"type": ["null", "number"]
},
"Assists": {
"description": "Number of times the ad assisted in converting a customer",
"type": ["null", "integer"]
},
"CostPerAssist": {
"description": "Cost per assist where the ad assisted in converting a customer",
"type": ["null", "number"]
},
"Revenue": {
"description": "Total revenue generated by the ad",
"type": ["null", "number"]
},
"CostPerConversion": {
"description": "Cost per conversion for the ad",
"type": ["null", "number"]
},
"RevenuePerConversion": {
"description": "Average revenue per conversion for the ad",
"type": ["null", "number"]
},
"RevenuePerAssist": {
"description": "Average revenue per assist where the ad assisted in converting a customer",
"type": ["null", "number"]
},
"CustomerId": {
"description": "ID of the customer",
"type": ["null", "integer"]
},
"CustomerName": {
"description": "Name of the customer",
"type": ["null", "string"]
},
"AssistedImpressions": {
"description": "Number of times the ad appeared in a measured position on the search results page but wasn't clicked",
"type": ["null", "integer"]
},
"AssistedClicks": {
"description": "Number of clicks in which the ad appeared in a measured position on the search results page but wasn't clicked",
"type": ["null", "integer"]
},
"AssistedConversions": {
"description": "Number of conversions in which the ad appeared in a measured position on the search results page but wasn't clicked",
"type": ["null", "integer"]
},
"AllConversions": {
"description": "Total number of all types of conversions",
"type": ["null", "integer"]
},
"AllRevenue": {
"description": "Total revenue generated from all types of conversions",
"type": ["null", "number"]
},
"AllConversionRate": {
"description": "The overall conversion rate for all types of conversions",
"type": ["null", "number"]
},
"AllCostPerConversion": {
"description": "Total cost per conversion for all types of conversions",
"type": ["null", "number"]
},
"AllRevenuePerConversion": {
"description": "Average revenue earned per conversion for all types of conversions",
"type": ["null", "number"]
},
"Goal": {
"description": "The goal associated with the ad",
"type": ["null", "string"]
},
"GoalType": {
"description": "Type of goal",
"type": ["null", "string"]
},
"AbsoluteTopImpressionRatePercent": {
"description": "The percentage of times this ad was shown in the top position on the search results page",
"type": ["null", "number"]
},
"AverageCpm": {
"description": "Average cost per thousand impressions",
"type": ["null", "number"]
},
"ConversionsQualified": {
"description": "Number of conversions that meet certain criteria",
"type": ["null", "number"]
},
"AssistedConversionsQualified": {
"description": "Number of conversions that meet certain criteria in which the ad appeared in a measured position on the search results page but wasn't clicked",
"type": ["null", "number"]
},
"AllConversionsQualified": {
"description": "Number of all types of conversions that meet certain criteria",
"type": ["null", "number"]
},
"CampaignType": {
"description": "Type of the campaign",
"type": ["null", "string"]
},
"AssetGroupId": {
"description": "ID of the asset group",
"type": ["null", "integer"]
},
"AssetGroupName": {
"description": "Name of the asset group",
"type": ["null", "string"]
}
}
}

239
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/schemas/search_query_performance_report.json
View File

@@ -1,239 +0,0 @@
{
"$schema": "https://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"AccountName": {
"description": "The name of the Bing Ads account.",
"type": ["null", "string"]
},
"AccountNumber": {
"description": "The account number associated with the Bing Ads account.",
"type": ["null", "string"]
},
"AccountId": {
"description": "The unique identifier of the Bing Ads account.",
"type": ["null", "integer"]
},
"TimePeriod": {
"description": "The time period of the data.",
"type": ["null", "string"],
"format": "date"
},
"CampaignName": {
"description": "The name of the campaign.",
"type": ["null", "string"]
},
"CampaignId": {
"description": "The unique identifier of the campaign.",
"type": ["null", "integer"]
},
"AdGroupName": {
"description": "The name of the ad group.",
"type": ["null", "string"]
},
"AdGroupId": {
"description": "The unique identifier of the ad group.",
"type": ["null", "integer"]
},
"AdId": {
"description": "The unique identifier of the ad.",
"type": ["null", "integer"]
},
"AdType": {
"description": "The type of ad (text ad, responsive ad, etc.).",
"type": ["null", "string"]
},
"DestinationUrl": {
"description": "The URL where the ad directs traffic.",
"type": ["null", "string"]
},
"BidMatchType": {
"description": "The type of match (bidded, auto, etc.) for the keyword bid.",
"type": ["null", "string"]
},
"DeliveredMatchType": {
"description": "The type of match (exact, broad, etc.) for the keyword delivery.",
"type": ["null", "string"]
},
"CampaignStatus": {
"description": "The status of the campaign.",
"type": ["null", "string"]
},
"AdStatus": {
"description": "The status of the ad.",
"type": ["null", "string"]
},
"Impressions": {
"description": "The total number of times the ad was shown.",
"type": ["null", "integer"]
},
"Clicks": {
"description": "The total number of clicks.",
"type": ["null", "integer"]
},
"Ctr": {
"description": "The click-through rate.",
"type": ["null", "number"]
},
"AverageCpc": {
"description": "The average cost per click.",
"type": ["null", "number"]
},
"Spend": {
"description": "The total spend on the ad.",
"type": ["null", "number"]
},
"AveragePosition": {
"description": "The average position of the ad on search results pages.",
"type": ["null", "number"]
},
"SearchQuery": {
"description": "The search query that triggered the ad.",
"type": ["null", "string"]
},
"Keyword": {
"description": "The keyword associated with the ad.",
"type": ["null", "string"]
},
"AdGroupCriterionId": {
"description": "The unique identifier of the ad group criterion.",
"type": ["null", "string"]
},
"Conversions": {
"description": "The total number of conversions.",
"type": ["null", "integer"]
},
"ConversionRate": {
"description": "The percentage of clicks that resulted in a conversion.",
"type": ["null", "number"]
},
"CostPerConversion": {
"description": "The cost per conversion.",
"type": ["null", "number"]
},
"Language": {
"description": "The language setting targeted by the ad.",
"type": ["null", "string"]
},
"KeywordId": {
"description": "The unique identifier of the keyword.",
"type": ["null", "integer"]
},
"Network": {
"description": "The network where the ad was shown (Bing, partner sites, etc.).",
"type": ["null", "string"]
},
"TopVsOther": {
"description": "Indicates if the ad was shown in the top position or elsewhere.",
"type": ["null", "string"]
},
"DeviceType": {
"description": "The type of device (desktop, mobile, tablet, etc.).",
"type": ["null", "string"]
},
"DeviceOS": {
"description": "The operating system of the device where the ad was displayed.",
"type": ["null", "string"]
},
"Assists": {
"description": "The number of assists on conversions.",
"type": ["null", "integer"]
},
"Revenue": {
"description": "The total revenue generated.",
"type": ["null", "number"]
},
"ReturnOnAdSpend": {
"description": "The return on ad spend.",
"type": ["null", "number"]
},
"CostPerAssist": {
"description": "The cost per assist on conversions.",
"type": ["null", "number"]
},
"RevenuePerConversion": {
"description": "The average revenue per conversion.",
"type": ["null", "number"]
},
"RevenuePerAssist": {
"description": "The average revenue per assist.",
"type": ["null", "number"]
},
"AccountStatus": {
"description": "The status of the Bing Ads account.",
"type": ["null", "string"]
},
"AdGroupStatus": {
"description": "The status of the ad group.",
"type": ["null", "string"]
},
"KeywordStatus": {
"description": "The status of the keyword.",
"type": ["null", "string"]
},
"CampaignType": {
"description": "The type of campaign (search, display, etc.).",
"type": ["null", "string"]
},
"CustomerId": {
"description": "The unique identifier of the customer.",
"type": ["null", "integer"]
},
"CustomerName": {
"description": "The name of the customer.",
"type": ["null", "string"]
},
"AllConversions": {
"description": "The total number of all conversions.",
"type": ["null", "integer"]
},
"AllRevenue": {
"description": "The total revenue generated from all conversions.",
"type": ["null", "number"]
},
"AllConversionRate": {
"description": "The percentage of all clicks that resulted in a conversion.",
"type": ["null", "number"]
},
"AllCostPerConversion": {
"description": "The cost per conversion for all conversions.",
"type": ["null", "number"]
},
"AllReturnOnAdSpend": {
"description": "The return on ad spend for all conversions.",
"type": ["null", "number"]
},
"AllRevenuePerConversion": {
"description": "The average revenue per conversion for all conversions.",
"type": ["null", "number"]
},
"Goal": {
"description": "The goal associated with the campaign.",
"type": ["null", "string"]
},
"GoalType": {
"description": "The type of goal (e.g., ROAS, CPA) for the campaign.",
"type": ["null", "string"]
},
"AbsoluteTopImpressionRatePercent": {
"description": "The percentage of impressions shown at the absolute top of the search results page.",
"type": ["null", "number"]
},
"TopImpressionRatePercent": {
"description": "The percentage of impressions shown at the top of the search results page.",
"type": ["null", "number"]
},
"AverageCpm": {
"description": "The average cost per thousand impressions.",
"type": ["null", "number"]
},
"ConversionsQualified": {
"description": "The total number of qualified conversions.",
"type": ["null", "number"]
},
"AllConversionsQualified": {
"description": "The total number of qualified conversions.",
"type": ["null", "number"]
}
}
}

240
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/schemas/search_query_performance_report_hourly.json
View File

@@ -1,240 +0,0 @@
{
"$schema": "https://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"AccountName": {
"description": "The name of the account.",
"type": ["null", "string"]
},
"AccountNumber": {
"description": "The number associated with the account.",
"type": ["null", "string"]
},
"AccountId": {
"description": "The unique identifier of the account.",
"type": ["null", "integer"]
},
"TimePeriod": {
"description": "The time period for the data.",
"type": ["null", "string"],
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
},
"CampaignName": {
"description": "The name of the campaign.",
"type": ["null", "string"]
},
"CampaignId": {
"description": "The unique identifier of the campaign.",
"type": ["null", "integer"]
},
"AdGroupName": {
"description": "The name of the ad group.",
"type": ["null", "string"]
},
"AdGroupId": {
"description": "The unique identifier of the ad group.",
"type": ["null", "integer"]
},
"AdId": {
"description": "The unique identifier of the ad.",
"type": ["null", "integer"]
},
"AdType": {
"description": "The type of the ad.",
"type": ["null", "string"]
},
"DestinationUrl": {
"description": "The URL where the user is directed to upon clicking.",
"type": ["null", "string"]
},
"BidMatchType": {
"description": "The type of bid match.",
"type": ["null", "string"]
},
"DeliveredMatchType": {
"description": "The type of delivered match.",
"type": ["null", "string"]
},
"CampaignStatus": {
"description": "The status of the campaign.",
"type": ["null", "string"]
},
"AdStatus": {
"description": "The status of the ad.",
"type": ["null", "string"]
},
"Impressions": {
"description": "The total number of impressions.",
"type": ["null", "integer"]
},
"Clicks": {
"description": "The total number of clicks.",
"type": ["null", "integer"]
},
"Ctr": {
"description": "The click-through rate.",
"type": ["null", "number"]
},
"AverageCpc": {
"description": "The average cost per click.",
"type": ["null", "number"]
},
"Spend": {
"description": "The total amount spent.",
"type": ["null", "number"]
},
"AveragePosition": {
"description": "The average position of the ad.",
"type": ["null", "number"]
},
"SearchQuery": {
"description": "The search query used by the user.",
"type": ["null", "string"]
},
"Keyword": {
"description": "The keyword associated with the ad.",
"type": ["null", "string"]
},
"AdGroupCriterionId": {
"description": "The unique identifier of the ad group criterion.",
"type": ["null", "string"]
},
"Conversions": {
"description": "The total number of conversions.",
"type": ["null", "integer"]
},
"ConversionRate": {
"description": "The conversion rate for conversions.",
"type": ["null", "number"]
},
"CostPerConversion": {
"description": "The cost per conversion.",
"type": ["null", "number"]
},
"Language": {
"description": "The language of the ad.",
"type": ["null", "string"]
},
"KeywordId": {
"description": "The unique identifier of the keyword.",
"type": ["null", "integer"]
},
"Network": {
"description": "The network where the ad is displayed.",
"type": ["null", "string"]
},
"TopVsOther": {
"description": "The comparison of top impression share versus other positions.",
"type": ["null", "string"]
},
"DeviceType": {
"description": "The type of device where the ad is displayed.",
"type": ["null", "string"]
},
"DeviceOS": {
"description": "The operating system of the device.",
"type": ["null", "string"]
},
"Assists": {
"description": "The number of assist conversions.",
"type": ["null", "integer"]
},
"Revenue": {
"description": "The total revenue.",
"type": ["null", "number"]
},
"ReturnOnAdSpend": {
"description": "The return on ad spend.",
"type": ["null", "number"]
},
"CostPerAssist": {
"description": "The cost per assist conversion.",
"type": ["null", "number"]
},
"RevenuePerConversion": {
"description": "The revenue per conversion.",
"type": ["null", "number"]
},
"RevenuePerAssist": {
"description": "The revenue per assist conversion.",
"type": ["null", "number"]
},
"AccountStatus": {
"description": "The status of the account.",
"type": ["null", "string"]
},
"AdGroupStatus": {
"description": "The status of the ad group.",
"type": ["null", "string"]
},
"KeywordStatus": {
"description": "The status of the keyword.",
"type": ["null", "string"]
},
"CampaignType": {
"description": "The type of the campaign.",
"type": ["null", "string"]
},
"CustomerId": {
"description": "The unique identifier of the customer.",
"type": ["null", "integer"]
},
"CustomerName": {
"description": "The name of the customer.",
"type": ["null", "string"]
},
"AllConversions": {
"description": "The total number of all conversions.",
"type": ["null", "integer"]
},
"AllRevenue": {
"description": "The total revenue from all conversions.",
"type": ["null", "number"]
},
"AllConversionRate": {
"description": "The conversion rate for all conversions.",
"type": ["null", "number"]
},
"AllCostPerConversion": {
"description": "The cost per conversion for all conversions.",
"type": ["null", "number"]
},
"AllReturnOnAdSpend": {
"description": "The return on ad spend for all conversions.",
"type": ["null", "number"]
},
"AllRevenuePerConversion": {
"description": "The revenue per conversion for all conversions.",
"type": ["null", "number"]
},
"Goal": {
"description": "The goal for the conversion.",
"type": ["null", "string"]
},
"GoalType": {
"description": "The type of goal for the conversion.",
"type": ["null", "string"]
},
"AbsoluteTopImpressionRatePercent": {
"description": "The percentage of absolute top impression share for the ad.",
"type": ["null", "number"]
},
"TopImpressionRatePercent": {
"description": "The percentage of top impression share for the ad.",
"type": ["null", "number"]
},
"AverageCpm": {
"description": "The average cost per thousand impressions.",
"type": ["null", "number"]
},
"ConversionsQualified": {
"description": "The total number of qualified conversions.",
"type": ["null", "number"]
},
"AllConversionsQualified": {
"description": "The total number of all qualified conversions.",
"type": ["null", "number"]
}
}
}

48
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/source.py
View File

@@ -1,48 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import logging
from typing import Any, List, Mapping, Optional, Tuple
from airbyte_cdk import TState, YamlDeclarativeSource
from airbyte_cdk.models import ConfiguredAirbyteCatalog, FailureType, SyncMode
from airbyte_cdk.sources.streams import Stream
from airbyte_cdk.utils import AirbyteTracedException
from source_bing_ads.base_streams import Accounts
from source_bing_ads.client import Client
from source_bing_ads.report_streams import ( # noqa: F401
BingAdsReportingServiceStream,
)
class SourceBingAds(YamlDeclarativeSource):
"""
Source implementation of Bing Ads API. Fetches advertising data from accounts
"""
def __init__(self, catalog: Optional[ConfiguredAirbyteCatalog], config: Optional[Mapping[str, Any]], state: TState, **kwargs):
super().__init__(catalog=catalog, config=config, state=state, **{"path_to_yaml": "manifest.yaml"})
def check_connection(self, logger: logging.Logger, config: Mapping[str, Any]) -> Tuple[bool, Any]:
try:
client = Client(**config)
accounts = Accounts(client, config)
account_ids = set()
for _slice in accounts.stream_slices():
account_ids.update({str(account["Id"]) for account in accounts.read_records(SyncMode.full_refresh, _slice)})
if account_ids:
return True, None
else:
raise AirbyteTracedException(
message="Config validation error: You don't have accounts assigned to this user. Please verify your developer token.",
internal_message="You don't have accounts assigned to this user.",
failure_type=FailureType.config_error,
)
except Exception as error:
return False, error
def _clear_reporting_object_name(self, report_object: str) -> str:
# reporting mixin adds it
if report_object.endswith("Request"):
return report_object.replace("Request", "")
return report_object

245
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/spec.json
View File

@@ -1,245 +0,0 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/sources/bing-ads",
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Bing Ads Spec",
"type": "object",
"required": ["developer_token", "client_id", "refresh_token"],
"additionalProperties": true,
"properties": {
"auth_method": {
"type": "string",
"const": "oauth2.0"
},
"tenant_id": {
"type": "string",
"title": "Tenant ID",
"description": "The Tenant ID of your Microsoft Advertising developer application. Set this to \"common\" unless you know you need a different value.",
"airbyte_secret": true,
"default": "common",
"order": 0
},
"client_id": {
"type": "string",
"title": "Client ID",
"description": "The Client ID of your Microsoft Advertising developer application.",
"airbyte_secret": true,
"order": 1
},
"client_secret": {
"type": "string",
"title": "Client Secret",
"description": "The Client Secret of your Microsoft Advertising developer application.",
"default": "",
"airbyte_secret": true,
"order": 2
},
"refresh_token": {
"type": "string",
"title": "Refresh Token",
"description": "Refresh Token to renew the expired Access Token.",
"airbyte_secret": true,
"order": 3
},
"developer_token": {
"type": "string",
"title": "Developer Token",
"description": "Developer token associated with user. See more info <a href=\"https://docs.microsoft.com/en-us/advertising/guides/get-started?view=bingads-13#get-developer-token\"> in the docs</a>.",
"airbyte_secret": true,
"order": 4
},
"account_names": {
"title": "Account Names Predicates",
"description": "Predicates that will be used to sync data by specific accounts.",
"type": "array",
"order": 5,
"items": {
"description": "Account Names Predicates Config.",
"type": "object",
"properties": {
"operator": {
"title": "Operator",
"description": "An Operator that will be used to filter accounts. The Contains predicate has features for matching words, matching inflectional forms of words, searching using wildcard characters, and searching using proximity. The Equals is used to return all rows where account name is equal(=) to the string that you provided",
"type": "string",
"enum": ["Contains", "Equals"]
},
"name": {
"title": "Account Name",
"description": "Account Name is a string value for comparing with the specified predicate.",
"type": "string"
}
},
"required": ["operator", "name"]
}
},
"reports_start_date": {
"type": "string",
"title": "Reports replication start date",
"format": "date",
"description": "The start date from which to begin replicating report data. Any data generated before this date will not be replicated in reports. This is a UTC date in YYYY-MM-DD format. If not set, data from previous and current calendar year will be replicated.",
"order": 6
},
"lookback_window": {
"title": "Lookback window",
"description": "Also known as attribution or conversion window. How far into the past to look for records (in days). If your conversion window has an hours/minutes granularity, round it up to the number of days exceeding. Used only for performance report streams in incremental mode without specified Reports Start Date.",
"type": "integer",
"default": 0,
"minimum": 0,
"maximum": 90,
"order": 7
},
"custom_reports": {
"title": "Custom Reports",
"description": "You can add your Custom Bing Ads report by creating one.",
"order": 8,
"type": "array",
"items": {
"title": "Custom Report Config",
"type": "object",
"properties": {
"name": {
"title": "Report Name",
"description": "The name of the custom report, this name would be used as stream name",
"type": "string",
"examples": [
"Account Performance",
"AdDynamicTextPerformanceReport",
"custom report"
]
},
"reporting_object": {
"title": "Reporting Data Object",
"description": "The name of the the object derives from the ReportRequest object. You can find it in Bing Ads Api docs - Reporting API - Reporting Data Objects.",
"type": "string",
"enum": [
"AccountPerformanceReportRequest",
"AdDynamicTextPerformanceReportRequest",
"AdExtensionByAdReportRequest",
"AdExtensionByKeywordReportRequest",
"AdExtensionDetailReportRequest",
"AdGroupPerformanceReportRequest",
"AdPerformanceReportRequest",
"AgeGenderAudienceReportRequest",
"AudiencePerformanceReportRequest",
"CallDetailReportRequest",
"CampaignPerformanceReportRequest",
"ConversionPerformanceReportRequest",
"DestinationUrlPerformanceReportRequest",
"DSAAutoTargetPerformanceReportRequest",
"DSACategoryPerformanceReportRequest",
"DSASearchQueryPerformanceReportRequest",
"GeographicPerformanceReportRequest",
"GoalsAndFunnelsReportRequest",
"HotelDimensionPerformanceReportRequest",
"HotelGroupPerformanceReportRequest",
"KeywordPerformanceReportRequest",
"NegativeKeywordConflictReportRequest",
"ProductDimensionPerformanceReportRequest",
"ProductMatchCountReportRequest",
"ProductNegativeKeywordConflictReportRequest",
"ProductPartitionPerformanceReportRequest",
"ProductPartitionUnitPerformanceReportRequest",
"ProductSearchQueryPerformanceReportRequest",
"ProfessionalDemographicsAudienceReportRequest",
"PublisherUsagePerformanceReportRequest",
"SearchCampaignChangeHistoryReportRequest",
"SearchQueryPerformanceReportRequest",
"ShareOfVoiceReportRequest",
"UserLocationPerformanceReportRequest"
]
},
"report_columns": {
"title": "Columns",
"description": "A list of available report object columns. You can find it in description of reporting object that you want to add to custom report.",
"type": "array",
"items": {
"description": "Name of report column.",
"type": "string"
},
"minItems": 1
},
"report_aggregation": {
"title": "Aggregation",
"description": "A list of available aggregations.",
"type": "string",
"items": {
"title": "ValidEnums",
"description": "An enumeration of aggregations.",
"enum": [
"Hourly",
"Daily",
"Weekly",
"Monthly",
"DayOfWeek",
"HourOfDay",
"WeeklyStartingMonday",
"Summary"
]
},
"default": ["Hourly"]
}
},
"required": [
"name",
"reporting_object",
"report_columns",
"report_aggregation"
]
}
}
}
},
"advanced_auth": {
"auth_flow_type": "oauth2.0",
"predicate_key": ["auth_method"],
"predicate_value": "oauth2.0",
"oauth_config_specification": {
"complete_oauth_output_specification": {
"type": "object",
"additionalProperties": false,
"properties": {
"refresh_token": {
"type": "string",
"path_in_connector_config": ["refresh_token"]
}
}
},
"complete_oauth_server_input_specification": {
"type": "object",
"additionalProperties": false,
"properties": {
"client_id": {
"type": "string"
},
"client_secret": {
"type": "string"
}
}
},
"complete_oauth_server_output_specification": {
"type": "object",
"additionalProperties": false,
"properties": {
"client_id": {
"type": "string",
"path_in_connector_config": ["client_id"]
},
"client_secret": {
"type": "string",
"path_in_connector_config": ["client_secret"]
}
}
},
"oauth_user_input_from_connector_config_specification": {
"type": "object",
"additionalProperties": false,
"properties": {
"tenant_id": {
"type": "string",
"path_in_connector_config": ["tenant_id"]
}
}
}
}
}
}

29
airbyte-integrations/connectors/source-bing-ads/source_bing_ads/utils.py
View File

@@ -1,29 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from datetime import datetime, timezone
def transform_bulk_datetime_format_to_rfc_3339(original_value: str) -> str:
"""
Bing Ads Bulk API provides datetime fields in custom format with milliseconds: "04/27/2023 18:00:14.970"
Return datetime in RFC3339 format: "2023-04-27T18:00:14.970+00:00"
"""
return datetime.strptime(original_value, "%m/%d/%Y %H:%M:%S.%f").replace(tzinfo=timezone.utc).isoformat(timespec="milliseconds")
def transform_date_format_to_rfc_3339(original_value: str) -> str:
"""
Bing Ads API provides date fields in custom format: "04/27/2023"
Return date in RFC3339 format: "2023-04-27"
"""
return datetime.strptime(original_value, "%m/%d/%Y").replace(tzinfo=timezone.utc).strftime("%Y-%m-%d")
def transform_report_hourly_datetime_format_to_rfc_3339(original_value: str) -> str:
"""
Bing Ads API reports with hourly aggregation provides date fields in custom format: "2023-11-04|11"
Return date in RFC3339 format: "2023-11-04T11:00:00+00:00"
"""
return datetime.strptime(original_value, "%Y-%m-%d|%H").replace(tzinfo=timezone.utc).isoformat(timespec="seconds")

376
airbyte-integrations/connectors/source-bing-ads/unit_tests/accountperformancereportmonthly_records.json
View File

@@ -1,376 +0,0 @@
[
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-06-01",
"DeviceType": "Computer",
"Network": "Bing and Yahoo! search",
"Impressions": 1380,
"Clicks": 22,
"Spend": 3.34,
"Ctr": 0.0159,
"AverageCpc": 0.15,
"ReturnOnAdSpend": 0.0,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-06-01",
"DeviceType": "Computer",
"Network": "Syndicated search partners",
"Impressions": 1992,
"Clicks": 25,
"Spend": 4.62,
"Ctr": 0.0126,
"AverageCpc": 0.18,
"ReturnOnAdSpend": 0.0,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-06-01",
"DeviceType": "Computer",
"Network": "AOL search",
"Impressions": 2,
"Clicks": 0,
"Spend": 0.0,
"Ctr": 0.0,
"AverageCpc": 0.0,
"ReturnOnAdSpend": null,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-06-01",
"DeviceType": "Computer",
"Network": "Audience",
"Impressions": 9,
"Clicks": 0,
"Spend": 0.0,
"Ctr": 0.0,
"AverageCpc": 0.0,
"ReturnOnAdSpend": null,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-06-01",
"DeviceType": "Smartphone",
"Network": "Bing and Yahoo! search",
"Impressions": 73,
"Clicks": 5,
"Spend": 0.98,
"Ctr": 0.06849999999999999,
"AverageCpc": 0.2,
"ReturnOnAdSpend": 0.0,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-06-01",
"DeviceType": "Smartphone",
"Network": "Syndicated search partners",
"Impressions": 5754,
"Clicks": 38,
"Spend": 7.1,
"Ctr": 0.0066,
"AverageCpc": 0.19,
"ReturnOnAdSpend": 0.0,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-06-01",
"DeviceType": "Tablet",
"Network": "Bing and Yahoo! search",
"Impressions": 1,
"Clicks": 0,
"Spend": 0.0,
"Ctr": 0.0,
"AverageCpc": 0.0,
"ReturnOnAdSpend": null,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-06-01",
"DeviceType": "Tablet",
"Network": "Syndicated search partners",
"Impressions": 82,
"Clicks": 1,
"Spend": 0.22,
"Ctr": 0.012199999999999999,
"AverageCpc": 0.22,
"ReturnOnAdSpend": 0.0,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-06-01",
"DeviceType": "Tablet",
"Network": "AOL search",
"Impressions": 0,
"Clicks": 0,
"Spend": 0.0,
"Ctr": null,
"AverageCpc": 0.0,
"ReturnOnAdSpend": null,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-07-01",
"DeviceType": "Computer",
"Network": "Bing and Yahoo! search",
"Impressions": 783,
"Clicks": 12,
"Spend": 1.23,
"Ctr": 0.015300000000000001,
"AverageCpc": 0.1,
"ReturnOnAdSpend": 0.0,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-07-01",
"DeviceType": "Computer",
"Network": "Syndicated search partners",
"Impressions": 1712,
"Clicks": 23,
"Spend": 2.72,
"Ctr": 0.0134,
"AverageCpc": 0.12,
"ReturnOnAdSpend": 0.0,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-07-01",
"DeviceType": "Computer",
"Network": "AOL search",
"Impressions": 0,
"Clicks": 0,
"Spend": 0.0,
"Ctr": null,
"AverageCpc": 0.0,
"ReturnOnAdSpend": null,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-07-01",
"DeviceType": "Computer",
"Network": "Audience",
"Impressions": 9,
"Clicks": 0,
"Spend": 0.0,
"Ctr": 0.0,
"AverageCpc": 0.0,
"ReturnOnAdSpend": null,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-07-01",
"DeviceType": "Smartphone",
"Network": "Bing and Yahoo! search",
"Impressions": 52,
"Clicks": 3,
"Spend": 0.19,
"Ctr": 0.057699999999999994,
"AverageCpc": 0.06,
"ReturnOnAdSpend": 0.0,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-07-01",
"DeviceType": "Smartphone",
"Network": "Syndicated search partners",
"Impressions": 38546,
"Clicks": 201,
"Spend": 18.1,
"Ctr": 0.0052,
"AverageCpc": 0.09,
"ReturnOnAdSpend": 0.0,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-07-01",
"DeviceType": "Tablet",
"Network": "Bing and Yahoo! search",
"Impressions": 1,
"Clicks": 0,
"Spend": 0.0,
"Ctr": 0.0,
"AverageCpc": 0.0,
"ReturnOnAdSpend": null,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-07-01",
"DeviceType": "Tablet",
"Network": "Syndicated search partners",
"Impressions": 729,
"Clicks": 7,
"Spend": 0.55,
"Ctr": 0.0096,
"AverageCpc": 0.08,
"ReturnOnAdSpend": 0.0,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-08-01",
"DeviceType": "Computer",
"Network": "Bing and Yahoo! search",
"Impressions": 22,
"Clicks": 0,
"Spend": 0.0,
"Ctr": 0.0,
"AverageCpc": 0.0,
"ReturnOnAdSpend": null,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-08-01",
"DeviceType": "Computer",
"Network": "Syndicated search partners",
"Impressions": 60,
"Clicks": 1,
"Spend": 0.11,
"Ctr": 0.0167,
"AverageCpc": 0.11,
"ReturnOnAdSpend": 0.0,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-08-01",
"DeviceType": "Smartphone",
"Network": "Bing and Yahoo! search",
"Impressions": 1,
"Clicks": 0,
"Spend": 0.0,
"Ctr": 0.0,
"AverageCpc": 0.0,
"ReturnOnAdSpend": null,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-08-01",
"DeviceType": "Smartphone",
"Network": "Syndicated search partners",
"Impressions": 1438,
"Clicks": 22,
"Spend": 1.62,
"Ctr": 0.015300000000000001,
"AverageCpc": 0.07,
"ReturnOnAdSpend": 0.0,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
},
{
"AccountName": "Daxtarity Inc.",
"AccountNumber": "F149GKV5",
"AccountId": 180278106,
"TimePeriod": "2021-08-01",
"DeviceType": "Tablet",
"Network": "Syndicated search partners",
"Impressions": 7,
"Clicks": 0,
"Spend": 0.0,
"Ctr": 0.0,
"AverageCpc": 0.0,
"ReturnOnAdSpend": null,
"RevenuePerConversion": null,
"ConversionRate": null,
"Conversions": 0.0
}
]

84
airbyte-integrations/connectors/source-bing-ads/unit_tests/accounts_records.json
View File

@@ -1,84 +0,0 @@
[
{
"BillToCustomerId": 251186883,
"CurrencyCode": "USD",
"AccountFinancialStatus": "ClearFinancialStatus",
"Id": 180519267,
"Language": "English",
"LastModifiedByUserId": 138225488,
"LastModifiedTime": "2021-07-09T13:16:43.337000",
"Name": "Airbyte",
"Number": "F149MJ18",
"ParentCustomerId": 251186883,
"PaymentMethodId": null,
"PaymentMethodType": null,
"PrimaryUserId": 138225488,
"AccountLifeCycleStatus": "Pending",
"TimeStamp": "AAAAAEpme9E=",
"TimeZone": "CentralTimeUSCanada",
"PauseReason": null,
"ForwardCompatibilityMap": null,
"LinkedAgencies": null,
"SalesHouseCustomerId": null,
"TaxInformation": null,
"BackUpPaymentInstrumentId": null,
"BillingThresholdAmount": null,
"BusinessAddress": {
"City": "San Francisco",
"CountryCode": "US",
"Id": 149649761,
"Line1": "350 29th Ave",
"Line2": null,
"Line3": null,
"Line4": null,
"PostalCode": "94121-1703",
"StateOrProvince": "CA",
"TimeStamp": null,
"BusinessName": "Airbyte"
},
"AutoTagType": "Preserve",
"SoldToPaymentInstrumentId": null,
"AccountMode": "Expert"
},
{
"BillToCustomerId": 251186883,
"CurrencyCode": "USD",
"AccountFinancialStatus": "ClearFinancialStatus",
"Id": 180278106,
"Language": "English",
"LastModifiedByUserId": 3,
"LastModifiedTime": "2021-08-23T07:06:19.147000",
"Name": "Daxtarity Inc.",
"Number": "F149GKV5",
"ParentCustomerId": 251186883,
"PaymentMethodId": 138188746,
"PaymentMethodType": "CreditCard",
"PrimaryUserId": 138225488,
"AccountLifeCycleStatus": "Active",
"TimeStamp": "AAAAAE0a41E=",
"TimeZone": "Arizona",
"PauseReason": null,
"ForwardCompatibilityMap": null,
"LinkedAgencies": null,
"SalesHouseCustomerId": null,
"TaxInformation": null,
"BackUpPaymentInstrumentId": null,
"BillingThresholdAmount": null,
"BusinessAddress": {
"City": "San Francisco",
"CountryCode": "US",
"Id": 149004358,
"Line1": "350 29th avenue",
"Line2": null,
"Line3": null,
"Line4": null,
"PostalCode": "94121",
"StateOrProvince": "CA",
"TimeStamp": null,
"BusinessName": "Daxtarity Inc."
},
"AutoTagType": "Inactive",
"SoldToPaymentInstrumentId": null,
"AccountMode": "Expert"
}
]

3
airbyte-integrations/connectors/source-bing-ads/unit_tests/app_install_ad_labels_base.csv
View File

@@ -1,3 +0,0 @@
Type,Status,Id,Parent Id,Campaign,Ad Group,Client Id,Modified Time,Name,Description,Label,Color
Format Version,,,,,,,,6.0,,,
App Install Ad Label,,-22,-11112,,,ClientIdGoesHere,,,,,
1 Type Status Id Parent Id Campaign Ad Group Client Id Modified Time Name Description Label Color
2 Format Version 6.0
3 App Install Ad Label -22 -11112 ClientIdGoesHere

3
airbyte-integrations/connectors/source-bing-ads/unit_tests/app_install_ads_base.csv
View File

@@ -1,3 +0,0 @@
Type,Status,Id,Parent Id,Campaign,Ad Group,Client Id,Modified Time,Title,Text,Display Url,Destination Url,Promotion,Device Preference,Name,App Platform,App Id,Final Url,Mobile Final Url,Tracking Template,Final Url Suffix,Custom Parameter
Format Version,,,,,,,,,,,,,,6.0,,,,,,,
App Install Ad,Active,,-1111,ParentCampaignNameGoesHere,AdGroupNameGoesHere,ClientIdGoesHere,,Contoso Quick Setup,Find New Customers & Increase Sales!,,,,All,,Android,AppStoreIdGoesHere,FinalUrlGoesHere,,,,{_promoCode}=PROMO1; {_season}=summer
1 Type Status Id Parent Id Campaign Ad Group Client Id Modified Time Title Text Display Url Destination Url Promotion Device Preference Name App Platform App Id Final Url Mobile Final Url Tracking Template Final Url Suffix Custom Parameter
2 Format Version 6.0
3 App Install Ad Active -1111 ParentCampaignNameGoesHere AdGroupNameGoesHere ClientIdGoesHere Contoso Quick Setup Find New Customers & Increase Sales! All Android AppStoreIdGoesHere FinalUrlGoesHere {_promoCode}=PROMO1; {_season}=summer

60
airbyte-integrations/connectors/source-bing-ads/unit_tests/conftest.py
View File

@@ -1,14 +1,34 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from unittest.mock import patch
import sys
import zipfile
from io import BytesIO
from pathlib import Path
from unittest.mock import MagicMock
import pytest
from source_bing_ads import SourceBingAds
from airbyte_cdk.sources.declarative.yaml_declarative_source import YamlDeclarativeSource
from airbyte_cdk.test.catalog_builder import CatalogBuilder
from airbyte_cdk.test.state_builder import StateBuilder
pytest_plugins = ["airbyte_cdk.test.utils.manifest_only_fixtures"]
def _get_manifest_path() -> Path:
source_declarative_manifest_path = Path("/airbyte/integration_code/source_declarative_manifest")
if source_declarative_manifest_path.exists():
return source_declarative_manifest_path
return Path(__file__).parent.parent
_SOURCE_FOLDER_PATH = _get_manifest_path()
_YAML_FILE_PATH = _SOURCE_FOLDER_PATH / "manifest.yaml"
sys.path.append(str(_SOURCE_FOLDER_PATH)) # to allow loading custom components
@pytest.fixture(autouse=True)
def patch_time(mocker):
mocker.patch("time.sleep")
@@ -93,7 +113,7 @@ def config_with_custom_reports_fixture():
@pytest.fixture(name="logger_mock")
def logger_mock_fixture():
return patch("source_bing_ads.source.logging.Logger")
return MagicMock()
@pytest.fixture(name="mock_auth_token")
@@ -129,10 +149,42 @@ def mock_account_query_fixture(requests_mock):
)
def get_source(config, state=None) -> YamlDeclarativeSource:
catalog = CatalogBuilder().build()
state = StateBuilder().build() if not state else state
return YamlDeclarativeSource(path_to_yaml=str(_YAML_FILE_PATH), catalog=catalog, config=config, state=state)
def find_stream(stream_name, config, state=None):
state = StateBuilder().build() if not state else state
streams = SourceBingAds(catalog=None, config=config, state=state).streams(config=config)
streams = get_source(config, state).streams(config=config)
for stream in streams:
if stream.name == stream_name:
return stream
raise ValueError(f"Stream {stream_name} not found")
def create_zip_from_csv(filename: str) -> bytes:
"""
Creates a zip file containing a CSV file from the resource/response folder.
The CSV is stored directly in the zip without additional gzip compression.
Args:
filename: The name of the CSV file without extension
Returns:
The zip file content as bytes
"""
# Build path to the CSV file in resource/response folder
csv_path = Path(__file__).parent / f"resource/response/{filename}.csv"
# Read the CSV content
with open(csv_path, "r") as csv_file:
csv_content = csv_file.read()
# Create a zip file containing the CSV file directly (without gzip compression)
zip_buffer = BytesIO()
with zipfile.ZipFile(zip_buffer, mode="w") as zip_file:
zip_file.writestr(f"{filename}.csv", csv_content)
return zip_buffer.getvalue()

20
airbyte-integrations/connectors/source-bing-ads/unit_tests/helpers.py
View File

@@ -1,20 +0,0 @@
#
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
#
from typing import Any, Dict, Optional
from source_bing_ads.source import SourceBingAds
from airbyte_cdk.models import ConfiguredAirbyteCatalog, SyncMode
from airbyte_cdk.test.catalog_builder import CatalogBuilder
from airbyte_cdk.test.entrypoint_wrapper import EntrypointOutput, discover, read
def source(
config: Dict[str, Any] = None, catalog: ConfiguredAirbyteCatalog = None, state: Optional[Dict[str, Any]] = None
) -> SourceBingAds:
if not catalog:
catalog = CatalogBuilder().with_stream("fake_stream", SyncMode.full_refresh).build()
return SourceBingAds(catalog=catalog, config=config, state=state)

28
airbyte-integrations/connectors/source-bing-ads/unit_tests/integrations/base_test.py
View File

@@ -3,22 +3,17 @@ import json
import zipfile
from io import BytesIO
from pathlib import Path
from typing import Any, Dict, Optional, Tuple, Union
from typing import Any, Dict, Optional, Union
from unittest import TestCase
from unittest.mock import MagicMock, patch
from bingads.v13.bulk import BulkServiceManager
from bingads.v13.reporting.reporting_service_manager import ReportingServiceManager
from client_builder import build_request, build_request_2, response_with_status
from config_builder import ConfigBuilder
from protocol_helpers import read_helper
from request_builder import RequestBuilder
from suds.transport.https import HttpAuthenticated
from suds_response_mock import mock_http_authenticated_send
from airbyte_cdk.models import AirbyteLogMessage, AirbyteMessage, AirbyteStateMessage, Level, SyncMode, Type
from airbyte_cdk.test.catalog_builder import CatalogBuilder
from airbyte_cdk.test.entrypoint_wrapper import EntrypointOutput, read
from airbyte_cdk.test.entrypoint_wrapper import EntrypointOutput
from airbyte_cdk.test.mock_http import HttpMocker, HttpResponse
from airbyte_cdk.test.mock_http.response_builder import find_template
from airbyte_cdk.test.state_builder import StateBuilder
@@ -60,13 +55,6 @@ class BaseTest(TestCase):
def tearDown(self) -> None:
self._http_mocker.__exit__(None, None, None)
@property
def service_manager(self) -> Union[ReportingServiceManager, BulkServiceManager]:
pass
def _download_file(self, file: Optional[str] = None) -> Path:
pass
@property
def _config(self) -> dict[str, Any]:
return ConfigBuilder().build()
@@ -118,14 +106,10 @@ class BaseTest(TestCase):
stream_data_file: str = None,
state: Optional[Dict[str, Any]] = None,
expecting_exception: bool = False,
) -> Tuple[EntrypointOutput, MagicMock]:
with patch.object(HttpAuthenticated, "send", mock_http_authenticated_send):
with patch.object(
self.service_manager, "download_file", return_value=self._download_file(stream_data_file)
) as service_call_mock:
self.mock_get_report_request_api(stream_data_file)
catalog = CatalogBuilder().with_stream(stream_name, sync_mode).build()
return read_helper(config, catalog, state, expecting_exception), service_call_mock
) -> EntrypointOutput:
self.mock_get_report_request_api(stream_data_file)
catalog = CatalogBuilder().with_stream(stream_name, sync_mode).build()
return read_helper(config, catalog, state, expecting_exception)
@property
def http_mocker(self) -> HttpMocker:

4
airbyte-integrations/connectors/source-bing-ads/unit_tests/integrations/config_builder.py
View File

@@ -5,7 +5,7 @@ from typing import Any, Dict, List
from airbyte_cdk.test.mock_http.response_builder import find_template
TENNANT_ID = "common"
TENANT_ID = "common"
DEVELOPER_TOKEN = "test-token"
REFRESH_TOKEN = "test-refresh-token"
CLIENT_ID = "test-client-id"
@@ -22,7 +22,7 @@ class ConfigBuilder:
self._client_secret: str = CLIENT_SECRET
self._refresh_token: str = REFRESH_TOKEN
self._developer_token: str = DEVELOPER_TOKEN
self._tenant_id: str = TENNANT_ID
self._tenant_id: str = TENANT_ID
self._report_start_date: str = None
self._custom_reports: List[str] = None
self._lookback_window: int = LOOKBACK_WINDOW

8
airbyte-integrations/connectors/source-bing-ads/unit_tests/integrations/protocol_helpers.py
View File

@@ -5,16 +5,12 @@
from typing import Any, Dict, Optional
from source_bing_ads.source import SourceBingAds
from unit_tests.conftest import get_source
from airbyte_cdk.models import ConfiguredAirbyteCatalog
from airbyte_cdk.test.entrypoint_wrapper import EntrypointOutput, read
def _source(catalog: ConfiguredAirbyteCatalog, config: Dict[str, Any], state: Optional[Dict[str, Any]]) -> SourceBingAds:
return SourceBingAds(catalog=catalog, config=config, state=state)
def read_helper(
config: Dict[str, Any],
catalog: ConfiguredAirbyteCatalog,
@@ -22,5 +18,5 @@ def read_helper(
expecting_exception: bool = False,
) -> EntrypointOutput:
source_state = state if state else {}
source = _source(catalog=catalog, config=config, state=source_state)
source = get_source(config, source_state)
return read(source, config, catalog, state, expecting_exception)

3
airbyte-integrations/connectors/source-bing-ads/unit_tests/integrations/request_builder.py
View File

@@ -12,6 +12,7 @@ from airbyte_cdk.test.mock_http.request import HttpRequest
CLIENT_CENTER_BASE_URL = "https://clientcenter.api.bingads.microsoft.com/CustomerManagement/v13"
REPORTING_BASE_URL = "https://reporting.api.bingads.microsoft.com/Reporting/v13"
BULK_BASE_URL = "https://bulk.api.bingads.microsoft.com"
REPORT_URL = "https://bingadsappsstorageprod.blob.core.windows.net:443/dumb/testing/potato/AdPerformanceReport.zip?skoid=dubb&sktid=testing&skt=potato&ske=potato&sks=b&skv=2019-12-12&sv=2023-11-03&st=potato&se=potato&sr=b&sp=r&sig=potato"
class RequestBuilder:
@@ -42,5 +43,5 @@ class RequestBuilder:
def build_report_url(self) -> HttpRequest:
return HttpRequest(
url="https://bingadsappsstorageprod.blob.core.windows.net:443/dumb/testing/potato/AdPerformanceReport.zip?skoid=dubb&sktid=testing&skt=potato&ske=potato&sks=b&skv=2019-12-12&sv=2023-11-03&st=potato&se=potato&sr=b&sp=r&sig=potato",
url=REPORT_URL,
)

157
airbyte-integrations/connectors/source-bing-ads/unit_tests/integrations/suds_response_mock.py
View File

@@ -1,157 +0,0 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
from suds.transport import Reply, Request
from suds.transport.https import HttpAuthenticated
SEARCH_ACCOUNTS_RESPONSE = b"""<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Header>
<h:TrackingId xmlns:h="https://bingads.microsoft.com/Customer/v13">6f0a329e-4cb4-4c79-9c08-2dfe601ba05a
</h:TrackingId>
</s:Header>
<s:Body>
<SearchAccountsResponse xmlns="https://bingads.microsoft.com/Customer/v13">
<Accounts xmlns:a="https://bingads.microsoft.com/Customer/v13/Entities"
xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<a:AdvertiserAccount>
<a:BillToCustomerId>251186883</a:BillToCustomerId>
<a:CurrencyCode>USD</a:CurrencyCode>
<a:AccountFinancialStatus>ClearFinancialStatus</a:AccountFinancialStatus>
<a:Id>180535609</a:Id>
<a:Language>English</a:Language>
<a:LastModifiedByUserId>0</a:LastModifiedByUserId>
<a:LastModifiedTime>2023-08-11T08:24:26.603</a:LastModifiedTime>
<a:Name>DEMO-ACCOUNT</a:Name>
<a:Number>F149W3B6</a:Number>
<a:ParentCustomerId>251186883</a:ParentCustomerId>
<a:PaymentMethodId i:nil="true"/>
<a:PaymentMethodType i:nil="true"/>
<a:PrimaryUserId>138225488</a:PrimaryUserId>
<a:AccountLifeCycleStatus>Pause</a:AccountLifeCycleStatus>
<a:TimeStamp>AAAAAH10c1A=</a:TimeStamp>
<a:TimeZone>Santiago</a:TimeZone>
<a:PauseReason>2</a:PauseReason>
<a:ForwardCompatibilityMap i:nil="true"
xmlns:b="http://schemas.datacontract.org/2004/07/System.Collections.Generic"/>
<a:LinkedAgencies/>
<a:SalesHouseCustomerId i:nil="true"/>
<a:TaxInformation xmlns:b="http://schemas.datacontract.org/2004/07/System.Collections.Generic"/>
<a:BackUpPaymentInstrumentId i:nil="true"/>
<a:BillingThresholdAmount i:nil="true"/>
<a:BusinessAddress>
<a:City>San Francisco</a:City>
<a:CountryCode>US</a:CountryCode>
<a:Id>149694999</a:Id>
<a:Line1>350 29th avenue</a:Line1>
<a:Line2 i:nil="true"/>
<a:Line3 i:nil="true"/>
<a:Line4 i:nil="true"/>
<a:PostalCode>94121</a:PostalCode>
<a:StateOrProvince>CA</a:StateOrProvince>
<a:TimeStamp i:nil="true"/>
<a:BusinessName>Daxtarity Inc.</a:BusinessName>
</a:BusinessAddress>
<a:AutoTagType>Inactive</a:AutoTagType>
<a:SoldToPaymentInstrumentId i:nil="true"/>
<a:TaxCertificate i:nil="false">
<a:TaxCertificateBlobContainerName i:nil="false">Test Container Name</a:TaxCertificateBlobContainerName>
<a:TaxCertificates xmlns:a="http://schemas.datacontract.org/2004/07/System.Collections.Generic" i:nil="false">
<a:KeyValuePairOfstringbase64Binary>
<a:key i:nil="false">test_key</a:key>
<a:value i:nil="false">test_value</a:value>
</a:KeyValuePairOfstringbase64Binary>
</a:TaxCertificates>
<a:Status i:nil="false">Active</a:Status>
</a:TaxCertificate>
<a:AccountMode>Expert</a:AccountMode>
</a:AdvertiserAccount>
</Accounts>
</SearchAccountsResponse>
</s:Body>
</s:Envelope>
"""
GET_USER_RESPONSE = b"""<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Header xmlns="https://bingads.microsoft.com/Customer/v13">
<TrackingId d3p1:nil="false" xmlns:d3p1="http://www.w3.org/2001/XMLSchema-instance">762354725472</TrackingId>
</s:Header>
<s:Body>
<GetUserResponse xmlns="https://bingads.microsoft.com/Customer/v13">
<User xmlns:e227="https://bingads.microsoft.com/Customer/v13/Entities" d4p1:nil="false" xmlns:d4p1="http://www.w3.org/2001/XMLSchema-instance">
<e227:ContactInfo d4p1:nil="false">
<e227:Address d4p1:nil="false">
<e227:City d4p1:nil="false">City</e227:City>
<e227:CountryCode d4p1:nil="false">USD</e227:CountryCode>
<e227:Id d4p1:nil="false">12345678</e227:Id>
<e227:Line1 d4p1:nil="false">Test Line</e227:Line1>
<e227:Line2 d4p1:nil="false">Test Line</e227:Line2>
<e227:Line3 d4p1:nil="false">Test Line</e227:Line3>
<e227:Line4 d4p1:nil="false">Test Line</e227:Line4>
<e227:PostalCode d4p1:nil="false">0671</e227:PostalCode>
<e227:StateOrProvince d4p1:nil="false">State</e227:StateOrProvince>
<e227:TimeStamp d4p1:nil="false">12327485</e227:TimeStamp>
<e227:BusinessName d4p1:nil="false">Test</e227:BusinessName>
</e227:Address>
<e227:ContactByPhone d4p1:nil="false">50005</e227:ContactByPhone>
<e227:ContactByPostalMail d4p1:nil="false">7365</e227:ContactByPostalMail>
<e227:Email d4p1:nil="false">test@mail.com</e227:Email>
<e227:EmailFormat d4p1:nil="false">test</e227:EmailFormat>
<e227:Fax d4p1:nil="false">73456-343</e227:Fax>
<e227:HomePhone d4p1:nil="false">83563</e227:HomePhone>
<e227:Id d4p1:nil="false">1232346573</e227:Id>
<e227:Mobile d4p1:nil="false">736537</e227:Mobile>
<e227:Phone1 d4p1:nil="false">2645</e227:Phone1>
<e227:Phone2 d4p1:nil="false">45353</e227:Phone2>
</e227:ContactInfo>
<e227:CustomerId d4p1:nil="false">234627</e227:CustomerId>
<e227:Id d4p1:nil="false">276342574</e227:Id>
<e227:JobTitle d4p1:nil="false">Title Job</e227:JobTitle>
<e227:LastModifiedByUserId d4p1:nil="false">234722342</e227:LastModifiedByUserId>
<e227:LastModifiedTime d4p1:nil="false">2024-01-01T01:01:10.327</e227:LastModifiedTime>
<e227:Lcid d4p1:nil="false">827462346</e227:Lcid>
<e227:Name d4p1:nil="false">
<e227:FirstName d4p1:nil="false">Name First</e227:FirstName>
<e227:LastName d4p1:nil="false">Name Last</e227:LastName>
<e227:MiddleInitial d4p1:nil="false">Test</e227:MiddleInitial>
</e227:Name>
<e227:Password d4p1:nil="false">test</e227:Password>
<e227:SecretAnswer d4p1:nil="false">test</e227:SecretAnswer>
<e227:SecretQuestion>test?</e227:SecretQuestion>
<e227:UserLifeCycleStatus d4p1:nil="false">test</e227:UserLifeCycleStatus>
<e227:TimeStamp d4p1:nil="false">2736452</e227:TimeStamp>
<e227:UserName d4p1:nil="false">test</e227:UserName>
<e227:ForwardCompatibilityMap xmlns:e228="http://schemas.datacontract.org/2004/07/System.Collections.Generic" d4p1:nil="false">
<e228:KeyValuePairOfstringstring>
<e228:key d4p1:nil="false">key</e228:key>
<e228:value d4p1:nil="false">value</e228:value>
</e228:KeyValuePairOfstringstring>
</e227:ForwardCompatibilityMap>
<e227:AuthenticationToken d4p1:nil="false">token</e227:AuthenticationToken>
</User>
<CustomerRoles xmlns:e229="https://bingads.microsoft.com/Customer/v13/Entities" d4p1:nil="false" xmlns:d4p1="http://www.w3.org/2001/XMLSchema-instance">
<e229:CustomerRole>
<e229:RoleId>8324628</e229:RoleId>
<e229:CustomerId>726542</e229:CustomerId>
<e229:AccountIds d4p1:nil="false" xmlns:a1="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
<a1:long>180535609</a1:long>
</e229:AccountIds>
<e229:LinkedAccountIds d4p1:nil="false" xmlns:a1="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
<a1:long>180535609</a1:long>
</e229:LinkedAccountIds>
<e229:CustomerLinkPermission d4p1:nil="false">http://link</e229:CustomerLinkPermission>
</e229:CustomerRole>
</CustomerRoles>
</GetUserResponse>
</s:Body>
</s:Envelope>
"""
def mock_http_authenticated_send(transport: HttpAuthenticated, request: Request) -> Reply:
if request.headers.get("SOAPAction").decode() == '"GetUser"':
return Reply(code=200, headers={}, message=GET_USER_RESPONSE)
if request.headers.get("SOAPAction").decode() == '"SearchAccounts"':
return Reply(code=200, headers={}, message=SEARCH_ACCOUNTS_RESPONSE)
raise Exception(f"Unexpected SOAPAction provided for mock SOAP client: {request.headers.get('SOAPAction').decode()}")

4
airbyte-integrations/connectors/source-bing-ads/unit_tests/integrations/test_accounts_stream.py
View File

@@ -8,8 +8,8 @@ from request_builder import RequestBuilder
from airbyte_cdk.models import SyncMode
from airbyte_cdk.test.catalog_builder import CatalogBuilder
from airbyte_cdk.test.entrypoint_wrapper import EntrypointOutput, read
from airbyte_cdk.test.mock_http import HttpMocker, HttpResponse
from airbyte_cdk.test.entrypoint_wrapper import EntrypointOutput
from airbyte_cdk.test.mock_http import HttpResponse
from airbyte_cdk.test.mock_http.response_builder import find_template

14
airbyte-integrations/connectors/source-bing-ads/unit_tests/integrations/test_app_install_ad_labels_stream.py
View File

@@ -14,19 +14,19 @@ class TestAppInstallAdLabelsStream(TestBulkStream):
def test_return_records_from_given_csv_file(self):
self.mock_apis(file=self.stream_name)
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "app_install_ad_labels")
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "app_install_ad_labels")
assert len(output.records) == 1
def test_return_logged_info_for_empty_csv_file(self):
def test_return_zero_record_from_empty_csv(self):
self.mock_apis(file="app_install_ad_labels_empty")
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "app_install_ad_labels_empty")
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "app_install_ad_labels_empty")
assert len(output.records) == 0
no_records_message = self.create_log_message(f"Read 0 records from {self.stream_name} stream")
assert no_records_message in output.logs
def test_transform_records(self):
self.mock_apis(file=self.stream_name)
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "app_install_ad_labels")
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "app_install_ad_labels")
assert output.records
for record in output.records:
assert "Account Id" in record.record.data.keys()
@@ -35,7 +35,7 @@ class TestAppInstallAdLabelsStream(TestBulkStream):
@freeze_time("2024-02-26")
def test_incremental_read_cursor_value_matches_value_from_most_recent_record(self):
self.mock_apis(file="app_install_ad_labels_with_cursor_value")
output, _ = self.read_stream(self.stream_name, SyncMode.incremental, self._config, "app_install_ad_labels_with_cursor_value")
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, "app_install_ad_labels_with_cursor_value")
assert len(output.records) == 4
assert output.most_recent_state.stream_state.states[0]["cursor"] == {self.cursor_field: "2024-01-04T12:12:12.028+0000"}
@@ -43,7 +43,5 @@ class TestAppInstallAdLabelsStream(TestBulkStream):
def test_incremental_read_with_state(self):
self.mock_apis(file="app_install_ad_labels_with_state", read_with_state=True)
state = self._state("app_install_ad_labels_state", self.stream_name)
output, service_call_mock = self.read_stream(
self.stream_name, SyncMode.incremental, self._config, "app_install_ad_labels_with_state", state
)
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, "app_install_ad_labels_with_state", state)
assert output.most_recent_state.stream_state.states[0]["cursor"] == {self.cursor_field: "2024-01-29T12:55:12.028+0000"}

14
airbyte-integrations/connectors/source-bing-ads/unit_tests/integrations/test_app_install_ads_stream.py
View File

@@ -14,19 +14,19 @@ class TestAppInstallAdsStream(TestBulkStream):
def test_return_records_from_given_csv_file(self):
self.mock_apis(file=self.stream_name)
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "app_install_ads")
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "app_install_ads")
assert len(output.records) == 1
def test_return_logged_info_for_empty_csv_file(self):
def test_return_zero_record_from_empty_csv(self):
self.mock_apis(file="app_install_ads_empty")
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "app_install_ads_empty")
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "app_install_ads_empty")
assert len(output.records) == 0
no_records_message = self.create_log_message(f"Read 0 records from {self.stream_name} stream")
assert no_records_message in output.logs
def test_transform_records(self):
self.mock_apis(file=self.stream_name)
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "app_install_ads")
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "app_install_ads")
assert output.records
for record in output.records:
assert "Account Id" in record.record.data.keys()
@@ -35,7 +35,7 @@ class TestAppInstallAdsStream(TestBulkStream):
@freeze_time("2024-02-26")
def test_incremental_read_cursor_value_matches_value_from_most_recent_record(self):
self.mock_apis(file="app_install_ads_with_cursor_value")
output, _ = self.read_stream(self.stream_name, SyncMode.incremental, self._config, "app_install_ads_with_cursor_value")
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, "app_install_ads_with_cursor_value")
assert len(output.records) == 4
assert output.most_recent_state.stream_state.states[0]["cursor"] == {self.cursor_field: "2024-03-01T12:49:12.028+0000"}
@@ -43,7 +43,5 @@ class TestAppInstallAdsStream(TestBulkStream):
def test_incremental_read_with_state(self):
self.mock_apis(file="app_install_ads_with_state", read_with_state=True)
state = self._state("app_install_ads_state", self.stream_name)
output, service_call_mock = self.read_stream(
self.stream_name, SyncMode.incremental, self._config, "app_install_ads_with_state", state
)
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, "app_install_ads_with_state", state)
assert output.most_recent_state.stream_state.states[0]["cursor"] == {self.cursor_field: "2024-01-29T12:55:12.028+0000"}

15
airbyte-integrations/connectors/source-bing-ads/unit_tests/integrations/test_budget_stream.py
View File

@@ -1,5 +1,4 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
import pendulum
from freezegun import freeze_time
from test_bulk_stream import TestBulkStream
@@ -14,19 +13,17 @@ class TestBudgetStream(TestBulkStream):
def test_return_records_from_given_csv_file(self):
self.mock_apis(file=self.stream_name)
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "budget")
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "budget")
assert len(output.records) == 1
def test_return_logged_info_for_empty_csv_file(self):
def test_return_zero_record_from_empty_csv(self):
self.mock_apis(file="budget_empty")
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "budget_empty")
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "budget_empty")
assert len(output.records) == 0
no_records_message = self.create_log_message(f"Read 0 records from {self.stream_name} stream")
assert no_records_message in output.logs
def test_transform_records(self):
self.mock_apis(file=self.stream_name)
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "budget")
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, "budget")
assert output.records
for record in output.records:
assert "Account Id" in record.record.data.keys()
@@ -35,7 +32,7 @@ class TestBudgetStream(TestBulkStream):
@freeze_time("2024-02-26")
def test_incremental_read_cursor_value_matches_value_from_most_recent_record(self):
self.mock_apis(file="budget_with_cursor_value")
output, _ = self.read_stream(self.stream_name, SyncMode.incremental, self._config, "budget_with_cursor_value")
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, "budget_with_cursor_value")
assert len(output.records) == 8
assert output.most_recent_state.stream_state.states[0]["cursor"] == {self.cursor_field: "2024-01-01T12:54:12.028+0000"}
@@ -43,6 +40,6 @@ class TestBudgetStream(TestBulkStream):
def test_incremental_read_with_state(self):
self.mock_apis(file="budget_with_state", read_with_state=True)
state = self._state("budget_state", self.stream_name)
output, service_call_mock = self.read_stream(self.stream_name, SyncMode.incremental, self._config, "budget_with_state", state)
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, "budget_with_state", state)
assert len(output.records) == 8
assert output.most_recent_state.stream_state.states[0]["cursor"] == {self.cursor_field: "2024-01-30T12:54:12.028+0000"}

21
airbyte-integrations/connectors/source-bing-ads/unit_tests/integrations/test_bulk_stream.py
View File

@@ -4,20 +4,15 @@ from pathlib import Path
from typing import Optional
from base_test import BaseTest
from bingads.v13.bulk.bulk_service_manager import BulkServiceManager
from request_builder import RequestBuilder
from airbyte_cdk.test.mock_http import HttpMocker, HttpResponse
from airbyte_cdk.test.mock_http import HttpResponse
from airbyte_cdk.test.mock_http.response_builder import find_template
class TestBulkStream(BaseTest):
download_entity: str = None
@property
def service_manager(self) -> BulkServiceManager:
return BulkServiceManager
def mock_apis(self, file: str, read_with_state: Optional[bool] = False):
self.mock_user_query_api(response_template="user_query")
self.mock_accounts_search_api(
@@ -72,17 +67,3 @@ class TestBulkStream(BaseTest):
RequestBuilder(resource="path/to/bulk/resultquery/url", api="bulk").build(),
HttpResponse(encoded_file_content, 200),
)
def _download_file(self, file: Optional[str] = None) -> Path:
"""
Returns path to temporary file of downloaded data that will be use in read.
Base file should be named as {file_name}.cvs in resource/response folder.
"""
if file:
path_to_tmp_file = Path(__file__).parent.parent / f"resource/response/{file}_tmp.csv"
path_to_file_base = Path(__file__).parent.parent / f"resource/response/{file}.csv"
with open(path_to_file_base, "r") as f1, open(path_to_tmp_file, "w") as f2:
for line in f1:
f2.write(line)
return path_to_tmp_file
return Path(__file__).parent.parent / "resource/response/non-existing-file.csv"

48
airbyte-integrations/connectors/source-bing-ads/unit_tests/integrations/test_custom_report.py
View File

@@ -3,11 +3,10 @@
from typing import Any
from base_test import BaseTest
from bingads.v13.reporting.reporting_service_manager import ReportingServiceManager
from config_builder import ConfigBuilder
from freezegun import freeze_time
from test_hourly_reports import HourlyReportsTestWithStateChangesAfterMigration, get_state_after_migration
from test_report_stream import SOURCE_BING_ADS, TestSuiteReportStream
from test_report_stream import TestSuiteReportStream
from airbyte_cdk.models import SyncMode
from airbyte_cdk.test.state_builder import StateBuilder
@@ -305,10 +304,6 @@ class CustomReportSummary(BaseTest):
.build()
)
@property
def service_manager(self) -> ReportingServiceManager:
return ReportingServiceManager
def mock_report_apis(self):
self.mock_user_query_api(response_template="user_query")
self.mock_accounts_search_api(
@@ -326,26 +321,23 @@ class CustomReportSummary(BaseTest):
@freeze_time("2024-05-06")
def test_return_records_from_given_csv_file(self):
assert SOURCE_BING_ADS
self.mock_report_apis()
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
assert len(output.records) == self.records_number
@freeze_time("2024-05-06")
def test_return_records_incrementally_from_given_csv_file(self):
assert SOURCE_BING_ADS
self.mock_report_apis()
output, _ = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file)
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file)
assert len(output.records) == self.records_number
# state is not updated as records don't have a cursor field
assert output.most_recent_state.stream_state.state["TimePeriod"] == self.start_date
@freeze_time("2024-05-06")
def test_return_records_incrementally_with_state_from_given_csv_file(self):
assert SOURCE_BING_ADS
self.mock_report_apis()
state = StateBuilder().with_stream_state(self.stream_name, {"TimePeriod": self.start_date}).build()
output, _ = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file, state)
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file, state)
assert len(output.records) == self.records_number
# state is not updated as records don't have a cursor field
assert output.most_recent_state.stream_state.state["TimePeriod"] == self.start_date
@@ -411,10 +403,6 @@ class CustomReportDayOfWeek(BaseTest):
.build()
)
@property
def service_manager(self) -> ReportingServiceManager:
return ReportingServiceManager
def mock_report_apis(self):
self.mock_user_query_api(response_template="user_query")
self.mock_accounts_search_api(
@@ -432,16 +420,14 @@ class CustomReportDayOfWeek(BaseTest):
@freeze_time("2024-05-06")
def test_return_records_from_given_csv_file(self):
assert SOURCE_BING_ADS
self.mock_report_apis()
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
assert len(output.records) == self.records_number
@freeze_time("2024-05-06")
def test_return_records_from_given_csv_file_transform_record(self):
assert SOURCE_BING_ADS
self.mock_report_apis()
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
assert len(output.records) == self.records_number
for record in output.records:
record = record.record.data
@@ -452,19 +438,17 @@ class CustomReportDayOfWeek(BaseTest):
@freeze_time("2024-05-06")
def test_return_records_incrementally_from_given_csv_file(self):
assert SOURCE_BING_ADS
self.mock_report_apis()
output, _ = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file)
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file)
assert len(output.records) == self.records_number
# state is not updated as records don't have a cursor field
assert output.most_recent_state.stream_state.state["TimePeriod"] == "2024-05-06"
@freeze_time("2024-05-06")
def test_return_records_incrementally_with_state_from_given_csv_file(self):
assert SOURCE_BING_ADS
self.mock_report_apis()
state = StateBuilder().with_stream_state(self.stream_name, {"TimePeriod": self.start_date}).build()
output, _ = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file, state)
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file, state)
assert len(output.records) == self.records_number
# state is not updated as records don't have a cursor field
assert output.most_recent_state.stream_state.state["TimePeriod"] == "2024-05-06"
@@ -530,10 +514,6 @@ class CustomReportHourOfDay(BaseTest):
.build()
)
@property
def service_manager(self) -> ReportingServiceManager:
return ReportingServiceManager
def mock_report_apis(self):
self.mock_user_query_api(response_template="user_query")
self.mock_accounts_search_api(
@@ -551,16 +531,14 @@ class CustomReportHourOfDay(BaseTest):
@freeze_time("2024-05-06")
def test_return_records_from_given_csv_file(self):
assert SOURCE_BING_ADS
self.mock_report_apis()
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
assert len(output.records) == self.records_number
@freeze_time("2024-05-06")
def test_return_records_from_given_csv_file_transform_record(self):
assert SOURCE_BING_ADS
self.mock_report_apis()
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
assert len(output.records) == self.records_number
for record in output.records:
record = record.record.data
@@ -571,19 +549,17 @@ class CustomReportHourOfDay(BaseTest):
@freeze_time("2024-05-06")
def test_return_records_incrementally_from_given_csv_file(self):
assert SOURCE_BING_ADS
self.mock_report_apis()
output, _ = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file)
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file)
assert len(output.records) == self.records_number
# state is not updated as records don't have a cursor field
assert output.most_recent_state.stream_state.state["TimePeriod"] == "2024-05-06"
@freeze_time("2024-05-06")
def test_return_records_incrementally_with_state_from_given_csv_file(self):
assert SOURCE_BING_ADS
self.mock_report_apis()
state = StateBuilder().with_stream_state(self.stream_name, {"TimePeriod": self.start_date}).build()
output, _ = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file, state)
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file, state)
assert len(output.records) == self.records_number
# state is not updated as records don't have a cursor field
assert output.most_recent_state.stream_state.state["TimePeriod"] == "2024-05-06"

112
airbyte-integrations/connectors/source-bing-ads/unit_tests/integrations/test_report_stream.py
View File

@@ -2,49 +2,29 @@
import re
from copy import deepcopy
from pathlib import Path
from typing import Any, Optional
from typing import Any, List, Optional
import pendulum
from base_test import BaseTest
from bingads.v13.reporting.reporting_service_manager import ReportingServiceManager
from config_builder import ConfigBuilder
from freezegun import freeze_time
from source_bing_ads.source import SourceBingAds
from airbyte_cdk.connector_builder.connector_builder_handler import resolve_manifest
from airbyte_cdk.models import SyncMode
SOURCE_BING_ADS = resolve_manifest(source=SourceBingAds(None, None, None)).record.data["manifest"]
MANIFEST_STREAMS = (
[stream["name"] for stream in SOURCE_BING_ADS["streams"]]
+ [
stream_params.get("name")
for stream_params in SOURCE_BING_ADS["dynamic_streams"][0]["components_resolver"]["stream_parameters"][
"list_of_parameters_for_stream"
]
]
+ ["custom_report"]
)
SECOND_READ_FREEZE_TIME = "2024-05-08"
class TestReportStream(BaseTest):
start_date = "2024-01-01"
@property
def service_manager(self) -> ReportingServiceManager:
return ReportingServiceManager
@property
def _config(self) -> dict[str, Any]:
return ConfigBuilder().with_reports_start_date(self.start_date).build()
def _download_file(self, file: Optional[str] = None) -> Path:
"""
Returns path to temporary file of downloaded data that will be use in read.
Base file should be named as {file_name}.csv in resource/response folder.
Returns path to a temporary file of downloaded data that will be use in read.
Base file should be named as {file_name}.csv in the resource/response folder.
"""
if file:
path_to_tmp_file = Path(__file__).parent.parent / f"resource/response/{file}.csv"
@@ -84,15 +64,14 @@ class TestSuiteReportStream(TestReportStream):
@freeze_time("2024-05-06")
def test_return_records_from_given_csv_file(self):
assert SOURCE_BING_ADS
self.mock_report_apis()
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
assert len(output.records) == self.records_number
@freeze_time("2024-05-06")
def test_transform_records_from_given_csv_file(self):
self.mock_report_apis()
output, _ = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
output = self.read_stream(self.stream_name, SyncMode.full_refresh, self._config, self.report_file)
assert len(output.records) == self.records_number
for record in output.records:
@@ -101,7 +80,7 @@ class TestSuiteReportStream(TestReportStream):
@freeze_time("2024-05-06")
def test_incremental_read_returns_records(self):
self.mock_report_apis()
output, _ = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file)
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file)
assert len(output.records) == self.records_number
assert output.most_recent_state.stream_state.__dict__ == self.first_read_state
@@ -110,60 +89,13 @@ class TestSuiteReportStream(TestReportStream):
"""
We validate the state cursor is set to the value of the latest record read.
"""
if self.stream_name not in MANIFEST_STREAMS:
self.skipTest(f"Skipping test_incremental_read_returns_records for NOT migrated to manifest stream: {self.stream_name}")
if not self.report_file_with_records_further_start_date or not self.first_read_state_for_records_further_start_date:
assert False, "test_incremental_read_returns_records_further_config_start_date is not correctly set"
self.mock_report_apis()
output, _ = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file_with_records_further_start_date)
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.report_file_with_records_further_start_date)
assert len(output.records) == self.records_number
assert output.most_recent_state.stream_state.__dict__ == self.first_read_state_for_records_further_start_date
@freeze_time("2024-05-06")
def test_incremental_read_with_state_returns_records(self):
if self.stream_name in MANIFEST_STREAMS:
self.skipTest(f"Skipping for migrated to manifest stream: : {self.stream_name}")
self.mock_report_apis()
state = self._state(self.state_file, self.stream_name)
output, service_call_mock = self.read_stream(
self.stream_name, SyncMode.incremental, self._config, self.incremental_report_file, state
)
if not self.second_read_records_number:
assert len(output.records) == self.records_number
else:
assert len(output.records) == self.second_read_records_number
most_recent_state = output.most_recent_state.stream_state.__dict__
actual_cursor = most_recent_state.get(self.account_id)
expected_cursor = self.second_read_state.get(self.account_id)
assert actual_cursor == expected_cursor
provided_state = state[0].stream.stream_state.__dict__[self.account_id][self.cursor_field]
# gets ReportDownloadParams object
request_start_date = service_call_mock.call_args.args[0].report_request.Time.CustomDateRangeStart
year = request_start_date.Year
month = request_start_date.Month
day = request_start_date.Day
assert pendulum.DateTime(year, month, day, tzinfo=pendulum.UTC) == pendulum.parse(provided_state)
def test_incremental_read_with_state_and_no_start_date_returns_records_once(self):
"""
Test that incremental read with state and no start date in config returns records only once.
We observed that if the start date is not provided in the config, and we don't parse correctly the account_id
from the state, the incremental read returns records multiple times as we yield the default_time_periods
for no start date scenario.
"""
if self.stream_name in MANIFEST_STREAMS:
self.skipTest(f"Skipping for migrated to manifest stream: : {self.stream_name}")
state = self._state(self.state_file, self.stream_name)
config = deepcopy(self._config)
del config["reports_start_date"] # Simulate no start date in config
output, service_call_mock = self.read_stream(self.stream_name, SyncMode.incremental, config, self.incremental_report_file, state)
if not self.second_read_records_number:
assert len(output.records) == self.records_number
else:
assert len(output.records) == self.second_read_records_number
@freeze_time(SECOND_READ_FREEZE_TIME)
def test_incremental_read_with_state_and_no_start_date_returns_records_once_after_migration(self):
"""
@@ -172,13 +104,11 @@ class TestSuiteReportStream(TestReportStream):
from the state, the incremental read returns records multiple times as we yield the default_time_periods
for no start date scenario.
"""
if self.stream_name not in MANIFEST_STREAMS:
self.skipTest(f"Skipping for NOT migrated to manifest stream: {self.stream_name}")
self.mock_report_apis()
state = self._state(self.state_file_legacy, self.stream_name)
config = deepcopy(self._config)
del config["reports_start_date"] # Simulate no start date in config
output, service_call_mock = self.read_stream(
output = self.read_stream(
self.stream_name, SyncMode.incremental, config, self.incremental_report_file_with_records_further_cursor, state
)
if not self.second_read_records_number:
@@ -191,13 +121,9 @@ class TestSuiteReportStream(TestReportStream):
"""
For this test the records are all with TimePeriod behind the config start date and the state TimePeriod cursor.
"""
if self.stream_name not in MANIFEST_STREAMS:
self.skipTest(f"Skipping for NOT migrated to manifest stream: {self.stream_name}")
self.mock_report_apis()
state = self._state(self.state_file_after_migration, self.stream_name)
output, service_call_mock = self.read_stream(
self.stream_name, SyncMode.incremental, self._config, self.incremental_report_file, state
)
output = self.read_stream(self.stream_name, SyncMode.incremental, self._config, self.incremental_report_file, state)
if not self.second_read_records_number:
assert len(output.records) == self.records_number
else:
@@ -225,11 +151,9 @@ class TestSuiteReportStream(TestReportStream):
So we validate that the cursor in the output.most_recent_state is moved to the value of the latest record read.
The state format before migration IS NOT involved in this test.
"""
if self.stream_name not in MANIFEST_STREAMS:
self.skipTest(f"Skipping for NOT migrated to manifest stream: : {self.stream_name}")
self.mock_report_apis()
provided_state = self._state(self.state_file_after_migration_with_cursor_further_config_start_date, self.stream_name)
output, service_call_mock = self.read_stream(
output = self.read_stream(
self.stream_name, SyncMode.incremental, self._config, self.incremental_report_file_with_records_further_cursor, provided_state
)
if not self.second_read_records_number:
@@ -290,18 +214,16 @@ class TestSuiteReportStream(TestReportStream):
@freeze_time(SECOND_READ_FREEZE_TIME)
def test_incremental_read_with_legacy_state_returns_records_after_migration_with_records_further_state_cursor(self):
"""
For this test we get records with TimePeriod further the config start date and the state TimePeriod cursor.
The provide state is taken from a previous run; with python stream; so, is already in legacy format, and
For this test, we get records with TimePeriod further the config start date and the state TimePeriod cursor.
The provided state is taken from a previous run; with python stream; so, is already in legacy format, and
where the resultant cursor was further the config start date.
So we validate that the cursor in the output.most_recent_state is moved to the value of the latest record read.
Also, the state is migrated to the new format, so we can validate that the partition is correctly set.
The state format before migration (legacy) IS involved in this test.
"""
if self.stream_name not in MANIFEST_STREAMS:
self.skipTest(f"Skipping for NOT migrated to manifest stream: {self.stream_name}")
self.mock_report_apis()
provided_state = self._state(self.state_file_legacy, self.stream_name)
output, service_call_mock = self.read_stream(
output = self.read_stream(
self.stream_name, SyncMode.incremental, self._config, self.incremental_report_file_with_records_further_cursor, provided_state
)
if not self.second_read_records_number:
@@ -326,7 +248,7 @@ class TestSuiteReportStream(TestReportStream):
assert False, f"Expected state is empty for account_id: {self.account_id}"
if not actual_partition or not expected_partition:
assert False, f"Expected state is empty for account_id: {self.account_id}"
# here the cursor moved to expected that is the latest record read
# here the cursor moved to expect that is the latest record read
assert actual_cursor == expected_cursor
assert actual_partition == expected_partition
@@ -364,8 +286,6 @@ class TestSuiteReportStream(TestReportStream):
If the field reports_start_date is blank, Airbyte will replicate all data from previous and current calendar years.
This test is to validate that the stream will return all records from the first day of the year 2023 (CustomDateRangeStart in mocked body).
"""
if self.stream_name not in MANIFEST_STREAMS:
self.skipTest(f"Skipping for NOT migrated to manifest stream: {self.stream_name}")
self.mock_report_apis()
# here we mock the report start date to be the first day of the year 2023
self.mock_generate_report_api(
@@ -375,11 +295,11 @@ class TestSuiteReportStream(TestReportStream):
)
config = deepcopy(self._config)
del config["reports_start_date"]
output, _ = self.read_stream(self.stream_name, SyncMode.incremental, config, self.report_file)
output = self.read_stream(self.stream_name, SyncMode.incremental, config, self.report_file)
assert len(output.records) == self.records_number
first_read_state = deepcopy(self.first_read_state)
# this corresponds to the last read record as we don't have started_date in the config
# the self.first_read_state is set using the config start date so it is not correct for this test
# the self.first_read_state is set using the config start date, so it is not correct for this test
if "hourly" in self.stream_name or (hasattr(self, "custom_report_aggregation") and self.custom_report_aggregation == "Hourly"):
first_read_state["state"][self.cursor_field] = "2023-11-12T00:00:00+00:00"
first_read_state["states"][0]["cursor"][self.cursor_field] = "2023-11-12T00:00:00+00:00"

710
airbyte-integrations/connectors/source-bing-ads/poetry.lock → airbyte-integrations/connectors/source-bing-ads/unit_tests/poetry.lock generated
View File

File diff suppressed because it is too large Load Diff

20
airbyte-integrations/connectors/source-bing-ads/unit_tests/pyproject.toml Normal file
View File

@@ -0,0 +1,20 @@
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.poetry]
name = "source-bing-ads"
version = "0.0.0"
description = "Unit tests for source-bing-ads"
authors = ["Airbyte <contact@airbyte.io>"]
[tool.poetry.dependencies]
python = "^3.10,<3.13"
airbyte-cdk = "^6"
freezegun = "^1.4.0"
pytest-mock = "^3.6.1"
pytest = "^8.0.0"
requests-mock = "^1.12.1"
mock = "^5.1.0"
[tool.pytest.ini_options]
filterwarnings = [
"ignore:This class is experimental*"
]

2
airbyte-integrations/connectors/source-bing-ads/unit_tests/resource/response/report_base_data.csv Normal file
View File

@@ -0,0 +1,2 @@
TimePeriod,AccountId
2023-01-01,1
1 TimePeriod AccountId
2 2023-01-01 1

2
airbyte-integrations/connectors/source-bing-ads/unit_tests/resource/response/report_base_data_hourly.csv Normal file
View File

@@ -0,0 +1,2 @@
TimePeriod,AccountId
2023-01-01|15,1
1 TimePeriod AccountId
2 2023-01-01|15 1

251
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_base_streams.py
View File

@@ -2,14 +2,16 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from unittest.mock import patch
import pytest
import source_bing_ads
from source_bing_ads.base_streams import Accounts
from conftest import find_stream, get_source
from airbyte_cdk.models import SyncMode
from airbyte_cdk.sources.declarative.types import StreamSlice
from airbyte_cdk.test.catalog_builder import CatalogBuilder
from airbyte_cdk.test.entrypoint_wrapper import read
from airbyte_cdk.test.state_builder import StateBuilder
@patch.object(source_bing_ads.source, "Client")
@pytest.mark.parametrize(
"record, expected",
[
@@ -21,6 +23,7 @@ from source_bing_ads.base_streams import Accounts
"Status": "Active",
"TaxCertificateBlobContainerName": "Test Container Name",
},
"LastModifiedTime": "2025-01-01",
},
{
"AccountId": 16253412,
@@ -29,6 +32,7 @@ from source_bing_ads.base_streams import Accounts
"Status": "Active",
"TaxCertificateBlobContainerName": "Test Container Name",
},
"LastModifiedTime": "2025-01-01",
},
),
(
@@ -39,6 +43,7 @@ from source_bing_ads.base_streams import Accounts
"Status": "Active",
"TaxCertificateBlobContainerName": "Test Container Name",
},
"LastModifiedTime": "2025-01-01",
},
{
"AccountId": 16253412,
@@ -47,19 +52,20 @@ from source_bing_ads.base_streams import Accounts
"Status": "Active",
"TaxCertificateBlobContainerName": "Test Container Name",
},
"LastModifiedTime": "2025-01-01",
},
),
(
{"AccountId": 16253412, "LastModifiedTime": "2025-01-01"},
{"AccountId": 16253412, "LastModifiedTime": "2025-01-01"},
),
(
{
"AccountId": 16253412,
"TaxCertificate": None,
"LastModifiedTime": "2025-01-01",
},
{
"AccountId": 16253412,
},
),
(
{"AccountId": 16253412, "TaxCertificate": None},
{"AccountId": 16253412, "TaxCertificate": None},
{"AccountId": 16253412, "TaxCertificate": None, "LastModifiedTime": "2025-01-01"},
),
],
ids=[
@@ -69,7 +75,220 @@ from source_bing_ads.base_streams import Accounts
"record_with_TaxCertificate_is_None",
],
)
def test_accounts_transform_tax_fields(mocked_client, config, record, expected):
stream = Accounts(mocked_client, config)
actual = stream._transform_tax_fields(record)
assert actual == expected
def test_accounts_transform_tax_fields(config, record, expected):
stream = find_stream("accounts", config)
transformed_record = list(
stream.retriever.record_selector.filter_and_transform(all_data=[record], stream_state={}, stream_slice={}, records_schema={})
)[0]
if expected.get("TaxCertificate"):
assert transformed_record["TaxCertificate"] == expected["TaxCertificate"]
else:
assert expected.get("TaxCertificate") is None
assert transformed_record.get("TaxCertificate") is None
def test_campaigns_request_params(config):
campaigns = find_stream("campaigns", config)
stream_slice = StreamSlice(partition={"account_id": "account_id"}, cursor_slice={})
request_params = campaigns.retriever.requester.get_request_body_json(stream_slice=stream_slice)
assert request_params
assert request_params["AccountId"] == "account_id"
assert request_params["CampaignType"] == "Audience,DynamicSearchAds,Search,Shopping,PerformanceMax"
assert (
request_params["ReturnAdditionalFields"]
== "AdScheduleUseSearcherTimeZone,BidStrategyId,CpvCpmBiddingScheme,DynamicDescriptionSetting,DynamicFeedSetting,MaxConversionValueBiddingScheme,MultimediaAdsBidAdjustment,TargetImpressionShareBiddingScheme,TargetSetting,VerifiedTrackingSetting"
)
def test_campaigns_stream_slices(config, logger_mock, mock_auth_token, mock_user_query, mock_account_query):
campaigns = find_stream("campaigns", config)
slices = campaigns.stream_slices(sync_mode=SyncMode.full_refresh, stream_state={})
assert list(slices) == [
{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}},
{"account_id": 2, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}},
{"account_id": 3, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}},
]
def test_adgroups_stream_slices(mock_auth_token, mock_user_query, mock_account_query, requests_mock, config):
requests_mock.post(
"https://campaign.api.bingads.microsoft.com/CampaignManagement/v13/Campaigns/QueryByAccountId",
status_code=200,
json={
"Campaigns": [
{"Id": 1, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 2, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 3, "LastModifiedTime": "2022-02-02T22:22:22"},
]
},
)
ad_groups = find_stream("ad_groups", config)
stream_slices = list(ad_groups.retriever.stream_slicer.stream_slices())
assert stream_slices == [
{"campaign_id": [1], "parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}]},
{"campaign_id": [2], "parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}]},
{"campaign_id": [3], "parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}]},
]
def test_ads_request_body_data(mock_auth_token, config):
ads = find_stream("ads", config)
stream_slice = {
"campaign_id": [1],
"parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "AccountName", "user_id": 1, "parent_slice": {}}}],
}
request_params = ads.retriever.requester.get_request_body_json(stream_slice=stream_slice)
assert request_params == {
"AdTypes": ["Text", "Image", "Product", "AppInstall", "ExpandedText", "DynamicSearch", "ResponsiveAd", "ResponsiveSearch"],
"ReturnAdditionalFields": "ImpressionTrackingUrls,Videos,LongHeadlines",
}
def test_ads_stream_slices(mock_auth_token, mock_user_query, mock_account_query, requests_mock, config):
requests_mock.post(
"https://campaign.api.bingads.microsoft.com/CampaignManagement/v13/Campaigns/QueryByAccountId",
status_code=200,
json={
"Campaigns": [
{"Id": 1, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 2, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 3, "LastModifiedTime": "2022-02-02T22:22:22"},
]
},
)
requests_mock.post(
"https://campaign.api.bingads.microsoft.com/CampaignManagement/v13/AdGroups/QueryByCampaignId",
status_code=200,
json={
"AdGroups": [
{"Id": 1, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 2, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 3, "LastModifiedTime": "2022-02-02T22:22:22"},
]
},
)
ads = find_stream("ads", config)
stream_slices = list(ads.retriever.stream_slicer.stream_slices())
assert stream_slices == [
{
"ad_group_id": [1],
"parent_slice": [
{
"campaign_id": [1],
"parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}],
}
],
},
{
"ad_group_id": [2],
"parent_slice": [
{
"campaign_id": [1],
"parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}],
}
],
},
{
"ad_group_id": [3],
"parent_slice": [
{
"campaign_id": [1],
"parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}],
}
],
},
]
@pytest.mark.parametrize(
"stream",
(
"accounts",
"ad_groups",
"ads",
"campaigns",
),
)
def test_streams_full_refresh(config, stream, mock_auth_token, mock_user_query, mock_account_query, requests_mock):
requests_mock.post(
"https://campaign.api.bingads.microsoft.com/CampaignManagement/v13/Campaigns/QueryByAccountId",
status_code=200,
json={
"Campaigns": [
{"Id": 1, "LastModifiedTime": "2022-02-02T22:22:22"},
]
},
)
requests_mock.post(
"https://campaign.api.bingads.microsoft.com/CampaignManagement/v13/AdGroups/QueryByCampaignId",
status_code=200,
json={
"AdGroups": [
{"Id": 1, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 2, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 3, "LastModifiedTime": "2022-02-02T22:22:22"},
]
},
)
requests_mock.post(
"https://campaign.api.bingads.microsoft.com/CampaignManagement/v13/Ads/QueryByAdGroupId",
status_code=200,
json={
"Ads": [
{"Id": 1, "LastModifiedTime": "2022-02-02T22:22:22"},
]
},
)
state = StateBuilder().with_stream_state(stream, {}).build()
catalog = CatalogBuilder().with_stream(stream, SyncMode.full_refresh).build()
source = get_source(config, state)
output = read(source, config, catalog, state)
assert len(output.records) == 3
def test_transform(mock_auth_token, config):
record = {"AdFormatPreference": "All", "DevicePreference": 0, "EditorialStatus": "ActiveLimited", "FinalAppUrls": None}
ads_stream = find_stream("ads", config)
expected_record = {
"AccountId": 909090,
"AdFormatPreference": "All",
"AdGroupId": 90909090,
"CustomerId": 9090909,
"Descriptions": None,
"DevicePreference": 0,
"EditorialStatus": "ActiveLimited",
"FinalAppUrls": None,
"FinalMobileUrls": None,
"FinalUrls": None,
"ForwardCompatibilityMap": None,
"Headlines": None,
"Images": None,
"LongHeadlines": None,
"Path1": None,
"Path2": None,
"TextPart2": None,
"TitlePart3": None,
"Videos": None,
}
transformed_record = list(
ads_stream.retriever.record_selector.filter_and_transform(
all_data=[
record,
],
stream_state=None,
records_schema={},
stream_slice={
"ad_group_id": [90909090],
"extra_fields": {"AccountId": [909090], "CustomerId": [9090909]},
"parent_slice": [
{
"campaign_id": [1],
"parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}],
}
],
},
)
)[0]
assert dict(sorted(transformed_record.items())) == dict(sorted(expected_record.items()))

192
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_client.py
View File

@@ -1,192 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import socket
from datetime import datetime, timedelta
from unittest import mock
from unittest.mock import patch
from urllib.error import URLError
import pytest
import source_bing_ads.client
from bingads.authorization import AuthorizationData, OAuthTokens
from bingads.v13.bulk import BulkServiceManager
from bingads.v13.reporting.exceptions import ReportingDownloadException
from suds import sudsobject
from airbyte_cdk.utils import AirbyteTracedException
def test_sudsobject_todict_primitive_types():
test_arr = ["1", "test", 1, [0, 0]]
test_dict = {"k1": {"k2": 2, "k3": [1, 2, 3]}}
test_date = datetime.utcnow()
suds_obj = sudsobject.Object()
suds_obj["int"] = 1
suds_obj["arr"] = test_arr
suds_obj["dict"] = test_dict
suds_obj["date"] = test_date
serialized_obj = source_bing_ads.client.Client.asdict(suds_obj)
assert serialized_obj["int"] == 1
assert serialized_obj["arr"] == test_arr
assert serialized_obj["dict"] == test_dict
assert serialized_obj["date"] == test_date.isoformat()
def test_sudsobject_todict_nested():
test_date = datetime.utcnow()
suds_obj = sudsobject.Object()
nested_suds_1, nested_suds_2, nested_suds_3, nested_suds_4 = (
sudsobject.Object(),
sudsobject.Object(),
sudsobject.Object(),
sudsobject.Object(),
)
nested_suds_1["value"] = test_date
nested_suds_2["value"] = 1
nested_suds_3["value"] = "str"
nested_suds_4["value"] = object()
suds_obj["obj1"] = nested_suds_1
suds_obj["arr"] = [nested_suds_2, nested_suds_3, nested_suds_4]
serialized_obj = source_bing_ads.client.Client.asdict(suds_obj)
assert serialized_obj["obj1"]["value"] == test_date.isoformat()
assert serialized_obj["arr"][0]["value"] == nested_suds_2["value"]
assert serialized_obj["arr"][1]["value"] == nested_suds_3["value"]
assert serialized_obj["arr"][2]["value"] == nested_suds_4["value"]
def test_is_expired_true():
def fake__init__(self, **kwargs):
self.oauth = OAuthTokens(access_token_expires_in_seconds=10)
self.oauth._access_token_received_datetime = datetime.utcnow() - timedelta(seconds=100)
with mock.patch.object(source_bing_ads.client.Client, "__init__", fake__init__):
client = source_bing_ads.client.Client()
assert client.is_token_expiring() is True
def test_is_expired_true_with_delta_threshold():
"""
Testing case when token still not expired actually, but refresh_token_safe_delta check is not passed
"""
def fake__init__(self, **kwargs):
expires_in = 100 + source_bing_ads.client.Client.refresh_token_safe_delta / 2
self.oauth = OAuthTokens(access_token_expires_in_seconds=expires_in)
self.oauth._access_token_received_datetime = datetime.utcnow() - timedelta(seconds=100)
with mock.patch.object(source_bing_ads.client.Client, "__init__", fake__init__):
client = source_bing_ads.client.Client()
assert client.is_token_expiring() is True
def test_is_expired_false():
def fake__init__(self, **kwargs):
self.oauth = OAuthTokens(access_token_expires_in_seconds=100)
self.oauth._access_token_received_datetime = datetime.utcnow() - timedelta(seconds=10)
with mock.patch.object(source_bing_ads.client.Client, "__init__", fake__init__):
client = source_bing_ads.client.Client()
assert client.is_token_expiring() is False
@patch("bingads.authorization.OAuthWebAuthCodeGrant.request_oauth_tokens_by_refresh_token")
def test_get_auth_client(patched_request_tokens):
client = source_bing_ads.client.Client("tenant_id", "2020-01-01", client_id="client_id", refresh_token="refresh_token")
client._get_auth_client("client_id", "tenant_id")
patched_request_tokens.assert_called_once_with("refresh_token")
@patch("bingads.authorization.OAuthWebAuthCodeGrant.request_oauth_tokens_by_refresh_token")
def test_get_auth_data(patched_request_tokens):
client = source_bing_ads.client.Client("tenant_id", "2020-01-01", client_id="client_id", refresh_token="refresh_token")
auth_data = client._get_auth_data()
assert isinstance(auth_data, AuthorizationData)
@patch("bingads.authorization.OAuthWebAuthCodeGrant.request_oauth_tokens_by_refresh_token")
def test_handling_ReportingDownloadException(patched_request_tokens):
client = source_bing_ads.client.Client("tenant_id", "2020-01-01", client_id="client_id", refresh_token="refresh_token")
give_up = client.should_give_up(ReportingDownloadException(message="test"))
assert False is give_up
assert client._download_timeout == 310000
client._download_timeout = 600000
client.should_give_up(ReportingDownloadException(message="test"))
assert client._download_timeout == 600000
def test_get_access_token(requests_mock):
requests_mock.post(
"https://login.microsoftonline.com/tenant_id/oauth2/v2.0/token",
status_code=400,
json={
"error": "invalid_grant",
"error_description": "AADSTS70000: The user could not be authenticated as the grant is expired. The user must sign in again.",
},
)
with pytest.raises(
AirbyteTracedException,
match="Failed to get OAuth access token by refresh token. The user could not be authenticated as the grant is expired. "
"The user must sign in again.",
):
source_bing_ads.client.Client("tenant_id", "2020-01-01", client_id="client_id", refresh_token="refresh_token")
def test_get_access_token_success(requests_mock):
requests_mock.post(
"https://login.microsoftonline.com/tenant_id/oauth2/v2.0/token",
status_code=200,
json={"access_token": "test", "expires_in": "900", "refresh_token": "test"},
)
source_bing_ads.client.Client("tenant_id", "2020-01-01", client_id="client_id", refresh_token="refresh_token")
assert requests_mock.call_count == 1
@patch("bingads.authorization.OAuthWebAuthCodeGrant.request_oauth_tokens_by_refresh_token")
def test_should_give_up(patched_request_tokens):
client = source_bing_ads.client.Client("tenant_id", "2020-01-01", client_id="client_id", refresh_token="refresh_token")
give_up = client.should_give_up(Exception())
assert True is give_up
give_up = client.should_give_up(URLError(reason="test"))
assert True is give_up
give_up = client.should_give_up(URLError(reason=socket.timeout()))
assert False is give_up
@patch("bingads.authorization.OAuthWebAuthCodeGrant.request_oauth_tokens_by_refresh_token")
def test_get_service(patched_request_tokens):
client = source_bing_ads.client.Client("tenant_id", "2020-01-01", client_id="client_id", refresh_token="refresh_token")
service = client.get_service(service_name="CustomerManagementService")
assert "customermanagement_service.xml" in service.service_url
@patch("bingads.authorization.OAuthWebAuthCodeGrant.request_oauth_tokens_by_refresh_token")
def test_get_reporting_service(patched_request_tokens):
client = source_bing_ads.client.Client("tenant_id", "2020-01-01", client_id="client_id", refresh_token="refresh_token")
service = client._get_reporting_service()
assert (service._poll_interval_in_milliseconds, service._environment) == (client.report_poll_interval, client.environment)
@patch("bingads.authorization.OAuthWebAuthCodeGrant.request_oauth_tokens_by_refresh_token")
def test_bulk_service_manager(patched_request_tokens):
client = source_bing_ads.client.Client("tenant_id", "2020-01-01", client_id="client_id", refresh_token="refresh_token")
service = client._bulk_service_manager()
assert (service._poll_interval_in_milliseconds, service._environment) == (5000, client.environment)
def test_get_bulk_entity(requests_mock):
requests_mock.post(
"https://login.microsoftonline.com/tenant_id/oauth2/v2.0/token",
status_code=200,
json={"access_token": "test", "expires_in": "9000", "refresh_token": "test"},
)
client = source_bing_ads.client.Client("tenant_id", "2020-01-01", client_id="client_id", refresh_token="refresh_token")
with patch.object(BulkServiceManager, "download_file", return_value="file.csv"):
bulk_entity = client.get_bulk_entity(data_scope=["EntityData"], download_entities=["AppInstallAds"])
assert bulk_entity == "file.csv"

311
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_components.py
View File

@@ -1,171 +1,164 @@
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
import pytest
from source_bing_ads.components import BingAdsCampaignsRecordTransformer, BulkStreamsStateMigration
class TestBingAdsCampaignsRecordTransformer:
"""Test cases for the BingAdsCampaignsRecordTransformer component."""
def setup_method(self):
"""Set up test fixtures."""
self.transformer = BingAdsCampaignsRecordTransformer()
@pytest.mark.parametrize(
"test_name,input_record,expected_settings",
[
(
"settings_with_target_setting_details",
{
"Id": 486441589,
"Settings": [{"Type": "TargetSetting", "Details": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]}],
},
{
"Setting": [
{
"Type": "TargetSetting",
"Details": {"TargetSettingDetail": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
}
]
},
),
(
"settings_with_performance_max_setting",
{"Id": 486441589, "Settings": [{"Type": "PerformanceMaxSetting", "FinalUrlExpansionOptOut": False}]},
{"Setting": [{"Type": "PerformanceMaxSetting", "FinalUrlExpansionOptOut": False}]},
),
(
"settings_with_mixed_types",
{
"Id": 486441589,
"Settings": [
{"Type": "PerformanceMaxSetting", "FinalUrlExpansionOptOut": False},
{"Type": "TargetSetting", "Details": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
],
},
{
"Setting": [
{"Type": "PerformanceMaxSetting", "FinalUrlExpansionOptOut": False},
{
"Type": "TargetSetting",
"Details": {"TargetSettingDetail": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
},
]
},
),
(
"settings_with_additional_properties",
{
"Id": 486441589,
"Settings": [
{
"Type": "TargetSetting",
"Details": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}],
"CustomProperty": "CustomValue",
"AnotherProperty": 123,
}
],
},
{
"Setting": [
{
"Type": "TargetSetting",
"Details": {"TargetSettingDetail": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
"CustomProperty": "CustomValue",
"AnotherProperty": 123,
}
]
},
),
(
"non_dict_setting_items",
{
"Id": 486441589,
"Settings": [
"invalid_setting",
{"Type": "TargetSetting", "Details": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
123,
],
},
{
"Setting": [
"invalid_setting",
{
"Type": "TargetSetting",
"Details": {"TargetSettingDetail": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
},
123,
]
},
),
(
"setting_with_empty_details",
{"Id": 486441589, "Settings": [{"Type": "TargetSetting", "Details": []}]},
{"Setting": [{"Type": "TargetSetting", "Details": {"TargetSettingDetail": []}}]},
),
(
"setting_with_none_details",
{"Id": 486441589, "Settings": [{"Type": "TargetSetting", "Details": None}]},
{"Setting": [{"Type": "TargetSetting", "Details": None}]},
),
(
"setting_without_type_field",
{
"Id": 486441589,
"Settings": [{"Details": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}], "CustomField": "value"}],
},
{
"Setting": [
{
"Type": None,
"Details": {"TargetSettingDetail": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
"CustomField": "value",
}
]
},
),
(
"complex_details_structure",
{
"Id": 486441589,
"Settings": [
{
"Type": "TargetSetting",
"Details": [
@pytest.mark.parametrize(
"test_name,input_record,expected_settings",
[
(
"settings_with_target_setting_details",
{
"Id": 486441589,
"Settings": [{"Type": "TargetSetting", "Details": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]}],
},
{
"Setting": [
{
"Type": "TargetSetting",
"Details": {"TargetSettingDetail": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
}
]
},
),
(
"settings_with_performance_max_setting",
{"Id": 486441589, "Settings": [{"Type": "PerformanceMaxSetting", "FinalUrlExpansionOptOut": False}]},
{"Setting": [{"Type": "PerformanceMaxSetting", "FinalUrlExpansionOptOut": False}]},
),
(
"settings_with_mixed_types",
{
"Id": 486441589,
"Settings": [
{"Type": "PerformanceMaxSetting", "FinalUrlExpansionOptOut": False},
{"Type": "TargetSetting", "Details": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
],
},
{
"Setting": [
{"Type": "PerformanceMaxSetting", "FinalUrlExpansionOptOut": False},
{
"Type": "TargetSetting",
"Details": {"TargetSettingDetail": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
},
]
},
),
(
"settings_with_additional_properties",
{
"Id": 486441589,
"Settings": [
{
"Type": "TargetSetting",
"Details": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}],
"CustomProperty": "CustomValue",
"AnotherProperty": 123,
}
],
},
{
"Setting": [
{
"Type": "TargetSetting",
"Details": {"TargetSettingDetail": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
"CustomProperty": "CustomValue",
"AnotherProperty": 123,
}
]
},
),
(
"non_dict_setting_items",
{
"Id": 486441589,
"Settings": [
"invalid_setting",
{"Type": "TargetSetting", "Details": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
123,
],
},
{
"Setting": [
"invalid_setting",
{
"Type": "TargetSetting",
"Details": {"TargetSettingDetail": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
},
123,
]
},
),
(
"setting_with_empty_details",
{"Id": 486441589, "Settings": [{"Type": "TargetSetting", "Details": []}]},
{"Setting": [{"Type": "TargetSetting", "Details": {"TargetSettingDetail": []}}]},
),
(
"setting_with_none_details",
{"Id": 486441589, "Settings": [{"Type": "TargetSetting", "Details": None}]},
{"Setting": [{"Type": "TargetSetting", "Details": None}]},
),
(
"setting_without_type_field",
{
"Id": 486441589,
"Settings": [{"Details": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}], "CustomField": "value"}],
},
{
"Setting": [
{
"Type": None,
"Details": {"TargetSettingDetail": [{"CriterionTypeGroup": "Audience", "TargetAndBid": False}]},
"CustomField": "value",
}
]
},
),
(
"complex_details_structure",
{
"Id": 486441589,
"Settings": [
{
"Type": "TargetSetting",
"Details": [
{
"CriterionTypeGroup": "Audience",
"TargetAndBid": False,
"NestedObject": {"Property1": "Value1", "Property2": ["item1", "item2"]},
},
{"CriterionTypeGroup": "Location", "TargetAndBid": True},
],
}
],
},
{
"Setting": [
{
"Type": "TargetSetting",
"Details": {
"TargetSettingDetail": [
{
"CriterionTypeGroup": "Audience",
"TargetAndBid": False,
"NestedObject": {"Property1": "Value1", "Property2": ["item1", "item2"]},
},
{"CriterionTypeGroup": "Location", "TargetAndBid": True},
],
}
],
},
{
"Setting": [
{
"Type": "TargetSetting",
"Details": {
"TargetSettingDetail": [
{
"CriterionTypeGroup": "Audience",
"TargetAndBid": False,
"NestedObject": {"Property1": "Value1", "Property2": ["item1", "item2"]},
},
{"CriterionTypeGroup": "Location", "TargetAndBid": True},
]
},
}
]
},
),
],
)
def test_settings_transformation(self, test_name, input_record, expected_settings):
"""Test various Settings field transformations."""
self.transformer.transform(input_record)
assert input_record["Settings"] == expected_settings
]
},
}
]
},
),
],
)
def test_settings_transformation(test_name, input_record, expected_settings, components_module):
"""Test various Settings field transformations."""
transformer = components_module.BingAdsCampaignsRecordTransformer()
transformer.transform(input_record)
assert input_record["Settings"] == expected_settings
@pytest.mark.parametrize(
"test_name,input_record,expected_settings",
@@ -818,6 +811,6 @@ class TestBingAdsCampaignsRecordTransformer:
),
),
)
def test_bulk_stream_state_migration(stream_state, expected_state):
migrator = BulkStreamsStateMigration()
def test_bulk_stream_state_migration(stream_state, expected_state, components_module):
migrator = components_module.BulkStreamsStateMigration()
assert migrator.migrate(stream_state) == expected_state

291
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_reports.py
View File

@@ -2,123 +2,109 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import _csv
import copy
import json
import xml.etree.ElementTree as ET
from pathlib import Path
from unittest.mock import MagicMock, Mock, patch
from urllib.parse import urlparse
import pytest
import source_bing_ads
from bingads.service_info import SERVICE_INFO_DICT_V13
from bingads.v13.internal.reporting.row_report import _RowReport
from conftest import find_stream
from helpers import source
from source_bing_ads.base_streams import Accounts
from source_bing_ads.report_streams import (
AccountPerformanceReportDaily,
AccountPerformanceReportHourly,
AccountPerformanceReportMonthly,
)
from source_bing_ads.reports import BingAdsReportingServicePerformanceStream, BingAdsReportingServiceStream
from source_bing_ads.reports.ad_performance_report import (
AdPerformanceReportDaily,
AdPerformanceReportHourly,
)
from source_bing_ads.source import SourceBingAds
from suds import WebFault
from conftest import create_zip_from_csv, find_stream, get_source
from freezegun import freeze_time
from airbyte_cdk.models import SyncMode
from airbyte_cdk.test.catalog_builder import CatalogBuilder
from airbyte_cdk.test.entrypoint_wrapper import read
from airbyte_cdk.test.state_builder import StateBuilder
TEST_CONFIG = {
"developer_token": "developer_token",
"client_id": "client_id",
"refresh_token": "refresh_token",
"reports_start_date": "2020-01-01T00:00:00Z",
"reports_start_date": "2020-01-01",
"tenant_id": "common",
}
class TestClient:
pass
class TestReport(BingAdsReportingServiceStream, SourceBingAds):
date_format, report_columns, report_name, cursor_field = "YYYY-MM-DD", None, None, "Time"
report_aggregation = "Monthly"
report_schema_name = "campaign_performance_report"
def __init__(self) -> None:
self.client = TestClient()
class TestPerformanceReport(BingAdsReportingServicePerformanceStream, SourceBingAds):
date_format, report_columns, report_name, cursor_field = "YYYY-MM-DD", None, None, "Time"
report_aggregation = "Monthly"
report_schema_name = "campaign_performance_report"
def __init__(self) -> None:
self.client = TestClient()
@patch.object(source_bing_ads.source, "Client")
def test_AccountPerformanceReportMonthly_request_params(mocked_client, config):
accountperformancereportmonthly = AccountPerformanceReportMonthly(mocked_client, config)
request_params = accountperformancereportmonthly.request_params(account_id=180278106, stream_slice={"time_period": "ThisYear"})
del request_params["report_request"]
assert request_params == {
"overwrite_result_file": True,
"result_file_directory": "/tmp",
"result_file_name": "AccountPerformanceReport",
"timeout_in_milliseconds": 300000,
}
def test_get_updated_state_init_state():
test_report = TestReport()
stream_state = {}
latest_record = {"AccountId": 123, "Time": "2020-01-02"}
new_state = test_report.get_updated_state(stream_state, latest_record)
assert new_state["123"]["Time"] == "2020-01-02"
def test_get_updated_state_new_state():
test_report = TestReport()
stream_state = {"123": {"Time": "2020-01-01"}}
latest_record = {"AccountId": 123, "Time": "2020-01-02"}
new_state = test_report.get_updated_state(stream_state, latest_record)
assert new_state["123"]["Time"] == "2020-01-02"
def test_get_updated_state_state_unchanged():
test_report = TestReport()
stream_state = {"123": {"Time": "2020-01-03"}}
latest_record = {"AccountId": 123, "Time": "2020-01-02"}
new_state = test_report.get_updated_state(copy.deepcopy(stream_state), latest_record)
assert stream_state == new_state
def test_get_updated_state_state_new_account():
test_report = TestReport()
stream_state = {"123": {"Time": "2020-01-03"}}
latest_record = {"AccountId": 234, "Time": "2020-01-02"}
new_state = test_report.get_updated_state(stream_state, latest_record)
assert "234" in new_state and "123" in new_state
assert new_state["234"]["Time"] == "2020-01-02"
@freeze_time("2024-01-01")
@pytest.mark.parametrize(
"stream_report_daily_cls",
(
AccountPerformanceReportDaily,
AdPerformanceReportDaily,
),
"stream_name",
[
"ad_performance_report_daily",
"ad_performance_report_weekly",
"ad_performance_report_monthly",
"age_gender_audience_report_daily",
"age_gender_audience_report_weekly",
"age_gender_audience_report_monthly",
"account_impression_performance_report_daily",
"account_impression_performance_report_weekly",
"account_impression_performance_report_monthly",
"account_performance_report_daily",
"account_performance_report_weekly",
"account_performance_report_monthly",
"audience_performance_report_daily",
"audience_performance_report_weekly",
"audience_performance_report_monthly",
"keyword_performance_report_daily",
"keyword_performance_report_weekly",
"keyword_performance_report_monthly",
"ad_group_performance_report_daily",
"ad_group_performance_report_weekly",
"ad_group_performance_report_monthly",
"ad_group_impression_performance_report_daily",
"ad_group_impression_performance_report_weekly",
"ad_group_impression_performance_report_monthly",
"campaign_performance_report_daily",
"campaign_performance_report_weekly",
"campaign_performance_report_monthly",
"campaign_impression_performance_report_daily",
"campaign_impression_performance_report_weekly",
"campaign_impression_performance_report_monthly",
"geographic_performance_report_daily",
"geographic_performance_report_weekly",
"geographic_performance_report_monthly",
"goals_and_funnels_report_daily",
"goals_and_funnels_report_weekly",
"goals_and_funnels_report_monthly",
"product_dimension_performance_report_daily",
"product_dimension_performance_report_weekly",
"product_dimension_performance_report_monthly",
"product_search_query_performance_report_daily",
"product_search_query_performance_report_weekly",
"product_search_query_performance_report_monthly",
"search_query_performance_report_daily",
"search_query_performance_report_weekly",
"search_query_performance_report_monthly",
"user_location_performance_report_daily",
"user_location_performance_report_weekly",
"user_location_performance_report_monthly",
],
)
def test_get_report_record_timestamp_daily(stream_report_daily_cls):
stream_report = stream_report_daily_cls(client=Mock(), config=TEST_CONFIG)
assert "2020-01-01" == stream_report.get_report_record_timestamp("2020-01-01")
def test_get_updated_state_new_state_daily_weekly_monthly(stream_name, mock_auth_token, mock_user_query, mock_account_query, requests_mock):
requests_mock.post(
"https://reporting.api.bingads.microsoft.com/Reporting/v13/GenerateReport/Submit",
status_code=200,
json={"ReportRequestId": "thisisthereport_requestid"},
)
requests_mock.post(
"https://reporting.api.bingads.microsoft.com/Reporting/v13/GenerateReport/Poll",
status_code=200,
json={
"ReportRequestStatus": {
"Status": "Success",
"ReportDownloadUrl": "https://bingadsappsstorageprod.blob.core.windows.net:443/download-url",
}
},
)
requests_mock.get(
"https://bingadsappsstorageprod.blob.core.windows.net:443/download-url",
status_code=200,
content=create_zip_from_csv("report_base_data"),
)
stream_state = {"1": {"TimePeriod": "2020-01-01"}}
state = StateBuilder().with_stream_state(stream_name, stream_state).build()
catalog = CatalogBuilder().with_stream(stream_name, SyncMode.full_refresh).build()
source = get_source(TEST_CONFIG, state)
output = read(source, TEST_CONFIG, catalog, state)
updated_state = output.most_recent_state.stream_state.state["TimePeriod"]
assert updated_state == "2023-01-01"
def test_get_report_record_timestamp_without_aggregation(config, mock_user_query, mock_auth_token):
@@ -131,63 +117,52 @@ def test_get_report_record_timestamp_without_aggregation(config, mock_user_query
assert transformed_record["Date"] == expected_record["Date"]
@freeze_time("2024-01-01")
@pytest.mark.parametrize(
"stream_report_hourly_cls",
"stream_name",
(
AccountPerformanceReportHourly,
AdPerformanceReportHourly,
"ad_performance_report_hourly",
"age_gender_audience_report_hourly",
"account_impression_performance_report_hourly",
"account_performance_report_hourly",
"audience_performance_report_hourly",
"keyword_performance_report_hourly",
"ad_group_performance_report_hourly",
"ad_group_impression_performance_report_hourly",
"campaign_performance_report_hourly",
"campaign_impression_performance_report_hourly",
"geographic_performance_report_hourly",
"goals_and_funnels_report_hourly",
"product_dimension_performance_report_hourly",
"product_search_query_performance_report_hourly",
"search_query_performance_report_hourly",
"user_location_performance_report_hourly",
),
)
def test_get_report_record_timestamp_hourly(stream_report_hourly_cls):
stream_report = stream_report_hourly_cls(client=Mock(), config=TEST_CONFIG)
assert "2020-01-01T15:00:00+00:00" == stream_report.get_report_record_timestamp("2020-01-01|15")
def test_report_parse_response_csv_error(caplog):
stream_report = AccountPerformanceReportHourly(client=Mock(), config=TEST_CONFIG)
fake_response = MagicMock()
fake_response.report_records.__iter__ = MagicMock(side_effect=_csv.Error)
list(stream_report.parse_response(fake_response))
assert (
"CSV report file for stream `account_performance_report_hourly` is broken or cannot be read correctly: , skipping ..."
in caplog.messages
def test_get_report_record_timestamp_hourly(stream_name, mock_auth_token, mock_user_query, mock_account_query, requests_mock):
requests_mock.post(
"https://reporting.api.bingads.microsoft.com/Reporting/v13/GenerateReport/Submit",
status_code=200,
json={"ReportRequestId": "thisisthereport_requestid"},
)
@patch.object(source_bing_ads.source, "Client")
def test_account_performance_report_monthly_stream_slices(mocked_client, config_without_start_date):
mocked_client.reports_start_date = None
account_performance_report_monthly = AccountPerformanceReportMonthly(mocked_client, config_without_start_date)
accounts_read_records = iter([{"Id": 180519267, "ParentCustomerId": 100}, {"Id": 180278106, "ParentCustomerId": 200}])
with patch.object(Accounts, "read_records", return_value=accounts_read_records):
stream_slice = list(account_performance_report_monthly.stream_slices(sync_mode=SyncMode.full_refresh))
assert stream_slice == [
{"account_id": 180519267, "customer_id": 100, "time_period": "LastYear"},
{"account_id": 180519267, "customer_id": 100, "time_period": "ThisYear"},
{"account_id": 180278106, "customer_id": 200, "time_period": "LastYear"},
{"account_id": 180278106, "customer_id": 200, "time_period": "ThisYear"},
]
@patch.object(source_bing_ads.source, "Client")
def test_account_performance_report_monthly_stream_slices_no_time_period(mocked_client, config):
account_performance_report_monthly = AccountPerformanceReportMonthly(mocked_client, config)
accounts_read_records = iter([{"Id": 180519267, "ParentCustomerId": 100}, {"Id": 180278106, "ParentCustomerId": 200}])
with patch.object(Accounts, "read_records", return_value=accounts_read_records):
stream_slice = list(account_performance_report_monthly.stream_slices(sync_mode=SyncMode.full_refresh))
assert stream_slice == [{"account_id": 180519267, "customer_id": 100}, {"account_id": 180278106, "customer_id": 200}]
@pytest.mark.parametrize(
"stream, response, records",
[
(AccountPerformanceReportHourly, "hourly_reports/account_performance.csv", "hourly_reports/account_performance_records.json"),
(AdPerformanceReportHourly, "hourly_reports/ad_performance.csv", "hourly_reports/ad_performance_records.json"),
],
)
@patch.object(source_bing_ads.source, "Client")
def test_hourly_reports(mocked_client, config, stream, response, records):
stream_object = stream(mocked_client, config)
with patch.object(stream, "send_request", return_value=_RowReport(file=Path(__file__).parent / response)):
with open(Path(__file__).parent / records, "r") as file:
assert list(stream_object.read_records(sync_mode=SyncMode.full_refresh, stream_slice={}, stream_state={})) == json.load(file)
requests_mock.post(
"https://reporting.api.bingads.microsoft.com/Reporting/v13/GenerateReport/Poll",
status_code=200,
json={
"ReportRequestStatus": {
"Status": "Success",
"ReportDownloadUrl": "https://bingadsappsstorageprod.blob.core.windows.net:443/download-url",
}
},
)
requests_mock.get(
"https://bingadsappsstorageprod.blob.core.windows.net:443/download-url",
status_code=200,
content=create_zip_from_csv("report_base_data_hourly"),
)
stream_state = {"1": {"TimePeriod": "2020-01-01T10:00:00+00:00"}}
state = StateBuilder().with_stream_state(stream_name, stream_state).build()
catalog = CatalogBuilder().with_stream(stream_name, SyncMode.full_refresh).build()
source = get_source(TEST_CONFIG, state)
output = read(source, TEST_CONFIG, catalog, state)
assert "2023-01-01T15:00:00+00:00" == output.records[0].record.data["TimePeriod"]

237
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_source.py
View File

@@ -2,232 +2,43 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from unittest.mock import MagicMock, patch
import pytest
import source_bing_ads
from bingads.service_info import SERVICE_INFO_DICT_V13
from conftest import find_stream
from helpers import source
from source_bing_ads.base_streams import Accounts, Campaigns
from airbyte_cdk.models import SyncMode
from airbyte_cdk.utils import AirbyteTracedException
from conftest import get_source
@patch.object(source_bing_ads.source, "Client")
def test_streams_config_based(mocked_client, config):
streams = source(config).streams(config)
def test_streams_config_based(config):
streams = get_source(config).streams(config)
assert len(streams) == 77
@patch.object(source_bing_ads.source, "Client")
def test_source_check_connection_ok(mocked_client, config, logger_mock):
with patch.object(Accounts, "read_records", return_value=iter([{"Id": 180519267}, {"Id": 180278106}])):
assert source(config).check_connection(logger_mock, config=config) == (True, None)
def test_source_check_connection_ok(config, logger_mock, mock_auth_token, mock_user_query, mock_account_query):
source = get_source(config)
assert source.check_connection(logger_mock, config=config) == (True, None)
@patch.object(source_bing_ads.source, "Client")
def test_source_check_connection_failed_user_do_not_have_accounts(mocked_client, config, logger_mock):
with patch.object(Accounts, "read_records", return_value=[]):
connected, reason = source(config).check_connection(logger_mock, config=config)
assert connected is False
assert (
reason.message == "Config validation error: You don't have accounts assigned to this user. Please verify your developer token."
)
def test_source_check_connection_failed_invalid_creds(config, logger_mock):
with patch.object(Accounts, "read_records", return_value=[]):
connected, reason = source(config).check_connection(logger_mock, config=config)
assert connected is False
@patch.object(source_bing_ads.source, "Client")
def test_campaigns_request_params(mocked_client, config):
campaigns = Campaigns(mocked_client, config)
request_params = campaigns.request_params(stream_slice={"account_id": "account_id"})
assert request_params == {
"AccountId": "account_id",
"CampaignType": "Audience DynamicSearchAds Search Shopping PerformanceMax",
"ReturnAdditionalFields": "AdScheduleUseSearcherTimeZone BidStrategyId CpvCpmBiddingScheme DynamicDescriptionSetting DynamicFeedSetting MaxConversionValueBiddingScheme MultimediaAdsBidAdjustment TargetImpressionShareBiddingScheme TargetSetting VerifiedTrackingSetting",
}
@patch.object(source_bing_ads.source, "Client")
def test_campaigns_stream_slices(mocked_client, config):
campaigns = Campaigns(mocked_client, config)
accounts_read_records = iter([{"Id": 180519267, "ParentCustomerId": 100}, {"Id": 180278106, "ParentCustomerId": 200}])
with patch.object(Accounts, "read_records", return_value=accounts_read_records):
slices = campaigns.stream_slices()
assert list(slices) == [
{"account_id": 180519267, "customer_id": 100},
{"account_id": 180278106, "customer_id": 200},
]
def test_adgroups_stream_slices(mock_auth_token, mock_user_query, mock_account_query, requests_mock, config):
def test_source_check_connection_ok_but_user_do_not_have_accounts(config, logger_mock, mock_auth_token, mock_user_query, requests_mock):
requests_mock.post(
"https://campaign.api.bingads.microsoft.com/CampaignManagement/v13/Campaigns/QueryByAccountId",
"https://clientcenter.api.bingads.microsoft.com/CustomerManagement/v13/Accounts/Search",
status_code=200,
json={
"Campaigns": [
{"Id": 1, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 2, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 3, "LastModifiedTime": "2022-02-02T22:22:22"},
]
},
json={"Accounts": []},
)
ad_groups = find_stream("ad_groups", config)
stream_slices = list(ad_groups.retriever.stream_slicer.stream_slices())
assert stream_slices == [
{"campaign_id": [1], "parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}]},
{"campaign_id": [2], "parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}]},
{"campaign_id": [3], "parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}]},
]
source = get_source(config)
connected, reason = source.check_connection(logger_mock, config=config)
assert connected is True
def test_ads_request_body_data(mock_auth_token, config):
ads = find_stream("ads", config)
stream_slice = {
"campaign_id": [1],
"parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "AccountName", "user_id": 1, "parent_slice": {}}}],
}
request_params = ads.retriever.requester.get_request_body_json(stream_slice=stream_slice)
assert request_params == {
"AdTypes": ["Text", "Image", "Product", "AppInstall", "ExpandedText", "DynamicSearch", "ResponsiveAd", "ResponsiveSearch"],
"ReturnAdditionalFields": "ImpressionTrackingUrls,Videos,LongHeadlines",
}
def test_ads_stream_slices(mock_auth_token, mock_user_query, mock_account_query, requests_mock, config):
def test_source_check_connection_failed_invalid_creds(config, logger_mock, mock_auth_token, mock_user_query, requests_mock):
requests_mock.post(
"https://campaign.api.bingads.microsoft.com/CampaignManagement/v13/Campaigns/QueryByAccountId",
status_code=200,
json={
"Campaigns": [
{"Id": 1, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 2, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 3, "LastModifiedTime": "2022-02-02T22:22:22"},
]
},
"https://clientcenter.api.bingads.microsoft.com/CustomerManagement/v13/Accounts/Search",
status_code=401,
json={"error": "invalid credentials"},
)
requests_mock.post(
"https://campaign.api.bingads.microsoft.com/CampaignManagement/v13/AdGroups/QueryByCampaignId",
status_code=200,
json={
"AdGroups": [
{"Id": 1, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 2, "LastModifiedTime": "2022-02-02T22:22:22"},
{"Id": 3, "LastModifiedTime": "2022-02-02T22:22:22"},
]
},
)
ads = find_stream("ads", config)
stream_slices = list(ads.retriever.stream_slicer.stream_slices())
assert stream_slices == [
{
"ad_group_id": [1],
"parent_slice": [
{
"campaign_id": [1],
"parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}],
}
],
},
{
"ad_group_id": [2],
"parent_slice": [
{
"campaign_id": [1],
"parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}],
}
],
},
{
"ad_group_id": [3],
"parent_slice": [
{
"campaign_id": [1],
"parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}],
}
],
},
]
source = get_source(config)
connected, reason = source.check_connection(logger_mock, config=config)
assert connected is False
@pytest.mark.parametrize(
("stream", "stream_slice"),
(
(
Accounts,
{
"predicates": {
"Predicate": [
{"Field": "UserId", "Operator": "Equals", "Value": "131313131"},
]
}
},
),
(Campaigns, {"account_id": "account_id"}),
),
)
@patch.object(source_bing_ads.source, "Client")
def test_streams_full_refresh(mocked_client, config, stream, stream_slice):
stream_instance = stream(mocked_client, config)
_ = list(stream_instance.read_records(SyncMode.full_refresh, stream_slice))
mocked_client.request.assert_called_once()
def test_transform(mock_auth_token, config):
record = {"AdFormatPreference": "All", "DevicePreference": 0, "EditorialStatus": "ActiveLimited", "FinalAppUrls": None}
ads_stream = find_stream("ads", config)
expected_record = {
"AccountId": 909090,
"AdFormatPreference": "All",
"AdGroupId": 90909090,
"CustomerId": 9090909,
"Descriptions": None,
"DevicePreference": 0,
"EditorialStatus": "ActiveLimited",
"FinalAppUrls": None,
"FinalMobileUrls": None,
"FinalUrls": None,
"ForwardCompatibilityMap": None,
"Headlines": None,
"Images": None,
"LongHeadlines": None,
"Path1": None,
"Path2": None,
"TextPart2": None,
"TitlePart3": None,
"Videos": None,
}
transformed_record = list(
ads_stream.retriever.record_selector.filter_and_transform(
all_data=[
record,
],
stream_state=None,
records_schema={},
stream_slice={
"ad_group_id": [90909090],
"extra_fields": {"AccountId": [909090], "CustomerId": [9090909]},
"parent_slice": [
{
"campaign_id": [1],
"parent_slice": [{"account_id": 1, "parent_slice": {"account_name": "", "user_id": 1, "parent_slice": {}}}],
}
],
},
)
)[0]
assert dict(sorted(transformed_record.items())) == dict(sorted(expected_record.items()))
@patch.object(source_bing_ads.source, "Client")
def test_check_connection_with_accounts_names_config(mocked_client, config_with_account_names, logger_mock):
with patch.object(Accounts, "read_records", return_value=iter([{"Id": 180519267}, {"Id": 180278106}])):
assert source(config=config_with_account_names).check_connection(logger_mock, config=config_with_account_names) == (True, None)
def test_check_connection_with_accounts_names_config(
config_with_account_names, logger_mock, mock_auth_token, mock_user_query, mock_account_query
):
source = get_source(config_with_account_names)
assert source.check_connection(logger_mock, config=config_with_account_names) == (True, None)

1
docs/integrations/sources/bing-ads.md
View File

@@ -262,6 +262,7 @@ The Bing Ads API limits the number of requests for all Microsoft Advertising cli
| Version | Date | Pull Request | Subject |
|:------------|:-----------|:---------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.23.0-rc.1 | 2025-07-07 | [62520](https://github.com/airbytehq/airbyte/pull/62520) | Migrate Source Bing Ads to manifest-only |
| 2.22.0 | 2025-07-02 | [62087](https://github.com/airbytehq/airbyte/pull/62087) | Migrate custom report streams to manifest |
| 2.21.0 | 2025-06-26 | [62079](https://github.com/airbytehq/airbyte/pull/62079) | Migrate `search_query_performance_report` and `product_search_query_performance_report` streams to manifest |
| 2.20.0 | 2025-06-26 | [62060](https://github.com/airbytehq/airbyte/pull/62060) | Migrate `budget_summary_report` stream to manifest, refactor `ad_performance_report` and `account_performance_report` to use dynamic definition |