jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

Source Iterable: update to use latest version of CDK (#3378)

Browse Source 
* Requirements updated to CDK.
airbyte-protocol and base-python requirements removed.

* Bugfix: BufferedStreamConsumer. (#3387)

* Format.

* Bump versions.

* main_dev.py renamed to main.py
README.md updated

* Source Stripe: Add Acceptance Tests to Stripe Connector (#3367)

* Add Acceptance Tests to Stripe Connector

* move configured_catalog.json to sample_files

* bump version

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>

* Legacy lib references removed

* FB Marketing source - lookback window logic not functioning correctly

* FB Marketing source #1390 - returning buffered record while incremental sync

* FB Marketing source #1390 - improving checking while syncing buffered record

* FB Marketing source #1390 - adding loop_back to IncrementalStreamAPI

* FB Marketing source #1390 - bump version

* FB Marketing source #1390 - add CHANGELOG.md

* Stop formatting python with spotless (#3388)

* add test that migration output schema same as source schema (#3356)

* Add updated architecture diagram to high level docs. (#3399)

* Add updated architecture doc to high level docs.

* Address review comments

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Correct GA readme error. (#3407)

* make shopify more resilient to timeouts (#3409)

* Update migration schema to include recent changes to the StandardSync object. (#3414)

* Update all of Pydantic to 1.6.2 per Dependabot. (#3408)

* Update all to 1.6.2.

* Publish new airbyte-cdk version.

* Use repr instead of str for exceptions.

* Use rc.

* Edit test.

* Bump for SAT.

* Format.

* Docker ignore update. Fix setup.py

* fixing ONLY problematic fields in freshdesk JSON schemas (#3376)

* bump airbyte-webapp version (#2266)

* add configuration for bumping webapp versionn

* set to current version

* Bump version: 0.16.0-alpha → 0.16.1-alpha

* Revert "Bump version: 0.16.0-alpha → 0.16.1-alpha"

Thiss reverts commit fdbf6dcaa6.

* also update package lock so we don't run into files changed errors

* use 0.19.0-alpha

* add npm webapp version

* Add a CDK speedrun tutorial doc (#3403)

* Add CDK Speedrun document.

* Finish speedrun doc.

* Address review comments

* Add to SUMMARY.md

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Add Rust as a connector specific dependency to source-file (#3426)

* Add Rust as a connector specific dependency to source-file

* Add more details about installation.

* Markdown lines are weird.

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* API update to latest airbyte-cdk version

* Add section Deploy Local on Windows (#3425)

* add deploy on windows steps

* correct minor

* change suggestions by @avaidyanatha

* GitBook: [master] 161 pages and 75 assets modified

* Display icons (#3140)

* Display icons

* Improve icons views

* MS SQL Server Destination implementation

Fixes issue #613.

Normalization is not yet enabled.  This will have to be added at a later point.

* Workflow to handle operations (custom transformation) (#3379)

* Keep normalization backward compatible with old settings from destination

* Bumpversion normalization image

* add npm install before all npm run generates' (#3442)

* restart containers if they fail automatically (#3423)

* Update link for contribution scheduling (#3443)

* Address issue with icon in onboarding (#3437)

* rename toy connector tutorial to "Build a connector the hard way"  (#3421)

* Upload test reports (from integration test slash commands) as GitHub artifacts (#3416)

* Archive test reports in github workflow

* Archive Test reports only when failures

* Fixing SqlServerOperations.java (#3454)

Fixing some issues with `SqlServerOperations`, which was out of sync with recent changes to `SqlOperations`.

* Add redirect to cdk tutorial page (#3456)

* add redirect to cdk tutorial page

* change path to cdk README.md

Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Yevhenii <34103125+yevhenii-ldv@users.noreply.github.com>
Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: vitaliizazmic <75620293+vitaliizazmic@users.noreply.github.com>
Co-authored-by: Charles <giardina.charles@gmail.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
Co-authored-by: Jared Rhizor <jared@dataline.io>
Co-authored-by: vovavovavovavova <39351371+vovavovavovavova@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcos@airbyte.io>
Co-authored-by: Artem Astapenko <3767150+Jamakase@users.noreply.github.com>
Co-authored-by: masonwheeler <masonwheeler@yahoo.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Michel Tricot <michel@dataline.io>
This commit is contained in:
Vadym
2021-05-18 18:26:58 +03:00
committed by GitHub
parent 0c11b27cf6
commit 588553e647
9 changed files with 32 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
airbyte-integrations/connectors/source-iterable/source_iterable/api.py
View File

@@ -30,8 +30,8 @@ from typing import Any, Iterable, Mapping, MutableMapping, Optional, Union
import pendulum
import requests
from airbyte_protocol import ConfiguredAirbyteStream
from base_python import HttpStream
from airbyte_cdk.models import SyncMode
from airbyte_cdk.sources.streams.http import HttpStream
class IterableStream(HttpStream, ABC):
@@ -39,6 +39,7 @@ class IterableStream(HttpStream, ABC):
# Hardcode the value because it is not returned from the API
BACKOFF_TIME_CONSTANT = 10.0
primary_key = "id"
def __init__(self, api_key, **kwargs):
super().__init__(**kwargs)
@@ -128,8 +129,13 @@ class ListUsers(IterableStream):
data_field = "getUsers"
name = "list_users"
def path(self, parent_stream_record, **kwargs) -> str:
return f"lists/{self.data_field}?listId={parent_stream_record['id']}"
def stream_slices(self, **kwargs) -> Iterable[Optional[Mapping[str, any]]]:
lists = Lists(api_key=self._api_key)
for list_record in lists.read_records(sync_mode=kwargs.get("sync_mode", SyncMode.full_refresh)):
yield {"list_id": list_record["id"]}
def path(self, stream_slice: Optional[Mapping[str, Any]] = None, **kwargs) -> str:
return f"lists/{self.data_field}?listId={stream_slice['list_id']}"
def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
list_id = self._get_list_id(response.url)
@@ -222,13 +228,11 @@ class Templates(IterableExportStream):
def path(self, **kwargs) -> str:
return "templates"
def read_stream(
self, configured_stream: ConfiguredAirbyteStream, stream_state: Mapping[str, Any] = None
) -> Iterable[Mapping[str, Any]]:
def read_records(self, stream_slice: Optional[Mapping[str, Any]] = None, **kwargs) -> Iterable[Mapping[str, Any]]:
for template in self.template_types:
for message in self.message_types:
self.stream_params = {"templateType": template, "messageMedium": message}
yield from super().read_stream(configured_stream=configured_stream, stream_state=stream_state)
yield from super().read_records(stream_slice=stream_slice, **kwargs)
def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
response_json = response.json()

13
airbyte-integrations/connectors/source-iterable/source_iterable/source.py
View File

@@ -25,7 +25,9 @@
from typing import Any, List, Mapping, Tuple
from base_python import AbstractSource, Stream
from airbyte_cdk.models import SyncMode
from airbyte_cdk.sources import AbstractSource
from airbyte_cdk.sources.streams import Stream
from .api import (
Campaigns,
@@ -50,14 +52,13 @@ from .api import (
class SourceIterable(AbstractSource):
def check_connection(self, logger, config) -> Tuple[bool, any]:
try:
list(Lists(api_key=config["api_key"])._list_records(stream_state={}))
list_gen = Lists(api_key=config["api_key"]).read_records(sync_mode=SyncMode.full_refresh)
next(list_gen)
return True, None
except Exception as e:
return False, f"Unable to connect to Iterable API with the provided credentials - {e}"
def streams(self, config: Mapping[str, Any]) -> List[Stream]:
lists = Lists(api_key=config["api_key"])
return [
Campaigns(api_key=config["api_key"]),
Channels(api_key=config["api_key"]),
@@ -69,8 +70,8 @@ class SourceIterable(AbstractSource):
EmailSendSkip(api_key=config["api_key"], start_date=config["start_date"]),
EmailSubscribe(api_key=config["api_key"], start_date=config["start_date"]),
EmailUnsubscribe(api_key=config["api_key"], start_date=config["start_date"]),
lists,
ListUsers(api_key=config["api_key"], parent_stream=lists),
Lists(api_key=config["api_key"]),
ListUsers(api_key=config["api_key"]),
MessageTypes(api_key=config["api_key"]),
Metadata(api_key=config["api_key"]),
Templates(api_key=config["api_key"], start_date=config["start_date"]),