jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-23 21:03:15 -05:00
Code Issues Packages Projects Releases Wiki Activity

✨Source My Hours: Migrate Python CDK to Low-code CDK (#36947)

Browse Source 
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
This commit is contained in:
btkcodedev
2024-04-16 06:09:15 +05:30
committed by GitHub
parent 466abc2968
commit 653113bd3f
36 changed files with 1909 additions and 1267 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
airbyte-integrations/connectors/source-my-hours/.dockerignore
View File

@@ -1,6 +0,0 @@
*
!Dockerfile
!main.py
!source_my_hours
!setup.py
!secrets

38
airbyte-integrations/connectors/source-my-hours/Dockerfile
View File

@@ -1,38 +0,0 @@
FROM python:3.9.11-alpine3.15 as base
# build and load all requirements
FROM base as builder
WORKDIR /airbyte/integration_code
# upgrade pip to the latest version
RUN apk --no-cache upgrade \
&& pip install --upgrade pip \
&& apk --no-cache add tzdata build-base
COPY setup.py ./
# install necessary packages to a temporary folder
RUN pip install --prefix=/install .
# build a clean environment
FROM base
WORKDIR /airbyte/integration_code
# copy all loaded and built libraries to a pure basic image
COPY --from=builder /install /usr/local
# add default timezone settings
COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
RUN echo "Etc/UTC" > /etc/timezone
# bash is installed for more convenient debugging.
RUN apk --no-cache add bash
# copy payload code only
COPY main.py ./
COPY source_my_hours ./source_my_hours
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
LABEL io.airbyte.version=0.1.2
LABEL io.airbyte.name=airbyte/source-my-hours

103
airbyte-integrations/connectors/source-my-hours/README.md
View File

@@ -1,69 +1,63 @@
# My Hours Source
This is the repository for the My Hours source connector, written in Python.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/my-hours).
This is the repository for the My Hours configuration based source connector.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/my-hours).
## Local development
### Prerequisites
**To iterate on this connector, make sure to complete this prerequisites section.**
#### Minimum Python version required `= 3.7.0`
* Python (`^3.9`)
* Poetry (`^1.7`) - installation instructions [here](https://python-poetry.org/docs/#installation)
#### Build & Activate Virtual Environment and install dependencies
From this connector directory, create a virtual environment:
```
python -m venv .venv
### Installing the connector
From this connector directory, run:
```bash
poetry install --with dev
```
This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
development environment of choice. To activate it from the terminal, run:
```
source .venv/bin/activate
pip install -r requirements.txt
pip install '.[tests]'
```
If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
should work as you expect.
### Create credentials
#### Create credentials
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/my-hours)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_my_hours/spec.json` file.
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/my-hours)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `spec` inside `manifest/yaml` file.
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
See `integration_tests/sample_config.json` for a sample config file.
See `sample_files/sample_config.json` for a sample config file.
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source my-hours test creds`
and place them into `secrets/config.json`.
### Locally running the connector
```
python main.py spec
python main.py check --config secrets/config.json
python main.py discover --config secrets/config.json
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
poetry run source-my-hours spec
poetry run source-my-hours check --config secrets/config.json
poetry run source-my-hours discover --config secrets/config.json
poetry run source-my-hours read --config secrets/config.json --catalog sample_files/configured_catalog.json
```
### Locally running the connector docker image
### Running tests
To run tests locally, from the connector directory run:
#### Build
**Via [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) (recommended):**
```
poetry run pytest tests
```
### Building the docker image
1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
2. Run the following command to build the docker image:
```bash
airbyte-ci connectors --name=source-my-hours build
```
An image will be built with the tag `airbyte/source-my-hours:dev`.
An image will be available on your host with the tag `airbyte/source-my-hours:dev`.
**Via `docker build`:**
```bash
docker build -t airbyte/source-my-hours:dev .
```
#### Run
### Running as a docker container
Then run any of the connector commands as follows:
```
docker run --rm airbyte/source-my-hours:dev spec
@@ -72,29 +66,38 @@ docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-my-hours:dev discover
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-my-hours:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```
## Testing
### Running our CI test suite
You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
```bash
airbyte-ci connectors --name=source-my-hours test
```
### Customizing acceptance Tests
Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
## Dependency Management
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
We split dependencies between two groups, dependencies that are:
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
* required for the testing need to go to `TEST_REQUIREMENTS` list
### Dependency Management
All of your dependencies should be managed via Poetry.
To add a new dependency, run:
```bash
poetry add <package-name>
```
Please commit the changes to `pyproject.toml` and `poetry.lock` files.
## Publishing a new version of the connector
### Publishing a new version of the connector
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-my-hours test`
2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
- bump the `dockerImageTag` value in in `metadata.yaml`
- bump the `version` value in `pyproject.toml`
3. Make sure the `metadata.yaml` content is up to date.
4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/my-hours.md`).
4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/my-hours.md`).
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
6. Pat yourself on the back for being an awesome contributor.
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.

3
airbyte-integrations/connectors/source-my-hours/__init__.py Normal file
View File

@@ -0,0 +1,3 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#

20
airbyte-integrations/connectors/source-my-hours/acceptance-test-config.yml
View File

@@ -1,10 +1,11 @@
# See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference)
# for more information about how to configure these tests
connector_image: airbyte/source-my-hours:dev
test_strictness_level: low
acceptance_tests:
spec:
tests:
- spec_path: "source_my_hours/spec.json"
- spec_path: "source_my_hours/spec.yaml"
connection:
tests:
- config_path: "secrets/config.json"
@@ -18,10 +19,19 @@ acceptance_tests:
tests:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
# time_logs stream contains a number of empty fields that are not
# documented in the API. Until we can verify the types on these fields,
# we need to disable this check.
fail_on_extra_columns: false
empty_streams: []
# TODO uncomment this block to specify that the tests should assert the connector outputs the records provided in the input file a file
# expect_records:
# path: "integration_tests/expected_records.jsonl"
# exact_order: no
incremental:
bypass_reason: "This connector does not implement incremental sync"
# TODO uncomment this block this block if your connector implements incremental sync:
# tests:
# - config_path: "secrets/config.json"
# configured_catalog_path: "integration_tests/configured_catalog.json"
# future_state:
# future_state_path: "integration_tests/abnormal_state.json"
full_refresh:
tests:
- config_path: "secrets/config.json"

2
airbyte-integrations/connectors/source-my-hours/integration_tests/__init__.py
View File

@@ -1,3 +1,3 @@
#
# Copyright (c) 2021 Airbyte, Inc., all rights reserved.
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#

5
airbyte-integrations/connectors/source-my-hours/integration_tests/abnormal_state.json Normal file
View File

@@ -0,0 +1,5 @@
{
"todo-stream-name": {
"todo-field-name": "todo-abnormal-value"
}
}

2
airbyte-integrations/connectors/source-my-hours/integration_tests/acceptance.py
View File

@@ -11,4 +11,6 @@ pytest_plugins = ("connector_acceptance_test.plugin",)
@pytest.fixture(scope="session", autouse=True)
def connector_setup():
"""This fixture is a placeholder for external resources that acceptance test might require."""
# TODO: setup test dependencies if needed. otherwise remove the TODO comments
yield
# TODO: clean up test dependencies

332
airbyte-integrations/connectors/source-my-hours/integration_tests/catalog.json
View File

@@ -1,332 +0,0 @@
{
"streams": [
{
"name": "clients",
"json_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"id": {
"type": ["number"]
},
"contactName": {
"type": ["null", "string"]
},
"contactEmail": {
"type": ["null", "string"]
},
"name": {
"type": ["null", "string"]
},
"archived": {
"type": ["null", "boolean"]
},
"dateArchived": {
"type": ["null", "string"]
}
}
},
"supported_sync_modes": ["full_refresh"],
"source_defined_primary_key": [["id"]]
},
{
"name": "projects",
"json_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"id": {
"type": ["number"]
},
"name": {
"type": ["null", "string"]
},
"archived": {
"type": ["null", "boolean"]
},
"dateArchived": {
"type": ["null", "string"]
},
"dateCreated": {
"type": ["null", "string"]
},
"clientName": {
"type": ["null", "string"]
},
"budgetAlertPercent": {
"type": ["null", "number"]
},
"budgetType": {
"type": ["null", "number"]
},
"totalTimeLogged": {
"type": ["null", "number"]
},
"budgetValue": {
"type": ["null", "number"]
},
"totalAmount": {
"type": ["null", "number"]
},
"totalExpense": {
"type": ["null", "number"]
},
"totalCost": {
"type": ["null", "number"]
},
"billableTimeLogged": {
"type": ["null", "number"]
},
"totalBillableAmount": {
"type": ["null", "number"]
},
"billable": {
"type": ["null", "boolean"]
},
"roundType": {
"type": ["null", "number"]
},
"roundInterval": {
"type": ["null", "number"]
},
"budgetSpentPercentage": {
"type": ["null", "number"]
},
"budgetTarget": {
"type": ["null", "number"]
},
"budgetPeriodType": {
"type": ["null", "number"]
},
"budgetSpent": {
"type": ["null", "number"]
}
}
},
"supported_sync_modes": ["full_refresh"],
"source_defined_primary_key": [["id"]]
},
{
"name": "tags",
"json_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"id": {
"type": ["number"]
},
"name": {
"type": ["null", "string"]
},
"hexColor": {
"type": ["null", "string"]
},
"archived": {
"type": ["null", "boolean"]
},
"dateArchived": {
"type": ["null", "string"]
}
}
},
"supported_sync_modes": ["full_refresh"],
"source_defined_primary_key": [["id"]]
},
{
"name": "time_logs",
"json_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"logId": {
"type": ["number"]
},
"userId": {
"type": ["null", "number"]
},
"date": {
"type": ["null", "string"]
},
"userName": {
"type": ["null", "string"]
},
"clientId": {
"type": ["null", "number"]
},
"clientName": {
"type": ["null", "string"]
},
"projectId": {
"type": ["null", "number"]
},
"projectName": {
"type": ["null", "string"]
},
"projectInvoiceMethod": {
"type": ["null", "number"]
},
"taskId": {
"type": ["null", "number"]
},
"taskName": {
"type": ["null", "string"]
},
"tags": {
"type": ["null", "string"]
},
"rate": {
"type": ["null", "number"]
},
"billable": {
"type": ["null", "boolean"]
},
"locked": {
"type": ["null", "boolean"]
},
"billableAmount": {
"type": ["null", "number"]
},
"amount": {
"type": ["null", "number"]
},
"laborCost": {
"type": ["null", "number"]
},
"laborRate": {
"type": ["null", "number"]
},
"laborDuration": {
"type": ["null", "number"]
},
"logduration": {
"type": ["null", "number"]
},
"expense": {
"type": ["null", "number"]
},
"cost": {
"type": ["null", "number"]
},
"note": {
"type": ["null", "string"]
},
"status": {
"type": ["null", "number"]
},
"invoiceId": {
"type": ["null", "number"]
},
"invoiced": {
"type": ["null", "boolean"]
},
"billableHours": {
"type": ["null", "number"]
},
"laborHours": {
"type": ["null", "number"]
},
"customField1": {
"type": ["null", "number"]
},
"customField2": {
"type": ["null", "string"]
},
"customField3": {
"type": ["null", "string"]
},
"monthOfYear": {
"type": ["null", "string"]
},
"weekOfYear": {
"type": ["null", "string"]
},
"times": {
"type": ["null", "object"],
"additionalProperties": false,
"properties": {
"id": {
"type": ["null", "number"]
},
"duration": {
"type": ["null", "number"]
},
"startTime": {
"type": ["null", "string"]
},
"endTime": {
"type": ["null", "string"]
},
"running": {
"type": ["null", "boolean"]
},
"deleted": {
"type": ["null", "boolean"]
}
}
},
"tagsData": {
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"attachments": {
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"roundtype": {
"type": ["null", "number"]
}
}
},
"supported_sync_modes": ["full_refresh"],
"source_defined_primary_key": [["logId"]]
},
{
"name": "users",
"json_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"id": {
"type": ["number"]
},
"name": {
"type": ["null", "string"]
},
"archived": {
"type": ["null", "boolean"]
},
"dateArchived": {
"type": ["null", "string"]
},
"active": {
"type": ["null", "boolean"]
},
"accountOwner": {
"type": ["null", "boolean"]
},
"email": {
"type": ["null", "string"]
},
"rate": {
"type": ["null", "number"]
},
"billableRate": {
"type": ["null", "number"]
},
"admin": {
"type": ["null", "boolean"]
},
"isProjectManager": {
"type": ["null", "boolean"]
}
}
},
"supported_sync_modes": ["full_refresh"],
"source_defined_primary_key": [["id"]]
}
]
}

4
airbyte-integrations/connectors/source-my-hours/integration_tests/invalid_config.json
View File

@@ -1,6 +1,6 @@
{
"email": "john@doe.com",
"password": "pw1234",
"start_date": "2016-01-01",
"logs_batch_size": 30
"logs_batch_size": 30,
"start_date": "2021-12-01"
}

4
airbyte-integrations/connectors/source-my-hours/integration_tests/sample_config.json
View File

@@ -1,6 +1,6 @@
{
"email": "email",
"password": "password",
"start_date": "2016-01-01",
"logs_batch_size": 30
"logs_batch_size": 30,
"start_date": "2021-12-01"
}

5
airbyte-integrations/connectors/source-my-hours/integration_tests/sample_state.json Normal file
View File

@@ -0,0 +1,5 @@
{
"todo-stream-name": {
"todo-field-name": "value"
}
}

31
airbyte-integrations/connectors/source-my-hours/metadata.yaml
View File

@@ -1,13 +1,7 @@
data:
connectorSubtype: api
connectorType: source
definitionId: 722ba4bf-06ec-45a4-8dd5-72e4a5cf3903
dockerImageTag: 0.1.2
dockerRepository: airbyte/source-my-hours
githubIssueLabel: source-my-hours
icon: my-hours.svg
license: MIT
name: My Hours
allowedHosts:
hosts:
- api2.myhours.com
remoteRegistries:
pypi:
enabled: true
@@ -17,13 +11,28 @@ data:
enabled: true
oss:
enabled: true
connectorBuildOptions:
# Please update to the latest version of the connector base image.
# https://hub.docker.com/r/airbyte/python-connector-base
# Please use the full address with sha256 hash to guarantee build reproducibility.
baseImage: docker.io/airbyte/python-connector-base:1.2.0@sha256:c22a9d97464b69d6ef01898edf3f8612dc11614f05a84984451dde195f337db9
connectorSubtype: api
connectorType: source
definitionId: 722ba4bf-06ec-45a4-8dd5-72e4a5cf3903
dockerImageTag: 0.2.0
dockerRepository: airbyte/source-my-hours
githubIssueLabel: source-my-hours
icon: my-hours.svg
license: MIT
name: My Hours
releaseDate: 2021-12-21
releaseStage: alpha
supportLevel: community
documentationUrl: https://docs.airbyte.com/integrations/sources/my-hours
tags:
- language:python
- cdk:python
- cdk:low-code
ab_internal:
sl: 100
ql: 100
supportLevel: community
metadataSpecVersion: "1.0"

1008
airbyte-integrations/connectors/source-my-hours/poetry.lock generated Normal file
View File

File diff suppressed because it is too large Load Diff

27
airbyte-integrations/connectors/source-my-hours/pyproject.toml Normal file
View File

@@ -0,0 +1,27 @@
[build-system]
requires = [ "poetry-core>=1.0.0",]
build-backend = "poetry.core.masonry.api"
[tool.poetry]
version = "0.2.0"
name = "source-my-hours"
description = "Source implementation for my-hours."
authors = [ "Airbyte <contact@airbyte.io>",]
license = "MIT"
readme = "README.md"
documentation = "https://docs.airbyte.com/integrations/sources/my-hours"
homepage = "https://airbyte.com"
repository = "https://github.com/airbytehq/airbyte"
packages = [ { include = "source_my_hours" }, {include = "main.py" } ]
[tool.poetry.dependencies]
python = "^3.9,<3.12"
airbyte-cdk = "^0"
[tool.poetry.scripts]
source-my-hours = "source_my_hours.run:run"
[tool.poetry.group.dev.dependencies]
requests-mock = "*"
pytest-mock = "*"
pytest = "*"

1
airbyte-integrations/connectors/source-my-hours/requirements.txt
View File

@@ -1 +0,0 @@
-e .

47
airbyte-integrations/connectors/source-my-hours/setup.py
View File

@@ -1,47 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from setuptools import find_packages, setup
MAIN_REQUIREMENTS = [
"airbyte-cdk",
]
TEST_REQUIREMENTS = [
"requests-mock~=1.9.3",
"pytest~=6.1",
"pytest-mock~=3.6.1",
"responses~=0.16.0",
]
setup(
entry_points={
"console_scripts": [
"source-my-hours=source_my_hours.run:run",
],
},
name="source_my_hours",
description="Source implementation for My Hours.",
author="Wisse Jelgersma",
author_email="wisse@vrowl.nl",
packages=find_packages(),
install_requires=MAIN_REQUIREMENTS,
package_data={
"": [
# Include yaml files in the package (if any)
"*.yml",
"*.yaml",
# Include all json files in the package, up to 4 levels deep
"*.json",
"*/*.json",
"*/*/*.json",
"*/*/*/*.json",
"*/*/*/*/*.json",
]
},
extras_require={
"tests": TEST_REQUIREMENTS,
},
)

2
airbyte-integrations/connectors/source-my-hours/source_my_hours/__init__.py
View File

@@ -1,5 +1,5 @@
#
# Copyright (c) 2021 Airbyte, Inc., all rights reserved.
# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
#

55
airbyte-integrations/connectors/source-my-hours/source_my_hours/auth.py
View File

@@ -1,55 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import json
from typing import Any, Mapping, MutableMapping, Tuple
import pendulum
import requests
from airbyte_cdk.sources.streams.http.requests_native_auth.oauth import Oauth2Authenticator
from .constants import REQUEST_HEADERS, URL_BASE
class MyHoursAuthenticator(Oauth2Authenticator):
def __init__(self, email: str, password: str):
super().__init__(
token_refresh_endpoint=f"{URL_BASE}/tokens/refresh",
client_id=None,
client_secret=None,
refresh_token=None,
access_token_name="accessToken",
expires_in_name="expiresIn",
)
self.retrieve_refresh_token(email, password)
def retrieve_refresh_token(self, email: str, password: str):
t0 = pendulum.now()
payload = json.dumps({"grantType": "password", "email": email, "password": password, "clientId": "api"})
response = requests.post(f"{URL_BASE}/tokens/login", headers=REQUEST_HEADERS, data=payload)
response.raise_for_status()
json_response = response.json()
self.refresh_token = json_response["refreshToken"]
self._access_token = json_response[self._access_token_name]
self._token_expiry_date = t0.add(seconds=json_response[self._expires_in_name])
def get_refresh_request_body(self) -> Mapping[str, Any]:
payload: MutableMapping[str, Any] = {
"grantType": "refresh_token",
"refreshToken": self.refresh_token,
}
return payload
def refresh_access_token(self) -> Tuple[str, int]:
try:
response = requests.request(method="POST", url=self._token_refresh_endpoint, data=self.get_refresh_request_body())
response.raise_for_status()
response_json = response.json()
self.refresh_token = response_json["refreshToken"]
return response_json[self._access_token_name], response_json[self._expires_in_name]
except Exception as e:
raise Exception(f"Error while refreshing access token: {e}") from e

85
airbyte-integrations/connectors/source-my-hours/source_my_hours/components.py Normal file
View File

@@ -0,0 +1,85 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from dataclasses import dataclass
from http import HTTPStatus
from typing import Any, Mapping, Union
import requests
from airbyte_cdk.sources.declarative.auth.declarative_authenticator import NoAuth
from airbyte_cdk.sources.declarative.interpolation import InterpolatedString
from airbyte_cdk.sources.declarative.types import Config
from requests import HTTPError
# https://docs.airbyte.com/integrations/sources/my-hours
# The Bearer token generated will expire in five days
@dataclass
class CustomAuthenticator(NoAuth):
config: Config
email: Union[InterpolatedString, str]
password: Union[InterpolatedString, str]
_access_token = None
_refreshToken = None
def __post_init__(self, parameters: Mapping[str, Any]):
self._email = InterpolatedString.create(self.email, parameters=parameters).eval(self.config)
self._password = InterpolatedString.create(self.password, parameters=parameters).eval(self.config)
def __call__(self, request: requests.PreparedRequest) -> requests.PreparedRequest:
"""Attach the page access token to params to authenticate on the HTTP request"""
if self._access_token is None or self._refreshToken is None:
self._access_token, self._refreshToken = self.generate_access_token()
headers = {self.auth_header: f"Bearer {self._access_token}", "Accept": "application/json", "api-version": "1.0"}
request.headers.update(headers)
return request
@property
def auth_header(self) -> str:
return "Authorization"
@property
def token(self) -> str:
return self._access_token
def _get_refresh_access_token_response(self):
url = f"https://api2.myhours.com/api/tokens/refresh"
headers = {"Content-Type": "application/json", "api-version": "1.0", self.auth_header: f"Bearer {self._access_token}"}
data = {
"refreshToken": self._refreshToken,
"grantType": "refresh_token",
}
try:
response = requests.post(url, headers=headers, json=data)
response.raise_for_status()
modified_response = {
"access_token": response.json().get("accessToken"),
"refresh_token": response.json().get("refreshToken"),
"expires_in": response.json().get("expiresIn"),
}
return modified_response
except Exception as e:
raise Exception(f"Error while refreshing access token: {e}") from e
def generate_access_token(self) -> tuple[str, str]:
try:
headers = {"Content-Type": "application/json", "api-version": "1.0"}
data = {
"email": self._email,
"password": self._password,
"grantType": "password",
"clientId": "api",
}
url = "https://api2.myhours.com/api/tokens/login"
rest = requests.post(url, headers=headers, json=data)
if rest.status_code != HTTPStatus.OK:
raise HTTPError(rest.text)
return (rest.json().get("accessToken"), rest.json().get("refreshToken"))
except Exception as e:
raise Exception(f"Error while generating access token: {e}") from e

6
airbyte-integrations/connectors/source-my-hours/source_my_hours/constants.py
View File

@@ -1,6 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
URL_BASE = "https://api2.myhours.com/api"
REQUEST_HEADERS = {"accept": "application/json", "api-version": "1.0", "Content-Type": "application/json"}

666
airbyte-integrations/connectors/source-my-hours/source_my_hours/manifest.yaml Normal file
View File

@@ -0,0 +1,666 @@
version: 0.44.0
type: DeclarativeSource
check:
type: CheckStream
stream_names:
- users
streams:
- type: DeclarativeStream
name: users
primary_key:
- id
schema_loader:
type: InlineSchemaLoader
schema:
$ref: "#/definitions/users_schema"
retriever:
type: SimpleRetriever
requester:
type: HttpRequester
url_base: https://api2.myhours.com/api
path: Users/getAll
http_method: GET
authenticator:
$ref: "#/definitions/custom_authenticator"
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path: []
paginator:
type: NoPagination
- type: DeclarativeStream
name: time_logs
primary_key:
- logId
schema_loader:
type: InlineSchemaLoader
schema:
$ref: "#/definitions/time_logs_schema"
retriever:
type: SimpleRetriever
requester:
type: HttpRequester
url_base: https://api2.myhours.com/api
path: Reports/activity
http_method: GET
request_parameters:
DateFrom: "{{ config['start_date'] }}"
DateTo: "{{ today_utc() }}"
authenticator:
$ref: "#/definitions/custom_authenticator"
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path: []
paginator:
type: NoPagination
- type: DeclarativeStream
name: tags
primary_key:
- id
schema_loader:
type: InlineSchemaLoader
schema:
$ref: "#/definitions/tags_schema"
retriever:
type: SimpleRetriever
requester:
type: HttpRequester
url_base: https://api2.myhours.com/api
path: Tags
http_method: GET
authenticator:
$ref: "#/definitions/custom_authenticator"
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path: []
paginator:
type: NoPagination
- type: DeclarativeStream
name: projects
primary_key:
- id
schema_loader:
type: InlineSchemaLoader
schema:
$ref: "#/definitions/projects_schema"
retriever:
type: SimpleRetriever
requester:
type: HttpRequester
url_base: https://api2.myhours.com/api
path: Projects/getAll
http_method: GET
authenticator:
$ref: "#/definitions/custom_authenticator"
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path: []
paginator:
type: NoPagination
- type: DeclarativeStream
name: clients
primary_key:
- id
schema_loader:
type: InlineSchemaLoader
schema:
$ref: "#/definitions/clients_schema"
retriever:
type: SimpleRetriever
requester:
type: HttpRequester
url_base: https://api2.myhours.com/api
path: Clients
http_method: GET
authenticator:
$ref: "#/definitions/custom_authenticator"
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path: []
paginator:
type: NoPagination
definitions:
custom_authenticator:
type: CustomAuthenticator
class_name: source_my_hours.components.CustomAuthenticator
email: "{{ config['email'] }}"
password: "{{ config['password'] }}"
users_schema:
$schema: http://json-schema.org/draft-07/schema#
type: object
additionalProperties: true
properties:
id:
type:
- "null"
- number
name:
type:
- "null"
- string
archived:
type:
- "null"
- boolean
dateArchived:
type:
- "null"
- string
active:
type:
- "null"
- boolean
accountOwner:
type:
- "null"
- boolean
email:
type:
- "null"
- string
rate:
type:
- "null"
- number
billableRate:
type:
- "null"
- number
admin:
type:
- "null"
- boolean
isProjectManager:
type:
- "null"
- boolean
roleType:
type:
- "null"
- integer
customId:
type:
- "null"
- string
clients_schema:
$schema: http://json-schema.org/draft-07/schema#
type: object
additionalProperties: true
properties:
id:
type:
- "null"
- number
customId:
type:
- "null"
- string
contactName:
type:
- "null"
- string
contactEmail:
type:
- "null"
- string
name:
type:
- "null"
- string
archived:
type:
- "null"
- boolean
dateArchived:
type:
- "null"
- string
projects_schema:
$schema: http://json-schema.org/draft-07/schema#
type: object
additionalProperties: true
properties:
id:
type:
- "null"
- number
name:
type:
- "null"
- string
archived:
type:
- "null"
- boolean
dateArchived:
type:
- "null"
- string
dateCreated:
type:
- "null"
- string
clientName:
type:
- "null"
- string
clientId:
type:
- "null"
- integer
customId:
type:
- "null"
- string
budgetAlertPercent:
type:
- "null"
- number
budgetType:
type:
- "null"
- number
laborCost:
type:
- "null"
- number
totalTimeLogged:
type:
- "null"
- number
budgetValue:
type:
- "null"
- number
totalAmount:
type:
- "null"
- number
totalExpense:
type:
- "null"
- number
totalCost:
type:
- "null"
- number
billableTimeLogged:
type:
- "null"
- number
totalBillableAmount:
type:
- "null"
- number
billable:
type:
- "null"
- boolean
roundType:
type:
- "null"
- number
roundInterval:
type:
- "null"
- number
budgetSpentPercentage:
type:
- "null"
- number
budgetTarget:
type:
- "null"
- number
budgetPeriodType:
type:
- "null"
- number
budgetSpent:
type:
- "null"
- number
tags_schema:
$schema: http://json-schema.org/draft-07/schema#
type: object
additionalProperties: true
properties:
id:
type:
- "null"
- number
name:
type:
- "null"
- string
hexColor:
type:
- "null"
- string
archived:
type:
- "null"
- boolean
dateArchived:
type:
- "null"
- string
time_logs_schema:
$schema: http://json-schema.org/draft-07/schema#
type: object
additionalProperties: true
properties:
logId:
type:
- "null"
- number
userId:
type:
- "null"
- number
date:
type:
- "null"
- string
userName:
type:
- "null"
- string
clientId:
type:
- "null"
- number
clientName:
type:
- "null"
- string
projectId:
type:
- "null"
- number
projectName:
type:
- "null"
- string
projectInvoiceMethod:
type:
- "null"
- number
taskId:
type:
- "null"
- number
taskName:
type:
- "null"
- string
tags:
type:
- "null"
- string
rate:
type:
- "null"
- number
billable:
type:
- "null"
- boolean
inLockedPeriod:
type:
- "null"
- boolean
billableAmount:
type:
- "null"
- number
amount:
type:
- "null"
- number
laborCost:
type:
- "null"
- number
laborRate:
type:
- "null"
- number
laborDuration:
type:
- "null"
- number
logDuration:
type:
- "null"
- number
expense:
type:
- "null"
- number
cost:
type:
- "null"
- number
note:
type:
- "null"
- string
status:
type:
- "null"
- number
invoiceId:
type:
- "null"
- number
invoiced:
type:
- "null"
- boolean
billableHours:
type:
- "null"
- number
laborHours:
type:
- "null"
- number
customField1:
type:
- "null"
- number
customField2:
type:
- "null"
- number
customField3:
type:
- "null"
- number
monthOfYear:
type:
- "null"
- string
weekNo:
type:
- "null"
- number
weekOfYear:
type:
- "null"
- string
times:
type:
- "null"
- array
items:
type:
- object
properties:
id:
type:
- "null"
- number
duration:
type:
- "null"
- number
startTime:
type:
- "null"
- string
endTime:
type:
- "null"
- string
running:
type:
- "null"
- boolean
deleted:
type:
- "null"
- boolean
roundtype:
type:
- "null"
- number
attachments:
type:
- "null"
- array
balance:
type:
- "null"
- number
billableExpense:
type:
- "null"
- number
billableHoursLogBillable:
type:
- "null"
- number
clientCustomId:
type:
- "null"
- string
endTime:
type:
- "null"
- string
invoicedAmount:
type:
- "null"
- number
logDurationBillable:
type:
- "null"
- number
startTime:
type:
- "null"
- string
format: date-time
startEndTime:
type:
- "null"
- string
teams:
type:
- "null"
- array
items:
type:
- "null"
- string
teamsNames:
type:
- "null"
- string
taskListName:
type:
- "null"
- string
taskDueDate:
type:
- "null"
- string
taskStartDate:
type:
- "null"
- string
tagsData:
type:
- "null"
- array
uninvoicedAmount:
type:
- "null"
- number
spec:
documentation_url: https://docs.airbyte.com/integrations/sources/my-hours
connection_specification:
$schema: http://json-schema.org/draft-07/schema#
title: My Hours Spec
type: object
required:
- email
- password
- start_date
additionalProperties: true
properties:
email:
title: Email
type: string
description: Your My Hours username
example: john@doe.com
password:
title: Password
type: string
description: The password associated to the username
airbyte_secret: true
start_date:
title: Start Date
description: Start date for collecting time logs
examples:
- "%Y-%m-%d"
- "2016-01-01"
type: string
pattern: "^[0-9]{4}-[0-9]{2}-[0-9]{2}$"
logs_batch_size:
title: Time logs batch size
description: Pagination size used for retrieving logs in days
examples:
- 30
type: integer
minimum: 1
maximum: 365
default: 30
type: Spec
metadata:
autoImportSchema:
users: false
time_logs: false
tags: false
projects: false
clients: false

3
airbyte-integrations/connectors/source-my-hours/source_my_hours/run.py
View File

@@ -6,7 +6,8 @@
import sys
from airbyte_cdk.entrypoint import launch
from source_my_hours import SourceMyHours
from .source import SourceMyHours
def run():

27
airbyte-integrations/connectors/source-my-hours/source_my_hours/schemas/clients.json
View File

@@ -1,27 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"id": {
"type": ["number"]
},
"contactName": {
"type": ["null", "string"]
},
"contactEmail": {
"type": ["null", "string"]
},
"customId": {
"type": ["null", "string"]
},
"name": {
"type": ["null", "string"]
},
"archived": {
"type": ["null", "boolean"]
},
"dateArchived": {
"type": ["null", "string"]
}
}
}

81
airbyte-integrations/connectors/source-my-hours/source_my_hours/schemas/projects.json
View File

@@ -1,81 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"id": {
"type": ["number"]
},
"name": {
"type": ["null", "string"]
},
"archived": {
"type": ["null", "boolean"]
},
"dateArchived": {
"type": ["null", "string"]
},
"dateCreated": {
"type": ["null", "string"]
},
"clientName": {
"type": ["null", "string"]
},
"clientId": {
"type": ["null", "integer"]
},
"customId": {
"type": ["null", "string"]
},
"budgetAlertPercent": {
"type": ["null", "number"]
},
"budgetType": {
"type": ["null", "number"]
},
"laborCost": {
"type": ["null", "number"]
},
"totalTimeLogged": {
"type": ["null", "number"]
},
"budgetValue": {
"type": ["null", "number"]
},
"totalAmount": {
"type": ["null", "number"]
},
"totalExpense": {
"type": ["null", "number"]
},
"totalCost": {
"type": ["null", "number"]
},
"billableTimeLogged": {
"type": ["null", "number"]
},
"totalBillableAmount": {
"type": ["null", "number"]
},
"billable": {
"type": ["null", "boolean"]
},
"roundType": {
"type": ["null", "number"]
},
"roundInterval": {
"type": ["null", "number"]
},
"budgetSpentPercentage": {
"type": ["null", "number"]
},
"budgetTarget": {
"type": ["null", "number"]
},
"budgetPeriodType": {
"type": ["null", "number"]
},
"budgetSpent": {
"type": ["null", "number"]
}
}
}

21
airbyte-integrations/connectors/source-my-hours/source_my_hours/schemas/tags.json
View File

@@ -1,21 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"id": {
"type": ["number"]
},
"name": {
"type": ["null", "string"]
},
"hexColor": {
"type": ["null", "string"]
},
"archived": {
"type": ["null", "boolean"]
},
"dateArchived": {
"type": ["null", "string"]
}
}
}

196
airbyte-integrations/connectors/source-my-hours/source_my_hours/schemas/time_logs.json
View File

@@ -1,196 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"additionalProperties": true,
"properties": {
"logId": {
"type": ["number"]
},
"userId": {
"type": ["null", "number"]
},
"date": {
"type": ["null", "string"]
},
"userName": {
"type": ["null", "string"]
},
"clientId": {
"type": ["null", "number"]
},
"clientName": {
"type": ["null", "string"]
},
"projectId": {
"type": ["null", "number"]
},
"projectName": {
"type": ["null", "string"]
},
"projectInvoiceMethod": {
"type": ["null", "number"]
},
"taskId": {
"type": ["null", "number"]
},
"taskName": {
"type": ["null", "string"]
},
"tags": {
"type": ["null", "string"]
},
"rate": {
"type": ["null", "number"]
},
"billable": {
"type": ["null", "boolean"]
},
"inLockedPeriod": {
"type": ["null", "boolean"]
},
"billableAmount": {
"type": ["null", "number"]
},
"amount": {
"type": ["null", "number"]
},
"laborCost": {
"type": ["null", "number"]
},
"laborRate": {
"type": ["null", "number"]
},
"laborDuration": {
"type": ["null", "number"]
},
"logDuration": {
"type": ["null", "number"]
},
"expense": {
"type": ["null", "number"]
},
"cost": {
"type": ["null", "number"]
},
"note": {
"type": ["null", "string"]
},
"status": {
"type": ["null", "number"]
},
"invoiceId": {
"type": ["null", "number"]
},
"invoiced": {
"type": ["null", "boolean"]
},
"billableHours": {
"type": ["null", "number"]
},
"laborHours": {
"type": ["null", "number"]
},
"customField1": {
"type": ["null", "number"]
},
"customField2": {
"type": ["null", "number"]
},
"customField3": {
"type": ["null", "number"]
},
"monthOfYear": {
"type": ["null", "string"]
},
"weekNo": {
"type": ["null", "number"]
},
"weekOfYear": {
"type": ["null", "string"]
},
"times": {
"type": ["null", "array"],
"items": {
"type": ["object"],
"properties": {
"id": {
"type": ["null", "number"]
},
"duration": {
"type": ["null", "number"]
},
"startTime": {
"type": ["null", "string"]
},
"endTime": {
"type": ["null", "string"]
},
"running": {
"type": ["null", "boolean"]
},
"deleted": {
"type": ["null", "boolean"]
}
}
}
},
"roundtype": {
"type": ["null", "number"]
},
"attachments": {
"type": ["null", "array"]
},
"balance": {
"type": ["null", "number"]
},
"billableExpense": {
"type": ["null", "number"]
},
"billableHoursLogBillable": {
"type": ["null", "number"]
},
"clientCustomId": {
"type": ["null", "string"]
},
"endTime": {
"type": ["null", "string"]
},
"invoicedAmount": {
"type": ["null", "number"]
},
"logDurationBillable": {
"type": ["null", "number"]
},
"startTime": {
"type": ["null", "string"],
"format": "date-time"
},
"startEndTime": {
"type": ["null", "string"]
},
"teams": {
"type": ["null", "array"],
"items": {
"type": ["null", "string"]
}
},
"teamsNames": {
"type": ["null", "string"]
},
"taskListName": {
"type": ["null", "string"]
},
"taskDueDate": {
"type": ["null", "string"]
},
"taskStartDate": {
"type": ["null", "string"]
},
"tagsData": {
"type": ["null", "array"]
},
"uninvoicedAmount": {
"type": ["null", "number"]
}
}
}

45
airbyte-integrations/connectors/source-my-hours/source_my_hours/schemas/users.json
View File

@@ -1,45 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"id": {
"type": ["number"]
},
"name": {
"type": ["null", "string"]
},
"archived": {
"type": ["null", "boolean"]
},
"dateArchived": {
"type": ["null", "string"]
},
"active": {
"type": ["null", "boolean"]
},
"accountOwner": {
"type": ["null", "boolean"]
},
"email": {
"type": ["null", "string"]
},
"rate": {
"type": ["null", "number"]
},
"billableRate": {
"type": ["null", "number"]
},
"admin": {
"type": ["null", "boolean"]
},
"isProjectManager": {
"type": ["null", "boolean"]
},
"roleType": {
"type": ["null", "integer"]
},
"customId": {
"type": ["null", "string"]
}
}
}

136
airbyte-integrations/connectors/source-my-hours/source_my_hours/source.py
View File

@@ -2,133 +2,17 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from typing import Any, List, Mapping, MutableMapping, Optional, Tuple
from urllib.parse import parse_qs, urlparse
from airbyte_cdk.sources.declarative.yaml_declarative_source import YamlDeclarativeSource
import pendulum
import requests
from airbyte_cdk.logger import AirbyteLogger
from airbyte_cdk.sources import AbstractSource
from airbyte_cdk.sources.streams import Stream
from airbyte_cdk.sources.streams.http.requests_native_auth import TokenAuthenticator
from source_my_hours.auth import MyHoursAuthenticator
from source_my_hours.stream import MyHoursStream
"""
This file provides the necessary constructs to interpret a provided declarative YAML configuration file into
source connector.
from .constants import REQUEST_HEADERS, URL_BASE
WARNING: Do not modify this file.
"""
class Clients(MyHoursStream):
primary_key = "id"
def path(
self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
) -> str:
return "Clients"
class Projects(MyHoursStream):
primary_key = "id"
def path(
self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
) -> str:
return "Projects/getAll"
class Tags(MyHoursStream):
primary_key = "id"
def path(
self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
) -> str:
return "Tags"
class TimeLogs(MyHoursStream):
primary_key = "logId"
def __init__(
self,
authenticator: TokenAuthenticator,
start_date: str,
batch_size: int,
**kwargs,
):
super().__init__(authenticator=authenticator)
self.start_date = pendulum.parse(start_date)
self.batch_size = batch_size
if self.start_date > pendulum.now():
self.logger.warn(f'Stream {self.name}: start_date "{start_date.isoformat()}" should be before today.')
def path(
self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
) -> str:
return "Reports/activity"
def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
previous_query = parse_qs(urlparse(response.request.url).query)
previous_end = pendulum.parse(previous_query["DateTo"][0])
new_from = previous_end.add(days=1)
new_to = new_from.add(days=self.batch_size - 1)
if new_from > pendulum.now():
return None
return {
"DateFrom": new_from.to_date_string(),
"DateTo": new_to.to_date_string(),
}
def request_params(
self,
stream_state: Mapping[str, Any],
stream_slice: Mapping[str, Any] = None,
next_page_token: Mapping[str, Any] = None,
) -> MutableMapping[str, Any]:
if next_page_token is None:
return {"DateFrom": self.start_date.to_date_string(), "DateTo": self.start_date.add(days=self.batch_size - 1).to_date_string()}
return next_page_token
class Users(MyHoursStream):
primary_key = "id"
def path(
self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
) -> str:
return "Users/getAll"
# Source
class SourceMyHours(AbstractSource):
def check_connection(self, logger: AirbyteLogger, config) -> Tuple[bool, any]:
url = f"{URL_BASE}/Clients"
try:
authenticator = self._make_authenticator(config)
headers = authenticator.get_auth_header()
headers.update(REQUEST_HEADERS)
response = requests.get(url, headers=headers)
response.raise_for_status()
return True, None
except Exception as e:
return False, e
def streams(self, config: Mapping[str, Any]) -> List[Stream]:
auth = self._make_authenticator(config)
return [
Clients(authenticator=auth),
Projects(authenticator=auth),
Tags(authenticator=auth),
TimeLogs(authenticator=auth, start_date=config["start_date"], batch_size=config["logs_batch_size"]),
Users(authenticator=auth),
]
@staticmethod
def _make_authenticator(config) -> MyHoursAuthenticator:
return MyHoursAuthenticator(config["email"], config["password"])
# Declarative Source
class SourceMyHours(YamlDeclarativeSource):
def __init__(self):
super().__init__(**{"path_to_yaml": "manifest.yaml"})

40
airbyte-integrations/connectors/source-my-hours/source_my_hours/spec.json
View File

@@ -1,40 +0,0 @@
{
"documentationUrl": "https://docs.airbyte.com/integrations/sources/my-hours",
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "My Hours Spec",
"type": "object",
"required": ["email", "password", "start_date"],
"additionalProperties": true,
"properties": {
"email": {
"title": "Email",
"type": "string",
"description": "Your My Hours username",
"example": "john@doe.com"
},
"password": {
"title": "Password",
"type": "string",
"description": "The password associated to the username",
"airbyte_secret": true
},
"start_date": {
"title": "Start Date",
"description": "Start date for collecting time logs",
"examples": ["%Y-%m-%d", "2016-01-01"],
"type": "string",
"pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}$"
},
"logs_batch_size": {
"title": "Time logs batch size",
"description": "Pagination size used for retrieving logs in days",
"examples": [30],
"type": "integer",
"minimum": 1,
"maximum": 365,
"default": 30
}
}
}
}

27
airbyte-integrations/connectors/source-my-hours/source_my_hours/stream.py
View File

@@ -1,27 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from abc import ABC
from typing import Any, Iterable, Mapping, Optional
import requests
from airbyte_cdk.sources.streams.http import HttpStream
from .constants import REQUEST_HEADERS, URL_BASE
class MyHoursStream(HttpStream, ABC):
url_base = URL_BASE + "/"
def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
return None
def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
for record in response.json():
yield record
def request_headers(
self, stream_state: Mapping[str, Any], stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
) -> Mapping[str, Any]:
return REQUEST_HEADERS

3
airbyte-integrations/connectors/source-my-hours/unit_tests/__init__.py
View File

@@ -1,3 +0,0 @@
#
# Copyright (c) 2021 Airbyte, Inc., all rights reserved.
#

30
airbyte-integrations/connectors/source-my-hours/unit_tests/test_auth.py
View File

@@ -1,30 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
import responses
from source_my_hours.auth import MyHoursAuthenticator
from source_my_hours.constants import URL_BASE
DEFAULT_CONFIG = {"email": "john@doe.com", "password": "pwd"}
@responses.activate
def test_init(mocker):
responses.add(responses.POST, f"{URL_BASE}/tokens/login", json={"accessToken": "at", "refreshToken": "rt", "expiresIn": 100})
authenticator = MyHoursAuthenticator(email="email", password="password")
authenticator._access_token
assert authenticator._access_token == "at"
@responses.activate
def test_refresh(mocker):
responses.add(responses.POST, f"{URL_BASE}/tokens/login", json={"accessToken": "at", "refreshToken": "rt", "expiresIn": 0})
responses.add(responses.POST, f"{URL_BASE}/tokens/refresh", json={"accessToken": "at2", "refreshToken": "rt2", "expiresIn": 100})
authenticator = MyHoursAuthenticator(email="email", password="password")
access_token = authenticator.get_access_token()
assert access_token == "at2"
assert authenticator.refresh_token == "rt2"

71
airbyte-integrations/connectors/source-my-hours/unit_tests/test_source.py
View File

@@ -1,71 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from unittest.mock import MagicMock
import responses
from source_my_hours.constants import URL_BASE
from source_my_hours.source import SourceMyHours, TimeLogs
DEFAULT_CONFIG = {"email": "john@doe.com", "password": "pwd"}
@responses.activate
def test_check_connection_success(mocker):
source = SourceMyHours()
logger_mock = MagicMock()
responses.add(responses.POST, f"{URL_BASE}/tokens/login", json={"accessToken": "at", "refreshToken": "rt", "expiresIn": 100})
responses.add(
responses.GET,
f"{URL_BASE}/Clients",
)
assert source.check_connection(logger_mock, DEFAULT_CONFIG) == (True, None)
@responses.activate
def test_check_connection_authentication_failure(mocker):
source = SourceMyHours()
logger_mock = MagicMock()
responses.add(responses.POST, f"{URL_BASE}/tokens/login", status=403, json={"message": "Incorrect email or password"})
success, exception = source.check_connection(logger_mock, DEFAULT_CONFIG)
assert success is False
assert exception is not None
@responses.activate
def test_check_connection_connection_failure(mocker):
source = SourceMyHours()
logger_mock = MagicMock()
responses.add(responses.POST, f"{URL_BASE}/tokens/login", json={"accessToken": "at", "refreshToken": "rt", "expiresIn": 100})
responses.add(responses.GET, f"{URL_BASE}/Clients", status=403)
success, exception = source.check_connection(logger_mock, DEFAULT_CONFIG)
assert success is False
assert exception is not None
@responses.activate
def test_streams(mocker):
source = SourceMyHours()
responses.add(responses.POST, f"{URL_BASE}/tokens/login", json={"accessToken": "at", "refreshToken": "rt", "expiresIn": 100})
config = {"email": "john@doe.com", "password": "pwd", "logs_batch_size": 30, "start_date": "2021-01-01"}
streams = source.streams(config)
expected_streams_number = 5
assert len(streams) == expected_streams_number
def test_time_logs_next_page_token(mocker):
stream = TimeLogs(authenticator=MagicMock(), start_date="2021-01-01", batch_size=10)
reponse_mock = MagicMock()
reponse_mock.request.url = "https://myhours.com/test?DateTo=2021-01-01"
inputs = {"response": reponse_mock}
expected_token = {"DateFrom": "2021-01-02", "DateTo": "2021-01-11"}
assert stream.next_page_token(**inputs) == expected_token

41
airbyte-integrations/connectors/source-my-hours/unit_tests/test_stream.py
View File

@@ -1,41 +0,0 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
from unittest.mock import MagicMock
import pytest
import requests
from source_my_hours.constants import REQUEST_HEADERS
from source_my_hours.stream import MyHoursStream
@pytest.fixture
def patch_base_class(mocker):
# Mock abstract methods to enable instantiating abstract class
mocker.patch.object(MyHoursStream, "path", "v0/example_endpoint")
mocker.patch.object(MyHoursStream, "primary_key", "test_primary_key")
mocker.patch.object(MyHoursStream, "__abstractmethods__", set())
def test_next_page_token(patch_base_class):
stream = MyHoursStream()
inputs = {"response": MagicMock()}
expected_token = None
assert stream.next_page_token(**inputs) == expected_token
def test_parse_response(patch_base_class, requests_mock):
stream = MyHoursStream()
requests_mock.get("https://dummy", json=[{"name": "test"}])
resp = requests.get("https://dummy")
inputs = {"response": resp}
expected_parsed_object = {"name": "test"}
assert next(stream.parse_response(**inputs)) == expected_parsed_object
def test_request_headers(patch_base_class):
stream = MyHoursStream()
inputs = {"stream_slice": None, "stream_state": None, "next_page_token": None}
expected_headers = REQUEST_HEADERS
assert stream.request_headers(**inputs) == expected_headers

3
docs/integrations/sources/my-hours.md
View File

@@ -22,7 +22,7 @@ This source allows you to synchronize the following data tables:
## Getting started
**Requirements**
In order to use the My Hours API you need to provide the credentials to an admin My Hours account.
- In order to use the My Hours API you need to provide the credentials to an admin My Hours account.
### Performance Considerations (Airbyte Open Source)
@@ -33,6 +33,7 @@ Depending on the amount of team members and time logs the source provides a prop
| Version | Date | Pull Request | Subject |
| :------ | :--------- | :------------------------------------------------------- | :----------------------------------- |
| 0.2.0 | 2024-03-15 | [36063](https://github.com/airbytehq/airbyte/pull/36063) | Migrate to Low Code |
| 0.1.2 | 2023-11-20 | [32680](https://github.com/airbytehq/airbyte/pull/32680) | Schema and CDK updates |
| 0.1.1 | 2022-06-08 | [12964](https://github.com/airbytehq/airbyte/pull/12964) | Update schema for time_logs stream |
| 0.1.0 | 2021-11-26 | [8270](https://github.com/airbytehq/airbyte/pull/8270) | New Source: My Hours |