jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-23 21:03:15 -05:00
Code Issues Packages Projects Releases Wiki Activity

🎉 Source greenhouse: added identification of accessible streams for API keys with limited permissions (#6238)

Browse Source 
* Added identification of accessible streams for API keys with limited permissions

* added credentials for API key with limited permissions (for user stream only)

* update docs and image version

* updated error message

* fixed formatting
This commit is contained in:
midavadim
2021-09-21 15:02:12 +03:00
committed by GitHub
parent c114ec85dc
commit 12cb3e3f9f
20 changed files with 262 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.github/workflows/publish-command.yml vendored
View File

@@ -117,6 +117,7 @@ jobs:
GOOGLE_SHEETS_TESTS_CREDS: ${{ secrets.GOOGLE_SHEETS_TESTS_CREDS }}
GOOGLE_WORKSPACE_ADMIN_REPORTS_TEST_CREDS: ${{ secrets.GOOGLE_WORKSPACE_ADMIN_REPORTS_TEST_CREDS }}
GREENHOUSE_TEST_CREDS: ${{ secrets.GREENHOUSE_TEST_CREDS }}
GREENHOUSE_TEST_CREDS_LIMITED: ${{ secrets.GREENHOUSE_TEST_CREDS_LIMITED }}
HARVEST_INTEGRATION_TESTS_CREDS: ${{ secrets.HARVEST_INTEGRATION_TESTS_CREDS }}
HUBSPOT_INTEGRATION_TESTS_CREDS: ${{ secrets.HUBSPOT_INTEGRATION_TESTS_CREDS }}
INSTAGRAM_INTEGRATION_TESTS_CREDS: ${{ secrets.INSTAGRAM_INTEGRATION_TESTS_CREDS }}

1
.github/workflows/test-command.yml vendored
View File

@@ -112,6 +112,7 @@ jobs:
GOOGLE_SHEETS_TESTS_CREDS: ${{ secrets.GOOGLE_SHEETS_TESTS_CREDS }}
GOOGLE_WORKSPACE_ADMIN_REPORTS_TEST_CREDS: ${{ secrets.GOOGLE_WORKSPACE_ADMIN_REPORTS_TEST_CREDS }}
GREENHOUSE_TEST_CREDS: ${{ secrets.GREENHOUSE_TEST_CREDS }}
GREENHOUSE_TEST_CREDS_LIMITED: ${{ secrets.GREENHOUSE_TEST_CREDS_LIMITED }}
HARVEST_INTEGRATION_TESTS_CREDS: ${{ secrets.HARVEST_INTEGRATION_TESTS_CREDS }}
HUBSPOT_INTEGRATION_TESTS_CREDS: ${{ secrets.HUBSPOT_INTEGRATION_TESTS_CREDS }}
INSTAGRAM_INTEGRATION_TESTS_CREDS: ${{ secrets.INSTAGRAM_INTEGRATION_TESTS_CREDS }}

2
airbyte-config/init/src/main/resources/config/STANDARD_SOURCE_DEFINITION/59f1e50a-331f-4f09-b3e8-2e8d4d355f44.json
View File

@@ -2,7 +2,7 @@
"sourceDefinitionId": "59f1e50a-331f-4f09-b3e8-2e8d4d355f44",
"name": "Greenhouse",
"dockerRepository": "airbyte/source-greenhouse",
"dockerImageTag": "0.2.3",
"dockerImageTag": "0.2.4",
"documentationUrl": "https://docs.airbyte.io/integrations/sources/greenhouse",
"icon": "greenhouse.svg"
}

2
airbyte-config/init/src/main/resources/seed/source_definitions.yaml
View File

@@ -214,7 +214,7 @@
- sourceDefinitionId: 59f1e50a-331f-4f09-b3e8-2e8d4d355f44
name: Greenhouse
dockerRepository: airbyte/source-greenhouse
dockerImageTag: 0.2.3
dockerImageTag: 0.2.4
documentationUrl: https://docs.airbyte.io/integrations/sources/greenhouse
icon: greenhouse.svg
sourceType: api

2
airbyte-integrations/connectors/source-greenhouse/Dockerfile
View File

@@ -14,5 +14,5 @@ RUN pip install .
ENV AIRBYTE_ENTRYPOINT "/airbyte/base.sh"
LABEL io.airbyte.version=0.2.3
LABEL io.airbyte.version=0.2.4
LABEL io.airbyte.name=airbyte/source-greenhouse

26
airbyte-integrations/connectors/source-greenhouse/acceptance-test-config.yml Normal file
View File

@@ -0,0 +1,26 @@
# See [Source Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/source-acceptance-tests-reference)
# for more information about how to configure these tests
connector_image: airbyte/source-greenhouse:dev
tests:
spec:
- spec_path: "source_greenhouse/spec.json"
connection:
- config_path: "secrets/config.json"
status: "succeed"
- config_path: "secrets/config_users_only.json"
status: "succeed"
- config_path: "integration_tests/config_invalid.json"
status: "failed"
discovery:
- config_path: "secrets/config.json"
- config_path: "secrets/config_users_only.json"
basic_read:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog_users_only.json"
full_refresh:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
- config_path: "secrets/config_users_only.json"
configured_catalog_path: "integration_tests/configured_catalog_users_only.json"

16
airbyte-integrations/connectors/source-greenhouse/acceptance-test-docker.sh Normal file
View File

@@ -0,0 +1,16 @@
#!/usr/bin/env sh
# Build latest connector image
docker build . -t $(cat acceptance-test-config.yml | grep "connector_image" | head -n 1 | cut -d: -f2)
# Pull latest acctest image
docker pull airbyte/source-acceptance-test:latest
# Run
docker run --rm -it \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp:/tmp \
-v $(pwd):/test_input \
airbyte/source-acceptance-test \
--acceptance-test-config /test_input

0
airbyte-integrations/connectors/source-greenhouse/integration_tests/__init__.py Normal file
View File

26
airbyte-integrations/connectors/source-greenhouse/unit_tests/test_client.py → airbyte-integrations/connectors/source-greenhouse/integration_tests/acceptance.py
View File

@@ -24,25 +24,11 @@
import pytest
from grnhse.exceptions import EndpointNotFound, HTTPError
from source_greenhouse.client import Client
pytest_plugins = ("source_acceptance_test.plugin",)
def test__heal_check_with_wrong_api_key():
client = Client(api_key="wrong_key")
alive, error = client.health_check()
assert not alive
assert error == '401 {"message":"Invalid Basic Auth credentials"}'
def test__custom_fields_with_wrong_api_key():
client = Client(api_key="wrong_key")
with pytest.raises(HTTPError, match='401 {"message":"Invalid Basic Auth credentials"}'):
list(client.list("custom_fields"))
def test_client_wrong_endpoint():
client = Client(api_key="wrong_key")
with pytest.raises(EndpointNotFound, match="unknown_endpoint"):
next(client.list("unknown_endpoint"))
@pytest.fixture(scope="session", autouse=True)
def connector_setup():
""" This fixture is a placeholder for external resources that acceptance test might require."""
yield

3
airbyte-integrations/connectors/source-greenhouse/integration_tests/config_invalid.json Normal file
View File

@@ -0,0 +1,3 @@
{
"api_key": "bla"
}

114
airbyte-integrations/connectors/source-greenhouse/integration_tests/configured_catalog.json Normal file
View File

@@ -0,0 +1,114 @@
{
"streams": [
{
"stream": {
"name": "applications",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": false
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "candidates",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": false
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "close_reasons",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": false
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "degrees",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": false
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "departments",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": false
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "job_posts",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": false
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "jobs",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": false
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "offers",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": false
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "scorecards",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": false
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "users",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": false
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "custom_fields",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": false
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
}
]
}

14
airbyte-integrations/connectors/source-greenhouse/integration_tests/configured_catalog_users_only.json Normal file
View File

@@ -0,0 +1,14 @@
{
"streams": [
{
"stream": {
"name": "users",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": false
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
}
]
}

1
airbyte-integrations/connectors/source-greenhouse/requirements.txt
View File

@@ -1,4 +1,5 @@
# This file is autogenerated -- only edit if you know what you are doing. Use setup.py for declaring dependencies.
-e ../../bases/airbyte-protocol
-e ../../bases/base-python
-e ../../bases/source-acceptance-test
-e .

9
airbyte-integrations/connectors/source-greenhouse/setup.py
View File

@@ -25,12 +25,19 @@
from setuptools import find_packages, setup
TEST_REQUIREMENTS = [
"pytest~=6.1",
"source-acceptance-test",
]
setup(
name="source_greenhouse",
description="Source implementation for Greenhouse.",
author="Airbyte",
author_email="contact@airbyte.io",
packages=find_packages(),
install_requires=["airbyte-protocol", "base-python", "six==1.15.0", "grnhse-api==0.1.1", "pytest==6.1.2"],
install_requires=["airbyte-protocol", "base-python", "six==1.15.0", "grnhse-api==0.1.1"],
package_data={"": ["*.json", "schemas/*.json"]},
extras_require={
"tests": TEST_REQUIREMENTS,
},
)

37
airbyte-integrations/connectors/source-greenhouse/source_greenhouse/client.py
View File

@@ -24,9 +24,10 @@
from functools import partial
from typing import Mapping, Tuple
from typing import Generator, List, Mapping, Tuple
from base_python import BaseClient
from airbyte_protocol import AirbyteStream
from base_python import AirbyteLogger, BaseClient
from grnhse import Harvest
from grnhse.exceptions import HTTPError
@@ -67,16 +68,40 @@ class Client(BaseClient):
def _enumerate_methods(self) -> Mapping[str, callable]:
return {entity: partial(self.list, name=entity) for entity in self.ENTITIES}
def get_accessible_endpoints(self) -> List[str]:
"""Try to read each supported endpoint and return accessible stream names"""
logger = AirbyteLogger()
accessible_endpoints = []
for entity in self.ENTITIES:
try:
getattr(self._client, entity).get()
accessible_endpoints.append(entity)
except HTTPError as error:
logger.warn(f"Endpoint '{entity}' error: {str(error)}")
if "This API Key does not have permission for this endpoint" not in str(error):
raise error
logger.info(f"API key has access to {len(accessible_endpoints)} endpoints: {accessible_endpoints}")
return accessible_endpoints
def health_check(self) -> Tuple[bool, str]:
alive = True
error_msg = None
try:
# because there is no good candidate to try our connection
# we use users endpoint as potentially smallest dataset
self._client.users.get()
accessible_endpoints = self.get_accessible_endpoints()
if not accessible_endpoints:
alive = False
error_msg = "Your API Key does not have permission for any existing endpoints. Please grant read permissions for required streams/endpoints"
except HTTPError as error:
alive = False
error_msg = str(error)
return alive, error_msg
@property
def streams(self) -> Generator[AirbyteStream, None, None]:
"""Process accessible streams only"""
accessible_endpoints = self.get_accessible_endpoints()
for stream in super().streams:
if stream.name in accessible_endpoints:
yield stream

12
airbyte-integrations/connectors/source-greenhouse/source_greenhouse/schemas/applications.json
View File

@@ -41,7 +41,7 @@
"type": ["null", "string"]
},
"prospect_owner": {
"type": "object",
"type": ["null", "object"],
"properties": {
"name": {
"type": "string"
@@ -69,7 +69,15 @@
"type": "integer"
},
"current_stage": {
"type": ["null", "string"]
"type": ["null", "object"],
"properties": {
"name": {
"type": "string"
},
"id": {
"type": "integer"
}
}
},
"credited_to": {
"type": "object",

12
airbyte-integrations/connectors/source-greenhouse/source_greenhouse/schemas/candidates.json
View File

@@ -109,7 +109,7 @@
"type": ["null", "string"]
},
"prospect_owner": {
"type": "object",
"type": ["null", "object"],
"properties": {
"name": {
"type": "string"
@@ -137,7 +137,15 @@
"type": "integer"
},
"current_stage": {
"type": ["null", "string"]
"type": ["null", "object"],
"properties": {
"name": {
"type": "string"
},
"id": {
"type": "integer"
}
}
},
"credited_to": {
"type": "object",

24
airbyte-integrations/connectors/source-greenhouse/source_greenhouse/schemas/users.json
View File

@@ -3,43 +3,43 @@
"type": "object",
"properties": {
"id": {
"type": "integer"
"type": ["null", "integer"]
},
"name": {
"type": "string"
"type": ["null", "string"]
},
"first_name": {
"type": "string"
"type": ["null", "string"]
},
"last_name": {
"type": "string"
"type": ["null", "string"]
},
"primary_email_address": {
"type": "string"
"type": ["null", "string"]
},
"updated_at": {
"type": "string"
"type": ["null", "string"]
},
"created_at": {
"type": "string"
"type": ["null", "string"]
},
"disabled": {
"type": "boolean"
"type": ["null", "boolean"]
},
"site_admin": {
"type": "boolean"
"type": ["null", "boolean"]
},
"emails": {
"type": "array",
"type": ["null", "array"],
"items": {
"type": "string"
"type": ["null", "string"]
}
},
"employee_id": {
"type": ["null", "integer"]
},
"linked_candidate_ids": {
"type": "array"
"type": ["null", "array"]
}
}
}

5
docs/integrations/sources/greenhouse.md
View File

@@ -46,3 +46,8 @@ The Greenhouse connector should not run into Greenhouse API limitations under no
Please follow the [Greenhouse documentation for generating an API key](https://developers.greenhouse.io/harvest.html#authentication).
## Changelog
| Version | Date | Pull Request | Subject |
| :------ | :-------- | :----- | :------ |
| 0.2.4 | 2021-09-15 | [6238](https://github.com/airbytehq/airbyte/pull/6238) | added identification of accessible streams for API keys with limited permissions |

1
tools/bin/ci_credentials.sh
View File

@@ -85,6 +85,7 @@ write_standard_creds source-google-search-console-singer "$GOOGLE_SEARCH_CONSOLE
write_standard_creds source-google-sheets "$GOOGLE_SHEETS_TESTS_CREDS"
write_standard_creds source-google-workspace-admin-reports "$GOOGLE_WORKSPACE_ADMIN_REPORTS_TEST_CREDS"
write_standard_creds source-greenhouse "$GREENHOUSE_TEST_CREDS"
write_standard_creds source-greenhouse "$GREENHOUSE_TEST_CREDS_LIMITED" "config_users_only.json"
write_standard_creds source-harvest "$HARVEST_INTEGRATION_TESTS_CREDS"
write_standard_creds source-hubspot "$HUBSPOT_INTEGRATION_TESTS_CREDS"
write_standard_creds source-instagram "$INSTAGRAM_INTEGRATION_TESTS_CREDS"