1
0
mirror of synced 2025-12-23 21:03:15 -05:00

🎉 New Source: SAP Fieldglass [low-code cdk] (#18656)

* Generated boilerplate

* Connector woring in sandbox

* Connector working with 1 record from stream

* Sample state

* Added docs

* Revert "Added docs"

This reverts commit 413dedff6d8eced2c6c07bf3aa849eb6dce5b30a.

* Added docs finally

* Updated config

* Fixed data.json and cleaned some files

* Changed name from sap to sap-fieldglass and fixed test errors

* Re-added Primary key for future use

* Removed primary key as test failed

* Update settings.json

Fixed formating

* add sap fieldglass to source def

* auto-bump connector version

* remove oss_catalog.json

* revert change to linting file

* run format

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
This commit is contained in:
Sid Lais
2022-11-11 07:16:50 +05:30
committed by GitHub
parent cbb2e986fb
commit 9eee819d5b
25 changed files with 618 additions and 0 deletions

View File

@@ -1256,6 +1256,13 @@
icon: salesforce.svg
sourceType: api
releaseStage: generally_available
- name: SAP Fieldglass
sourceDefinitionId: ec5f3102-fb31-4916-99ae-864faf8e7e25
dockerRepository: airbyte/source-sap-fieldglass
dockerImageTag: 0.1.0
documentationUrl: https://docs.airbyte.com/integrations/sources/sap-fieldglass
sourceType: api
releaseStage: alpha
- name: SearchMetrics
sourceDefinitionId: 8d7ef552-2c0f-11ec-8d3d-0242ac130003
dockerRepository: airbyte/source-search-metrics

View File

@@ -12059,6 +12059,24 @@
type: "string"
path_in_connector_config:
- "client_secret"
- dockerImage: "airbyte/source-sap-fieldglass:0.1.0"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/sap-fieldglass"
connectionSpecification:
$schema: "http://json-schema.org/draft-07/schema#"
title: "Sap Fieldglass Spec"
type: "object"
required:
- "api_key"
additionalProperties: true
properties:
api_key:
type: "string"
description: "API Key"
airbyte_secret: true
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-search-metrics:0.1.1"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/seacrh-metrics"

View File

@@ -0,0 +1,6 @@
*
!Dockerfile
!main.py
!source_sap_fieldglass
!setup.py
!secrets

View File

@@ -0,0 +1,38 @@
FROM python:3.9.11-alpine3.15 as base
# build and load all requirements
FROM base as builder
WORKDIR /airbyte/integration_code
# upgrade pip to the latest version
RUN apk --no-cache upgrade \
&& pip install --upgrade pip \
&& apk --no-cache add tzdata build-base
COPY setup.py ./
# install necessary packages to a temporary folder
RUN pip install --prefix=/install .
# build a clean environment
FROM base
WORKDIR /airbyte/integration_code
# copy all loaded and built libraries to a pure basic image
COPY --from=builder /install /usr/local
# add default timezone settings
COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
RUN echo "Etc/UTC" > /etc/timezone
# bash is installed for more convenient debugging.
RUN apk --no-cache add bash
# copy payload code only
COPY main.py ./
COPY source_sap_fieldglass ./source_sap_fieldglass
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
LABEL io.airbyte.version=0.1.0
LABEL io.airbyte.name=airbyte/source-sap-fieldglass

View File

@@ -0,0 +1,79 @@
# Sap Fieldglass Source
This is the repository for the Sap Fieldglass configuration based source connector.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/sap-fieldglass).
## Local development
#### Building via Gradle
You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.
To build using Gradle, from the Airbyte repository root, run:
```
./gradlew :airbyte-integrations:connectors:source-sap-fieldglass:build
```
#### Create credentials
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/sap-fieldglass)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_sap_fieldglass/spec.yaml` file.
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
See `integration_tests/sample_config.json` for a sample config file.
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source sap-fieldglass test creds`
and place them into `secrets/config.json`.
### Locally running the connector docker image
#### Build
First, make sure you build the latest Docker image:
```
docker build . -t airbyte/source-sap-fieldglass:dev
```
You can also build the connector image via Gradle:
```
./gradlew :airbyte-integrations:connectors:source-sap-fieldglass:airbyteDocker
```
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
the Dockerfile.
#### Run
Then run any of the connector commands as follows:
```
docker run --rm airbyte/source-sap-fieldglass:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-sap-fieldglass:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-sap-fieldglass:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-sap-fieldglass:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```
## Testing
#### Acceptance Tests
Customize `acceptance-test-config.yml` file to configure tests. See [Source Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/source-acceptance-tests-reference) for more information.
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
To run your integration tests with docker
### Using gradle to run tests
All commands should be run from airbyte project root.
To run unit tests:
```
./gradlew :airbyte-integrations:connectors:source-sap-fieldglass:unitTest
```
To run acceptance and custom integration tests:
```
./gradlew :airbyte-integrations:connectors:source-sap-fieldglass:integrationTest
```
## Dependency Management
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
We split dependencies between two groups, dependencies that are:
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
* required for the testing need to go to `TEST_REQUIREMENTS` list
### Publishing a new version of the connector
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
1. Make sure your changes are passing unit and integration tests.
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
1. Create a Pull Request.
1. Pat yourself on the back for being an awesome contributor.
1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

View File

@@ -0,0 +1,3 @@
#
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
#

View File

@@ -0,0 +1,38 @@
# See [Source Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/source-acceptance-tests-reference)
# for more information about how to configure these tests
connector_image: airbyte/source-sap-fieldglass:dev
acceptance_tests:
spec:
tests:
- spec_path: "source_sap_fieldglass/spec.yaml"
connection:
tests:
- config_path: "secrets/config.json"
status: "succeed"
- config_path: "integration_tests/invalid_config.json"
status: "failed"
discovery:
tests:
- config_path: "secrets/config.json"
basic_read:
tests:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
empty_streams: []
# TODO uncomment this block to specify that the tests should assert the connector outputs the records provided in the input file a file
# expect_records:
# path: "integration_tests/expected_records.txt"
# extra_fields: no
# exact_order: no
# extra_records: yes
incremental:
bypass_reason: "This connector does not implement incremental sync"
# TODO uncomment this block this block if your connector implements incremental sync:
# tests:
# - config_path: "secrets/config.json"
# configured_catalog_path: "integration_tests/configured_catalog.json"
# future_state_path: "integration_tests/abnormal_state.json"
full_refresh:
tests:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"

View File

@@ -0,0 +1,16 @@
#!/usr/bin/env sh
# Build latest connector image
docker build . -t $(cat acceptance-test-config.yml | grep "connector_image" | head -n 1 | cut -d: -f2-)
# Pull latest acctest image
docker pull airbyte/source-acceptance-test:latest
# Run
docker run --rm -it \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp:/tmp \
-v $(pwd):/test_input \
airbyte/source-acceptance-test \
--acceptance-test-config /test_input

View File

@@ -0,0 +1,9 @@
plugins {
id 'airbyte-python'
id 'airbyte-docker'
id 'airbyte-source-acceptance-test'
}
airbytePython {
moduleDirectory 'source_sap_fieldglass'
}

View File

@@ -0,0 +1,3 @@
#
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
#

View File

@@ -0,0 +1,5 @@
{
"data": {
"date": "31/12/2999"
}
}

View File

@@ -0,0 +1,16 @@
#
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
#
import pytest
pytest_plugins = ("source_acceptance_test.plugin",)
@pytest.fixture(scope="session", autouse=True)
def connector_setup():
"""This fixture is a placeholder for external resources that acceptance test might require."""
# TODO: setup test dependencies if needed. otherwise remove the TODO comments
yield
# TODO: clean up test dependencies

View File

@@ -0,0 +1,13 @@
{
"streams": [
{
"stream": {
"name": "data",
"json_schema": {},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
}
]
}

View File

@@ -0,0 +1,3 @@
{
"api_key": "<invalid_key>"
}

View File

@@ -0,0 +1,3 @@
{
"api_key": "<api_key>"
}

View File

@@ -0,0 +1,5 @@
{
"todo-stream-name": {
"todo-field-name": "value"
}
}

View File

@@ -0,0 +1,13 @@
#
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
#
import sys
from airbyte_cdk.entrypoint import launch
from source_sap_fieldglass import SourceSapFieldglass
if __name__ == "__main__":
source = SourceSapFieldglass()
launch(source, sys.argv[1:])

View File

@@ -0,0 +1,2 @@
-e ../../bases/source-acceptance-test
-e .

View File

@@ -0,0 +1,29 @@
#
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
#
from setuptools import find_packages, setup
MAIN_REQUIREMENTS = [
"airbyte-cdk~=0.1",
]
TEST_REQUIREMENTS = [
"pytest~=6.1",
"pytest-mock~=3.6.1",
"source-acceptance-test",
]
setup(
name="source_sap_fieldglass",
description="Source implementation for Sap Fieldglass.",
author="Airbyte",
author_email="contact@airbyte.io",
packages=find_packages(),
install_requires=MAIN_REQUIREMENTS,
package_data={"": ["*.json", "*.yaml", "schemas/*.json", "schemas/shared/*.json"]},
extras_require={
"tests": TEST_REQUIREMENTS,
},
)

View File

@@ -0,0 +1,8 @@
#
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
#
from .source import SourceSapFieldglass
__all__ = ["SourceSapFieldglass"]

View File

@@ -0,0 +1,38 @@
version: "0.1.0"
definitions:
selector:
extractor:
field_pointer: []
requester:
url_base: "https://sandbox.api.sap.com/fieldglass/api/vc/connector"
http_method: "GET"
authenticator:
type: ApiKeyAuthenticator
header: "apikey"
api_token: "{{ config['api_key'] }}"
requrest_options_provider:
request_parameters:
base: "{{ config['base'] }}"
retriever:
record_selector:
$ref: "*ref(definitions.selector)"
paginator:
type: NoPagination
requester:
$ref: "*ref(definitions.requester)"
base_stream:
retriever:
$ref: "*ref(definitions.retriever)"
data_stream:
$ref: "*ref(definitions.base_stream)"
$options:
name: "data"
path: "/Active Worker Download"
streams:
- "*ref(definitions.data_stream)"
check:
stream_names:
- "data"

View File

@@ -0,0 +1,210 @@
{
"Data": {
"type": "array",
"items": {
"type": "object",
"required": [
"Bill Rate",
"Business Unit Code",
"Business Unit Name",
"Buyer Code",
"Cost Center Code",
"Currency",
"End Date",
"First Name",
"Job Posting Title",
"Job Seeker ID",
"Last Name",
"Pay Rate",
"Rate Category /UOM",
"Sequence",
"Site Code",
"Site Name",
"Start Date",
"Status",
"Vendor Name",
"Vendor Number",
"Work Order ID",
"Work Order/Work Order Revision Owner",
"Worker Email",
"Worker ID"
],
"properties": {
"[c]Worker Custom Fields": {
"type": "string",
"description": "Custom fields found on the worker.\nIf there are many custom fields, there will be many\ncolumns.Column header format will be: \n\"[c]modulename_custom field name\" (i.e. [c] followed by\nmodule custom text lowercase with no spaces, followed by\nunderscore, followed by the custom field name text as\ndefined by users in the SAP Fieldglass application.)",
"maxLength": 4000
},
"Job Seeker ID": {
"type": "string",
"description": "14-character alphanumeric SAP Fieldglass Job Seeker ID.",
"maxLength": 14
},
"[c]Buyer or Supplier custom fields": {
"type": "string",
"description": "Custom fields found on the supplier. (For supplier side\ndownload, only those fields required by buyer for supplier\nto be entered and viewed are downloaded.)If there are many\ncustom fields, there will be many columns.Column header\nformat will be: \n?[c]modulename_custom field name? (i.e. [c] followed by\nmodule custom text lowercase with no spaces, followed by\nunderscore, followed by the custom field name text as\ndefined by users in the SAP Fieldglass application.)",
"maxLength": 4000
},
"Billable Per Diem": {
"type": "integer",
"format": "float",
"description": ""
},
"First Name": {
"type": "string",
"description": "First name of the Worker as defined in SAP Fieldglass.",
"maxLength": 100
},
"Remit To address Code": {
"type": "string",
"description": "Code as assigned to the Remit To Address by supplier.",
"maxLength": 100
},
"Cost Center Name": {
"type": "string",
"description": "Cost Center Name in SAP Fieldglass.",
"maxLength": 200
},
"Work Order/Work Order Revision Owner": {
"type": "string",
"description": "Work Order Owner?s name.",
"maxLength": 100
},
"Vendor Name": {
"type": "string",
"description": "Supplier name.",
"maxLength": 200
},
"Work Order ID": {
"type": "string",
"description": "Work Order ID.",
"maxLength": 14
},
"Bill Rate": {
"type": "integer",
"format": "float",
"description": "Worker?s Bill Rate that is associated to the Rate\nCategory/UOM listed in the previous field."
},
"Start Date": {
"type": "string",
"format": "date",
"description": "Work order start date."
},
"Security ID": {
"type": "string",
"description": "This column will only appear in the file if the security ID\nfield functionality is activated in SAP Fieldglass for the\nbuyer. If the functionality is not activated, this column\nwill not be in the file.",
"maxLength": 100
},
"Currency": {
"type": "string",
"description": "ISO currency designation (ex. USD).",
"maxLength": 100
},
"[c]Work Order Custom Fields": {
"type": "string",
"description": "Custom fields found on the work order/work order revision. \nIf there are many custom fields, there will be many columns.\nColumn header format will be:\n?[c]modulename_custom field name? (i.e. [c] followed by\nmodule custom text lowercase with no spaces, followed by\nunderscore, followed by the custom field name text as\ndefined by users in the SAP Fieldglass application.)",
"maxLength": 4000
},
"Job Posting Title": {
"type": "string",
"description": "Job Posting?s Title or SOW Name for SOW Workers.",
"maxLength": 100
},
"Worker ID": {
"type": "string",
"description": "14-character alphanumeric SAP Fieldglass Worker ID.",
"maxLength": 14
},
"Business Unit Name": {
"type": "string",
"description": "Business Unit Name.",
"maxLength": 100
},
"Site Name": {
"type": "string",
"description": "Site Name.",
"maxLength": 100
},
"Buyer Code": {
"type": "string",
"description": "Buyer Company Code.",
"maxLength": 4
},
"Status": {
"type": "string",
"description": "Status of the worker.",
"maxLength": 50
},
"Pay Rate": {
"type": "integer",
"format": "float",
"description": "Worker?s Pay Rate that is associated to the Rate Category/UOM\nlisted in the previous field. \nIf Supplier did not enter a Pay Rate, 0.00 will be\ndisplayed."
},
"Vendor Number": {
"type": "string",
"description": "SAP Fieldglass Supplier code for buyer.",
"maxLength": 4
},
"Sequence": {
"type": "number",
"description": "Work Order Revision Number."
},
"Person ID": {
"type": "string",
"description": "Unique identifier for the worker as a person.",
"maxLength": 24
},
"Cost Center Code": {
"type": "string",
"description": "Cost Center Code in SAP Fieldglass.",
"maxLength": 200
},
"[c]Worker User Person Custom Fields": {
"type": "string",
"description": "Custom fields found on the worker user person\nIf there are many custom fields, there will be many columns.\nColumn header format will be: \n?[c]modulename_custom field name? (i.e. [c] followed by\nmodule custom text lowercase with no spaces, followed by\nunderscore, followed by the custom field name text as\ndefined by users in the SAP Fieldglass application.)",
"maxLength": 4000
},
"Rate Category /UOM": {
"type": "string",
"description": "Worker?s Rate Category/Unit Of Measure for which the\ncorresponding rate will be displayed. \nMore than one Rate Category/UOM may exist for a single\nWorker. If this is the case a new line will appear for each\nassociated Rate Category/UOM in this file.",
"maxLength": 100
},
"Segmented Object Detail": {
"type": "string",
"description": "Segmented Object Detail string. Segments will be separated by\ndelimiter on the Segmented Object.\nValid delimiters are colon (:), semi-colon (;), pipe (|), and\ndash (-).\nRequired if ?Enable Segmented Object Detail? configuration is\nset.\nThis field is mutually exclusive with GL Account fields.\nMore than one string can appear for a cost center.",
"maxLength": 2000
},
"End Date": {
"type": "string",
"format": "date",
"description": "Work order end date."
},
"Worker Email": {
"type": "string",
"description": "Worker?s e-mail address.",
"maxLength": 100
},
"Work Order/Work Order Revision Owner Employee ID": {
"type": "string",
"description": "Work Order Owner?s Employee ID.",
"maxLength": 50
},
"Last Name": {
"type": "string",
"description": "Last name of the Worker as defined in SAP Fieldglass.",
"maxLength": 100
},
"Site Code": {
"type": "string",
"description": "Site Code.",
"maxLength": 100
},
"Business Unit Code": {
"type": "string",
"description": "Business Unit Code.",
"maxLength": 100
}
}
}
}
}

View File

@@ -0,0 +1,18 @@
#
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
#
from airbyte_cdk.sources.declarative.yaml_declarative_source import YamlDeclarativeSource
"""
This file provides the necessary constructs to interpret a provided declarative YAML configuration file into
source connector.
WARNING: Do not modify this file.
"""
# Declarative Source
class SourceSapFieldglass(YamlDeclarativeSource):
def __init__(self):
super().__init__(**{"path_to_yaml": "sap_fieldglass.yaml"})

View File

@@ -0,0 +1,13 @@
documentationUrl: https://docs.airbyte.com/integrations/sources/sap-fieldglass
connectionSpecification:
$schema: http://json-schema.org/draft-07/schema#
title: Sap Fieldglass Spec
type: object
required:
- api_key
additionalProperties: true
properties:
api_key:
type: string
description: API Key
airbyte_secret: true

View File

@@ -0,0 +1,25 @@
## SAP Fieldglass Active Worker Download Source
This page contains the setup guide and reference information for the SAP Fieldglass Active Worker Download source connector built in `low-code cdk`.
## Prerequisites
- A [Fieldglass Account](https://www.fieldglass.net/)
- An [api key from SAP](https://api.sap.com/)
## Supported sync modes
| Feature | Supported?\(Yes/No\) | Notes |
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental Sync | No | |
## Supported Streams
* [Active Worker Download](https://api.sap.com/api/activeWorkerDownload/resource)
## Changelog
| Version | Date | Pull Request | Subject |
|:--------|:-----------| :----------- |:-----------------------------------------------------------|
| 0.1.0 | 2022-10-22 | https://github.com/airbytehq/airbyte/pull/18656 | Initial commit |