add shopify integration (#861)

* shopify integration

* fix for spotless

* gcc

* get syncs working

* don't include pii in replication for testing

* add documetation about private app api keys

* switch to api password

* fix credential checking

* remove unused gitignore

* finish configuring standard test

* remove orders from integration test catalog

* Update airbyte-integrations/connectors/source-shopify-singer/setup.py

Co-authored-by: Charles <giardina.charles@gmail.com>

* add secret for CI

* allow py files to start with def to fix generated test case

* fmt

* add gcc to test image

* limit catalog for faster tests

* fmt

Co-authored-by: Charles <giardina.charles@gmail.com>

.github/workflows/gradle.yml

@@ -55,6 +55,7 @@ jobs:
           ADWORDS_INTEGRATION_TEST_CREDS: ${{ secrets.ADWORDS_INTEGRATION_TEST_CREDS }}
           FACEBOOK_MARKETING_API_TEST_INTEGRATION_CREDS: ${{ secrets.FACEBOOK_MARKETING_API_TEST_INTEGRATION_CREDS }}
           SOURCE_MARKETO_SINGER_INTEGRATION_TEST_CONFIG: ${{ secrets.SOURCE_MARKETO_SINGER_INTEGRATION_TEST_CONFIG }}
+          SHOPIFY_INTEGRATION_TEST_CREDS: ${{ secrets.SHOPIFY_INTEGRATION_TEST_CREDS }}
           AWS_S3_INTEGRATION_TEST_CREDS: ${{ secrets.AWS_S3_INTEGRATION_TEST_CREDS }}
 
       - name: Build

airbyte-config/init/src/main/resources/config/STANDARD_SOURCE_DEFINITION/b1892b11-788d-44bd-b9ec-3a436f7b54ce.json

@@ -0,0 +1,7 @@
+{
+  "sourceDefinitionId": "b1892b11-788d-44bd-b9ec-3a436f7b54ce",
+  "name": "Shopify",
+  "dockerRepository": "airbyte/source-shopify-singer",
+  "dockerImageTag": "0.1.0",
+  "documentationUrl": "https://hub.docker.com/r/airbyte/source-shopify-singer"
+}

airbyte-integrations/connectors/source-shopify-singer/.dockerignore

@@ -0,0 +1,7 @@
+*
+!Dockerfile
+!Dockerfile.test
+!source_shopify_singer
+!setup.py
+!integration_tests
+!secrets

airbyte-integrations/connectors/source-shopify-singer/Dockerfile

@@ -0,0 +1,16 @@
+FROM airbyte/integration-base-singer:dev
+
+# Bash is installed for more convenient debugging.
+RUN apt-get update && apt-get install -y bash gcc && rm -rf /var/lib/apt/lists/*
+
+ENV CODE_PATH="source_shopify_singer"
+ENV AIRBYTE_IMPL_MODULE="source_shopify_singer"
+ENV AIRBYTE_IMPL_PATH="SourceShopifySinger"
+
+WORKDIR /airbyte/integration_code
+COPY $CODE_PATH ./$CODE_PATH
+COPY setup.py ./
+RUN pip install ".[main]"
+
+LABEL io.airbyte.version=0.1.0
+LABEL io.airbyte.name=airbyte/source-shopify-singer

airbyte-integrations/connectors/source-shopify-singer/Dockerfile.test

@@ -0,0 +1,23 @@
+FROM airbyte/base-python-test:dev
+
+# Bash is installed for convenient debugging.
+RUN apt-get update && apt-get install -y gcc bash && rm -rf /var/lib/apt/lists/*
+
+ENV MODULE_NAME="source_shopify_singer"
+ENV CODE_PATH="integration_tests"
+ENV AIRBYTE_TEST_MODULE="integration_tests"
+ENV AIRBYTE_TEST_PATH="SourceShopifySingerStandardTest"
+
+LABEL io.airbyte.version=0.1.0
+LABEL io.airbyte.name=airbyte/source-shopify-singer-standard-test
+
+WORKDIR /airbyte/integration_code
+COPY $MODULE_NAME $MODULE_NAME
+COPY $CODE_PATH $CODE_PATH
+COPY secrets/config.json $CODE_PATH/config.json
+COPY $MODULE_NAME/*.json $CODE_PATH/
+COPY setup.py ./
+
+RUN pip install ".[tests]"
+
+WORKDIR /airbyte

airbyte-integrations/connectors/source-shopify-singer/README.md

@@ -0,0 +1,62 @@
+# Source Shopify Singer
+
+This is the repository for the Shopify source connector, based on a Singer tap.
+For information about how to use this connector within Airbyte, see [the User Documentation](https://docs.airbyte.io/integrations/sources/shopify).
+
+## Local development
+### Prerequisites
+**To iterate on this connector, make sure to complete this prerequisites section.**
+
+#### Build & Activate Virtual Environment
+First, build the module by running the following from the `airbyte` project root directory: 
+```
+./gradlew :airbyte-integrations:connectors:source-shopify-singer:build
+```
+
+This will generate a virtualenv for this module in `source-shopify-singer/.venv`. Make sure this venv is active in your
+development environment of choice. To activate the venv from the terminal, run:
+```
+cd airbyte-integrations/connectors/source-shopify # cd into the connector directory
+source .venv/bin/activate
+```
+If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
+
+#### Create credentials
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/shopify)
+to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_shopify_singer/spec.json` file.
+See `sample_files/sample_config.json` for a sample config file.
+
+**If you are an Airbyte core member**, copy the credentials in RPass under the secret name `source-shopify-singer-integration-test-config`
+and place them into `secrets/config.json`.
+
+### Locally running the connector
+```
+python main_dev.py spec
+python main_dev.py check --config secrets/config.json
+python main_dev.py discover --config secrets/config.json
+python main_dev.py read --config secrets/config.json --catalog sample_files/sample_catalog.json
+```
+
+### Unit Tests
+To run unit tests locally, from the connector root run:
+```
+pytest unit_tests
+```
+
+### Locally running the connector docker image
+```
+# in airbyte root directory
+./gradlew :airbyte-integrations:connectors:source-shopify-singer:buildImage
+docker run --rm airbyte/source-shopify-singer:dev spec
+docker run --rm -v $(pwd)/airbyte-integrations/connectors/source-shopify-singer/secrets:/secrets airbyte/source-shopify-singer:dev check --config /secrets/config.json
+docker run --rm -v $(pwd)/airbyte-integrations/connectors/source-shopify-singer/secrets:/secrets airbyte/source-shopify-singer:dev discover --config /secrets/config.json
+docker run --rm -v $(pwd)/airbyte-integrations/connectors/source-shopify-singer/secrets:/secrets -v $(pwd)/airbyte-integrations/connectors/source-shopify-singer/sample_files:/sample_files airbyte/source-shopify-singer:dev read --config /secrets/config.json --catalog /sample_files/sample_catalog.json
+```
+
+### Integration Tests
+1. From the airbyte project root, run `./gradlew :airbyte-integrations:connectors:source-shopify-singer:standardSourceTestPython` to run the standard integration test suite.
+1. To run additional integration tests, place your integration tests in the `integration_tests` directory and run them with `pytest integration_tests`.
+   Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
+
+## Dependency Management
+All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.

airbyte-integrations/connectors/source-shopify-singer/airbyte_protocol

@@ -0,0 +1 @@
+../../bases/airbyte-protocol/airbyte_protocol

airbyte-integrations/connectors/source-shopify-singer/base_python

@@ -0,0 +1 @@
+../../bases/base-python/base_python

airbyte-integrations/connectors/source-shopify-singer/base_singer

@@ -0,0 +1 @@
+../../bases/base-singer/base_singer

airbyte-integrations/connectors/source-shopify-singer/build.gradle

@@ -0,0 +1,13 @@
+plugins {
+    id 'airbyte-python'
+    id 'airbyte-docker'
+    id 'airbyte-source-test'
+}
+
+airbytePython {
+    moduleDirectory 'source_shopify_singer'
+}
+
+dependencies {
+    implementation files(project(':airbyte-integrations:bases:base-singer').airbyteDocker.outputs)
+}

airbyte-integrations/connectors/source-shopify-singer/integration_tests/__init__.py

@@ -0,0 +1,27 @@
+"""
+MIT License
+
+Copyright (c) 2020 Airbyte
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+"""
+
+from .standard_source_test import SourceShopifySingerStandardTest
+
+__all__ = ["SourceShopifySingerStandardTest"]

airbyte-integrations/connectors/source-shopify-singer/integration_tests/catalog.json

@@ -0,0 +1,253 @@
+{
+  "streams": [
+    {
+      "name": "products",
+      "json_schema": {
+        "properties": {
+          "published_at": {
+            "type": ["null", "string"],
+            "format": "date-time"
+          },
+          "created_at": {
+            "type": ["null", "string"],
+            "format": "date-time"
+          },
+          "published_scope": {
+            "type": ["null", "string"]
+          },
+          "vendor": {
+            "type": ["null", "string"]
+          },
+          "updated_at": {
+            "type": ["null", "string"],
+            "format": "date-time"
+          },
+          "body_html": {
+            "type": ["null", "string"]
+          },
+          "product_type": {
+            "type": ["null", "string"]
+          },
+          "tags": {
+            "type": ["null", "string"]
+          },
+          "options": {
+            "type": ["null", "array"],
+            "items": {
+              "properties": {
+                "name": {
+                  "type": ["null", "string"]
+                },
+                "product_id": {
+                  "type": ["null", "integer"]
+                },
+                "values": {
+                  "type": ["null", "array"],
+                  "items": {
+                    "type": ["null", "string"]
+                  }
+                },
+                "id": {
+                  "type": ["null", "integer"]
+                },
+                "position": {
+                  "type": ["null", "integer"]
+                }
+              },
+              "type": ["null", "object"]
+            }
+          },
+          "image": {
+            "properties": {
+              "updated_at": {
+                "type": ["null", "string"],
+                "format": "date-time"
+              },
+              "created_at": {
+                "type": ["null", "string"],
+                "format": "date-time"
+              },
+              "variant_ids": {
+                "type": ["null", "array"],
+                "items": {
+                  "type": ["null", "integer"]
+                }
+              },
+              "height": {
+                "type": ["null", "integer"]
+              },
+              "alt": {
+                "type": ["null", "string"]
+              },
+              "src": {
+                "type": ["null", "string"]
+              },
+              "position": {
+                "type": ["null", "integer"]
+              },
+              "id": {
+                "type": ["null", "integer"]
+              },
+              "admin_graphql_api_id": {
+                "type": ["null", "string"]
+              },
+              "width": {
+                "type": ["null", "integer"]
+              }
+            },
+            "type": ["null", "object"]
+          },
+          "handle": {
+            "type": ["null", "string"]
+          },
+          "images": {
+            "type": ["null", "array"],
+            "items": {
+              "properties": {
+                "updated_at": {
+                  "type": ["null", "string"],
+                  "format": "date-time"
+                },
+                "created_at": {
+                  "type": ["null", "string"],
+                  "format": "date-time"
+                },
+                "variant_ids": {
+                  "type": ["null", "array"],
+                  "items": {
+                    "type": ["null", "integer"]
+                  }
+                },
+                "height": {
+                  "type": ["null", "integer"]
+                },
+                "alt": {
+                  "type": ["null", "string"]
+                },
+                "src": {
+                  "type": ["null", "string"]
+                },
+                "position": {
+                  "type": ["null", "integer"]
+                },
+                "id": {
+                  "type": ["null", "integer"]
+                },
+                "admin_graphql_api_id": {
+                  "type": ["null", "string"]
+                },
+                "width": {
+                  "type": ["null", "integer"]
+                }
+              },
+              "type": ["null", "object"]
+            }
+          },
+          "template_suffix": {
+            "type": ["null", "string"]
+          },
+          "title": {
+            "type": ["null", "string"]
+          },
+          "variants": {
+            "type": ["null", "array"],
+            "items": {
+              "properties": {
+                "barcode": {
+                  "type": ["null", "string"]
+                },
+                "tax_code": {
+                  "type": ["null", "string"]
+                },
+                "created_at": {
+                  "type": ["null", "string"],
+                  "format": "date-time"
+                },
+                "weight_unit": {
+                  "type": ["null", "string"]
+                },
+                "id": {
+                  "type": ["null", "integer"]
+                },
+                "position": {
+                  "type": ["null", "integer"]
+                },
+                "price": {
+                  "type": ["null", "number"],
+                  "multipleOf": 1e-10
+                },
+                "image_id": {
+                  "type": ["null", "integer"]
+                },
+                "inventory_policy": {
+                  "type": ["null", "string"]
+                },
+                "sku": {
+                  "type": ["null", "string"]
+                },
+                "inventory_item_id": {
+                  "type": ["null", "integer"]
+                },
+                "fulfillment_service": {
+                  "type": ["null", "string"]
+                },
+                "title": {
+                  "type": ["null", "string"]
+                },
+                "weight": {
+                  "type": ["null", "number"]
+                },
+                "inventory_management": {
+                  "type": ["null", "string"]
+                },
+                "taxable": {
+                  "type": ["null", "boolean"]
+                },
+                "admin_graphql_api_id": {
+                  "type": ["null", "string"]
+                },
+                "option1": {
+                  "type": ["null", "string"]
+                },
+                "compare_at_price": {
+                  "type": ["null", "number"],
+                  "multipleOf": 1e-10
+                },
+                "updated_at": {
+                  "type": ["null", "string"],
+                  "format": "date-time"
+                },
+                "option2": {
+                  "type": ["null", "string"]
+                },
+                "old_inventory_quantity": {
+                  "type": ["null", "integer"]
+                },
+                "requires_shipping": {
+                  "type": ["null", "boolean"]
+                },
+                "inventory_quantity": {
+                  "type": ["null", "integer"]
+                },
+                "grams": {
+                  "type": ["null", "integer"]
+                },
+                "option3": {
+                  "type": ["null", "string"]
+                }
+              },
+              "type": ["null", "object"]
+            }
+          },
+          "admin_graphql_api_id": {
+            "type": ["null", "string"]
+          },
+          "id": {
+            "type": ["null", "integer"]
+          }
+        },
+        "type": "object"
+      }
+    }
+  ]
+}

airbyte-integrations/connectors/source-shopify-singer/integration_tests/standard_source_test.py

@@ -0,0 +1,51 @@
+"""
+MIT License
+
+Copyright (c) 2020 Airbyte
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+"""
+
+import json
+import pkgutil
+
+from airbyte_protocol import AirbyteCatalog, ConnectorSpecification
+from base_python_test import StandardSourceTestIface
+
+
+class SourceShopifySingerStandardTest(StandardSourceTestIface):
+    def __init__(self):
+        pass
+
+    def get_spec(self) -> ConnectorSpecification:
+        raw_spec = pkgutil.get_data(self.__class__.__module__.split(".")[0], "spec.json")
+        return ConnectorSpecification.parse_obj(json.loads(raw_spec))
+
+    def get_config(self) -> object:
+        return json.loads(pkgutil.get_data(self.__class__.__module__.split(".")[0], "config.json"))
+
+    def get_catalog(self) -> AirbyteCatalog:
+        raw_catalog = pkgutil.get_data(self.__class__.__module__.split(".")[0], "catalog.json")
+        return AirbyteCatalog.parse_obj(json.loads(raw_catalog))
+
+    def setup(self) -> None:
+        pass
+
+    def teardown(self) -> None:
+        pass

airbyte-integrations/connectors/source-shopify-singer/main_dev.py

@@ -0,0 +1,32 @@
+"""
+MIT License
+
+Copyright (c) 2020 Airbyte
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+"""
+
+import sys
+
+from base_python.entrypoint import launch
+from source_shopify_singer import SourceShopifySinger
+
+if __name__ == "__main__":
+    source = SourceShopifySinger()
+    launch(source, sys.argv[1:])

airbyte-integrations/connectors/source-shopify-singer/requirements.txt

@@ -0,0 +1,5 @@
+-e ../../bases/airbyte-protocol
+-e ../../bases/base-singer
+-e ../../bases/base-python
+-e ../../bases/base-python-test
+-e .

airbyte-integrations/connectors/source-shopify-singer/sample_files/config.json

@@ -0,0 +1,5 @@
+{
+  "start_date": "2020-01-30",
+  "api_password": "<Shopify API Password>",
+  "shop": "<Shop Name>"
+}

airbyte-integrations/connectors/source-shopify-singer/setup.py

@@ -0,0 +1,44 @@
+"""
+MIT License
+
+Copyright (c) 2020 Airbyte
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+"""
+
+from setuptools import find_packages, setup
+
+setup(
+    name="source_shopify_singer",
+    description="Source implementation for Shopify, built on the Singer tap implementation.",
+    author="Airbyte",
+    author_email="contact@airbyte.io",
+    packages=find_packages(),
+    install_requires=["airbyte-protocol", "tap-shopify==1.2.6", "ShopifyAPI==8.0.4"],
+    package_data={"": ["*.json"]},
+    setup_requires=["pytest-runner"],
+    tests_require=["pytest"],
+    extras_require={
+        # Dependencies required by the main package but not integration tests should go in main. Deps required by
+        # integration tests but not the main package go in integration_tests. Deps required by both should go in
+        # install_requires.
+        "main": ["base-singer", "base-python"],
+        "tests": ["airbyte-python-test", "pytest"],
+    },
+)

airbyte-integrations/connectors/source-shopify-singer/source_shopify_singer/__init__.py

@@ -0,0 +1,27 @@
+"""
+MIT License
+
+Copyright (c) 2020 Airbyte
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+"""
+
+from .source import SourceShopifySinger
+
+__all__ = ["SourceShopifySinger"]

airbyte-integrations/connectors/source-shopify-singer/source_shopify_singer/source.py

@@ -0,0 +1,60 @@
+"""
+MIT License
+
+Copyright (c) 2020 Airbyte
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+"""
+
+import shopify
+from airbyte_protocol import AirbyteConnectionStatus, Status
+from base_python import AirbyteLogger, ConfigContainer
+from base_singer import SingerSource
+
+TAP_CMD = "tap-shopify"
+
+
+class SourceShopifySinger(SingerSource):
+    def __init__(self):
+        super().__init__()
+
+    def transform_config(self, raw_config):
+        return {"start_date": raw_config["start_date"], "api_key": raw_config["api_password"], "shop": raw_config["shop"]}
+
+    def check(self, logger: AirbyteLogger, config_container: ConfigContainer) -> AirbyteConnectionStatus:
+        try:
+            config = config_container.rendered_config
+            session = shopify.Session(f"{config['shop']}.myshopify.com", "2020-10", config["api_key"])
+            shopify.ShopifyResource.activate_session(session)
+            # try to read the name of the shop, which should be available with any level of permissions
+            shopify.GraphQL().execute("{ shop { name id } }")
+            shopify.ShopifyResource.clear_session()
+            return AirbyteConnectionStatus(status=Status.SUCCEEDED)
+        except Exception as e:
+            logger.error(f"Exception connecting to Shopify: ${e}")
+            return AirbyteConnectionStatus(
+                status=Status.FAILED, message="Unable to connect to the Shopify API with the provided credentials."
+            )
+
+    def discover_cmd(self, logger: AirbyteLogger, config_path: str) -> str:
+        return f"{TAP_CMD} -c {config_path} --discover"
+
+    def read_cmd(self, logger: AirbyteLogger, config_path: str, catalog_path: str, state_path: str = None) -> str:
+        state_path = f"--state {state_path}" if state_path else ""
+        return f"{TAP_CMD} -c {config_path} --catalog {catalog_path} {state_path}"

airbyte-integrations/connectors/source-shopify-singer/source_shopify_singer/spec.json

@@ -0,0 +1,24 @@
+{
+  "documentationUrl": "https://docs.airbyte.io/integrations/sources/shopify",
+  "connectionSpecification": {
+    "$schema": "http://json-schema.org/draft-07/schema#",
+    "title": "Shopify Source Spec",
+    "type": "object",
+    "required": ["start_date", "api_password", "shop"],
+    "additionalProperties": false,
+    "properties": {
+      "start_date": {
+        "type": "string",
+        "description": "The start date for data replication in the format YYYY-MM-DD."
+      },
+      "api_password": {
+        "type": "string",
+        "description": "The API password for a private application in Shopify shop."
+      },
+      "shop": {
+        "type": "string",
+        "description": "The name of the shopify store. For https://EXAMPLE.myshopify.com, the shop name is EXAMPLE."
+      }
+    }
+  }
+}

airbyte-integrations/connectors/source-shopify-singer/unit_tests/unit_test.py

@@ -0,0 +1,26 @@
+"""
+MIT License
+
+Copyright (c) 2020 Airbyte
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+"""
+
+def test_example_method():
+    assert True

build.gradle

@@ -77,7 +77,7 @@ spotless {
     python {
         target createSpotlessTarget('**/*.py')
 
-        licenseHeaderFile createPythonLicenseWith(rootProject.file('LICENSE')), '(from|import|# generated)'
+        licenseHeaderFile createPythonLicenseWith(rootProject.file('LICENSE')), '(from|import|def|# generated)'
     }
     format 'styling', {
         target createSpotlessTarget(['**/*.yaml', '**/*.json'])

docs/integrations/sources/shopify.md

@@ -0,0 +1,50 @@
+# Shopify API
+
+## Sync overview
+
+This source can sync data for the [Shopify API](https://help.shopify.com/en/api/reference).
+
+This Source Connector is based on a [Singer Tap](https://github.com/singer-io/tap-shopify).
+
+### Output schema
+
+This Source is capable of syncing the following core Streams:
+
+* [Abandoned Checkouts](https://help.shopify.com/en/api/reference/orders/abandoned_checkouts)
+* [Collects](https://help.shopify.com/en/api/reference/products/collect)
+* [Custom Collections](https://help.shopify.com/en/api/reference/products/customcollection)
+* [Customers](https://help.shopify.com/en/api/reference/customers)
+* [Metafields](https://help.shopify.com/en/api/reference/metafield)
+* [Orders](https://help.shopify.com/en/api/reference/orders)
+* [Products](https://help.shopify.com/en/api/reference/products)
+* [Transactions](https://help.shopify.com/en/api/reference/orders/transaction)
+
+### Data type mapping
+
+| Integration Type | Airbyte Type | Notes |
+| :--- | :--- | :--- |
+| `string` | `string` |  |
+| `number` | `number` |  |
+| `array` | `array` |  |
+| `object` | `object` |  |
+
+### Features
+
+| Feature | Supported?\(Yes/No\) | Notes |
+| :--- | :--- | :--- |
+| Full Refresh Sync | yes |  |
+| Incremental Sync | no |  |
+
+### Performance considerations
+
+Shopify has some [rate limit restrictions](https://shopify.dev/concepts/about-apis/rate-limits).
+
+## Getting started
+
+1. Go to `https://YOURSTORE.myshopify.com/admin/apps/private`
+1. Enable private development if it isn't enabled.
+1. Create a private application.
+1. Select the resources you want to allow access to. Airbyte only needs read-level access. 
+    * Note: The UI will show all possible data sources and will show errors when syncing if it doesn't have permissions to access a resource. 
+1. The password under the `Admin API` section is what you'll use as the `api_password` for the integration.
+1. You're ready to set up Shopify in Airbyte!

tools/bin/ci_credentials.sh

@@ -39,6 +39,10 @@ MKTO_SECRETS_DIR=airbyte-integrations/connectors/source-marketo-singer/secrets
 mkdir $MKTO_SECRETS_DIR
 echo "$SOURCE_MARKETO_SINGER_INTEGRATION_TEST_CONFIG" > "${MKTO_SECRETS_DIR}/config.json"
 
+
+mkdir airbyte-integrations/connectors/source-shopify-singer/secrets
+echo "$SHOPIFY_INTEGRATION_TEST_CREDS" > airbyte-integrations/connectors/source-shopify-singer/secrets/config.json
+
 SOURCEFILE_DIR=airbyte-integrations/connectors/source-file/secrets
 mkdir $SOURCEFILE_DIR
 echo "$BIGQUERY_INTEGRATION_TEST_CREDS" > "${SOURCEFILE_DIR}/gcs.json"