🚨Source The Guardian API: Migrate to poetry (#41049)
Co-authored-by: Natik Gadzhi <natik@respawn.io> Co-authored-by: ChristoGrab <christo.grab@gmail.com>
This commit is contained in:
btkcodedev
@@ -1,6 +0,0 @@
|
||||
*
|
||||
!Dockerfile
|
||||
!main.py
|
||||
!source_the_guardian_api
|
||||
!setup.py
|
||||
!secrets
|
||||
@@ -1,38 +0,0 @@
|
||||
FROM python:3.9.11-alpine3.15 as base
|
||||
|
||||
# build and load all requirements
|
||||
FROM base as builder
|
||||
WORKDIR /airbyte/integration_code
|
||||
|
||||
# upgrade pip to the latest version
|
||||
RUN apk --no-cache upgrade \
|
||||
&& pip install --upgrade pip \
|
||||
&& apk --no-cache add tzdata build-base
|
||||
|
||||
|
||||
COPY setup.py ./
|
||||
# install necessary packages to a temporary folder
|
||||
RUN pip install --prefix=/install .
|
||||
|
||||
# build a clean environment
|
||||
FROM base
|
||||
WORKDIR /airbyte/integration_code
|
||||
|
||||
# copy all loaded and built libraries to a pure basic image
|
||||
COPY --from=builder /install /usr/local
|
||||
# add default timezone settings
|
||||
COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
|
||||
RUN echo "Etc/UTC" > /etc/timezone
|
||||
|
||||
# bash is installed for more convenient debugging.
|
||||
RUN apk --no-cache add bash
|
||||
|
||||
# copy payload code only
|
||||
COPY main.py ./
|
||||
COPY source_the_guardian_api ./source_the_guardian_api
|
||||
|
||||
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
|
||||
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
|
||||
|
||||
LABEL io.airbyte.version=0.1.0
|
||||
LABEL io.airbyte.name=airbyte/source-the-guardian-api
|
||||
@@ -17,22 +17,70 @@ and place them into `secrets/config.json`.
|
||||
|
||||
### Locally running the connector docker image
|
||||
|
||||
#### Build
|
||||
|
||||
**Via [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) (recommended):**
|
||||
|
||||
#### Use `airbyte-ci` to build your connector
|
||||
The Airbyte way of building this connector is to use our `airbyte-ci` tool.
|
||||
You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1).
|
||||
Then running the following command will build your connector:
|
||||
|
||||
```bash
|
||||
airbyte-ci connectors --name=source-the-guardian-api build
|
||||
airbyte-ci connectors --name source-the-guardian-api build
|
||||
```
|
||||
Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-the-guardian-api:dev`.
|
||||
|
||||
##### Customizing our build process
|
||||
When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
|
||||
You can customize our build process by adding a `build_customization.py` module to your connector.
|
||||
This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively.
|
||||
It will be imported at runtime by our build process and the functions will be called if they exist.
|
||||
|
||||
Here is an example of a `build_customization.py` module:
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
# Feel free to check the dagger documentation for more information on the Container object and its methods.
|
||||
# https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/
|
||||
from dagger import Container
|
||||
|
||||
|
||||
async def pre_connector_install(base_image_container: Container) -> Container:
|
||||
return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")
|
||||
|
||||
async def post_connector_install(connector_container: Container) -> Container:
|
||||
return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")
|
||||
```
|
||||
|
||||
An image will be built with the tag `airbyte/source-the-guardian-api:dev`.
|
||||
#### Build your own connector image
|
||||
This connector is built using our dynamic built process in `airbyte-ci`.
|
||||
The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`.
|
||||
The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py).
|
||||
It does not rely on a Dockerfile.
|
||||
|
||||
**Via `docker build`:**
|
||||
If you would like to patch our connector and build your own a simple approach would be to:
|
||||
|
||||
1. Create your own Dockerfile based on the latest version of the connector image.
|
||||
```Dockerfile
|
||||
FROM airbyte/source-the-guardian-api:latest
|
||||
|
||||
COPY . ./airbyte/integration_code
|
||||
RUN pip install ./airbyte/integration_code
|
||||
|
||||
# The entrypoint and default env vars are already set in the base image
|
||||
# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
|
||||
# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
|
||||
```
|
||||
Please use this as an example. This is not optimized.
|
||||
|
||||
2. Build your image:
|
||||
```bash
|
||||
docker build -t airbyte/source-the-guardian-api:dev .
|
||||
# Running the spec command against your patched connector
|
||||
docker run airbyte/source-the-guardian-api:dev spec
|
||||
```
|
||||
|
||||
#### Run
|
||||
|
||||
Then run any of the connector commands as follows:
|
||||
@@ -75,4 +123,4 @@ You've checked out the repo, implemented a million dollar feature, and you're re
|
||||
4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/the-guardian-api.md`).
|
||||
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
|
||||
6. Pat yourself on the back for being an awesome contributor.
|
||||
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
|
||||
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
|
||||
@@ -1,5 +1,13 @@
|
||||
{
|
||||
"content": {
|
||||
"webPublicationDate": "2123-10-31T10:10:10Z"
|
||||
[
|
||||
{
|
||||
"type": "STREAM",
|
||||
"stream": {
|
||||
"stream_state": {
|
||||
"webPublicationDate": "2123-10-31T10:10:10Z"
|
||||
},
|
||||
"stream_descriptor": {
|
||||
"name": "content"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
@@ -1,37 +1,39 @@
|
||||
data:
|
||||
ab_internal:
|
||||
ql: 100
|
||||
sl: 100
|
||||
connectorBuildOptions:
|
||||
baseImage: docker.io/airbyte/python-connector-base:1.2.2@sha256:57703de3b4c4204bd68a7b13c9300f8e03c0189bffddaffc796f1da25d2dbea0
|
||||
connectorSubtype: api
|
||||
connectorTestSuitesOptions:
|
||||
- suite: acceptanceTests
|
||||
testSecrets:
|
||||
- fileName: config.json
|
||||
name: SECRET_SOURCE-THE-GUARDIAN-API__CREDS
|
||||
secretStore:
|
||||
alias: airbyte-connector-testing-secret-store
|
||||
type: GSM
|
||||
connectorType: source
|
||||
definitionId: d42bd69f-6bf0-4d0b-9209-16231af07a92
|
||||
dockerImageTag: 0.1.0
|
||||
dockerImageTag: 0.1.1
|
||||
dockerRepository: airbyte/source-the-guardian-api
|
||||
documentationUrl: https://docs.airbyte.com/integrations/sources/the-guardian-api
|
||||
githubIssueLabel: source-the-guardian-api
|
||||
icon: theguardian.svg
|
||||
license: MIT
|
||||
name: The Guardian API
|
||||
remoteRegistries:
|
||||
pypi:
|
||||
enabled: true
|
||||
packageName: airbyte-source-the-guardian-api
|
||||
registries:
|
||||
cloud:
|
||||
enabled: true
|
||||
oss:
|
||||
enabled: true
|
||||
releaseStage: alpha
|
||||
documentationUrl: https://docs.airbyte.com/integrations/sources/the-guardian-api
|
||||
remoteRegistries:
|
||||
pypi:
|
||||
enabled: true
|
||||
packageName: airbyte-source-the-guardian-api
|
||||
supportLevel: community
|
||||
tags:
|
||||
- language:python
|
||||
- cdk:low-code
|
||||
ab_internal:
|
||||
sl: 100
|
||||
ql: 100
|
||||
supportLevel: community
|
||||
connectorTestSuitesOptions:
|
||||
- suite: acceptanceTests
|
||||
testSecrets:
|
||||
- name: SECRET_SOURCE-THE-GUARDIAN-API__CREDS
|
||||
fileName: config.json
|
||||
secretStore:
|
||||
type: GSM
|
||||
alias: airbyte-connector-testing-secret-store
|
||||
metadataSpecVersion: "1.0"
|
||||
|
||||
1030
airbyte-integrations/connectors/source-the-guardian-api/poetry.lock
generated
Normal file
1030
airbyte-integrations/connectors/source-the-guardian-api/poetry.lock
generated
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,29 @@
|
||||
[build-system]
|
||||
requires = [ "poetry-core>=1.0.0",]
|
||||
build-backend = "poetry.core.masonry.api"
|
||||
|
||||
[tool.poetry]
|
||||
version = "0.1.1"
|
||||
name = "source-the-guardian-api"
|
||||
description = "Source implementation for the guardian api."
|
||||
authors = [ "Airbyte <contact@airbyte.io>",]
|
||||
license = "MIT"
|
||||
readme = "README.md"
|
||||
documentation = "https://docs.airbyte.com/integrations/sources/the-guardian-api"
|
||||
homepage = "https://airbyte.com"
|
||||
repository = "https://github.com/airbytehq/airbyte"
|
||||
[[tool.poetry.packages]]
|
||||
include = "source_the_guardian_api"
|
||||
|
||||
[tool.poetry.dependencies]
|
||||
python = "^3.9,<3.12"
|
||||
airbyte-cdk = "0.60.0"
|
||||
|
||||
[tool.poetry.scripts]
|
||||
source-the-guardian-api = "source_the_guardian_api.run:run"
|
||||
|
||||
[tool.poetry.group.dev.dependencies]
|
||||
requests-mock = "^1.11.0"
|
||||
pytest = "^8.0.0"
|
||||
freezegun = "^1.4.0"
|
||||
pytest-mock = "^3.6.1"
|
||||
@@ -1 +0,0 @@
|
||||
-e .
|
||||
@@ -1,46 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
|
||||
from setuptools import find_packages, setup
|
||||
|
||||
MAIN_REQUIREMENTS = [
|
||||
"airbyte-cdk~=0.1",
|
||||
]
|
||||
|
||||
TEST_REQUIREMENTS = [
|
||||
"requests-mock~=1.9.3",
|
||||
"pytest~=6.1",
|
||||
"pytest-mock~=3.6.1",
|
||||
]
|
||||
|
||||
setup(
|
||||
entry_points={
|
||||
"console_scripts": [
|
||||
"source-the-guardian-api=source_the_guardian_api.run:run",
|
||||
],
|
||||
},
|
||||
name="source_the_guardian_api",
|
||||
description="Source implementation for The Guardian Api.",
|
||||
author="Airbyte",
|
||||
author_email="contact@airbyte.io",
|
||||
packages=find_packages(),
|
||||
install_requires=MAIN_REQUIREMENTS,
|
||||
package_data={
|
||||
"": [
|
||||
# Include yaml files in the package (if any)
|
||||
"*.yml",
|
||||
"*.yaml",
|
||||
# Include all json files in the package, up to 4 levels deep
|
||||
"*.json",
|
||||
"*/*.json",
|
||||
"*/*/*.json",
|
||||
"*/*/*/*.json",
|
||||
"*/*/*/*/*.json",
|
||||
]
|
||||
},
|
||||
extras_require={
|
||||
"tests": TEST_REQUIREMENTS,
|
||||
},
|
||||
)
|
||||
@@ -18,7 +18,7 @@ definitions:
|
||||
# from-date: "{{ config['start_date'] }}"
|
||||
# to-date: "{{ config['end_date'] or now_utc().strftime('%Y-%m-%d') }}"
|
||||
incremental_sync:
|
||||
type: "DatetimeBasedCursor"
|
||||
type: DatetimeBasedCursor
|
||||
start_datetime:
|
||||
datetime: "{{ config['start_date'] }}"
|
||||
datetime_format: "%Y-%m-%d"
|
||||
|
||||
@@ -113,6 +113,7 @@ The key that you are assigned is rate-limited and as such any applications that
|
||||
|
||||
| Version | Date | Pull Request | Subject |
|
||||
| :------ | :--------- | :-------------------------------------------------------- | :--------------------------------------------- |
|
||||
| 0.1.1 | 2024-07-10 | [41049](https://github.com/airbytehq/airbyte/pull/41049) | Migrate to poetry |
|
||||
| 0.1.0 | 2022-10-30 | [#18654](https://github.com/airbytehq/airbyte/pull/18654) | 🎉 New Source: The Guardian API [low-code CDK] |
|
||||
|
||||
</details>
|
||||
Reference in New Issue
Block a user