chore: delete Airbyte CI Base Images Package (#64135)
This commit is contained in:
David Gold
@@ -34,7 +34,6 @@ jobs:
|
||||
# This list is duplicated in `pipelines/airbyte_ci/test/__init__.py`
|
||||
internal_poetry_packages:
|
||||
- airbyte-ci/connectors/pipelines/**
|
||||
- airbyte-ci/connectors/base_images/**
|
||||
- airbyte-ci/connectors/connectors_insights/**
|
||||
- airbyte-ci/connectors/connector_ops/**
|
||||
- airbyte-ci/connectors/connectors_qa/**
|
||||
|
||||
@@ -9,7 +9,6 @@ The installation instructions for the `airbyte-ci` CLI tool cal be found here
|
||||
|
||||
| Directory | Description |
|
||||
| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
|
||||
| [`base_images`](connectors/base_images) | A set of tools to build and publish Airbyte base connector images. |
|
||||
| [`ci_credentials`](connectors/ci_credentials) | A CLI tool to fetch connector secrets from GCP Secrets Manager. |
|
||||
| [`connector_ops`](connectors/connector_ops) | A python package with utils reused in internal packages. |
|
||||
| [`connectors_qa`](connectors/connectors_qa/) | A tool to verify connectors have sounds assets and metadata. |
|
||||
|
||||
@@ -1,158 +0,0 @@
|
||||
# airbyte-connectors-base-images
|
||||
|
||||
This python package contains the base images used by Airbyte connectors.
|
||||
It is intended to be used as a python library.
|
||||
Our connector build pipeline ([`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1)) uses this library to build the connector images.
|
||||
Our base images are declared in code, using the [Dagger Python SDK](https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/).
|
||||
|
||||
- [Python base image code declaration](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/base_images/base_images/python/bases.py)
|
||||
- [Java base image code declaration](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/base_images/base_images/java/bases.py)
|
||||
|
||||
|
||||
## Where are the Dockerfiles?
|
||||
Our base images are not declared using Dockerfiles.
|
||||
They are declared in code using the [Dagger Python SDK](https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/).
|
||||
We prefer this approach because it allows us to interact with base images container as code: we can use python to declare the base images and use the full power of the language to build and test them.
|
||||
However, we do artificially generate Dockerfiles for debugging and documentation purposes.
|
||||
|
||||
|
||||
|
||||
### Example for `airbyte/python-connector-base`:
|
||||
```dockerfile
|
||||
FROM docker.io/python:3.11.13-slim-bookworm@sha256:139020233cc412efe4c8135b0efe1c7569dc8b28ddd88bddb109b764f8977e30
|
||||
RUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime
|
||||
RUN adduser --uid 1000 --system --group --no-create-home airbyte
|
||||
RUN mkdir --mode 755 /custom_cache
|
||||
RUN mkdir --mode 755 /airbyte
|
||||
RUN chown airbyte:airbyte /airbyte
|
||||
ENV PIP_CACHE_DIR=/custom_cache/pip
|
||||
RUN pip install --upgrade pip==24.0 setuptools==70.0.0
|
||||
ENV POETRY_VIRTUALENVS_CREATE=false
|
||||
ENV POETRY_VIRTUALENVS_IN_PROJECT=false
|
||||
ENV POETRY_NO_INTERACTION=1
|
||||
RUN pip install poetry==1.8.4
|
||||
RUN sh -c apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y && apt-get clean
|
||||
RUN sh -c apt-get install -y socat=1.7.4.4-2
|
||||
RUN sh -c apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+deb12u1
|
||||
RUN mkdir -p 755 /usr/share/nltk_data
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Example for `airbyte/java-connector-base`:
|
||||
```dockerfile
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Base images
|
||||
|
||||
|
||||
### `airbyte/python-connector-base`
|
||||
|
||||
| Version | Published | Docker Image Address | Changelog |
|
||||
|---------|-----------|--------------|-----------|
|
||||
| 4.0.2 | ✅| | Bump to Python 3.11.13 to patch security vulnerabilities |
|
||||
| 4.0.1-rc.1 | ✅| docker.io/airbyte/python-connector-base:4.0.1-rc.1@sha256:5d1d8216467b29cf8a65d42e9c3316cdb5e62529ec3bcb14cfa993f5221316ed | Bump to Python 3.11.13 to patch security vulnerabilities |
|
||||
| 4.0.0 | ✅| docker.io/airbyte/python-connector-base:4.0.0@sha256:d9894b6895923b379f3006fa251147806919c62b7d9021b5cd125bb67d7bbe22 | Python 3.11.11 |
|
||||
| 4.0.0-rc.1 | ✅| docker.io/airbyte/python-connector-base:4.0.0-rc.1@sha256:0f3b3baab5ee7a040dd9f01b436afabf023b86616e21215ef7467e0501b463ea | |
|
||||
| 3.0.2 | ✅| docker.io/airbyte/python-connector-base:3.0.2@sha256:73697fbe1c0e2ebb8ed58e2268484bb4bfb2cb56b653808e1680cbc50bafef75 | |
|
||||
| 3.0.1-rc.1 | ✅| docker.io/airbyte/python-connector-base:3.0.1-rc.1@sha256:5b5cbe613691cc61d643a347ee4ba21d6358252f274ebb040ef820c7b9f4a5a7 | |
|
||||
| 3.0.0 | ✅| docker.io/airbyte/python-connector-base:3.0.0@sha256:1a0845ff2b30eafa793c6eee4e8f4283c2e52e1bbd44eed6cb9e9abd5d34d844 | Create airbyte user |
|
||||
| 3.0.0-rc.1 | ✅| docker.io/airbyte/python-connector-base:3.0.0-rc.1@sha256:ee046486af9ad90b1b248afe5e92846b51375a21463dff1cd377c4f06abb55b5 | Update Python 3.10.4 image + create airbyte user |
|
||||
| 2.0.0 | ✅| docker.io/airbyte/python-connector-base:2.0.0@sha256:c44839ba84406116e8ba68722a0f30e8f6e7056c726f447681bb9e9ece8bd916 | Use Python 3.10 |
|
||||
| 1.2.3 | ✅| docker.io/airbyte/python-connector-base:1.2.3@sha256:a8abfdc75f8e22931657a1ae15069e7b925e74bb7b5ef36371a85e4caeae5696 | Use latest root image version and update system packages |
|
||||
| 1.2.2 | ✅| docker.io/airbyte/python-connector-base:1.2.2@sha256:57703de3b4c4204bd68a7b13c9300f8e03c0189bffddaffc796f1da25d2dbea0 | Fix Python 3.9.19 image digest |
|
||||
| 1.2.2-rc.1 | ✅| docker.io/airbyte/python-connector-base:1.2.2-rc.1@sha256:a8abfdc75f8e22931657a1ae15069e7b925e74bb7b5ef36371a85e4caeae5696 | Create an airbyte user and use it |
|
||||
| 1.2.1 | ✅| docker.io/airbyte/python-connector-base:1.2.1@sha256:4a4255e2bccab71fa5912487e42d9755cdecffae77273fed8be01a081cd6e795 | Upgrade to Python 3.9.19 + update pip and setuptools |
|
||||
| 1.2.0 | ✅| docker.io/airbyte/python-connector-base:1.2.0@sha256:c22a9d97464b69d6ef01898edf3f8612dc11614f05a84984451dde195f337db9 | Add CDK system dependencies: nltk data, tesseract, poppler. |
|
||||
| 1.2.0-rc.1 | ✅| docker.io/airbyte/python-connector-base:1.2.0-rc.1@sha256:f6467768b75fb09125f6e6b892b6b48c98d9fe085125f3ff4adc722afb1e5b30 | |
|
||||
| 1.1.0 | ✅| docker.io/airbyte/python-connector-base:1.1.0@sha256:bd98f6505c6764b1b5f99d3aedc23dfc9e9af631a62533f60eb32b1d3dbab20c | Install socat |
|
||||
| 1.0.0 | ✅| docker.io/airbyte/python-connector-base:1.0.0@sha256:dd17e347fbda94f7c3abff539be298a65af2d7fc27a307d89297df1081a45c27 | Initial release: based on Python 3.9.18, on slim-bookworm system, with pip==23.2.1 and poetry==1.6.1 |
|
||||
|
||||
|
||||
### `airbyte/java-connector-base`
|
||||
|
||||
| Version | Published | Docker Image Address | Changelog |
|
||||
|---------|-----------|--------------|-----------|
|
||||
| 2.1.0-test-1 | ✅| docker.io/airbyte/java-connector-base:2.1.0-test-1@sha256:68ae2f8041efa3df55395013a192e35480f7a8129c9c11ec8924f8a39d55ffe6 | |
|
||||
| 2.1.0-test | ✅| docker.io/airbyte/java-connector-base:2.1.0-test@sha256:14c7ea45c1f83e5a16f02fef54d3b7d855882eb050360c10ce2dcececfdad373 | |
|
||||
| 2.0.3 | ✅| docker.io/airbyte/java-connector-base:2.0.3@sha256:119b8506bca069bbc8357a275936c7e2b0994e6947b81f1bf8d6ce9e16db7d47 | |
|
||||
| 2.0.2 | ✅| docker.io/airbyte/java-connector-base:2.0.2@sha256:f8e47304842a2c4d75ac223cf4b3c4117aa1c5c9207149369d296616815fe5b0 | |
|
||||
| 2.0.1 | ✅| docker.io/airbyte/java-connector-base:2.0.1@sha256:ec89bd1a89e825514dd2fc8730ba299a3ae1544580a078df0e35c5202c2085b3 | Bump Amazon Coretto image version for compatibility with Apple M4 architecture. |
|
||||
| 2.0.0 | ✅| docker.io/airbyte/java-connector-base:2.0.0@sha256:5a1a21c75c5e1282606de9fa539ba136520abe2fbd013058e988bb0297a9f454 | ~Release non root base image~ |
|
||||
| 2.0.0-rc.2 | ✅| docker.io/airbyte/java-connector-base:2.0.0-rc.2@sha256:e5543b3de4c38e9ef45dba886bad5ee319b0d7bfe921f310c788f1d4466e25eb | Fine tune permissions and reproduce platform java base implementation |
|
||||
| 2.0.0-rc.1 | ✅| docker.io/airbyte/java-connector-base:2.0.0-rc.1@sha256:484b929684b9e4f60d06cde171ee0b8238802cb434403293fcede81c1e73c537 | Make the java base image non root |
|
||||
| 1.0.0 | ✅| docker.io/airbyte/java-connector-base:1.0.0@sha256:be86e5684e1e6d9280512d3d8071b47153698fe08ad990949c8eeff02803201a | Create a base image for our java connectors based on Amazon Corretto. |
|
||||
| 1.0.0-rc.4 | ✅| docker.io/airbyte/java-connector-base:1.0.0-rc.4@sha256:be86e5684e1e6d9280512d3d8071b47153698fe08ad990949c8eeff02803201a | Bundle yum calls in a single RUN |
|
||||
| 1.0.0-rc.3 | ✅| docker.io/airbyte/java-connector-base:1.0.0-rc.3@sha256:be86e5684e1e6d9280512d3d8071b47153698fe08ad990949c8eeff02803201a | |
|
||||
| 1.0.0-rc.2 | ✅| docker.io/airbyte/java-connector-base:1.0.0-rc.2@sha256:fca66e81b4d2e4869a03b57b1b34beb048e74f5d08deb2046c3bb9919e7e2273 | Set entrypoint to base.sh |
|
||||
| 1.0.0-rc.1 | ✅| docker.io/airbyte/java-connector-base:1.0.0-rc.1@sha256:886a7ce7eccfe3c8fb303511d0e46b83b7edb4f28e3705818c090185ba511fe7 | Create a base image for our java connectors. |
|
||||
|
||||
|
||||
## How to release a new base image version (example for Python)
|
||||
|
||||
### Requirements
|
||||
* [Docker](https://docs.docker.com/get-docker/)
|
||||
* [Poetry](https://python-poetry.org/docs/#installation)
|
||||
* Dockerhub logins
|
||||
|
||||
### Steps
|
||||
1. `poetry install`
|
||||
2. Open `base_images/python/bases.py`.
|
||||
3. Make changes to the `AirbytePythonConnectorBaseImage`, you're likely going to change the `get_container` method to change the base image.
|
||||
4. Implement the `container` property which must return a `dagger.Container` object.
|
||||
5. **Recommended**: Add new sanity checks to `run_sanity_check` to confirm that the new version is working as expected.
|
||||
6. Cut a new base image version by running `poetry run generate-release`. You'll need your DockerHub credentials.
|
||||
|
||||
It will:
|
||||
- Prompt you to pick which base image you'd like to publish.
|
||||
- Prompt you for a major/minor/patch/pre-release version bump.
|
||||
- Prompt you for a changelog message.
|
||||
- Run the sanity checks on the new version.
|
||||
- Optional: Publish the new version to DockerHub.
|
||||
- Regenerate the docs and the registry json file.
|
||||
7. Commit and push your changes.
|
||||
8. Create a PR and ask for a review from the Connector Operations team.
|
||||
|
||||
**Please note that if you don't publish your image while cutting the new version you can publish it later with `poetry run publish <repository> <version>`.**
|
||||
No connector will use the new base image version until its metadata is updated to use it.
|
||||
If you're not fully confident with the new base image version please:
|
||||
- please publish it as a pre-release version
|
||||
- try out the new version on a couple of connectors
|
||||
- cut a new version with a major/minor/patch bump and publish it
|
||||
- This steps can happen in different PRs.
|
||||
|
||||
|
||||
## Running tests locally
|
||||
```bash
|
||||
poetry run pytest
|
||||
# Static typing checks
|
||||
poetry run mypy base_images --check-untyped-defs
|
||||
```
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### 1.4.0
|
||||
- Declare a base image for our java connectors.
|
||||
|
||||
### 1.3.1
|
||||
- Update the crane image address. The previous address was deleted by the maintainer.
|
||||
|
||||
### 1.2.0
|
||||
- Improve new version prompt to pick bump type with optional pre-release version.
|
||||
|
||||
### 1.1.0
|
||||
- Add a cache ttl for base image listing to avoid DockerHub rate limiting.
|
||||
|
||||
### 1.0.4
|
||||
- Upgrade Dagger to `0.13.3`
|
||||
|
||||
### 1.0.2
|
||||
|
||||
- Improved support for images with non-semantic-versioned tags.
|
||||
|
||||
### 1.0.1
|
||||
|
||||
- Bumped dependencies ([#42581](https://github.com/airbytehq/airbyte/pull/42581))
|
||||
@@ -1,7 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
from rich.console import Console
|
||||
|
||||
console = Console()
|
||||
@@ -1,121 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
"""This module declares common (abstract) classes and methods used by all base images."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import final
|
||||
|
||||
import dagger
|
||||
import semver
|
||||
|
||||
from .published_image import PublishedImage
|
||||
|
||||
|
||||
class AirbyteConnectorBaseImage(ABC):
|
||||
"""An abstract class that represents an Airbyte base image.
|
||||
Please do not declare any Dagger with_exec instruction in this class as in the abstract class context we have no guarantee about the underlying system used in the base image.
|
||||
"""
|
||||
|
||||
USER: str = "airbyte"
|
||||
USER_ID: int = 1000
|
||||
CACHE_DIR_PATH: str = "/custom_cache"
|
||||
AIRBYTE_DIR_PATH: str = "/airbyte"
|
||||
|
||||
@final
|
||||
def __init__(self, dagger_client: dagger.Client, version: semver.VersionInfo):
|
||||
"""Initializes the Airbyte base image.
|
||||
|
||||
Args:
|
||||
dagger_client (dagger.Client): The dagger client used to build the base image.
|
||||
version (semver.VersionInfo): The version of the base image.
|
||||
"""
|
||||
self.dagger_client = dagger_client
|
||||
self.version = version
|
||||
|
||||
# INSTANCE PROPERTIES:
|
||||
|
||||
@property
|
||||
def name_with_tag(self) -> str:
|
||||
"""Returns the full name of the Airbyte base image, with its tag.
|
||||
|
||||
Returns:
|
||||
str: The full name of the Airbyte base image, with its tag.
|
||||
"""
|
||||
return f"{self.repository}:{self.version}"
|
||||
|
||||
# Child classes should define their root image if the image is indeed managed by base_images.
|
||||
@property
|
||||
def root_image(self) -> PublishedImage:
|
||||
"""Returns the base image used to build the Airbyte base image.
|
||||
|
||||
Raises:
|
||||
NotImplementedError: Raised if a subclass does not define a 'root_image' attribute.
|
||||
|
||||
Returns:
|
||||
PublishedImage: The base image used to build the Airbyte base image.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must define a 'root_image' attribute.")
|
||||
|
||||
# MANDATORY SUBCLASSES ATTRIBUTES / PROPERTIES:
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def repository(self) -> str:
|
||||
"""This is the name of the repository where the image will be hosted.
|
||||
e.g: airbyte/python-connector-base
|
||||
|
||||
Raises:
|
||||
NotImplementedError: Raised if a subclass does not define an 'repository' attribute.
|
||||
|
||||
Returns:
|
||||
str: The repository name where the image will be hosted.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must define an 'repository' attribute.")
|
||||
|
||||
# MANDATORY SUBCLASSES METHODS:
|
||||
|
||||
@abstractmethod
|
||||
def get_container(self, platform: dagger.Platform) -> dagger.Container:
|
||||
"""Returns the container of the Airbyte connector base image."""
|
||||
raise NotImplementedError("Subclasses must define a 'get_container' method.")
|
||||
|
||||
@abstractmethod
|
||||
async def run_sanity_checks(self, platform: dagger.Platform):
|
||||
"""Runs sanity checks on the base image container.
|
||||
This method is called before image publication.
|
||||
|
||||
Args:
|
||||
base_image_version (AirbyteConnectorBaseImage): The base image version on which the sanity checks should run.
|
||||
|
||||
Raises:
|
||||
SanityCheckError: Raised if a sanity check fails.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must define a 'run_sanity_checks' method.")
|
||||
|
||||
# INSTANCE METHODS:
|
||||
@final
|
||||
def get_base_container(self, platform: dagger.Platform) -> dagger.Container:
|
||||
"""Returns a container using the base image.
|
||||
This container is used to build the Airbyte base image.
|
||||
We create the user 'airbyte' with the UID 1000 and GID 1000.
|
||||
|
||||
Returns:
|
||||
dagger.Container: The container using the base python image.
|
||||
"""
|
||||
return (
|
||||
self.dagger_client.container(platform=platform)
|
||||
.from_(self.root_image.address)
|
||||
# Set the timezone to UTC
|
||||
.with_exec(["ln", "-snf", "/usr/share/zoneinfo/Etc/UTC", "/etc/localtime"])
|
||||
# Create the user 'airbyte' with the UID 1000 and GID 1000
|
||||
.with_exec(["adduser", "--uid", str(self.USER_ID), "--system", "--group", "--no-create-home", self.USER])
|
||||
# Create the cache airbyte directories and set the right permissions
|
||||
.with_exec(["mkdir", "--mode", "755", self.CACHE_DIR_PATH])
|
||||
.with_exec(["mkdir", "--mode", "755", self.AIRBYTE_DIR_PATH])
|
||||
# Change the owner of the airbyte directory to the user 'airbyte'
|
||||
.with_exec(["chown", f"{self.USER}:{self.USER}", self.AIRBYTE_DIR_PATH])
|
||||
)
|
||||
@@ -1,198 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from typing import Callable, Type
|
||||
|
||||
import anyio
|
||||
import dagger
|
||||
import inquirer # type: ignore
|
||||
import semver
|
||||
from jinja2 import Environment, FileSystemLoader
|
||||
|
||||
from base_images import bases, console, consts, errors, hacks, publish, utils, version_registry
|
||||
|
||||
|
||||
async def _generate_docs(dagger_client: dagger.Client):
|
||||
"""This function will generate the README.md file from the templates/README.md.j2 template.
|
||||
It will first load all the registries to render the template with up to date information.
|
||||
"""
|
||||
docker_credentials = utils.docker.get_credentials()
|
||||
env = Environment(loader=FileSystemLoader("base_images/templates"))
|
||||
template = env.get_template("README.md.j2")
|
||||
rendered_template = template.render({"registries": await version_registry.get_all_registries(dagger_client, docker_credentials)})
|
||||
with open("README.md", "w") as readme:
|
||||
readme.write(rendered_template)
|
||||
console.log("README.md generated successfully.")
|
||||
|
||||
|
||||
async def _generate_release(dagger_client: dagger.Client):
|
||||
"""This function will cut a new version on top of the previous one. It will prompt the user for release details: version bump, changelog entry.
|
||||
The user can optionally publish the new version to our remote registry.
|
||||
If the version is not published its changelog entry is still persisted.
|
||||
It can later be published by running the publish command.
|
||||
In the future we might only allow publishing new pre-release versions from this flow.
|
||||
"""
|
||||
docker_credentials = utils.docker.get_credentials()
|
||||
select_base_image_class_answers = inquirer.prompt(
|
||||
[
|
||||
inquirer.List(
|
||||
"BaseImageClass",
|
||||
message="Which base image would you like to release a new version for?",
|
||||
choices=[(BaseImageClass.repository, BaseImageClass) for BaseImageClass in version_registry.MANAGED_BASE_IMAGES],
|
||||
)
|
||||
]
|
||||
)
|
||||
BaseImageClass = select_base_image_class_answers["BaseImageClass"]
|
||||
registry = await version_registry.VersionRegistry.load(BaseImageClass, dagger_client, docker_credentials)
|
||||
latest_entry = registry.latest_entry
|
||||
|
||||
# If theres in no latest entry, it means we have no version yet: the registry is empty
|
||||
# New version will be cut on top of 0.0.0 so this one will actually never be published
|
||||
seed_version = semver.VersionInfo.parse("0.0.0")
|
||||
if latest_entry is None:
|
||||
latest_version = seed_version
|
||||
else:
|
||||
latest_version = latest_entry.version
|
||||
|
||||
if latest_version != seed_version and not latest_entry.published: # type: ignore
|
||||
console.log(
|
||||
f"The latest version of {BaseImageClass.repository} ({latest_version}) has not been published yet. Please publish it first before cutting a new version."
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
new_version_answers = inquirer.prompt(
|
||||
[
|
||||
inquirer.List(
|
||||
"new_version",
|
||||
message=f"Which kind of new version would you like to cut? (latest version is {latest_version}))",
|
||||
choices=[
|
||||
("prerelease", latest_version.bump_prerelease()),
|
||||
("finalize", latest_version.finalize_version()),
|
||||
("patch", latest_version.bump_patch()),
|
||||
("patch-prerelease", latest_version.bump_patch().bump_prerelease()),
|
||||
("minor", latest_version.bump_minor()),
|
||||
("minor-prerelease", latest_version.bump_minor().bump_prerelease()),
|
||||
("major", latest_version.bump_major()),
|
||||
("major-prerelease", latest_version.bump_major().bump_prerelease()),
|
||||
],
|
||||
),
|
||||
inquirer.Text("changelog_entry", message="What should the changelog entry be?", validate=lambda _, entry: len(entry) > 0),
|
||||
inquirer.Confirm("publish_now", message="Would you like to publish it to our remote registry now?"),
|
||||
]
|
||||
)
|
||||
|
||||
new_version, changelog_entry, publish_now = (
|
||||
new_version_answers["new_version"],
|
||||
new_version_answers["changelog_entry"],
|
||||
new_version_answers["publish_now"],
|
||||
)
|
||||
|
||||
base_image_version = BaseImageClass(dagger_client, new_version)
|
||||
|
||||
try:
|
||||
await publish.run_sanity_checks(base_image_version)
|
||||
console.log("Sanity checks passed.")
|
||||
except errors.SanityCheckError as e:
|
||||
console.log(f"Sanity checks failed: {e}")
|
||||
console.log("Aborting.")
|
||||
sys.exit(1)
|
||||
dockerfile_example = hacks.get_container_dockerfile(base_image_version.get_container(consts.PLATFORMS_WE_PUBLISH_FOR[0]))
|
||||
|
||||
# Add this step we can create a changelog entry: sanity checks passed, image built successfully and sanity checks passed.
|
||||
changelog_entry = version_registry.ChangelogEntry(new_version, changelog_entry, dockerfile_example)
|
||||
if publish_now:
|
||||
published_docker_image = await publish.publish_to_remote_registry(base_image_version)
|
||||
console.log(f"Published {published_docker_image.address} successfully.")
|
||||
else:
|
||||
published_docker_image = None
|
||||
console.log(
|
||||
f"Skipping publication. You can publish it later by running `poetry run publish {base_image_version.repository} {new_version}`."
|
||||
)
|
||||
|
||||
new_registry_entry = version_registry.VersionRegistryEntry(published_docker_image, changelog_entry, new_version)
|
||||
registry.add_entry(new_registry_entry)
|
||||
console.log(f"Added {new_version} to the registry.")
|
||||
await _generate_docs(dagger_client)
|
||||
console.log("Generated docs successfully.")
|
||||
|
||||
|
||||
async def _publish(
|
||||
dagger_client: dagger.Client, BaseImageClassToPublish: Type[bases.AirbyteConnectorBaseImage], version: semver.VersionInfo
|
||||
):
|
||||
"""This function will publish a specific version of a base image to our remote registry.
|
||||
Users are prompted for confirmation before overwriting an existing version.
|
||||
If the version does not exist in the registry, the flow is aborted and user is suggested to cut a new version first.
|
||||
"""
|
||||
docker_credentials = utils.docker.get_credentials()
|
||||
registry = await version_registry.VersionRegistry.load(BaseImageClassToPublish, dagger_client, docker_credentials)
|
||||
registry_entry = registry.get_entry_for_version(version)
|
||||
if not registry_entry:
|
||||
console.log(f"No entry found for version {version} in the registry. Please cut a new version first: `poetry run generate-release`")
|
||||
sys.exit(1)
|
||||
if registry_entry.published:
|
||||
force_answers = inquirer.prompt(
|
||||
[
|
||||
inquirer.Confirm(
|
||||
"force", message="This version has already been published to our remote registry. Would you like to overwrite it?"
|
||||
),
|
||||
]
|
||||
)
|
||||
if not force_answers["force"]:
|
||||
console.log("Not overwriting the already exiting image.")
|
||||
sys.exit(0)
|
||||
|
||||
base_image_version = BaseImageClassToPublish(dagger_client, version)
|
||||
published_docker_image = await publish.publish_to_remote_registry(base_image_version)
|
||||
console.log(f"Published {published_docker_image.address} successfully.")
|
||||
await _generate_docs(dagger_client)
|
||||
console.log("Generated docs successfully.")
|
||||
|
||||
|
||||
async def execute_async_command(command_fn: Callable, *args, **kwargs):
|
||||
"""This is a helper function that will execute a command function in an async context, required by the use of Dagger."""
|
||||
# NOTE: Dagger logs using Rich now, and two rich apps don't play well with each other.
|
||||
# Logging into a file makes the CLI experience tolerable.
|
||||
async with dagger.Connection(dagger.Config(log_output=open("dagger.log", "w"))) as dagger_client:
|
||||
await command_fn(dagger_client, *args, **kwargs)
|
||||
|
||||
|
||||
def generate_docs():
|
||||
"""This command will generate the README.md file from the templates/README.md.j2 template.
|
||||
It will first load all the registries to render the template with up to date information.
|
||||
"""
|
||||
anyio.run(execute_async_command, _generate_docs)
|
||||
|
||||
|
||||
def generate_release():
|
||||
"""This command will cut a new version on top of the previous one. It will prompt the user for release details: version bump, changelog entry.
|
||||
The user can optionally publish the new version to our remote registry.
|
||||
If the version is not published its changelog entry is still persisted.
|
||||
It can later be published by running the publish command.
|
||||
In the future we might only allow publishing new pre-release versions from this flow.
|
||||
"""
|
||||
anyio.run(execute_async_command, _generate_release)
|
||||
|
||||
|
||||
def publish_existing_version():
|
||||
"""This command is intended to be used when:
|
||||
- We have a changelog entry for a new version but it's not published yet (for future publish on merge flows).
|
||||
- We have a good reason to overwrite an existing version in the remote registry.
|
||||
"""
|
||||
parser = argparse.ArgumentParser(description="Publish a specific version of a base image to our remote registry.")
|
||||
parser.add_argument("repository", help="The base image repository name")
|
||||
parser.add_argument("version", help="The version to publish")
|
||||
args = parser.parse_args()
|
||||
|
||||
version = semver.VersionInfo.parse(args.version)
|
||||
BaseImageClassToPublish = None
|
||||
for BaseImageClass in version_registry.MANAGED_BASE_IMAGES:
|
||||
if BaseImageClass.repository == args.repository:
|
||||
BaseImageClassToPublish = BaseImageClass
|
||||
if BaseImageClassToPublish is None:
|
||||
console.log(f"Unknown base image name: {args.repository}")
|
||||
sys.exit(1)
|
||||
|
||||
anyio.run(execute_async_command, _publish, BaseImageClassToPublish, version)
|
||||
@@ -1,11 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
"""This module declares constants used by the base_images module."""
|
||||
|
||||
import dagger
|
||||
|
||||
|
||||
REMOTE_REGISTRY = "docker.io"
|
||||
PLATFORMS_WE_PUBLISH_FOR = (dagger.Platform("linux/amd64"), dagger.Platform("linux/arm64"))
|
||||
@@ -1,16 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
"""This module contains the exceptions used by the base_images module."""
|
||||
|
||||
from typing import Union
|
||||
|
||||
import dagger
|
||||
|
||||
|
||||
class SanityCheckError(Exception):
|
||||
"""Raised when a sanity check fails."""
|
||||
|
||||
def __init__(self, error: Union[str, dagger.ExecError], *args: object) -> None:
|
||||
super().__init__(error, *args)
|
||||
@@ -1,39 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
import dagger
|
||||
|
||||
|
||||
# If we perform addition dagger operations on the container, we need to make sure that a mapping exists for the new field name.
|
||||
DAGGER_FIELD_NAME_TO_DOCKERFILE_INSTRUCTION = {
|
||||
"from": lambda field: f'FROM {field.args.get("address")}',
|
||||
"withExec": lambda field: f'RUN {" ".join(field.args.get("args"))}',
|
||||
"withEnvVariable": lambda field: f'ENV {field.args.get("name")}={field.args.get("value")}',
|
||||
"withLabel": lambda field: f'LABEL {field.args.get("name")}={field.args.get("value")}',
|
||||
}
|
||||
|
||||
|
||||
def get_container_dockerfile(container) -> str:
|
||||
"""Returns the Dockerfile of the base image container.
|
||||
Disclaimer: THIS IS HIGHLY EXPERIMENTAL, HACKY AND BRITTLE.
|
||||
TODO: CONFIRM WITH THE DAGGER TEAM WHAT CAN GO WRONG HERE.
|
||||
Returns:
|
||||
str: The Dockerfile of the base image container.
|
||||
"""
|
||||
lineage = [
|
||||
field
|
||||
for field in list(container._ctx.selections)
|
||||
if isinstance(field, dagger.client._core.Field) and field.type_name == "Container"
|
||||
]
|
||||
dockerfile = []
|
||||
|
||||
for field in lineage:
|
||||
if field.name in DAGGER_FIELD_NAME_TO_DOCKERFILE_INSTRUCTION:
|
||||
try:
|
||||
dockerfile.append(DAGGER_FIELD_NAME_TO_DOCKERFILE_INSTRUCTION[field.name](field))
|
||||
except KeyError:
|
||||
raise KeyError(
|
||||
f"Unknown field name: {field.name}, please add it to the DAGGER_FIELD_NAME_TO_DOCKERFILE_INSTRUCTION mapping."
|
||||
)
|
||||
return "\n".join(dockerfile)
|
||||
@@ -1,3 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
@@ -1,129 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Callable, Final
|
||||
|
||||
import dagger
|
||||
|
||||
from base_images import bases, published_image
|
||||
from base_images import sanity_checks as base_sanity_checks
|
||||
from base_images.root_images import AMAZON_CORRETTO_21_AL_2023
|
||||
from base_images.utils.dagger import sh_dash_c
|
||||
|
||||
|
||||
class AirbyteJavaConnectorBaseImage(bases.AirbyteConnectorBaseImage):
|
||||
root_image: Final[published_image.PublishedImage] = AMAZON_CORRETTO_21_AL_2023
|
||||
repository: Final[str] = "airbyte/java-connector-base"
|
||||
|
||||
DD_AGENT_JAR_URL: Final[str] = "https://dtdg.co/latest-java-tracer"
|
||||
BASE_SCRIPT_URL = "https://raw.githubusercontent.com/airbytehq/airbyte/6d8a3a2bc4f4ca79f10164447a90fdce5c9ad6f9/airbyte-integrations/bases/base/base.sh"
|
||||
JAVA_BASE_SCRIPT_URL: Final[str] = (
|
||||
"https://raw.githubusercontent.com/airbytehq/airbyte/6d8a3a2bc4f4ca79f10164447a90fdce5c9ad6f9/airbyte-integrations/bases/base-java/javabase.sh"
|
||||
)
|
||||
|
||||
def get_container(self, platform: dagger.Platform) -> dagger.Container:
|
||||
"""Returns the container used to build the base image for java connectors
|
||||
We currently use the Amazon coretto image as a base.
|
||||
We install some packages required to build java connectors.
|
||||
We also download the datadog java agent jar and the javabase.sh script.
|
||||
We set some env variables used by the javabase.sh script.
|
||||
|
||||
Args:
|
||||
platform (dagger.Platform): The platform this container should be built for.
|
||||
|
||||
Returns:
|
||||
dagger.Container: The container used to build the base image.
|
||||
"""
|
||||
|
||||
return (
|
||||
self.dagger_client.container(platform=platform)
|
||||
.from_(self.root_image.address)
|
||||
# Bundle RUN commands together to reduce the number of layers.
|
||||
.with_exec(
|
||||
sh_dash_c(
|
||||
[
|
||||
# Shadow-utils is required to add a user with a specific UID and GID.
|
||||
# tar is equired to untar java connector binary distributions.
|
||||
# openssl is required because we need to ssh and scp sometimes.
|
||||
# findutils is required for xargs, which is shipped as part of findutils.
|
||||
f"yum install -y shadow-utils tar openssl findutils",
|
||||
# Update first, but in the same .with_exec step as the package installation.
|
||||
# Otherwise, we risk caching stale package URLs.
|
||||
"yum update -y --security",
|
||||
# Remove any dangly bits.
|
||||
"yum clean all",
|
||||
# Remove the yum cache to reduce the image size.
|
||||
"rm -rf /var/cache/yum",
|
||||
# Create the group 'airbyte' with the GID 1000
|
||||
f"groupadd --gid {self.USER_ID} {self.USER}",
|
||||
# Create the user 'airbyte' with the UID 1000
|
||||
f"useradd --uid {self.USER_ID} --gid {self.USER} --shell /bin/bash --create-home {self.USER}",
|
||||
# Create mount point for secrets and configs
|
||||
"mkdir /secrets",
|
||||
"mkdir /config",
|
||||
# Create the cache airbyte directories and set the right permissions
|
||||
f"mkdir --mode 755 {self.AIRBYTE_DIR_PATH}",
|
||||
f"mkdir --mode 755 {self.CACHE_DIR_PATH}",
|
||||
# Change the owner of the airbyte directory to the user 'airbyte'
|
||||
f"chown -R {self.USER}:{self.USER} {self.AIRBYTE_DIR_PATH}",
|
||||
f"chown -R {self.USER}:{self.USER} {self.CACHE_DIR_PATH}",
|
||||
f"chown -R {self.USER}:{self.USER} /secrets",
|
||||
f"chown -R {self.USER}:{self.USER} /config",
|
||||
f"chown -R {self.USER}:{self.USER} /usr/share/pki/ca-trust-source",
|
||||
f"chown -R {self.USER}:{self.USER} /etc/pki/ca-trust",
|
||||
f"chown -R {self.USER}:{self.USER} /tmp",
|
||||
]
|
||||
)
|
||||
)
|
||||
.with_workdir(self.AIRBYTE_DIR_PATH)
|
||||
# Copy the datadog java agent jar from the internet.
|
||||
.with_file("dd-java-agent.jar", self.dagger_client.http(self.DD_AGENT_JAR_URL), owner=self.USER)
|
||||
# Copy base.sh from the git repo.
|
||||
.with_file("base.sh", self.dagger_client.http(self.BASE_SCRIPT_URL), owner=self.USER)
|
||||
# Copy javabase.sh from the git repo.
|
||||
.with_file("javabase.sh", self.dagger_client.http(self.JAVA_BASE_SCRIPT_URL), owner=self.USER)
|
||||
# Set a bunch of env variables used by base.sh.
|
||||
.with_env_variable("AIRBYTE_SPEC_CMD", "/airbyte/javabase.sh --spec")
|
||||
.with_env_variable("AIRBYTE_CHECK_CMD", "/airbyte/javabase.sh --check")
|
||||
.with_env_variable("AIRBYTE_DISCOVER_CMD", "/airbyte/javabase.sh --discover")
|
||||
.with_env_variable("AIRBYTE_READ_CMD", "/airbyte/javabase.sh --read")
|
||||
.with_env_variable("AIRBYTE_WRITE_CMD", "/airbyte/javabase.sh --write")
|
||||
.with_env_variable("AIRBYTE_ENTRYPOINT", "/airbyte/base.sh")
|
||||
.with_entrypoint(["/airbyte/base.sh"])
|
||||
.with_user(self.USER)
|
||||
)
|
||||
|
||||
async def run_sanity_checks(self, platform: dagger.Platform):
|
||||
"""Runs sanity checks on the base image container.
|
||||
This method is called before image publication.
|
||||
Consider it like a pre-flight check before take-off to the remote registry.
|
||||
|
||||
Args:
|
||||
platform (dagger.Platform): The platform on which the sanity checks should run.
|
||||
"""
|
||||
container = await self.get_container(platform)
|
||||
for expected_rw_dir in [
|
||||
self.AIRBYTE_DIR_PATH,
|
||||
self.CACHE_DIR_PATH,
|
||||
"/tmp",
|
||||
"/secrets",
|
||||
"/config",
|
||||
"/usr/share/pki/ca-trust-source",
|
||||
"/etc/pki/ca-trust",
|
||||
]:
|
||||
await base_sanity_checks.check_user_can_write_dir(container, self.USER, expected_rw_dir)
|
||||
await base_sanity_checks.check_user_can_read_dir(container, self.USER, expected_rw_dir)
|
||||
await base_sanity_checks.check_user_uid_guid(container, self.USER, self.USER_ID, self.USER_ID)
|
||||
await base_sanity_checks.check_file_exists(container, "/airbyte/dd-java-agent.jar")
|
||||
await base_sanity_checks.check_file_exists(container, "/airbyte/base.sh")
|
||||
await base_sanity_checks.check_file_exists(container, "/airbyte/javabase.sh")
|
||||
await base_sanity_checks.check_env_var_with_printenv(container, "AIRBYTE_SPEC_CMD", "/airbyte/javabase.sh --spec")
|
||||
await base_sanity_checks.check_env_var_with_printenv(container, "AIRBYTE_CHECK_CMD", "/airbyte/javabase.sh --check")
|
||||
await base_sanity_checks.check_env_var_with_printenv(container, "AIRBYTE_DISCOVER_CMD", "/airbyte/javabase.sh --discover")
|
||||
await base_sanity_checks.check_env_var_with_printenv(container, "AIRBYTE_READ_CMD", "/airbyte/javabase.sh --read")
|
||||
await base_sanity_checks.check_env_var_with_printenv(container, "AIRBYTE_WRITE_CMD", "/airbyte/javabase.sh --write")
|
||||
await base_sanity_checks.check_env_var_with_printenv(container, "AIRBYTE_ENTRYPOINT", "/airbyte/base.sh")
|
||||
await base_sanity_checks.check_a_command_is_available_using_version_option(container, "tar")
|
||||
await base_sanity_checks.check_a_command_is_available_using_version_option(container, "openssl", "version")
|
||||
@@ -1,35 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
|
||||
import dagger
|
||||
|
||||
from base_images import bases, consts, published_image
|
||||
|
||||
|
||||
async def run_sanity_checks(base_image_version: bases.AirbyteConnectorBaseImage):
|
||||
for platform in consts.PLATFORMS_WE_PUBLISH_FOR:
|
||||
await base_image_version.run_sanity_checks(platform)
|
||||
|
||||
|
||||
async def publish_to_remote_registry(base_image_version: bases.AirbyteConnectorBaseImage) -> published_image.PublishedImage:
|
||||
"""Publishes a base image to the remote registry.
|
||||
|
||||
Args:
|
||||
base_image_version (common.AirbyteConnectorBaseImage): The base image to publish.
|
||||
|
||||
Returns:
|
||||
models.PublishedImage: The published image as a PublishedImage instance.
|
||||
"""
|
||||
|
||||
address = f"{consts.REMOTE_REGISTRY}/{base_image_version.repository}:{base_image_version.version}"
|
||||
variants_to_publish = []
|
||||
for platform in consts.PLATFORMS_WE_PUBLISH_FOR:
|
||||
await base_image_version.run_sanity_checks(platform)
|
||||
variants_to_publish.append(base_image_version.get_container(platform))
|
||||
# Publish with forced compression to ensure backward compatibility with older versions of docker
|
||||
published_address = await variants_to_publish[0].publish(
|
||||
address, platform_variants=variants_to_publish[1:], forced_compression=dagger.ImageLayerCompression.Gzip
|
||||
)
|
||||
return published_image.PublishedImage.from_address(published_address)
|
||||
@@ -1,47 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import semver
|
||||
|
||||
|
||||
@dataclass
|
||||
class PublishedImage:
|
||||
registry: str
|
||||
repository: str
|
||||
tag: str
|
||||
sha: str
|
||||
|
||||
@property
|
||||
def address(self) -> str:
|
||||
return f"{self.registry}/{self.repository}:{self.tag}@sha256:{self.sha}"
|
||||
|
||||
@classmethod
|
||||
def from_address(cls, address: str) -> PublishedImage:
|
||||
"""Creates a PublishedImage instance from a docker image address.
|
||||
A docker image address is a string of the form:
|
||||
registry/repository:tag@sha256:sha
|
||||
|
||||
Args:
|
||||
address (str): _description_
|
||||
|
||||
Returns:
|
||||
PublishedImage: _description_
|
||||
"""
|
||||
parts = address.split("/")
|
||||
registry = parts.pop(0)
|
||||
without_registry = "/".join(parts)
|
||||
repository, tag, sha = without_registry.replace("@sha256", "").split(":")
|
||||
return cls(registry, repository, tag, sha)
|
||||
|
||||
@property
|
||||
def name_with_tag(self) -> str:
|
||||
return f"{self.repository}:{self.tag}"
|
||||
|
||||
@property
|
||||
def version(self) -> semver.VersionInfo:
|
||||
return semver.VersionInfo.parse(self.tag)
|
||||
@@ -1,3 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
@@ -1,139 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Callable, Final
|
||||
|
||||
import dagger
|
||||
|
||||
from base_images import bases, published_image
|
||||
from base_images import sanity_checks as base_sanity_checks
|
||||
from base_images.python import sanity_checks as python_sanity_checks
|
||||
from base_images.root_images import PYTHON_3_11_13
|
||||
|
||||
|
||||
class AirbyteManifestOnlyConnectorBaseImage(bases.AirbyteConnectorBaseImage):
|
||||
"""ManifestOnly base image class, only used to fetch the registry."""
|
||||
|
||||
repository: Final[str] = "airbyte/source-declarative-manifest"
|
||||
|
||||
|
||||
class AirbytePythonConnectorBaseImage(bases.AirbyteConnectorBaseImage):
|
||||
root_image: Final[published_image.PublishedImage] = PYTHON_3_11_13
|
||||
repository: Final[str] = "airbyte/python-connector-base"
|
||||
pip_cache_name: Final[str] = "pip_cache"
|
||||
nltk_data_path: Final[str] = "/usr/share/nltk_data"
|
||||
ntlk_data = {
|
||||
"tokenizers": {"https://github.com/nltk/nltk_data/raw/5db857e6f7df11eabb5e5665836db9ec8df07e28/packages/tokenizers/punkt.zip"},
|
||||
"taggers": {
|
||||
"https://github.com/nltk/nltk_data/raw/5db857e6f7df11eabb5e5665836db9ec8df07e28/packages/taggers/averaged_perceptron_tagger.zip"
|
||||
},
|
||||
}
|
||||
|
||||
@property
|
||||
def pip_cache_path(self) -> str:
|
||||
return f"{self.CACHE_DIR_PATH}/pip"
|
||||
|
||||
def install_cdk_system_dependencies(self) -> Callable:
|
||||
def get_nltk_data_dir() -> dagger.Directory:
|
||||
"""Returns a dagger directory containing the nltk data.
|
||||
|
||||
Returns:
|
||||
dagger.Directory: A dagger directory containing the nltk data.
|
||||
"""
|
||||
data_container = self.dagger_client.container().from_("bash:latest")
|
||||
|
||||
for nltk_data_subfolder, nltk_data_urls in self.ntlk_data.items():
|
||||
full_nltk_data_path = f"{self.nltk_data_path}/{nltk_data_subfolder}"
|
||||
for nltk_data_url in nltk_data_urls:
|
||||
zip_file = self.dagger_client.http(nltk_data_url)
|
||||
data_container = (
|
||||
data_container.with_file("/tmp/data.zip", zip_file)
|
||||
.with_exec(["mkdir", "-p", full_nltk_data_path])
|
||||
.with_exec(["unzip", "-o", "/tmp/data.zip", "-d", full_nltk_data_path])
|
||||
.with_exec(["rm", "/tmp/data.zip"])
|
||||
)
|
||||
return data_container.directory(self.nltk_data_path)
|
||||
|
||||
def with_tesseract_and_poppler(container: dagger.Container) -> dagger.Container:
|
||||
"""
|
||||
Installs Tesseract-OCR and Poppler-utils in the base image.
|
||||
These tools are necessary for OCR (Optical Character Recognition) processes and working with PDFs, respectively.
|
||||
"""
|
||||
|
||||
container = container.with_exec(
|
||||
["sh", "-c", "apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+deb12u1"]
|
||||
)
|
||||
|
||||
return container
|
||||
|
||||
def with_file_based_connector_dependencies(container: dagger.Container) -> dagger.Container:
|
||||
"""
|
||||
Installs the dependencies for file-based connectors. This includes:
|
||||
- tesseract-ocr
|
||||
- poppler-utils
|
||||
- nltk data
|
||||
"""
|
||||
container = with_tesseract_and_poppler(container)
|
||||
container = container.with_exec(["mkdir", "-p", "755", self.nltk_data_path]).with_directory(
|
||||
self.nltk_data_path, get_nltk_data_dir()
|
||||
)
|
||||
return container
|
||||
|
||||
return with_file_based_connector_dependencies
|
||||
|
||||
def get_container(self, platform: dagger.Platform) -> dagger.Container:
|
||||
"""Returns the container used to build the base image.
|
||||
We currently use the python:3.9.18-slim-bookworm image as a base.
|
||||
We set the container system timezone to UTC.
|
||||
We then upgrade pip and install poetry.
|
||||
|
||||
Args:
|
||||
platform (dagger.Platform): The platform this container should be built for.
|
||||
|
||||
Returns:
|
||||
dagger.Container: The container used to build the base image.
|
||||
"""
|
||||
pip_cache_volume: dagger.CacheVolume = self.dagger_client.cache_volume(AirbytePythonConnectorBaseImage.pip_cache_name)
|
||||
|
||||
return (
|
||||
self.get_base_container(platform)
|
||||
.with_mounted_cache(self.pip_cache_path, pip_cache_volume, owner=self.USER)
|
||||
.with_env_variable("PIP_CACHE_DIR", self.pip_cache_path)
|
||||
# Upgrade pip to the expected version
|
||||
.with_exec(["pip", "install", "--upgrade", "pip==24.0", "setuptools==70.0.0"])
|
||||
# Declare poetry specific environment variables
|
||||
.with_env_variable("POETRY_VIRTUALENVS_CREATE", "false")
|
||||
.with_env_variable("POETRY_VIRTUALENVS_IN_PROJECT", "false")
|
||||
.with_env_variable("POETRY_NO_INTERACTION", "1")
|
||||
.with_exec(["pip", "install", "poetry==1.8.4"])
|
||||
.with_exec(["sh", "-c", "apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y && apt-get clean"])
|
||||
.with_exec(["sh", "-c", "apt-get install -y socat=1.7.4.4-2"])
|
||||
# Install CDK system dependencies
|
||||
.with_(self.install_cdk_system_dependencies())
|
||||
)
|
||||
|
||||
async def run_sanity_checks(self, platform: dagger.Platform):
|
||||
"""Runs sanity checks on the base image container.
|
||||
This method is called before image publication.
|
||||
Consider it like a pre-flight check before take-off to the remote registry.
|
||||
|
||||
Args:
|
||||
platform (dagger.Platform): The platform on which the sanity checks should run.
|
||||
"""
|
||||
container = self.get_container(platform)
|
||||
await base_sanity_checks.check_timezone_is_utc(container)
|
||||
await base_sanity_checks.check_a_command_is_available_using_version_option(container, "bash")
|
||||
await python_sanity_checks.check_python_version(container, "3.11.13")
|
||||
await python_sanity_checks.check_pip_version(container, "24.0")
|
||||
await base_sanity_checks.check_user_exists(container, self.USER, expected_uid=self.USER_ID, expected_gid=self.USER_ID)
|
||||
await base_sanity_checks.check_user_can_read_dir(container, self.USER, self.AIRBYTE_DIR_PATH)
|
||||
await base_sanity_checks.check_user_can_read_dir(container, self.USER, self.nltk_data_path)
|
||||
await base_sanity_checks.check_user_can_read_dir(container, self.USER, self.CACHE_DIR_PATH)
|
||||
await base_sanity_checks.check_user_can_write_dir(container, self.USER, self.AIRBYTE_DIR_PATH)
|
||||
await python_sanity_checks.check_poetry_version(container, "1.8.4")
|
||||
await python_sanity_checks.check_python_image_has_expected_env_vars(container)
|
||||
await base_sanity_checks.check_a_command_is_available_using_version_option(container, "socat", "-V")
|
||||
await base_sanity_checks.check_socat_version(container, "1.7.4.4")
|
||||
await python_sanity_checks.check_cdk_system_dependencies(container)
|
||||
@@ -1,145 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
import dagger
|
||||
|
||||
from base_images import errors
|
||||
from base_images import sanity_checks as base_sanity_checks
|
||||
|
||||
|
||||
async def check_python_version(container: dagger.Container, expected_python_version: str):
|
||||
"""Checks that the python version is the expected one.
|
||||
|
||||
Args:
|
||||
container (dagger.Container): The container on which the sanity checks should run.
|
||||
expected_python_version (str): The expected python version.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the python --version command could not be executed or if the outputted version is not the expected one.
|
||||
"""
|
||||
try:
|
||||
python_version_output: str = await container.with_exec(["python", "--version"]).stdout()
|
||||
except dagger.ExecError as e:
|
||||
raise errors.SanityCheckError(e)
|
||||
if python_version_output != f"Python {expected_python_version}\n":
|
||||
raise errors.SanityCheckError(f"unexpected python version: {python_version_output}")
|
||||
|
||||
|
||||
async def check_pip_version(container: dagger.Container, expected_pip_version: str):
|
||||
"""Checks that the pip version is the expected one.
|
||||
|
||||
Args:
|
||||
container (dagger.Container): The container on which the sanity checks should run.
|
||||
expected_pip_version (str): The expected pip version.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the pip --version command could not be executed or if the outputted version is not the expected one.
|
||||
"""
|
||||
try:
|
||||
pip_version_output: str = await container.with_exec(["pip", "--version"]).stdout()
|
||||
except dagger.ExecError as e:
|
||||
raise errors.SanityCheckError(e)
|
||||
if not pip_version_output.startswith(f"pip {expected_pip_version}"):
|
||||
raise errors.SanityCheckError(f"unexpected pip version: {pip_version_output}")
|
||||
|
||||
|
||||
async def check_poetry_version(container: dagger.Container, expected_poetry_version: str):
|
||||
"""Checks that the poetry version is the expected one.
|
||||
|
||||
Args:
|
||||
container (dagger.Container): The container on which the sanity checks should run.
|
||||
expected_poetry_version (str): The expected poetry version.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the poetry --version command could not be executed or if the outputted version is not the expected one.
|
||||
"""
|
||||
try:
|
||||
poetry_version_output: str = await container.with_exec(["poetry", "--version"]).stdout()
|
||||
except dagger.ExecError as e:
|
||||
raise errors.SanityCheckError(e)
|
||||
if not poetry_version_output.startswith(f"Poetry (version {expected_poetry_version}"):
|
||||
raise errors.SanityCheckError(f"unexpected poetry version: {poetry_version_output}")
|
||||
|
||||
|
||||
async def check_python_image_has_expected_env_vars(python_image_container: dagger.Container):
|
||||
"""Check a python container has the set of env var we always expect on python images.
|
||||
|
||||
Args:
|
||||
python_image_container (dagger.Container): The container on which the sanity checks should run.
|
||||
"""
|
||||
expected_env_vars = {
|
||||
"PYTHON_VERSION",
|
||||
"HOME",
|
||||
"PATH",
|
||||
"LANG",
|
||||
"GPG_KEY",
|
||||
}
|
||||
# It's not suboptimal to call printenv multiple times because the printenv output is cached.
|
||||
for expected_env_var in expected_env_vars:
|
||||
await base_sanity_checks.check_env_var_with_printenv(python_image_container, expected_env_var)
|
||||
|
||||
|
||||
async def check_nltk_data(python_image_container: dagger.Container):
|
||||
"""Install nltk and check that the required data is available.
|
||||
As of today the required data is:
|
||||
- taggers/averaged_perceptron_tagger
|
||||
- tokenizers/punkt
|
||||
|
||||
Args:
|
||||
python_image_container (dagger.Container): The container on which the sanity checks should run.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the nltk data is not available.
|
||||
"""
|
||||
with_nltk = await python_image_container.with_exec(["pip", "install", "nltk==3.8.1"])
|
||||
try:
|
||||
await with_nltk.with_exec(
|
||||
["python", "-c", 'import nltk;nltk.data.find("taggers/averaged_perceptron_tagger");nltk.data.find("tokenizers/punkt")']
|
||||
)
|
||||
except dagger.ExecError as e:
|
||||
raise errors.SanityCheckError(e)
|
||||
|
||||
|
||||
async def check_tesseract_version(python_image_container: dagger.Container, tesseract_version: str):
|
||||
"""Check that the tesseract version is the expected one.
|
||||
|
||||
Args:
|
||||
python_image_container (dagger.Container): The container on which the sanity checks should run.
|
||||
tesseract_version (str): The expected tesseract version.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the tesseract --version command could not be executed or if the outputted version is not the expected one.
|
||||
"""
|
||||
try:
|
||||
tesseract_version_output = await python_image_container.with_exec(["tesseract", "--version"]).stdout()
|
||||
except dagger.ExecError as e:
|
||||
raise errors.SanityCheckError(e)
|
||||
if not tesseract_version_output.startswith(f"tesseract {tesseract_version}"):
|
||||
raise errors.SanityCheckError(f"unexpected tesseract version: {tesseract_version_output}")
|
||||
|
||||
|
||||
async def check_poppler_utils_version(python_image_container: dagger.Container, poppler_version: str):
|
||||
"""Check that the poppler version is the expected one.
|
||||
The poppler version can be checked by running a pdftotext -v command.
|
||||
|
||||
Args:
|
||||
python_image_container (dagger.Container): The container on which the sanity checks should run.
|
||||
poppler_version (str): The expected poppler version.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the pdftotext -v command could not be executed or if the outputted version is not the expected one.
|
||||
"""
|
||||
try:
|
||||
pdf_to_text_version_output = await python_image_container.with_exec(["pdftotext", "-v"]).stderr()
|
||||
except dagger.ExecError as e:
|
||||
raise errors.SanityCheckError(e)
|
||||
|
||||
if f"pdftotext version {poppler_version}" not in pdf_to_text_version_output:
|
||||
raise errors.SanityCheckError(f"unexpected poppler version: {pdf_to_text_version_output}")
|
||||
|
||||
|
||||
async def check_cdk_system_dependencies(python_image_container: dagger.Container):
|
||||
await check_nltk_data(python_image_container)
|
||||
await check_tesseract_version(python_image_container, "5.3.0")
|
||||
await check_poppler_utils_version(python_image_container, "22.12.0")
|
||||
@@ -1,48 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
from .published_image import PublishedImage
|
||||
|
||||
|
||||
PYTHON_3_9_18 = PublishedImage(
|
||||
registry="docker.io",
|
||||
repository="python",
|
||||
tag="3.9.18-slim-bookworm",
|
||||
sha="44b7f161ed03f85e96d423b9916cdc8cb0509fb970fd643bdbc9896d49e1cad0",
|
||||
)
|
||||
|
||||
PYTHON_3_9_19 = PublishedImage(
|
||||
registry="docker.io",
|
||||
repository="python",
|
||||
tag="3.9.19-slim-bookworm",
|
||||
sha="e6941b744e8eb9df6cf6baf323d2b9ad1dfe17d118f5efee14634aff4d47c76f",
|
||||
)
|
||||
|
||||
PYTHON_3_10_14 = PublishedImage(
|
||||
registry="docker.io",
|
||||
repository="python",
|
||||
tag="3.10.14-slim-bookworm",
|
||||
sha="2407c61b1a18067393fecd8a22cf6fceede893b6aaca817bf9fbfe65e33614a3",
|
||||
)
|
||||
|
||||
PYTHON_3_11_11 = PublishedImage(
|
||||
registry="docker.io",
|
||||
repository="python",
|
||||
tag="3.11.11-slim-bookworm",
|
||||
sha="6ed5bff4d7d377e2a27d9285553b8c21cfccc4f00881de1b24c9bc8d90016e82",
|
||||
)
|
||||
|
||||
PYTHON_3_11_13 = PublishedImage(
|
||||
registry="docker.io",
|
||||
repository="python",
|
||||
tag="3.11.13-slim-bookworm",
|
||||
sha="139020233cc412efe4c8135b0efe1c7569dc8b28ddd88bddb109b764f8977e30",
|
||||
)
|
||||
|
||||
AMAZON_CORRETTO_21_AL_2023 = PublishedImage(
|
||||
registry="docker.io",
|
||||
repository="amazoncorretto",
|
||||
tag="21-al2023",
|
||||
sha="c90f38f8a5c4494cb773a984dc9fa9a727b3e6c2f2ee2cba27c834a6e101af0d",
|
||||
)
|
||||
@@ -1,202 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
import dagger
|
||||
|
||||
from base_images import errors
|
||||
|
||||
|
||||
async def check_env_var_with_printenv(
|
||||
container: dagger.Container, expected_env_var_name: str, expected_env_var_value: Optional[str] = None
|
||||
):
|
||||
"""This checks if an environment variable is correctly defined by calling the printenv command in a container.
|
||||
|
||||
Args:
|
||||
container (dagger.Container): The container on which the sanity checks should run.
|
||||
expected_env_var_name (str): The name of the environment variable to check.
|
||||
expected_env_var_value (Optional[str], optional): The expected value of the environment variable. Defaults to None.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the environment variable is not defined or if it has an unexpected value.
|
||||
"""
|
||||
try:
|
||||
printenv_output = await container.with_exec(["printenv"]).stdout()
|
||||
except dagger.ExecError as e:
|
||||
raise errors.SanityCheckError(e)
|
||||
env_vars = {line.split("=")[0]: line.split("=")[1] for line in printenv_output.splitlines()}
|
||||
if expected_env_var_name not in env_vars:
|
||||
raise errors.SanityCheckError(f"the {expected_env_var_name} environment variable is not defined.")
|
||||
if expected_env_var_value is not None and env_vars[expected_env_var_name] != expected_env_var_value:
|
||||
raise errors.SanityCheckError(
|
||||
f"the {expected_env_var_name} environment variable is defined but has an unexpected value: {env_vars[expected_env_var_name]}."
|
||||
)
|
||||
|
||||
|
||||
async def check_timezone_is_utc(container: dagger.Container):
|
||||
"""Check that the system timezone is UTC.
|
||||
|
||||
Args:
|
||||
container (dagger.Container): The container on which the sanity checks should run.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the date command could not be executed or if the outputted timezone is not UTC.
|
||||
"""
|
||||
try:
|
||||
tz_output: str = await container.with_exec(["date"]).stdout()
|
||||
except dagger.ExecError as e:
|
||||
raise errors.SanityCheckError(e)
|
||||
if "UTC" not in tz_output:
|
||||
raise errors.SanityCheckError(f"unexpected timezone: {tz_output}")
|
||||
|
||||
|
||||
async def check_a_command_is_available_using_version_option(container: dagger.Container, command: str, version_option: str = "--version"):
|
||||
"""Checks that a command is available in the container by calling it with the --version option.
|
||||
|
||||
Args:
|
||||
container (dagger.Container): The container on which the sanity checks should run.
|
||||
command (str): The command to check.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the command could not be executed or if the outputted version is not the expected one.
|
||||
"""
|
||||
try:
|
||||
command_version_output: str = await container.with_exec([command, version_option]).stdout()
|
||||
except dagger.ExecError as e:
|
||||
raise errors.SanityCheckError(e)
|
||||
if command_version_output == "":
|
||||
raise errors.SanityCheckError(f"unexpected {command} version: {command_version_output}")
|
||||
|
||||
|
||||
async def check_socat_version(container: dagger.Container, expected_socat_version: str):
|
||||
"""Checks that the socat version is the expected one.
|
||||
|
||||
Args:
|
||||
container (dagger.Container): The container on which the sanity checks should run.
|
||||
expected_socat_version (str): The expected socat version.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the socat --version command could not be executed or if the outputted version is not the expected one.
|
||||
"""
|
||||
try:
|
||||
socat_version_output: str = await container.with_exec(["socat", "-V"]).stdout()
|
||||
except dagger.ExecError as e:
|
||||
raise errors.SanityCheckError(e)
|
||||
socat_version_line = None
|
||||
for line in socat_version_output.splitlines():
|
||||
if line.startswith("socat version"):
|
||||
socat_version_line = line
|
||||
break
|
||||
if socat_version_line is None:
|
||||
raise errors.SanityCheckError(f"Could not parse the socat version from the output: {socat_version_output}")
|
||||
version_pattern = r"version (\d+\.\d+\.\d+\.\d+)"
|
||||
match = re.search(version_pattern, socat_version_line)
|
||||
if match:
|
||||
version_number = match.group(1)
|
||||
if version_number != expected_socat_version:
|
||||
raise errors.SanityCheckError(f"unexpected socat version: {version_number}")
|
||||
else:
|
||||
raise errors.SanityCheckError(f"Could not find the socat version in the version output: {socat_version_line}")
|
||||
|
||||
|
||||
async def check_user_exists(container: dagger.Container, user: str, expected_uid: int, expected_gid: int):
|
||||
"""Check that a user exists in the container, can be impersonated and has the expected user id and group id.
|
||||
|
||||
Args:
|
||||
container (dagger.Container): The container on which the sanity checks should run.
|
||||
user (str): The user to impersonate.
|
||||
expected_uid (int): The expected user id.
|
||||
expected_gid (int): The expected group id.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the id command could not be executed or if the user does not exist.
|
||||
"""
|
||||
container = container.with_user(user)
|
||||
try:
|
||||
whoami_output = (await container.with_exec(["whoami"]).stdout()).strip()
|
||||
except dagger.ExecError as e:
|
||||
raise errors.SanityCheckError(e)
|
||||
if whoami_output != user:
|
||||
raise errors.SanityCheckError(f"The user {user} does not exist in the container.")
|
||||
user_id = (await container.with_exec(["id", "-u"]).stdout()).strip()
|
||||
if int(user_id) != expected_uid:
|
||||
raise errors.SanityCheckError(f"Unexpected user id: {user_id}")
|
||||
group_id = (await container.with_exec(["id", "-g"]).stdout()).strip()
|
||||
if int(group_id) != expected_gid:
|
||||
raise errors.SanityCheckError(f"Unexpected group id: {group_id}")
|
||||
|
||||
|
||||
async def check_user_can_read_dir(container: dagger.Container, user: str, dir_path: str):
|
||||
"""Check that the given user has read permissions on files in a given directory.
|
||||
|
||||
Args:
|
||||
container (dagger.Container): The container on which the sanity checks should run.
|
||||
user (str): The user to impersonate.
|
||||
dir_path (str): The directory path to check.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the given user could not read a file created in the given directory.
|
||||
"""
|
||||
try:
|
||||
await container.with_exec(["touch", f"{dir_path}/foo.txt"]).with_user(user).with_exec(["cat", f"{dir_path}/foo.txt"])
|
||||
except dagger.ExecError:
|
||||
raise errors.SanityCheckError(f"{dir_path} is not readable by {user}.")
|
||||
|
||||
|
||||
async def check_user_can_write_dir(container: dagger.Container, user: str, dir_path: str):
|
||||
"""Check that the given user has write permissions on files in a given directory.
|
||||
|
||||
Args:
|
||||
container (dagger.Container): The container on which the sanity checks should run.
|
||||
user (str): The user to impersonate.
|
||||
dir_path (str): The directory path to check.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the user could write a file in the given directory.
|
||||
"""
|
||||
try:
|
||||
await container.with_user(user).with_exec(["touch", f"{dir_path}/foo.txt"])
|
||||
except dagger.ExecError:
|
||||
raise errors.SanityCheckError(f"{dir_path} is not writable by the {user}.")
|
||||
|
||||
|
||||
async def check_file_exists(container: dagger.Container, file_path: str):
|
||||
"""Check that a file exists in the container.
|
||||
|
||||
Args:
|
||||
container (dagger.Container): The container on which the sanity checks should run.
|
||||
file_path (str): The file path to check.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the file does not exist.
|
||||
"""
|
||||
try:
|
||||
await container.with_exec(["test", "-f", file_path])
|
||||
except dagger.ExecError:
|
||||
raise errors.SanityCheckError(f"{file_path} does not exist.")
|
||||
|
||||
|
||||
async def check_user_uid_guid(container: dagger.Container, user: str, expected_uid: int, expected_gid: int):
|
||||
"""Check that the given user has the expected user id and group id.
|
||||
|
||||
Args:
|
||||
container (dagger.Container): The container on which the sanity checks should run.
|
||||
user (str): The user to impersonate.
|
||||
expected_uid (int): The expected user id.
|
||||
expected_gid (int): The expected group id.
|
||||
|
||||
Raises:
|
||||
errors.SanityCheckError: Raised if the user does not have the expected user id or group id.
|
||||
"""
|
||||
try:
|
||||
user_id = (await container.with_user(user).with_exec(["id", "-u"]).stdout()).strip()
|
||||
if int(user_id) != expected_uid:
|
||||
raise errors.SanityCheckError(f"Unexpected user id: {user_id}")
|
||||
group_id = (await container.with_user(user).with_exec(["id", "-g"]).stdout()).strip()
|
||||
if int(group_id) != expected_gid:
|
||||
raise errors.SanityCheckError(f"Unexpected group id: {group_id}")
|
||||
except dagger.ExecError as e:
|
||||
raise errors.SanityCheckError(e)
|
||||
@@ -1,103 +0,0 @@
|
||||
# airbyte-connectors-base-images
|
||||
|
||||
This python package contains the base images used by Airbyte connectors.
|
||||
It is intended to be used as a python library.
|
||||
Our connector build pipeline ([`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1)) uses this library to build the connector images.
|
||||
Our base images are declared in code, using the [Dagger Python SDK](https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/).
|
||||
|
||||
- [Python base image code declaration](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/base_images/base_images/python/bases.py)
|
||||
- [Java base image code declaration](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/base_images/base_images/java/bases.py)
|
||||
|
||||
|
||||
## Where are the Dockerfiles?
|
||||
Our base images are not declared using Dockerfiles.
|
||||
They are declared in code using the [Dagger Python SDK](https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/).
|
||||
We prefer this approach because it allows us to interact with base images container as code: we can use python to declare the base images and use the full power of the language to build and test them.
|
||||
However, we do artificially generate Dockerfiles for debugging and documentation purposes.
|
||||
|
||||
{% for registry in registries %}
|
||||
{% if registry.entries %}
|
||||
### Example for `{{ registry.ConnectorBaseImageClass.repository }}`:
|
||||
```dockerfile
|
||||
{{ registry.entries[0].changelog_entry.dockerfile_example }}
|
||||
```
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
|
||||
## Base images
|
||||
{% for registry in registries %}
|
||||
|
||||
### `{{ registry.ConnectorBaseImageClass.repository }}`
|
||||
|
||||
| Version | Published | Docker Image Address | Changelog |
|
||||
|---------|-----------|--------------|-----------|
|
||||
{%- for entry in registry.entries %}
|
||||
| {{ entry.version }} | {{ "✅" if entry.published else "❌" }}| {{ entry.published_docker_image.address }} | {{ entry.changelog_entry.changelog_entry }} |
|
||||
{%- endfor %}
|
||||
{% endfor %}
|
||||
|
||||
## How to release a new base image version (example for Python)
|
||||
|
||||
### Requirements
|
||||
* [Docker](https://docs.docker.com/get-docker/)
|
||||
* [Poetry](https://python-poetry.org/docs/#installation)
|
||||
* Dockerhub logins
|
||||
|
||||
### Steps
|
||||
1. `poetry install`
|
||||
2. Open `base_images/python/bases.py`.
|
||||
3. Make changes to the `AirbytePythonConnectorBaseImage`, you're likely going to change the `get_container` method to change the base image.
|
||||
4. Implement the `container` property which must return a `dagger.Container` object.
|
||||
5. **Recommended**: Add new sanity checks to `run_sanity_check` to confirm that the new version is working as expected.
|
||||
6. Cut a new base image version by running `poetry run generate-release`. You'll need your DockerHub credentials.
|
||||
|
||||
It will:
|
||||
- Prompt you to pick which base image you'd like to publish.
|
||||
- Prompt you for a major/minor/patch/pre-release version bump.
|
||||
- Prompt you for a changelog message.
|
||||
- Run the sanity checks on the new version.
|
||||
- Optional: Publish the new version to DockerHub.
|
||||
- Regenerate the docs and the registry json file.
|
||||
7. Commit and push your changes.
|
||||
8. Create a PR and ask for a review from the Connector Operations team.
|
||||
|
||||
**Please note that if you don't publish your image while cutting the new version you can publish it later with `poetry run publish <repository> <version>`.**
|
||||
No connector will use the new base image version until its metadata is updated to use it.
|
||||
If you're not fully confident with the new base image version please:
|
||||
- please publish it as a pre-release version
|
||||
- try out the new version on a couple of connectors
|
||||
- cut a new version with a major/minor/patch bump and publish it
|
||||
- This steps can happen in different PRs.
|
||||
|
||||
|
||||
## Running tests locally
|
||||
```bash
|
||||
poetry run pytest
|
||||
# Static typing checks
|
||||
poetry run mypy base_images --check-untyped-defs
|
||||
```
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### 1.4.0
|
||||
- Declare a base image for our java connectors.
|
||||
|
||||
### 1.3.1
|
||||
- Update the crane image address. The previous address was deleted by the maintainer.
|
||||
|
||||
### 1.2.0
|
||||
- Improve new version prompt to pick bump type with optional pre-release version.
|
||||
|
||||
### 1.1.0
|
||||
- Add a cache ttl for base image listing to avoid DockerHub rate limiting.
|
||||
|
||||
### 1.0.4
|
||||
- Upgrade Dagger to `0.13.3`
|
||||
|
||||
### 1.0.2
|
||||
|
||||
- Improved support for images with non-semantic-versioned tags.
|
||||
|
||||
### 1.0.1
|
||||
|
||||
- Bumped dependencies ([#42581](https://github.com/airbytehq/airbyte/pull/42581))
|
||||
@@ -1,3 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
@@ -1,6 +0,0 @@
|
||||
# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
|
||||
|
||||
|
||||
def sh_dash_c(lines: list[str]) -> list[str]:
|
||||
"""Wrap sequence of commands in shell for safe usage of dagger Container's with_exec method."""
|
||||
return ["sh", "-c", " && ".join(["set -o xtrace"] + lines)]
|
||||
@@ -1,94 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
import getpass
|
||||
import os
|
||||
import time
|
||||
import uuid
|
||||
from typing import List, Tuple
|
||||
|
||||
import dagger
|
||||
|
||||
from base_images import console, published_image
|
||||
|
||||
|
||||
def get_credentials() -> Tuple[str, str]:
|
||||
"""This function will prompt the user for docker credentials.
|
||||
If the user has set the DOCKER_HUB_USERNAME and DOCKER_HUB_PASSWORD environment variables, it will use those instead.
|
||||
Returns:
|
||||
Tuple[str, str]: (username, password)
|
||||
"""
|
||||
if os.environ.get("DOCKER_HUB_USERNAME") and os.environ.get("DOCKER_HUB_PASSWORD"):
|
||||
console.log("Using docker credentials from environment variables.")
|
||||
return os.environ["DOCKER_HUB_USERNAME"], os.environ["DOCKER_HUB_PASSWORD"]
|
||||
else:
|
||||
console.log("Please enter your docker credentials.")
|
||||
console.log("You can set them as environment variables to avoid being prompted again: DOCKER_HUB_USERNAME, DOCKER_HUB_PASSWORD")
|
||||
# Not using inquirer here because of the sensitive nature of the information
|
||||
docker_username = input("Dockerhub username: ")
|
||||
docker_password = getpass.getpass("Dockerhub Password: ")
|
||||
return docker_username, docker_password
|
||||
|
||||
|
||||
class CraneClient:
|
||||
CRANE_IMAGE_ADDRESS = "gcr.io/go-containerregistry/crane/debug:c195f151efe3369874c72662cd69ad43ee485128@sha256:94f61956845714bea3b788445454ae4827f49a90dcd9dac28255c4cccb6220ad"
|
||||
|
||||
def __init__(self, dagger_client: dagger.Client, docker_credentials: Tuple[str, str], cache_ttl_seconds: int = 0):
|
||||
self.docker_hub_username_secret = dagger_client.set_secret("DOCKER_HUB_USERNAME", docker_credentials[0])
|
||||
self.docker_hub_username_password = dagger_client.set_secret("DOCKER_HUB_PASSWORD", docker_credentials[1])
|
||||
|
||||
if cache_ttl_seconds == 0:
|
||||
cache_buster = str(uuid.uuid4())
|
||||
else:
|
||||
cache_buster = str(int(time.time()) // cache_ttl_seconds)
|
||||
|
||||
self.bare_container = dagger_client.container().from_(self.CRANE_IMAGE_ADDRESS).with_env_variable("CACHE_BUSTER", cache_buster)
|
||||
|
||||
self.authenticated_container = self.login()
|
||||
|
||||
def login(self) -> dagger.Container:
|
||||
return (
|
||||
self.bare_container.with_secret_variable("DOCKER_HUB_USERNAME", self.docker_hub_username_secret)
|
||||
.with_secret_variable("DOCKER_HUB_PASSWORD", self.docker_hub_username_password)
|
||||
.with_exec(["sh", "-c", "crane auth login index.docker.io -u $DOCKER_HUB_USERNAME -p $DOCKER_HUB_PASSWORD"])
|
||||
)
|
||||
|
||||
async def digest(self, repository_and_tag: str) -> str:
|
||||
return (await self.authenticated_container.with_exec(["digest", repository_and_tag], use_entrypoint=True).stdout()).strip()
|
||||
|
||||
async def ls(self, registry_name: str, repository_name: str) -> List[str]:
|
||||
repository_address = f"{registry_name}/{repository_name}"
|
||||
console.log(f"Fetching published images in {repository_address}...")
|
||||
try:
|
||||
crane_ls_output = await self.authenticated_container.with_exec(["ls", repository_address], use_entrypoint=True).stdout()
|
||||
return crane_ls_output.splitlines()
|
||||
except dagger.ExecError as exec_error:
|
||||
# When the repository does not exist, crane ls returns an error with NAME_UNKNOWN in the stderr.
|
||||
if "NAME_UNKNOWN" in exec_error.stderr:
|
||||
console.log(f"Repository {repository_address} does not exist. Returning an empty list.")
|
||||
return []
|
||||
else:
|
||||
raise exec_error
|
||||
|
||||
|
||||
class RemoteRepository:
|
||||
def __init__(self, crane_client: CraneClient, registry_name: str, repository_name: str):
|
||||
self.crane_client = crane_client
|
||||
self.registry_name = registry_name
|
||||
self.repository_name = repository_name
|
||||
|
||||
async def get_all_images(self) -> List[published_image.PublishedImage]:
|
||||
repository_address = f"{self.registry_name}/{self.repository_name}"
|
||||
all_tags = await self.crane_client.ls(self.registry_name, self.repository_name)
|
||||
# CraneClient ls lists the tags available for a repository, but not the digests.
|
||||
# We want the digest to uniquely identify the image, so we need to fetch it separately with `crane digest`
|
||||
available_addresses_without_digest = [f"{repository_address}:{tag}" for tag in all_tags]
|
||||
available_addresses_with_digest = []
|
||||
console.log(f"Fetching digests for {len(available_addresses_without_digest)} images...")
|
||||
# TODO: This is a bottleneck, we should parallelize this
|
||||
for address in available_addresses_without_digest:
|
||||
digest = await self.crane_client.digest(address)
|
||||
available_addresses_with_digest.append(f"{address}@{digest}")
|
||||
console.log(f"Found digests for {len(available_addresses_with_digest)} images in {repository_address}")
|
||||
return [published_image.PublishedImage.from_address(address) for address in available_addresses_with_digest]
|
||||
@@ -1,314 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Mapping, Optional, Tuple, Type
|
||||
|
||||
import dagger
|
||||
import semver
|
||||
|
||||
from base_images import consts, published_image
|
||||
from base_images.bases import AirbyteConnectorBaseImage
|
||||
from base_images.java.bases import AirbyteJavaConnectorBaseImage
|
||||
from base_images.python.bases import AirbyteManifestOnlyConnectorBaseImage, AirbytePythonConnectorBaseImage
|
||||
from base_images.utils import docker
|
||||
from connector_ops.utils import ConnectorLanguage # type: ignore
|
||||
|
||||
|
||||
MANAGED_BASE_IMAGES = [AirbytePythonConnectorBaseImage, AirbyteJavaConnectorBaseImage]
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChangelogEntry:
|
||||
version: semver.VersionInfo
|
||||
changelog_entry: str
|
||||
dockerfile_example: str
|
||||
|
||||
def to_serializable_dict(self):
|
||||
return {
|
||||
"version": str(self.version),
|
||||
"changelog_entry": self.changelog_entry,
|
||||
"dockerfile_example": self.dockerfile_example,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def from_dict(entry_dict: Dict):
|
||||
return ChangelogEntry(
|
||||
version=semver.VersionInfo.parse(entry_dict["version"]),
|
||||
changelog_entry=entry_dict["changelog_entry"],
|
||||
dockerfile_example=entry_dict["dockerfile_example"],
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class VersionRegistryEntry:
|
||||
published_docker_image: Optional[published_image.PublishedImage]
|
||||
changelog_entry: Optional[ChangelogEntry]
|
||||
version: semver.VersionInfo
|
||||
|
||||
@property
|
||||
def published(self) -> bool:
|
||||
return self.published_docker_image is not None
|
||||
|
||||
|
||||
class VersionRegistry:
|
||||
def __init__(
|
||||
self,
|
||||
ConnectorBaseImageClass: Type[AirbyteConnectorBaseImage],
|
||||
entries: List[VersionRegistryEntry],
|
||||
) -> None:
|
||||
self.ConnectorBaseImageClass: Type[AirbyteConnectorBaseImage] = ConnectorBaseImageClass
|
||||
self._entries: List[VersionRegistryEntry] = entries
|
||||
|
||||
@staticmethod
|
||||
def get_changelog_dump_path(ConnectorBaseImageClass: Type[AirbyteConnectorBaseImage]) -> Path:
|
||||
"""Returns the path where the changelog is dumped to disk.
|
||||
|
||||
Args:
|
||||
ConnectorBaseImageClass (Type[AirbyteConnectorBaseImage]): The base image version class bound to the registry.
|
||||
|
||||
Returns:
|
||||
Path: The path where the changelog JSON is dumped to disk.
|
||||
"""
|
||||
registries_dir = Path("generated/changelogs")
|
||||
registries_dir.mkdir(exist_ok=True, parents=True)
|
||||
return registries_dir / f'{ConnectorBaseImageClass.repository.replace("-", "_").replace("/", "_")}.json' # type: ignore
|
||||
|
||||
@property
|
||||
def changelog_dump_path(self) -> Path:
|
||||
"""Returns the path where the changelog JSON is dumped to disk.
|
||||
|
||||
Returns:
|
||||
Path: The path where the changelog JSON is dumped to disk.
|
||||
"""
|
||||
return self.get_changelog_dump_path(self.ConnectorBaseImageClass)
|
||||
|
||||
@staticmethod
|
||||
def get_changelog_entries(ConnectorBaseImageClass: Type[AirbyteConnectorBaseImage]) -> List[ChangelogEntry]:
|
||||
"""Returns the changelog entries for a given base image version class.
|
||||
The changelog entries are loaded from the checked in changelog dump JSON file.
|
||||
|
||||
Args:
|
||||
ConnectorBaseImageClass (Type[AirbyteConnectorBaseImage]): The base image version class bound to the registry.
|
||||
|
||||
Returns:
|
||||
List[ChangelogEntry]: The changelog entries for a given base image version class.
|
||||
"""
|
||||
change_log_dump_path = VersionRegistry.get_changelog_dump_path(ConnectorBaseImageClass)
|
||||
if not change_log_dump_path.exists():
|
||||
changelog_entries = []
|
||||
else:
|
||||
changelog_entries = [ChangelogEntry.from_dict(raw_entry) for raw_entry in json.loads(change_log_dump_path.read_text())]
|
||||
return changelog_entries
|
||||
|
||||
@staticmethod
|
||||
async def get_all_published_base_images(
|
||||
dagger_client: dagger.Client,
|
||||
docker_credentials: Tuple[str, str],
|
||||
ConnectorBaseImageClass: Type[AirbyteConnectorBaseImage],
|
||||
cache_ttl_seconds: int = 0,
|
||||
) -> List[published_image.PublishedImage]:
|
||||
"""Returns all the published base images for a given base image version class.
|
||||
|
||||
Args:
|
||||
dagger_client (dagger.Client): The dagger client used to build the registry.
|
||||
docker_credentials (Tuple[str, str]): The docker credentials used to fetch published images from DockerHub.
|
||||
ConnectorBaseImageClass (Type[AirbyteConnectorBaseImage]): The base image version class bound to the registry.
|
||||
cache_ttl_seconds (int, optional): The cache time to live in seconds for crane output. Defaults to 0.
|
||||
Returns:
|
||||
List[published_image.PublishedImage]: The published base images for a given base image version class.
|
||||
"""
|
||||
crane_client = docker.CraneClient(dagger_client, docker_credentials, cache_ttl_seconds=cache_ttl_seconds)
|
||||
remote_registry = docker.RemoteRepository(crane_client, consts.REMOTE_REGISTRY, ConnectorBaseImageClass.repository) # type: ignore
|
||||
return await remote_registry.get_all_images()
|
||||
|
||||
@staticmethod
|
||||
async def load(
|
||||
ConnectorBaseImageClass: Type[AirbyteConnectorBaseImage],
|
||||
dagger_client: dagger.Client,
|
||||
docker_credentials: Tuple[str, str],
|
||||
cache_ttl_seconds: int = 0,
|
||||
) -> VersionRegistry:
|
||||
"""Instantiates a registry by fetching available versions from the remote registry and loading the changelog from disk.
|
||||
|
||||
Args:
|
||||
ConnectorBaseImageClass (Type[AirbyteConnectorBaseImage]): The base image version class bound to the registry.
|
||||
dagger_client (dagger.Client): The dagger client used to build the registry.
|
||||
docker_credentials (Tuple[str, str]): The docker credentials used to fetch published images from DockerHub.
|
||||
cache_ttl_seconds (int, optional): The cache time to live in seconds for crane output. Defaults to 0.
|
||||
Returns:
|
||||
VersionRegistry: The registry.
|
||||
"""
|
||||
# Loading the local structured changelog file which is stored as a json file.
|
||||
changelog_entries = VersionRegistry.get_changelog_entries(ConnectorBaseImageClass)
|
||||
|
||||
# Build a dict of changelog entries by version number for easier lookup
|
||||
changelog_entries_by_version = {entry.version: entry for entry in changelog_entries}
|
||||
|
||||
# Instantiate a crane client and a remote registry to fetch published images from DockerHub
|
||||
published_docker_images = await VersionRegistry.get_all_published_base_images(
|
||||
dagger_client, docker_credentials, ConnectorBaseImageClass, cache_ttl_seconds=cache_ttl_seconds
|
||||
)
|
||||
|
||||
# Build a dict of published images by version number for easier lookup
|
||||
published_docker_images_by_version = dict()
|
||||
for image in published_docker_images:
|
||||
try:
|
||||
published_docker_images_by_version[image.version] = image
|
||||
# Skip any images with invalid version tags (i.e. "test_build")
|
||||
except ValueError:
|
||||
continue
|
||||
|
||||
# We union the set of versions from the changelog and the published images to get all the versions we have to consider
|
||||
all_versions = set(changelog_entries_by_version.keys()) | set(published_docker_images_by_version.keys())
|
||||
|
||||
registry_entries = []
|
||||
# Iterate over all the versions we have to consider and build a registry entry for each of them
|
||||
# The registry entry will contain the published image if available, and the changelog entry if available
|
||||
# If the version is not published, the published image will be None
|
||||
# If the version is not in the changelog, the changelog entry will be None
|
||||
for version in all_versions:
|
||||
published_docker_image = published_docker_images_by_version.get(version)
|
||||
changelog_entry = changelog_entries_by_version.get(version)
|
||||
registry_entries.append(VersionRegistryEntry(published_docker_image, changelog_entry, version))
|
||||
return VersionRegistry(ConnectorBaseImageClass, registry_entries)
|
||||
|
||||
def save_changelog(self):
|
||||
"""Writes the changelog to disk. The changelog is dumped as a json file with a list of ChangelogEntry objects."""
|
||||
as_json = json.dumps([entry.changelog_entry.to_serializable_dict() for entry in self.entries if entry.changelog_entry])
|
||||
self.changelog_dump_path.write_text(as_json)
|
||||
|
||||
def add_entry(self, new_entry: VersionRegistryEntry) -> List[VersionRegistryEntry]:
|
||||
"""Registers a new entry in the registry and saves the changelog locally.
|
||||
|
||||
Args:
|
||||
new_entry (VersionRegistryEntry): The new entry to register.
|
||||
|
||||
Returns:
|
||||
List[VersionRegistryEntry]: All the entries sorted by version number in descending order.
|
||||
"""
|
||||
self._entries.append(new_entry)
|
||||
self.save_changelog()
|
||||
return self.entries
|
||||
|
||||
@property
|
||||
def entries(self) -> List[VersionRegistryEntry]:
|
||||
"""Returns all the base image versions sorted by version number in descending order.
|
||||
|
||||
Returns:
|
||||
List[Type[VersionRegistryEntry]]: All the published versions sorted by version number in descending order.
|
||||
"""
|
||||
return sorted(self._entries, key=lambda entry: entry.version, reverse=True)
|
||||
|
||||
@property
|
||||
def latest_entry(self) -> Optional[VersionRegistryEntry]:
|
||||
"""Returns the latest entry this registry.
|
||||
The latest entry is the one with the highest version number.
|
||||
If no entry is available, returns None.
|
||||
Returns:
|
||||
Optional[VersionRegistryEntry]: The latest registry entry, or None if no entry is available.
|
||||
"""
|
||||
try:
|
||||
return self.entries[0]
|
||||
except IndexError:
|
||||
return None
|
||||
|
||||
@property
|
||||
def latest_published_entry(self) -> Optional[VersionRegistryEntry]:
|
||||
"""Returns the latest published entry this registry.
|
||||
The latest published entry is the one with the highest version number among the published entries.
|
||||
If no entry is available, returns None.
|
||||
Returns:
|
||||
Optional[VersionRegistryEntry]: The latest published registry entry, or None if no entry is available.
|
||||
"""
|
||||
try:
|
||||
return [entry for entry in self.entries if entry.published][0]
|
||||
except IndexError:
|
||||
return None
|
||||
|
||||
def get_entry_for_version(self, version: semver.VersionInfo) -> Optional[VersionRegistryEntry]:
|
||||
"""Returns the entry for a given version.
|
||||
If no entry is available, returns None.
|
||||
Returns:
|
||||
Optional[VersionRegistryEntry]: The registry entry for the given version, or None if no entry is available.
|
||||
"""
|
||||
for entry in self.entries:
|
||||
if entry.version == version:
|
||||
return entry
|
||||
return None
|
||||
|
||||
@property
|
||||
def latest_not_pre_released_published_entry(self) -> Optional[VersionRegistryEntry]:
|
||||
"""Returns the latest entry with a not pre-released version in this registry which is published.
|
||||
If no entry is available, returns None.
|
||||
It is meant to be used externally to get the latest published version.
|
||||
Returns:
|
||||
Optional[VersionRegistryEntry]: The latest registry entry with a not pre-released version, or None if no entry is available.
|
||||
"""
|
||||
try:
|
||||
not_pre_release_published_entries = [entry for entry in self.entries if not entry.version.prerelease and entry.published]
|
||||
return not_pre_release_published_entries[0]
|
||||
except IndexError:
|
||||
return None
|
||||
|
||||
|
||||
async def get_python_registry(
|
||||
dagger_client: dagger.Client, docker_credentials: Tuple[str, str], cache_ttl_seconds: int = 0
|
||||
) -> VersionRegistry:
|
||||
return await VersionRegistry.load(
|
||||
AirbytePythonConnectorBaseImage, dagger_client, docker_credentials, cache_ttl_seconds=cache_ttl_seconds
|
||||
)
|
||||
|
||||
|
||||
async def get_manifest_only_registry(
|
||||
dagger_client: dagger.Client, docker_credentials: Tuple[str, str], cache_ttl_seconds: int = 0
|
||||
) -> VersionRegistry:
|
||||
return await VersionRegistry.load(
|
||||
AirbyteManifestOnlyConnectorBaseImage, dagger_client, docker_credentials, cache_ttl_seconds=cache_ttl_seconds
|
||||
)
|
||||
|
||||
|
||||
async def get_java_registry(
|
||||
dagger_client: dagger.Client, docker_credentials: Tuple[str, str], cache_ttl_seconds: int = 0
|
||||
) -> VersionRegistry:
|
||||
return await VersionRegistry.load(AirbyteJavaConnectorBaseImage, dagger_client, docker_credentials, cache_ttl_seconds=cache_ttl_seconds)
|
||||
|
||||
|
||||
async def get_registry_for_language(
|
||||
dagger_client: dagger.Client, language: ConnectorLanguage, docker_credentials: Tuple[str, str], cache_ttl_seconds: int = 0
|
||||
) -> VersionRegistry:
|
||||
"""Returns the registry for a given language.
|
||||
It is meant to be used externally to get the registry for a given connector language.
|
||||
|
||||
Args:
|
||||
dagger_client (dagger.Client): The dagger client used to build the registry.
|
||||
language (ConnectorLanguage): The connector language.
|
||||
docker_credentials (Tuple[str, str]): The docker credentials used to fetch published images from DockerHub.
|
||||
cache_ttl_seconds (int, optional): The cache time to live in seconds for crane output. Defaults to 0.
|
||||
Raises:
|
||||
NotImplementedError: Raised if the registry for the given language is not implemented yet.
|
||||
|
||||
Returns:
|
||||
VersionRegistry: The registry for the given language.
|
||||
"""
|
||||
if language in [ConnectorLanguage.PYTHON, ConnectorLanguage.LOW_CODE]:
|
||||
return await get_python_registry(dagger_client, docker_credentials, cache_ttl_seconds=cache_ttl_seconds)
|
||||
elif language is ConnectorLanguage.MANIFEST_ONLY:
|
||||
return await get_manifest_only_registry(dagger_client, docker_credentials, cache_ttl_seconds=cache_ttl_seconds)
|
||||
elif language is ConnectorLanguage.JAVA:
|
||||
return await get_java_registry(dagger_client, docker_credentials, cache_ttl_seconds=cache_ttl_seconds)
|
||||
else:
|
||||
raise NotImplementedError(f"Registry for language {language} is not implemented yet.")
|
||||
|
||||
|
||||
async def get_all_registries(dagger_client: dagger.Client, docker_credentials: Tuple[str, str]) -> List[VersionRegistry]:
|
||||
return [
|
||||
await get_python_registry(dagger_client, docker_credentials),
|
||||
await get_java_registry(dagger_client, docker_credentials),
|
||||
# await get_manifest_only_registry(dagger_client, docker_credentials),
|
||||
]
|
||||
@@ -1,42 +0,0 @@
|
||||
[
|
||||
{
|
||||
"version": "2.0.1",
|
||||
"changelog_entry": "Bump Amazon Coretto image version for compatibility with Apple M4 architecture.",
|
||||
"dockerfile_example": "FROM docker.io/amazoncorretto:21-al2023@sha256:c90f38f8a5c4494cb773a984dc9fa9a727b3e6c2f2ee2cba27c834a6e101af0d\nRUN sh -c set -o xtrace && yum install -y shadow-utils tar openssl findutils && yum update -y --security && yum clean all && rm -rf /var/cache/yum && groupadd --gid 1000 airbyte && useradd --uid 1000 --gid airbyte --shell /bin/bash --create-home airbyte && mkdir /secrets && mkdir /config && mkdir --mode 755 /airbyte && mkdir --mode 755 /custom_cache && chown -R airbyte:airbyte /airbyte && chown -R airbyte:airbyte /custom_cache && chown -R airbyte:airbyte /secrets && chown -R airbyte:airbyte /config && chown -R airbyte:airbyte /usr/share/pki/ca-trust-source && chown -R airbyte:airbyte /etc/pki/ca-trust && chown -R airbyte:airbyte /tmp\nENV AIRBYTE_SPEC_CMD=/airbyte/javabase.sh --spec\nENV AIRBYTE_CHECK_CMD=/airbyte/javabase.sh --check\nENV AIRBYTE_DISCOVER_CMD=/airbyte/javabase.sh --discover\nENV AIRBYTE_READ_CMD=/airbyte/javabase.sh --read\nENV AIRBYTE_WRITE_CMD=/airbyte/javabase.sh --write\nENV AIRBYTE_ENTRYPOINT=/airbyte/base.sh"
|
||||
},
|
||||
{
|
||||
"version": "2.0.0",
|
||||
"changelog_entry": "~Release non root base image~",
|
||||
"dockerfile_example": "FROM docker.io/amazoncorretto:21-al2023@sha256:5454cb606e803fce56861fdbc9eab365eaa2ab4f357ceb8c1d56f4f8c8a7bc33\nRUN sh -c set -o xtrace && yum install -y shadow-utils tar openssl findutils && yum update -y --security && yum clean all && rm -rf /var/cache/yum && groupadd --gid 1000 airbyte && useradd --uid 1000 --gid airbyte --shell /bin/bash --create-home airbyte && mkdir /secrets && mkdir /config && mkdir --mode 755 /airbyte && mkdir --mode 755 /custom_cache && chown -R airbyte:airbyte /airbyte && chown -R airbyte:airbyte /custom_cache && chown -R airbyte:airbyte /secrets && chown -R airbyte:airbyte /config && chown -R airbyte:airbyte /usr/share/pki/ca-trust-source && chown -R airbyte:airbyte /etc/pki/ca-trust && chown -R airbyte:airbyte /tmp\nENV AIRBYTE_SPEC_CMD=/airbyte/javabase.sh --spec\nENV AIRBYTE_CHECK_CMD=/airbyte/javabase.sh --check\nENV AIRBYTE_DISCOVER_CMD=/airbyte/javabase.sh --discover\nENV AIRBYTE_READ_CMD=/airbyte/javabase.sh --read\nENV AIRBYTE_WRITE_CMD=/airbyte/javabase.sh --write\nENV AIRBYTE_ENTRYPOINT=/airbyte/base.sh"
|
||||
},
|
||||
{
|
||||
"version": "2.0.0-rc.2",
|
||||
"changelog_entry": "Fine tune permissions and reproduce platform java base implementation",
|
||||
"dockerfile_example": "FROM docker.io/amazoncorretto:21-al2023@sha256:5454cb606e803fce56861fdbc9eab365eaa2ab4f357ceb8c1d56f4f8c8a7bc33\nRUN sh -c set -o xtrace && yum install -y shadow-utils tar openssl findutils && yum update -y --security && yum clean all && rm -rf /var/cache/yum && groupadd --gid 1000 airbyte && useradd --uid 1000 --gid airbyte --shell /bin/bash --create-home airbyte && mkdir /secrets && mkdir /config && mkdir --mode 755 /airbyte && mkdir --mode 755 /custom_cache && chown -R airbyte:airbyte /airbyte && chown -R airbyte:airbyte /custom_cache && chown -R airbyte:airbyte /secrets && chown -R airbyte:airbyte /config && chown -R airbyte:airbyte /usr/share/pki/ca-trust-source && chown -R airbyte:airbyte /etc/pki/ca-trust && chown -R airbyte:airbyte /tmp\nENV AIRBYTE_SPEC_CMD=/airbyte/javabase.sh --spec\nENV AIRBYTE_CHECK_CMD=/airbyte/javabase.sh --check\nENV AIRBYTE_DISCOVER_CMD=/airbyte/javabase.sh --discover\nENV AIRBYTE_READ_CMD=/airbyte/javabase.sh --read\nENV AIRBYTE_WRITE_CMD=/airbyte/javabase.sh --write\nENV AIRBYTE_ENTRYPOINT=/airbyte/base.sh"
|
||||
},
|
||||
{
|
||||
"version": "2.0.0-rc.1",
|
||||
"changelog_entry": " Make the java base image non root",
|
||||
"dockerfile_example": "FROM docker.io/amazoncorretto:21-al2023@sha256:5454cb606e803fce56861fdbc9eab365eaa2ab4f357ceb8c1d56f4f8c8a7bc33\nRUN sh -c set -o xtrace && yum update -y --security && yum install -y /usr/sbin/adduser tar openssl findutils && yum clean all && adduser --base-dir /airbyte --uid 1000 --user-group --system airbyte && mkdir --mode 755 /airbyte && mkdir --mode 755 /custom_cache && chown -R airbyte:airbyte /airbyte\nENV AIRBYTE_SPEC_CMD=/airbyte/javabase.sh --spec\nENV AIRBYTE_CHECK_CMD=/airbyte/javabase.sh --check\nENV AIRBYTE_DISCOVER_CMD=/airbyte/javabase.sh --discover\nENV AIRBYTE_READ_CMD=/airbyte/javabase.sh --read\nENV AIRBYTE_WRITE_CMD=/airbyte/javabase.sh --write\nENV AIRBYTE_ENTRYPOINT=/airbyte/base.sh"
|
||||
},
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"changelog_entry": "Create a base image for our java connectors based on Amazon Corretto.",
|
||||
"dockerfile_example": "FROM docker.io/amazoncorretto:21-al2023@sha256:5454cb606e803fce56861fdbc9eab365eaa2ab4f357ceb8c1d56f4f8c8a7bc33\nRUN sh -c set -o xtrace && yum update -y --security && yum install -y tar openssl findutils && yum clean all\nENV AIRBYTE_SPEC_CMD=/airbyte/javabase.sh --spec\nENV AIRBYTE_CHECK_CMD=/airbyte/javabase.sh --check\nENV AIRBYTE_DISCOVER_CMD=/airbyte/javabase.sh --discover\nENV AIRBYTE_READ_CMD=/airbyte/javabase.sh --read\nENV AIRBYTE_WRITE_CMD=/airbyte/javabase.sh --write\nENV AIRBYTE_ENTRYPOINT=/airbyte/base.sh"
|
||||
},
|
||||
{
|
||||
"version": "1.0.0-rc.4",
|
||||
"changelog_entry": "Bundle yum calls in a single RUN",
|
||||
"dockerfile_example": "FROM docker.io/amazoncorretto:21-al2023@sha256:5454cb606e803fce56861fdbc9eab365eaa2ab4f357ceb8c1d56f4f8c8a7bc33\nRUN sh -c set -o xtrace && yum update -y --security && yum install -y tar openssl findutils && yum clean all\nENV AIRBYTE_SPEC_CMD=/airbyte/javabase.sh --spec\nENV AIRBYTE_CHECK_CMD=/airbyte/javabase.sh --check\nENV AIRBYTE_DISCOVER_CMD=/airbyte/javabase.sh --discover\nENV AIRBYTE_READ_CMD=/airbyte/javabase.sh --read\nENV AIRBYTE_WRITE_CMD=/airbyte/javabase.sh --write\nENV AIRBYTE_ENTRYPOINT=/airbyte/base.sh"
|
||||
},
|
||||
{
|
||||
"version": "1.0.0-rc.2",
|
||||
"changelog_entry": "Set entrypoint to base.sh",
|
||||
"dockerfile_example": "FROM docker.io/amazoncorretto:21-al2023@sha256:5454cb606e803fce56861fdbc9eab365eaa2ab4f357ceb8c1d56f4f8c8a7bc33\nRUN yum update -y --security\nRUN yum install -y tar openssl findutils\nENV AIRBYTE_SPEC_CMD=/airbyte/javabase.sh --spec\nENV AIRBYTE_CHECK_CMD=/airbyte/javabase.sh --check\nENV AIRBYTE_DISCOVER_CMD=/airbyte/javabase.sh --discover\nENV AIRBYTE_READ_CMD=/airbyte/javabase.sh --read\nENV AIRBYTE_WRITE_CMD=/airbyte/javabase.sh --write\nENV AIRBYTE_ENTRYPOINT=/airbyte/base.sh"
|
||||
},
|
||||
{
|
||||
"version": "1.0.0-rc.1",
|
||||
"changelog_entry": "Create a base image for our java connectors.",
|
||||
"dockerfile_example": "FROM docker.io/amazoncorretto:21-al2023@sha256:5454cb606e803fce56861fdbc9eab365eaa2ab4f357ceb8c1d56f4f8c8a7bc33\nRUN yum update -y --security\nRUN yum install -y tar openssl findutils\nENV AIRBYTE_SPEC_CMD=/airbyte/javabase.sh --spec\nENV AIRBYTE_CHECK_CMD=/airbyte/javabase.sh --check\nENV AIRBYTE_DISCOVER_CMD=/airbyte/javabase.sh --discover\nENV AIRBYTE_READ_CMD=/airbyte/javabase.sh --read\nENV AIRBYTE_WRITE_CMD=/airbyte/javabase.sh --write\nENV AIRBYTE_ENTRYPOINT=/airbyte/base.sh"
|
||||
}
|
||||
]
|
||||
@@ -1,67 +0,0 @@
|
||||
[
|
||||
{
|
||||
"version": "4.0.2",
|
||||
"changelog_entry": "Bump to Python 3.11.13 to patch security vulnerabilities",
|
||||
"dockerfile_example": "FROM docker.io/python:3.11.13-slim-bookworm@sha256:139020233cc412efe4c8135b0efe1c7569dc8b28ddd88bddb109b764f8977e30\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN adduser --uid 1000 --system --group --no-create-home airbyte\nRUN mkdir --mode 755 /custom_cache\nRUN mkdir --mode 755 /airbyte\nRUN chown airbyte:airbyte /airbyte\nENV PIP_CACHE_DIR=/custom_cache/pip\nRUN pip install --upgrade pip==24.0 setuptools==70.0.0\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.8.4\nRUN sh -c apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y && apt-get clean\nRUN sh -c apt-get install -y socat=1.7.4.4-2\nRUN sh -c apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+deb12u1\nRUN mkdir -p 755 /usr/share/nltk_data"
|
||||
},
|
||||
{
|
||||
"version": "4.0.1-rc.1",
|
||||
"changelog_entry": "Bump to Python 3.11.13 to patch security vulnerabilities",
|
||||
"dockerfile_example": "FROM docker.io/python:3.11.13-slim-bookworm@sha256:139020233cc412efe4c8135b0efe1c7569dc8b28ddd88bddb109b764f8977e30\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN adduser --uid 1000 --system --group --no-create-home airbyte\nRUN mkdir --mode 755 /custom_cache\nRUN mkdir --mode 755 /airbyte\nRUN chown airbyte:airbyte /airbyte\nENV PIP_CACHE_DIR=/custom_cache/pip\nRUN pip install --upgrade pip==24.0 setuptools==70.0.0\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.8.4\nRUN sh -c apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y && apt-get clean\nRUN sh -c apt-get install -y socat=1.7.4.4-2\nRUN sh -c apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+deb12u1\nRUN mkdir -p 755 /usr/share/nltk_data"
|
||||
},
|
||||
{
|
||||
"version": "4.0.0",
|
||||
"changelog_entry": "Python 3.11.11",
|
||||
"dockerfile_example": "FROM docker.io/python:3.11.11-slim-bookworm@sha256:6ed5bff4d7d377e2a27d9285553b8c21cfccc4f00881de1b24c9bc8d90016e82\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN adduser --uid 1000 --system --group --no-create-home airbyte\nRUN mkdir --mode 755 /custom_cache\nRUN mkdir --mode 755 /airbyte\nRUN chown airbyte:airbyte /airbyte\nENV PIP_CACHE_DIR=/custom_cache/pip\nRUN pip install --upgrade pip==24.0 setuptools==70.0.0\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.8.4\nRUN sh -c apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y && apt-get clean\nRUN sh -c apt-get install -y socat=1.7.4.4-2\nRUN sh -c apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+b1\nRUN mkdir -p 755 /usr/share/nltk_data"
|
||||
},
|
||||
{
|
||||
"version": "3.0.0",
|
||||
"changelog_entry": "Create airbyte user",
|
||||
"dockerfile_example": "FROM docker.io/python:3.10.14-slim-bookworm@sha256:2407c61b1a18067393fecd8a22cf6fceede893b6aaca817bf9fbfe65e33614a3\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN adduser --uid 1000 --system --group --no-create-home airbyte\nRUN mkdir --mode 755 /custom_cache\nRUN mkdir --mode 755 /airbyte\nRUN chown airbyte:airbyte /airbyte\nENV PIP_CACHE_DIR=/custom_cache/pip\nRUN pip install --upgrade pip==24.0 setuptools==70.0.0\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.6.1\nRUN sh -c apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y && apt-get clean\nRUN sh -c apt-get install -y socat=1.7.4.4-2\nRUN sh -c apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+b1\nRUN mkdir -p 755 /usr/share/nltk_data"
|
||||
},
|
||||
{
|
||||
"version": "3.0.0-rc.1",
|
||||
"changelog_entry": "Update Python 3.10.4 image + create airbyte user",
|
||||
"dockerfile_example": "FROM docker.io/python:3.10.14-slim-bookworm@sha256:2407c61b1a18067393fecd8a22cf6fceede893b6aaca817bf9fbfe65e33614a3\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN adduser --uid 1000 --system --group --no-create-home airbyte\nRUN mkdir --mode 755 /custom_cache\nRUN mkdir --mode 755 /airbyte\nRUN chown airbyte:airbyte /airbyte\nENV PIP_CACHE_DIR=/custom_cache/pip\nRUN pip install --upgrade pip==24.0 setuptools==70.0.0\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.6.1\nRUN sh -c apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y && apt-get clean\nRUN sh -c apt-get install -y socat=1.7.4.4-2\nRUN sh -c apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+b1\nRUN mkdir -p 755 /usr/share/nltk_data"
|
||||
},
|
||||
{
|
||||
"version": "2.0.0",
|
||||
"changelog_entry": "Use Python 3.10",
|
||||
"dockerfile_example": "FROM docker.io/python:3.10.14-slim-bookworm@sha256:3b37199fbc5a730a551909b3efa7b29105c859668b7502451c163f2a4a7ae1ed\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN pip install --upgrade pip==24.0 setuptools==70.0.0\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.6.1\nRUN sh -c apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y && apt-get clean\nRUN sh -c apt-get install -y socat=1.7.4.4-2\nRUN sh -c apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+b1\nRUN mkdir /usr/share/nltk_data"
|
||||
},
|
||||
{
|
||||
"version": "1.2.3",
|
||||
"changelog_entry": "Use latest root image version and update system packages",
|
||||
"dockerfile_example": "FROM docker.io/python:3.9.19-slim-bookworm@sha256:e6941b744e8eb9df6cf6baf323d2b9ad1dfe17d118f5efee14634aff4d47c76f\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN pip install --upgrade pip==24.0 setuptools==70.0.0\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.6.1\nRUN sh -c apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y && apt-get clean\nRUN sh -c apt-get install -y socat=1.7.4.4-2\nRUN sh -c apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+b1\nRUN mkdir /usr/share/nltk_data"
|
||||
},
|
||||
{
|
||||
"version": "1.2.2",
|
||||
"changelog_entry": "Fix Python 3.9.19 image digest",
|
||||
"dockerfile_example": "FROM docker.io/python:3.9.19-slim-bookworm@sha256:088d9217202188598aac37f8db0929345e124a82134ac66b8bb50ee9750b045b\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN pip install --upgrade pip==24.0 setuptools==70.0.0\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.6.1\nRUN sh -c apt update && apt-get install -y socat=1.7.4.4-2\nRUN sh -c apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+b1\nRUN mkdir /usr/share/nltk_data"
|
||||
},
|
||||
{
|
||||
"version": "1.2.2-rc.1",
|
||||
"changelog_entry": "Create an airbyte user and use it",
|
||||
"dockerfile_example": "FROM docker.io/python:3.9.19-slim-bookworm@sha256:088d9217202188598aac37f8db0929345e124a82134ac66b8bb50ee9750b045b\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN sh -c apt update && apt-get install -y socat=1.7.4.4-2\nRUN adduser --uid 1000 --system --group --no-create-home airbyte\nRUN mkdir --mode 755 /custom_cache\nRUN mkdir --mode 755 /airbyte\nRUN chown airbyte:airbyte /airbyte\nENV PIP_CACHE_DIR=/custom_cache/pip\nRUN pip install --upgrade pip==24.0 setuptools==70.0.0\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.6.1\nRUN sh -c apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+b1\nRUN mkdir -p 755 /usr/share/nltk_data"
|
||||
},
|
||||
{
|
||||
"version": "1.2.1",
|
||||
"changelog_entry": "Upgrade to Python 3.9.19 + update pip and setuptools",
|
||||
"dockerfile_example": "FROM docker.io/python:3.9.19-slim-bookworm@sha256:b92e6f45b58d9cafacc38563e946f8d249d850db862cbbd8befcf7f49eef8209\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN pip install --upgrade pip==24.0 setuptools==70.0.0\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.6.1\nRUN sh -c apt update && apt-get install -y socat=1.7.4.4-2\nRUN sh -c apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+b1\nRUN mkdir /usr/share/nltk_data"
|
||||
},
|
||||
{
|
||||
"version": "1.2.0",
|
||||
"changelog_entry": "Add CDK system dependencies: nltk data, tesseract, poppler.",
|
||||
"dockerfile_example": "FROM docker.io/python:3.9.18-slim-bookworm@sha256:44b7f161ed03f85e96d423b9916cdc8cb0509fb970fd643bdbc9896d49e1cad0\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN pip install --upgrade pip==23.2.1\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.6.1\nRUN sh -c apt update && apt-get install -y socat=1.7.4.4-2\nRUN sh -c apt-get update && apt-get install -y tesseract-ocr=5.3.0-2 poppler-utils=22.12.0-2+b1\nRUN mkdir /usr/share/nltk_data"
|
||||
},
|
||||
{
|
||||
"version": "1.1.0",
|
||||
"changelog_entry": "Install socat",
|
||||
"dockerfile_example": "FROM docker.io/python:3.9.18-slim-bookworm@sha256:44b7f161ed03f85e96d423b9916cdc8cb0509fb970fd643bdbc9896d49e1cad0\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN pip install --upgrade pip==23.2.1\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.6.1\nRUN sh -c apt update && apt-get install -y socat=1.7.4.4-2"
|
||||
},
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"changelog_entry": "Initial release: based on Python 3.9.18, on slim-bookworm system, with pip==23.2.1 and poetry==1.6.1",
|
||||
"dockerfile_example": "FROM docker.io/python:3.9.18-slim-bookworm@sha256:44b7f161ed03f85e96d423b9916cdc8cb0509fb970fd643bdbc9896d49e1cad0\nRUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime\nRUN pip install --upgrade pip==23.2.1\nENV POETRY_VIRTUALENVS_CREATE=false\nENV POETRY_VIRTUALENVS_IN_PROJECT=false\nENV POETRY_NO_INTERACTION=1\nRUN pip install poetry==1.6.1"
|
||||
}
|
||||
]
|
||||
2456
airbyte-ci/connectors/base_images/poetry.lock
generated
2456
airbyte-ci/connectors/base_images/poetry.lock
generated
File diff suppressed because it is too large
Load Diff
@@ -1,43 +0,0 @@
|
||||
[tool.poetry]
|
||||
name = "airbyte-connectors-base-images"
|
||||
version = "1.6"
|
||||
description = "This package is used to generate and publish the base images for Airbyte Connectors."
|
||||
authors = ["Augustin Lafanechere <augustin@airbyte.io>"]
|
||||
readme = "README.md"
|
||||
packages = [{ include = "base_images" }]
|
||||
include = ["generated"]
|
||||
[tool.poetry.dependencies]
|
||||
python = "^3.11"
|
||||
dagger-io = "==0.13.3"
|
||||
beartype = ">=0.18.2"
|
||||
gitpython = "^3.1.35"
|
||||
rich = "^13.5.2"
|
||||
semver = "^3.0.1"
|
||||
connector-ops = { path = "../connector_ops", develop = true }
|
||||
inquirer = "^3.1.3"
|
||||
jinja2 = "^3.1.2"
|
||||
|
||||
[tool.poetry.group.dev.dependencies]
|
||||
pytest = "^8"
|
||||
pytest-mock = "^3.10.0"
|
||||
pytest-cov = "^4.1.0"
|
||||
mypy = "^1.5.1"
|
||||
vulture = "^2.9.1"
|
||||
|
||||
[build-system]
|
||||
requires = ["poetry-core"]
|
||||
build-backend = "poetry.core.masonry.api"
|
||||
|
||||
[tool.poetry.scripts]
|
||||
generate-docs = "base_images.commands:generate_docs"
|
||||
generate-release = "base_images.commands:generate_release"
|
||||
publish = "base_images.commands:publish_existing_version"
|
||||
|
||||
[tool.poe.tasks]
|
||||
test = "pytest tests"
|
||||
|
||||
[tool.airbyte_ci]
|
||||
python_versions = ["3.11"]
|
||||
optional_poetry_groups = ["dev"]
|
||||
poe_tasks = ["test"]
|
||||
mount_docker_socket = true
|
||||
@@ -1,2 +0,0 @@
|
||||
[pytest]
|
||||
addopts = --cov=base_images --cov-report=term-missing
|
||||
@@ -1,3 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
@@ -1,25 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
import platform
|
||||
import sys
|
||||
|
||||
import dagger
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def anyio_backend():
|
||||
return "asyncio"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
async def dagger_client():
|
||||
async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def current_platform():
|
||||
return dagger.Platform(f"linux/{platform.machine()}")
|
||||
@@ -1,3 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
@@ -1,40 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
import pytest
|
||||
import semver
|
||||
from base_images import root_images
|
||||
from base_images.python import bases
|
||||
|
||||
|
||||
pytestmark = [
|
||||
pytest.mark.anyio,
|
||||
]
|
||||
|
||||
|
||||
class TestAirbytePythonConnectorBaseImage:
|
||||
@pytest.fixture
|
||||
def dummy_version(self):
|
||||
return semver.VersionInfo.parse("0.0.0-rc.1")
|
||||
|
||||
def test_class_attributes(self):
|
||||
"""Spot any regression in the class attributes."""
|
||||
assert bases.AirbytePythonConnectorBaseImage.root_image == root_images.PYTHON_3_11_13
|
||||
assert bases.AirbytePythonConnectorBaseImage.repository == "airbyte/python-connector-base"
|
||||
assert bases.AirbytePythonConnectorBaseImage.pip_cache_name == "pip_cache"
|
||||
|
||||
async def test_run_sanity_checks(self, dagger_client, current_platform, dummy_version):
|
||||
base_image_version = bases.AirbytePythonConnectorBaseImage(dagger_client, dummy_version)
|
||||
await base_image_version.run_sanity_checks(current_platform)
|
||||
|
||||
async def test_pip_cache_volume(self, dagger_client, current_platform, dummy_version):
|
||||
base_image_version = bases.AirbytePythonConnectorBaseImage(dagger_client, dummy_version)
|
||||
container = base_image_version.get_container(current_platform)
|
||||
assert "/custom_cache/pip" in await container.mounts()
|
||||
|
||||
async def test_is_using_bookworm(self, dagger_client, current_platform, dummy_version):
|
||||
base_image_version = bases.AirbytePythonConnectorBaseImage(dagger_client, dummy_version)
|
||||
container = base_image_version.get_container(current_platform)
|
||||
cat_output = await container.with_exec(["cat", "/etc/os-release"]).stdout()
|
||||
assert "Debian GNU/Linux 12 (bookworm)" in [kv.split("=")[1].replace('"', "") for kv in cat_output.splitlines()]
|
||||
@@ -1,56 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
from contextlib import nullcontext as does_not_raise
|
||||
|
||||
import pytest
|
||||
from base_images import root_images
|
||||
from base_images.errors import SanityCheckError
|
||||
from base_images.python import sanity_checks
|
||||
|
||||
pytestmark = [
|
||||
pytest.mark.anyio,
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"docker_image, python_version, expected_error",
|
||||
[
|
||||
(root_images.PYTHON_3_9_18.address, "3.9.18", does_not_raise()),
|
||||
(root_images.PYTHON_3_9_18.address, "3.9.19", pytest.raises(SanityCheckError)),
|
||||
("hello-world:latest", "3.9.19", pytest.raises(SanityCheckError)),
|
||||
],
|
||||
)
|
||||
async def test_check_python_version(dagger_client, docker_image, python_version, expected_error):
|
||||
container_with_python = dagger_client.container().from_(docker_image)
|
||||
with expected_error:
|
||||
await sanity_checks.check_python_version(container_with_python, python_version)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"docker_image, pip_version, expected_error",
|
||||
[
|
||||
(root_images.PYTHON_3_9_18.address, "23.0.1", does_not_raise()),
|
||||
(root_images.PYTHON_3_9_18.address, "23.0.2", pytest.raises(SanityCheckError)),
|
||||
("hello-world:latest", "23.0.1", pytest.raises(SanityCheckError)),
|
||||
],
|
||||
)
|
||||
async def test_check_pip_version(dagger_client, docker_image, pip_version, expected_error):
|
||||
container_with_python = dagger_client.container().from_(docker_image)
|
||||
with expected_error:
|
||||
await sanity_checks.check_pip_version(container_with_python, pip_version)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"docker_image, poetry_version, expected_error",
|
||||
[
|
||||
("pfeiffermax/python-poetry:1.6.0-poetry1.6.1-python3.9.18-slim-bookworm", "1.6.1", does_not_raise()),
|
||||
("pfeiffermax/python-poetry:1.6.0-poetry1.4.2-python3.9.18-bookworm", "1.6.1", pytest.raises(SanityCheckError)),
|
||||
(root_images.PYTHON_3_9_18.address, "23.0.2", pytest.raises(SanityCheckError)),
|
||||
],
|
||||
)
|
||||
async def test_check_poetry_version(dagger_client, docker_image, poetry_version, expected_error):
|
||||
container_with_python = dagger_client.container().from_(docker_image)
|
||||
with expected_error:
|
||||
await sanity_checks.check_poetry_version(container_with_python, poetry_version)
|
||||
@@ -1,112 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
from contextlib import nullcontext as does_not_raise
|
||||
|
||||
import pytest
|
||||
from base_images import root_images, sanity_checks
|
||||
from base_images.errors import SanityCheckError
|
||||
|
||||
pytestmark = [
|
||||
pytest.mark.anyio,
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"docker_image, expected_env_var_name, expected_env_var_value, expected_error",
|
||||
[
|
||||
(root_images.PYTHON_3_9_18.address, "PYTHON_VERSION", "3.9.18", does_not_raise()),
|
||||
(root_images.PYTHON_3_9_18.address, "PYTHON_VERSION", "3.9.19", pytest.raises(SanityCheckError)),
|
||||
(root_images.PYTHON_3_9_18.address, "NOT_EXISTING_ENV_VAR", "3.9.19", pytest.raises(SanityCheckError)),
|
||||
],
|
||||
)
|
||||
async def test_check_env_var_with_printenv(dagger_client, docker_image, expected_env_var_name, expected_env_var_value, expected_error):
|
||||
container = dagger_client.container().from_(docker_image)
|
||||
with expected_error:
|
||||
await sanity_checks.check_env_var_with_printenv(container, expected_env_var_name, expected_env_var_value)
|
||||
container_without_printenv = container.with_exec(["rm", "/usr/bin/printenv"])
|
||||
with pytest.raises(SanityCheckError):
|
||||
await sanity_checks.check_env_var_with_printenv(container_without_printenv, expected_env_var_name, expected_env_var_value)
|
||||
|
||||
|
||||
async def test_check_timezone_is_utc(dagger_client):
|
||||
container = dagger_client.container().from_(root_images.PYTHON_3_9_18.address)
|
||||
# This containers has UTC as timezone by default
|
||||
await sanity_checks.check_timezone_is_utc(container)
|
||||
container_not_on_utc = container.with_exec(["ln", "-sf", "/usr/share/zoneinfo/Europe/Paris", "/etc/localtime"])
|
||||
with pytest.raises(SanityCheckError):
|
||||
await sanity_checks.check_timezone_is_utc(container_not_on_utc)
|
||||
container_without_date = container.with_exec(["rm", "/usr/bin/date"])
|
||||
with pytest.raises(SanityCheckError):
|
||||
await sanity_checks.check_timezone_is_utc(container_without_date)
|
||||
|
||||
|
||||
async def test_check_a_command_is_available_using_version_option(dagger_client):
|
||||
container = dagger_client.container().from_(root_images.PYTHON_3_9_18.address)
|
||||
await sanity_checks.check_a_command_is_available_using_version_option(container, "bash")
|
||||
container_without_bash = container.with_exec(["rm", "/usr/bin/bash"])
|
||||
with pytest.raises(SanityCheckError):
|
||||
await sanity_checks.check_a_command_is_available_using_version_option(container_without_bash, "bash")
|
||||
container_without_ls = container.with_exec(["rm", "/usr/bin/ls"])
|
||||
with pytest.raises(SanityCheckError):
|
||||
await sanity_checks.check_a_command_is_available_using_version_option(container_without_ls, "ls")
|
||||
container_without_date = container.with_exec(["rm", "/usr/bin/date"])
|
||||
with pytest.raises(SanityCheckError):
|
||||
await sanity_checks.check_a_command_is_available_using_version_option(container_without_date, "date")
|
||||
container_without_printenv = container.with_exec(["rm", "/usr/bin/printenv"])
|
||||
with pytest.raises(SanityCheckError):
|
||||
await sanity_checks.check_a_command_is_available_using_version_option(container_without_printenv, "printenv")
|
||||
|
||||
|
||||
async def test_check_socat_version(mocker):
|
||||
# Mocking is used in this test because it's hard to install a different socat version in the PYTHON_3_9_18 container
|
||||
|
||||
# Mock the container and its 'with_exec' method
|
||||
mock_container = mocker.Mock()
|
||||
# Set the expected version
|
||||
expected_version = "1.2.3.4"
|
||||
|
||||
# Mock the 'stdout' method and return an output different from the socat -V command
|
||||
mock_stdout = mocker.AsyncMock(return_value="foobar")
|
||||
mock_container.with_exec.return_value.stdout = mock_stdout
|
||||
|
||||
# Run the function
|
||||
with pytest.raises(SanityCheckError) as exc_info:
|
||||
await sanity_checks.check_socat_version(mock_container, expected_version)
|
||||
|
||||
# Check the error message
|
||||
assert str(exc_info.value) == "Could not parse the socat version from the output: foobar"
|
||||
|
||||
# Mock the 'stdout' method and return a "socat version" line but with a version structure not matching the pattern from the socat -V command
|
||||
mock_stdout = mocker.AsyncMock(
|
||||
return_value="socat by Gerhard Rieger and contributors - see www.dest-unreach.org\nsocat version 1.1 on 06 Nov 2022 08:15:51"
|
||||
)
|
||||
mock_container.with_exec.return_value.stdout = mock_stdout
|
||||
|
||||
# Run the function
|
||||
with pytest.raises(SanityCheckError) as exc_info:
|
||||
await sanity_checks.check_socat_version(mock_container, expected_version)
|
||||
# Check the error message
|
||||
assert str(exc_info.value) == "Could not find the socat version in the version output: socat version 1.1 on 06 Nov 2022 08:15:51"
|
||||
|
||||
# Mock the 'stdout' method and return a correct "socat version" line but with a version different from the expected one
|
||||
mock_stdout = mocker.AsyncMock(
|
||||
return_value="socat by Gerhard Rieger and contributors - see www.dest-unreach.org\nsocat version 1.7.4.4 on 06 Nov 2022 08:15:51"
|
||||
)
|
||||
mock_container.with_exec.return_value.stdout = mock_stdout
|
||||
|
||||
# Run the function
|
||||
with pytest.raises(SanityCheckError) as exc_info:
|
||||
await sanity_checks.check_socat_version(mock_container, expected_version)
|
||||
# Check the error message
|
||||
assert str(exc_info.value) == "unexpected socat version: 1.7.4.4"
|
||||
|
||||
# Mock the 'stdout' method and return a correct "socat version" matching the expected one
|
||||
mock_stdout = mocker.AsyncMock(
|
||||
return_value=f"socat by Gerhard Rieger and contributors - see www.dest-unreach.org\nsocat version {expected_version} on 06 Nov 2022 08:15:51"
|
||||
)
|
||||
mock_container.with_exec.return_value.stdout = mock_stdout
|
||||
|
||||
# No exception should be raised by this function call
|
||||
await sanity_checks.check_socat_version(mock_container, expected_version)
|
||||
@@ -1,152 +0,0 @@
|
||||
#
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
#
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import semver
|
||||
from base_images import version_registry
|
||||
from base_images.python.bases import AirbytePythonConnectorBaseImage
|
||||
|
||||
|
||||
class TestChangelogEntry:
|
||||
def test_to_serializable_dict(self):
|
||||
changelog_entry = version_registry.ChangelogEntry(semver.VersionInfo.parse("1.0.0"), "first version", "Dockerfile example")
|
||||
assert (
|
||||
json.dumps(changelog_entry.to_serializable_dict())
|
||||
== '{"version": "1.0.0", "changelog_entry": "first version", "dockerfile_example": "Dockerfile example"}'
|
||||
), "The changelog entry should be serializable to JSON"
|
||||
|
||||
|
||||
class TestVersionRegistry:
|
||||
@pytest.fixture
|
||||
def fake_entries(self, mocker):
|
||||
# Please keep this list ordered by version
|
||||
return [
|
||||
version_registry.VersionRegistryEntry(
|
||||
published_docker_image=mocker.Mock(), changelog_entry="first version", version=semver.VersionInfo.parse("1.0.0")
|
||||
),
|
||||
version_registry.VersionRegistryEntry(
|
||||
published_docker_image=mocker.Mock(), changelog_entry="second version", version=semver.VersionInfo.parse("2.0.0")
|
||||
),
|
||||
version_registry.VersionRegistryEntry(
|
||||
published_docker_image=mocker.Mock(), changelog_entry="pre-release", version=semver.VersionInfo.parse("3.0.0-rc.1")
|
||||
),
|
||||
version_registry.VersionRegistryEntry(
|
||||
published_docker_image=None, changelog_entry="third version", version=semver.VersionInfo.parse("3.0.0")
|
||||
),
|
||||
]
|
||||
|
||||
def test_entries(self, fake_entries):
|
||||
entries = version_registry.VersionRegistry(AirbytePythonConnectorBaseImage, fake_entries).entries
|
||||
versions = [entry.version for entry in entries]
|
||||
assert set(versions) == set(
|
||||
[entry.version for entry in fake_entries]
|
||||
), "The entries should be unique by version and contain all the entries passed as argument"
|
||||
assert versions == sorted(versions, reverse=True), "The entries should be sorted by version in descending order"
|
||||
|
||||
def test_latest_entry(self, fake_entries):
|
||||
vr = version_registry.VersionRegistry(AirbytePythonConnectorBaseImage, fake_entries)
|
||||
assert vr.latest_entry == fake_entries[-1]
|
||||
|
||||
def test_get_entry_for_version(self, fake_entries):
|
||||
vr = version_registry.VersionRegistry(AirbytePythonConnectorBaseImage, fake_entries)
|
||||
entry = vr.get_entry_for_version(semver.VersionInfo.parse("1.0.0"))
|
||||
assert entry.version == semver.VersionInfo.parse("1.0.0")
|
||||
|
||||
def test_latest_published_entry(self, fake_entries):
|
||||
vr = version_registry.VersionRegistry(AirbytePythonConnectorBaseImage, fake_entries)
|
||||
assert vr.latest_published_entry == fake_entries[-2]
|
||||
|
||||
def latest_not_pre_released_published_entry(self, fake_entries):
|
||||
vr = version_registry.VersionRegistry(AirbytePythonConnectorBaseImage, fake_entries)
|
||||
assert vr.latest_not_pre_released_published_entry == fake_entries[1]
|
||||
|
||||
def test_get_changelog_dump_path(self, mocker):
|
||||
mock_connector_class = mocker.Mock()
|
||||
mock_connector_class.repository = "example-repo"
|
||||
|
||||
path = version_registry.VersionRegistry.get_changelog_dump_path(mock_connector_class)
|
||||
expected_changelog_dump_path = Path("generated/changelogs/example_repo.json")
|
||||
assert path == expected_changelog_dump_path
|
||||
assert version_registry.VersionRegistry(mock_connector_class, []).changelog_dump_path == expected_changelog_dump_path
|
||||
|
||||
def test_get_changelog_entries_with_existing_json(self, mocker, tmp_path):
|
||||
dummy_change_log_path = tmp_path / "changelog.json"
|
||||
dummy_changelog_entry = version_registry.ChangelogEntry(semver.VersionInfo.parse("1.0.0"), "Initial release", "")
|
||||
dummy_change_log_path.write_text(json.dumps([dummy_changelog_entry.to_serializable_dict()]))
|
||||
|
||||
mock_connector_class = mocker.Mock()
|
||||
mocker.patch.object(version_registry.VersionRegistry, "get_changelog_dump_path", return_value=dummy_change_log_path)
|
||||
|
||||
changelog_entries = version_registry.VersionRegistry.get_changelog_entries(mock_connector_class)
|
||||
|
||||
assert len(changelog_entries) == 1
|
||||
assert isinstance(changelog_entries[0], version_registry.ChangelogEntry)
|
||||
assert changelog_entries[0].version == semver.VersionInfo.parse("1.0.0")
|
||||
assert changelog_entries[0].changelog_entry == "Initial release"
|
||||
|
||||
def test_get_changelog_entries_without_json(self, mocker, tmp_path):
|
||||
dummy_change_log_path = tmp_path / "changelog.json"
|
||||
|
||||
mock_connector_class = mocker.Mock()
|
||||
mocker.patch.object(version_registry.VersionRegistry, "get_changelog_dump_path", return_value=dummy_change_log_path)
|
||||
|
||||
changelog_entries = version_registry.VersionRegistry.get_changelog_entries(mock_connector_class)
|
||||
assert len(changelog_entries) == 0
|
||||
|
||||
@pytest.fixture
|
||||
def mock_dagger_client(self, mocker):
|
||||
return mocker.Mock()
|
||||
|
||||
@pytest.fixture
|
||||
def fake_docker_credentials(self):
|
||||
return ("username", "password")
|
||||
|
||||
@pytest.fixture
|
||||
def fake_changelog_entries(self):
|
||||
return [
|
||||
version_registry.ChangelogEntry(semver.VersionInfo.parse("1.0.0"), "first version", ""),
|
||||
version_registry.ChangelogEntry(semver.VersionInfo.parse("2.0.0"), "second unpublished version", ""),
|
||||
]
|
||||
|
||||
@pytest.fixture
|
||||
def fake_published_images(self, mocker):
|
||||
# Mock the published images to include only one version (2.0.0)
|
||||
return [mocker.Mock(version=semver.VersionInfo.parse("1.0.0"))]
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_get_all_published_base_images(self, mocker, mock_dagger_client, fake_docker_credentials):
|
||||
mock_crane_client = mocker.Mock()
|
||||
mocker.patch.object(version_registry.docker, "CraneClient", return_value=mock_crane_client)
|
||||
|
||||
mock_remote_registry = mocker.AsyncMock()
|
||||
mocker.patch.object(version_registry.docker, "RemoteRepository", return_value=mock_remote_registry)
|
||||
|
||||
sample_published_images = [mocker.Mock(), mocker.Mock()]
|
||||
mock_remote_registry.get_all_images.return_value = sample_published_images
|
||||
|
||||
published_images = await version_registry.VersionRegistry.get_all_published_base_images(
|
||||
mock_dagger_client, fake_docker_credentials, mocker.Mock()
|
||||
)
|
||||
|
||||
assert published_images == sample_published_images
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_load_with_mocks(
|
||||
self, mocker, mock_dagger_client, fake_docker_credentials, fake_changelog_entries, fake_published_images
|
||||
):
|
||||
mocker.patch.object(version_registry.VersionRegistry, "get_changelog_entries", return_value=fake_changelog_entries)
|
||||
mocker.patch.object(version_registry.VersionRegistry, "get_all_published_base_images", return_value=fake_published_images)
|
||||
|
||||
registry = await version_registry.VersionRegistry.load(mocker.Mock(), mock_dagger_client, fake_docker_credentials)
|
||||
|
||||
assert len(registry.entries) == 2, "Two entries should be in the registry"
|
||||
version_1_entry = registry.get_entry_for_version(semver.VersionInfo.parse("1.0.0"))
|
||||
assert version_1_entry is not None, "The version 1.0.0 should be in the registry even if not published"
|
||||
assert version_1_entry.published, "The version 1.0.0 should be published"
|
||||
version_2_entry = registry.get_entry_for_version(semver.VersionInfo.parse("2.0.0"))
|
||||
assert version_2_entry is not None, "The version 2.0.0 should be in the registry even if not published"
|
||||
assert version_2_entry.published is False, "The version 2.0.0 should not be published"
|
||||
947
airbyte-ci/connectors/connectors_qa/poetry.lock
generated
947
airbyte-ci/connectors/connectors_qa/poetry.lock
generated
File diff suppressed because it is too large
Load Diff
@@ -12,7 +12,6 @@ build-backend = "poetry.core.masonry.api"
|
||||
|
||||
[tool.poetry.dependencies]
|
||||
python = "^3.11"
|
||||
airbyte-connectors-base-images = { path = "../base_images", develop = false }
|
||||
connector-ops = { path = "../connector_ops", develop = false }
|
||||
metadata-service = { path = "../metadata_service/lib", develop = false }
|
||||
pydash = "^6.0.2"
|
||||
|
||||
@@ -1,7 +1,5 @@
|
||||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
|
||||
|
||||
from base_images.python.bases import AirbytePythonConnectorBaseImage # type: ignore
|
||||
|
||||
CONNECTORS_QA_DOC_TEMPLATE_NAME = "qa_checks.md.j2"
|
||||
DOCKER_HUB_PASSWORD_ENV_VAR_NAME = "DOCKER_HUB_PASSWORD"
|
||||
DOCKER_HUB_USERNAME_ENV_VAR_NAME = "DOCKER_HUB_USERNAME"
|
||||
@@ -22,4 +20,4 @@ SETUP_PY_FILE_NAME = "setup.py"
|
||||
VALID_LICENSES = {"MIT", "ELV2", "AIRBYTE ENTERPRISE"}
|
||||
|
||||
# Derived from other constants
|
||||
AIRBYTE_PYTHON_CONNECTOR_BASE_IMAGE_NAME = f"{DOCKER_INDEX}/{AirbytePythonConnectorBaseImage.repository}"
|
||||
AIRBYTE_PYTHON_CONNECTOR_BASE_IMAGE_NAME = f"{DOCKER_INDEX}/airbyte/python-connector-base"
|
||||
|
||||
@@ -122,7 +122,7 @@ Maintaining business logic in smaller projects also increases velocity, as intro
|
||||
#### Good examples of this principle
|
||||
|
||||
- `connectors-qa`: We want to run specific static checks on all our connectors: we introduced a specific python package ([`connectors-qa`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/connectors_qa/README.md#L1))which declares and run the checks on connectors. We orchestrate the run of this package inside the [QaChecks](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/test/steps/common.py#L122) step. This class is just aware of the tool location, its entry point, and what has to be mounted to the container for the command to run.
|
||||
- Internal package testing: We expose an `airbyte-ci test` command which can run a CI pipeline on an internal poetry package. The pipeline logic is declared at the package level with `poe` tasks in the package `pyproject.toml`. `airbyte-ci` is made aware about what is has to run by parsing the content of the `[tool.airbyte_ci]` section of the `pyproject.toml`file. [Example](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/base_images/pyproject.toml#L39)
|
||||
- Internal package testing: We expose an `airbyte-ci test` command which can run a CI pipeline on an internal poetry package. The pipeline logic is declared at the package level with `poe` tasks in the package `pyproject.toml`. `airbyte-ci` is made aware about what is has to run by parsing the content of the `[tool.airbyte_ci]` section of the `pyproject.toml`file. [Example](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/ci_credentials/pyproject.toml#L35)
|
||||
|
||||
### No command or pipeline should be language specific
|
||||
|
||||
|
||||
@@ -7,7 +7,6 @@ from abc import ABC
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import docker # type: ignore
|
||||
from base_images.bases import AirbyteConnectorBaseImage # type: ignore
|
||||
from click import UsageError
|
||||
from connector_ops.utils import Connector # type: ignore
|
||||
from dagger import Container, ExecError, Platform, QueryError
|
||||
@@ -34,7 +33,7 @@ class BuildConnectorImagesBase(Step, ABC):
|
||||
"""
|
||||
|
||||
context: ConnectorContext
|
||||
USER = AirbyteConnectorBaseImage.USER
|
||||
USER = "airbyte"
|
||||
|
||||
@property
|
||||
def title(self) -> str:
|
||||
|
||||
@@ -1,20 +1,20 @@
|
||||
# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
|
||||
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
from typing import TYPE_CHECKING, List, Optional
|
||||
|
||||
import dagger
|
||||
import semver
|
||||
import yaml
|
||||
from base_images import version_registry # type: ignore
|
||||
from connector_ops.utils import METADATA_FILE_NAME # type: ignore
|
||||
from connector_ops.utils import METADATA_FILE_NAME, ConnectorLanguage # type: ignore
|
||||
|
||||
from pipelines.airbyte_ci.connectors.context import ConnectorContext
|
||||
from pipelines.dagger.actions.system.docker import with_crane
|
||||
from pipelines.helpers.connectors.dagger_fs import dagger_read_file, dagger_write_file
|
||||
from pipelines.models.steps import StepModifyingFiles, StepResult, StepStatus
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from typing import Optional
|
||||
|
||||
import dagger
|
||||
|
||||
|
||||
@@ -40,17 +40,51 @@ class UpdateBaseImageMetadata(StepModifyingFiles):
|
||||
self.modified_files = []
|
||||
self.connector_directory = connector_directory
|
||||
|
||||
def _get_repository_for_language(self, language: ConnectorLanguage) -> str:
|
||||
"""Map connector language to DockerHub repository."""
|
||||
if language in [ConnectorLanguage.PYTHON, ConnectorLanguage.LOW_CODE]:
|
||||
return "airbyte/python-connector-base"
|
||||
elif language is ConnectorLanguage.MANIFEST_ONLY:
|
||||
return "airbyte/source-declarative-manifest"
|
||||
elif language is ConnectorLanguage.JAVA:
|
||||
return "airbyte/java-connector-base"
|
||||
else:
|
||||
raise NotImplementedError(f"Registry for language {language} is not implemented yet.")
|
||||
|
||||
def _parse_latest_stable_tag(self, tags: List[str]) -> Optional[str]:
|
||||
"""Parse tags to find latest stable (non-prerelease) version."""
|
||||
valid_versions = []
|
||||
for tag in tags:
|
||||
try:
|
||||
version = semver.VersionInfo.parse(tag)
|
||||
if not version.prerelease: # Exclude pre-release versions
|
||||
valid_versions.append(version)
|
||||
except ValueError:
|
||||
continue # Skip non-semver tags
|
||||
|
||||
if valid_versions:
|
||||
return str(max(valid_versions))
|
||||
return None
|
||||
|
||||
async def get_latest_base_image_address(self) -> Optional[str]:
|
||||
try:
|
||||
if self.context.docker_hub_username is None or self.context.docker_hub_password is None:
|
||||
if not (self.context.docker_hub_username and self.context.docker_hub_password):
|
||||
raise ValueError("Docker Hub credentials are required to get the latest base image address")
|
||||
version_registry_for_language = await version_registry.get_registry_for_language(
|
||||
self.dagger_client,
|
||||
self.context.connector.language,
|
||||
(self.context.docker_hub_username.value, self.context.docker_hub_password.value),
|
||||
cache_ttl_seconds=self.BASE_IMAGE_LIST_CACHE_TTL_SECONDS,
|
||||
)
|
||||
return version_registry_for_language.latest_not_pre_released_published_entry.published_docker_image.address
|
||||
|
||||
repository = self._get_repository_for_language(self.context.connector.language)
|
||||
crane_container = with_crane(self.context)
|
||||
|
||||
# List all tags
|
||||
tags_output = await crane_container.with_exec(["crane", "ls", f"docker.io/{repository}"]).stdout()
|
||||
tags = [tag.strip() for tag in tags_output.strip().split("\n") if tag.strip()]
|
||||
|
||||
latest_tag = self._parse_latest_stable_tag(tags)
|
||||
if latest_tag:
|
||||
# Get the digest for the specific tag to ensure immutable reference
|
||||
digest_output = await crane_container.with_exec(["crane", "digest", f"docker.io/{repository}:{latest_tag}"]).stdout()
|
||||
digest = digest_output.strip()
|
||||
return f"docker.io/{repository}:{latest_tag}@{digest}"
|
||||
return None
|
||||
except NotImplementedError:
|
||||
return None
|
||||
|
||||
|
||||
@@ -7,7 +7,6 @@ from pathlib import Path
|
||||
INTERNAL_POETRY_PACKAGES = [
|
||||
"airbyte-ci/connectors/auto_merge",
|
||||
"airbyte-ci/connectors/pipelines",
|
||||
"airbyte-ci/connectors/base_images",
|
||||
"airbyte-ci/connectors/connectors_insights",
|
||||
"airbyte-ci/connectors/connector_ops",
|
||||
"airbyte-ci/connectors/connectors_qa",
|
||||
|
||||
232
airbyte-ci/connectors/pipelines/poetry.lock
generated
232
airbyte-ci/connectors/pipelines/poetry.lock
generated
@@ -1,28 +1,4 @@
|
||||
# This file is automatically @generated by Poetry 2.0.1 and should not be changed by hand.
|
||||
|
||||
[[package]]
|
||||
name = "airbyte-connectors-base-images"
|
||||
version = "1.6"
|
||||
description = "This package is used to generate and publish the base images for Airbyte Connectors."
|
||||
optional = false
|
||||
python-versions = "^3.11"
|
||||
groups = ["main"]
|
||||
files = []
|
||||
develop = true
|
||||
|
||||
[package.dependencies]
|
||||
beartype = ">=0.18.2"
|
||||
connector-ops = {path = "../connector_ops", develop = true}
|
||||
dagger-io = "==0.13.3"
|
||||
gitpython = "^3.1.35"
|
||||
inquirer = "^3.1.3"
|
||||
jinja2 = "^3.1.2"
|
||||
rich = "^13.5.2"
|
||||
semver = "^3.0.1"
|
||||
|
||||
[package.source]
|
||||
type = "directory"
|
||||
url = "../base_images"
|
||||
# This file is automatically @generated by Poetry 2.1.3 and should not be changed by hand.
|
||||
|
||||
[[package]]
|
||||
name = "airbyte-protocol-models"
|
||||
@@ -51,19 +27,6 @@ files = [
|
||||
{file = "altgraph-0.17.4.tar.gz", hash = "sha256:1b5afbb98f6c4dcadb2e2ae6ab9fa994bbb8c1d75f4fa96d340f9437ae454406"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ansicon"
|
||||
version = "1.89.0"
|
||||
description = "Python wrapper for loading Jason Hood's ANSICON"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
groups = ["main"]
|
||||
markers = "platform_system == \"Windows\""
|
||||
files = [
|
||||
{file = "ansicon-1.89.0-py2.py3-none-any.whl", hash = "sha256:f1def52d17f65c2c9682cf8370c03f541f410c1752d6a14029f97318e4b9dfec"},
|
||||
{file = "ansicon-1.89.0.tar.gz", hash = "sha256:e4d039def5768a47e4afec8e89e83ec3ae5a26bf00ad851f914d1240b444d2b1"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "anyio"
|
||||
version = "4.9.0"
|
||||
@@ -83,7 +46,7 @@ typing_extensions = {version = ">=4.5", markers = "python_version < \"3.13\""}
|
||||
|
||||
[package.extras]
|
||||
doc = ["Sphinx (>=8.2,<9.0)", "packaging", "sphinx-autodoc-typehints (>=1.2.0)", "sphinx_rtd_theme"]
|
||||
test = ["anyio[trio]", "blockbuster (>=1.5.23)", "coverage[toml] (>=7)", "exceptiongroup (>=1.2.0)", "hypothesis (>=4.0)", "psutil (>=5.9)", "pytest (>=7.0)", "trustme", "truststore (>=0.9.1)", "uvloop (>=0.21)"]
|
||||
test = ["anyio[trio]", "blockbuster (>=1.5.23)", "coverage[toml] (>=7)", "exceptiongroup (>=1.2.0)", "hypothesis (>=4.0)", "psutil (>=5.9)", "pytest (>=7.0)", "trustme", "truststore (>=0.9.1) ; python_version >= \"3.10\"", "uvloop (>=0.21) ; platform_python_implementation == \"CPython\" and platform_system != \"Windows\" and python_version < \"3.14\""]
|
||||
trio = ["trio (>=0.26.1)"]
|
||||
|
||||
[[package]]
|
||||
@@ -131,12 +94,12 @@ files = [
|
||||
]
|
||||
|
||||
[package.extras]
|
||||
benchmark = ["cloudpickle", "hypothesis", "mypy (>=1.11.1)", "pympler", "pytest (>=4.3.0)", "pytest-codspeed", "pytest-mypy-plugins", "pytest-xdist[psutil]"]
|
||||
cov = ["cloudpickle", "coverage[toml] (>=5.3)", "hypothesis", "mypy (>=1.11.1)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "pytest-xdist[psutil]"]
|
||||
dev = ["cloudpickle", "hypothesis", "mypy (>=1.11.1)", "pre-commit-uv", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "pytest-xdist[psutil]"]
|
||||
benchmark = ["cloudpickle ; platform_python_implementation == \"CPython\"", "hypothesis", "mypy (>=1.11.1) ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pympler", "pytest (>=4.3.0)", "pytest-codspeed", "pytest-mypy-plugins ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pytest-xdist[psutil]"]
|
||||
cov = ["cloudpickle ; platform_python_implementation == \"CPython\"", "coverage[toml] (>=5.3)", "hypothesis", "mypy (>=1.11.1) ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pytest-xdist[psutil]"]
|
||||
dev = ["cloudpickle ; platform_python_implementation == \"CPython\"", "hypothesis", "mypy (>=1.11.1) ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pre-commit-uv", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pytest-xdist[psutil]"]
|
||||
docs = ["cogapp", "furo", "myst-parser", "sphinx", "sphinx-notfound-page", "sphinxcontrib-towncrier", "towncrier"]
|
||||
tests = ["cloudpickle", "hypothesis", "mypy (>=1.11.1)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "pytest-xdist[psutil]"]
|
||||
tests-mypy = ["mypy (>=1.11.1)", "pytest-mypy-plugins"]
|
||||
tests = ["cloudpickle ; platform_python_implementation == \"CPython\"", "hypothesis", "mypy (>=1.11.1) ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pytest-xdist[psutil]"]
|
||||
tests-mypy = ["mypy (>=1.11.1) ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\"", "pytest-mypy-plugins ; platform_python_implementation == \"CPython\" and python_version >= \"3.10\""]
|
||||
|
||||
[[package]]
|
||||
name = "auto-merge"
|
||||
@@ -181,28 +144,12 @@ files = [
|
||||
]
|
||||
|
||||
[package.extras]
|
||||
dev = ["autoapi (>=0.9.0)", "click", "coverage (>=5.5)", "equinox", "jax[cpu]", "jaxtyping", "langchain", "mypy (>=0.800)", "nuitka (>=1.2.6)", "numba", "numpy", "pandera", "pydata-sphinx-theme (<=0.7.2)", "pygments", "pyright (>=1.1.370)", "pytest (>=4.0.0)", "rich-click", "sphinx", "sphinx (>=4.2.0,<6.0.0)", "sphinxext-opengraph (>=0.7.5)", "tox (>=3.20.1)", "typing-extensions (>=3.10.0.0)", "xarray"]
|
||||
dev = ["autoapi (>=0.9.0)", "click", "coverage (>=5.5)", "equinox ; sys_platform == \"linux\"", "jax[cpu] ; sys_platform == \"linux\"", "jaxtyping ; sys_platform == \"linux\"", "langchain", "mypy (>=0.800) ; platform_python_implementation != \"PyPy\"", "nuitka (>=1.2.6) ; sys_platform == \"linux\"", "numba ; python_version < \"3.13.0\"", "numpy ; sys_platform != \"darwin\" and platform_python_implementation != \"PyPy\"", "pandera", "pydata-sphinx-theme (<=0.7.2)", "pygments", "pyright (>=1.1.370)", "pytest (>=4.0.0)", "rich-click", "sphinx", "sphinx (>=4.2.0,<6.0.0)", "sphinxext-opengraph (>=0.7.5)", "tox (>=3.20.1)", "typing-extensions (>=3.10.0.0)", "xarray"]
|
||||
doc-rtd = ["autoapi (>=0.9.0)", "pydata-sphinx-theme (<=0.7.2)", "sphinx (>=4.2.0,<6.0.0)", "sphinxext-opengraph (>=0.7.5)"]
|
||||
test = ["click", "coverage (>=5.5)", "equinox", "jax[cpu]", "jaxtyping", "langchain", "mypy (>=0.800)", "nuitka (>=1.2.6)", "numba", "numpy", "pandera", "pygments", "pyright (>=1.1.370)", "pytest (>=4.0.0)", "rich-click", "sphinx", "tox (>=3.20.1)", "typing-extensions (>=3.10.0.0)", "xarray"]
|
||||
test-tox = ["click", "equinox", "jax[cpu]", "jaxtyping", "langchain", "mypy (>=0.800)", "nuitka (>=1.2.6)", "numba", "numpy", "pandera", "pygments", "pyright (>=1.1.370)", "pytest (>=4.0.0)", "rich-click", "sphinx", "typing-extensions (>=3.10.0.0)", "xarray"]
|
||||
test = ["click", "coverage (>=5.5)", "equinox ; sys_platform == \"linux\"", "jax[cpu] ; sys_platform == \"linux\"", "jaxtyping ; sys_platform == \"linux\"", "langchain", "mypy (>=0.800) ; platform_python_implementation != \"PyPy\"", "nuitka (>=1.2.6) ; sys_platform == \"linux\"", "numba ; python_version < \"3.13.0\"", "numpy ; sys_platform != \"darwin\" and platform_python_implementation != \"PyPy\"", "pandera", "pygments", "pyright (>=1.1.370)", "pytest (>=4.0.0)", "rich-click", "sphinx", "tox (>=3.20.1)", "typing-extensions (>=3.10.0.0)", "xarray"]
|
||||
test-tox = ["click", "equinox ; sys_platform == \"linux\"", "jax[cpu] ; sys_platform == \"linux\"", "jaxtyping ; sys_platform == \"linux\"", "langchain", "mypy (>=0.800) ; platform_python_implementation != \"PyPy\"", "nuitka (>=1.2.6) ; sys_platform == \"linux\"", "numba ; python_version < \"3.13.0\"", "numpy ; sys_platform != \"darwin\" and platform_python_implementation != \"PyPy\"", "pandera", "pygments", "pyright (>=1.1.370)", "pytest (>=4.0.0)", "rich-click", "sphinx", "typing-extensions (>=3.10.0.0)", "xarray"]
|
||||
test-tox-coverage = ["coverage (>=5.5)"]
|
||||
|
||||
[[package]]
|
||||
name = "blessed"
|
||||
version = "1.21.0"
|
||||
description = "Easy, practical library for making terminal apps, by providing an elegant, well-documented interface to Colors, Keyboard input, and screen Positioning capabilities."
|
||||
optional = false
|
||||
python-versions = ">=2.7"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "blessed-1.21.0-py2.py3-none-any.whl", hash = "sha256:f831e847396f5a2eac6c106f4dfadedf46c4f804733574b15fe86d2ed45a9588"},
|
||||
{file = "blessed-1.21.0.tar.gz", hash = "sha256:ece8bbc4758ab9176452f4e3a719d70088eb5739798cd5582c9e05f2a28337ec"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
jinxed = {version = ">=1.1.0", markers = "platform_system == \"Windows\""}
|
||||
wcwidth = ">=0.1.4"
|
||||
|
||||
[[package]]
|
||||
name = "cachetools"
|
||||
version = "5.5.2"
|
||||
@@ -234,8 +181,8 @@ attrs = ">=23.1.0"
|
||||
bson = ["pymongo (>=4.4.0)"]
|
||||
cbor2 = ["cbor2 (>=5.4.6)"]
|
||||
msgpack = ["msgpack (>=1.0.5)"]
|
||||
msgspec = ["msgspec (>=0.18.5)"]
|
||||
orjson = ["orjson (>=3.9.2)"]
|
||||
msgspec = ["msgspec (>=0.18.5) ; implementation_name == \"cpython\""]
|
||||
orjson = ["orjson (>=3.9.2) ; implementation_name == \"cpython\""]
|
||||
pyyaml = ["pyyaml (>=6.0)"]
|
||||
tomlkit = ["tomlkit (>=0.11.8)"]
|
||||
ujson = ["ujson (>=5.7.0)"]
|
||||
@@ -586,7 +533,7 @@ files = [
|
||||
]
|
||||
|
||||
[package.extras]
|
||||
toml = ["tomli"]
|
||||
toml = ["tomli ; python_full_version <= \"3.11.0a6\""]
|
||||
|
||||
[[package]]
|
||||
name = "cryptography"
|
||||
@@ -637,10 +584,10 @@ files = [
|
||||
cffi = {version = ">=1.12", markers = "platform_python_implementation != \"PyPy\""}
|
||||
|
||||
[package.extras]
|
||||
docs = ["sphinx (>=5.3.0)", "sphinx-rtd-theme (>=3.0.0)"]
|
||||
docs = ["sphinx (>=5.3.0)", "sphinx-rtd-theme (>=3.0.0) ; python_version >= \"3.8\""]
|
||||
docstest = ["pyenchant (>=3)", "readme-renderer (>=30.0)", "sphinxcontrib-spelling (>=7.3.1)"]
|
||||
nox = ["nox (>=2024.4.15)", "nox[uv] (>=2024.3.2)"]
|
||||
pep8test = ["check-sdist", "click (>=8.0.1)", "mypy (>=1.4)", "ruff (>=0.3.6)"]
|
||||
nox = ["nox (>=2024.4.15)", "nox[uv] (>=2024.3.2) ; python_version >= \"3.8\""]
|
||||
pep8test = ["check-sdist ; python_version >= \"3.8\"", "click (>=8.0.1)", "mypy (>=1.4)", "ruff (>=0.3.6)"]
|
||||
sdist = ["build (>=1.0.0)"]
|
||||
ssh = ["bcrypt (>=3.1.5)"]
|
||||
test = ["certifi (>=2024)", "cryptography-vectors (==44.0.2)", "pretend (>=0.7)", "pytest (>=7.4.0)", "pytest-benchmark (>=4.0)", "pytest-cov (>=2.10.1)", "pytest-xdist (>=3.5.0)"]
|
||||
@@ -704,7 +651,7 @@ files = [
|
||||
wrapt = ">=1.10,<2"
|
||||
|
||||
[package.extras]
|
||||
dev = ["PyTest", "PyTest-Cov", "bump2version (<1)", "setuptools", "tox"]
|
||||
dev = ["PyTest", "PyTest-Cov", "bump2version (<1)", "setuptools ; python_version >= \"3.12\"", "tox"]
|
||||
|
||||
[[package]]
|
||||
name = "docker"
|
||||
@@ -741,22 +688,6 @@ files = [
|
||||
{file = "dpath-2.2.0.tar.gz", hash = "sha256:34f7e630dc55ea3f219e555726f5da4b4b25f2200319c8e6902c394258dd6a3e"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "editor"
|
||||
version = "1.6.6"
|
||||
description = "🖋 Open the default text editor 🖋"
|
||||
optional = false
|
||||
python-versions = ">=3.8"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "editor-1.6.6-py3-none-any.whl", hash = "sha256:e818e6913f26c2a81eadef503a2741d7cca7f235d20e217274a009ecd5a74abf"},
|
||||
{file = "editor-1.6.6.tar.gz", hash = "sha256:bb6989e872638cd119db9a4fce284cd8e13c553886a1c044c6b8d8a160c871f8"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
runs = "*"
|
||||
xmod = "*"
|
||||
|
||||
[[package]]
|
||||
name = "exceptiongroup"
|
||||
version = "1.2.2"
|
||||
@@ -819,7 +750,7 @@ gitdb = ">=4.0.1,<5"
|
||||
|
||||
[package.extras]
|
||||
doc = ["sphinx (>=7.1.2,<7.2)", "sphinx-autodoc-typehints", "sphinx_rtd_theme"]
|
||||
test = ["coverage[toml]", "ddt (>=1.1.1,!=1.4.3)", "mock", "mypy", "pre-commit", "pytest (>=7.3.1)", "pytest-cov", "pytest-instafail", "pytest-mock", "pytest-sugar", "typing-extensions"]
|
||||
test = ["coverage[toml]", "ddt (>=1.1.1,!=1.4.3)", "mock ; python_version < \"3.8\"", "mypy", "pre-commit", "pytest (>=7.3.1)", "pytest-cov", "pytest-instafail", "pytest-mock", "pytest-sugar", "typing-extensions ; python_version < \"3.11\""]
|
||||
|
||||
[[package]]
|
||||
name = "google-api-core"
|
||||
@@ -844,7 +775,7 @@ requests = ">=2.18.0,<3.0.0"
|
||||
|
||||
[package.extras]
|
||||
async-rest = ["google-auth[aiohttp] (>=2.35.0,<3.0.dev0)"]
|
||||
grpc = ["grpcio (>=1.33.2,<2.0dev)", "grpcio (>=1.49.1,<2.0dev)", "grpcio-status (>=1.33.2,<2.0.dev0)", "grpcio-status (>=1.49.1,<2.0.dev0)"]
|
||||
grpc = ["grpcio (>=1.33.2,<2.0dev)", "grpcio (>=1.49.1,<2.0dev) ; python_version >= \"3.11\"", "grpcio-status (>=1.33.2,<2.0.dev0)", "grpcio-status (>=1.49.1,<2.0.dev0) ; python_version >= \"3.11\""]
|
||||
grpcgcp = ["grpcio-gcp (>=0.2.2,<1.0.dev0)"]
|
||||
grpcio-gcp = ["grpcio-gcp (>=0.2.2,<1.0.dev0)"]
|
||||
|
||||
@@ -868,11 +799,11 @@ rsa = ">=3.1.4,<5"
|
||||
[package.extras]
|
||||
aiohttp = ["aiohttp (>=3.6.2,<4.0.0)", "requests (>=2.20.0,<3.0.0)"]
|
||||
enterprise-cert = ["cryptography", "pyopenssl"]
|
||||
pyjwt = ["cryptography (<39.0.0)", "cryptography (>=38.0.3)", "pyjwt (>=2.0)"]
|
||||
pyopenssl = ["cryptography (<39.0.0)", "cryptography (>=38.0.3)", "pyopenssl (>=20.0.0)"]
|
||||
pyjwt = ["cryptography (<39.0.0) ; python_version < \"3.8\"", "cryptography (>=38.0.3)", "pyjwt (>=2.0)"]
|
||||
pyopenssl = ["cryptography (<39.0.0) ; python_version < \"3.8\"", "cryptography (>=38.0.3)", "pyopenssl (>=20.0.0)"]
|
||||
reauth = ["pyu2f (>=0.1.5)"]
|
||||
requests = ["requests (>=2.20.0,<3.0.0)"]
|
||||
testing = ["aiohttp (<3.10.0)", "aiohttp (>=3.6.2,<4.0.0)", "aioresponses", "cryptography (<39.0.0)", "cryptography (>=38.0.3)", "flask", "freezegun", "grpcio", "mock", "oauth2client", "packaging", "pyjwt (>=2.0)", "pyopenssl (<24.3.0)", "pyopenssl (>=20.0.0)", "pytest", "pytest-asyncio", "pytest-cov", "pytest-localserver", "pyu2f (>=0.1.5)", "requests (>=2.20.0,<3.0.0)", "responses", "urllib3"]
|
||||
testing = ["aiohttp (<3.10.0)", "aiohttp (>=3.6.2,<4.0.0)", "aioresponses", "cryptography (<39.0.0) ; python_version < \"3.8\"", "cryptography (>=38.0.3)", "flask", "freezegun", "grpcio", "mock", "oauth2client", "packaging", "pyjwt (>=2.0)", "pyopenssl (<24.3.0)", "pyopenssl (>=20.0.0)", "pytest", "pytest-asyncio", "pytest-cov", "pytest-localserver", "pyu2f (>=0.1.5)", "requests (>=2.20.0,<3.0.0)", "responses", "urllib3"]
|
||||
urllib3 = ["packaging", "urllib3"]
|
||||
|
||||
[[package]]
|
||||
@@ -1042,14 +973,14 @@ httpx = {version = ">=0.23.1,<1", optional = true, markers = "extra == \"httpx\"
|
||||
yarl = ">=1.6,<2.0"
|
||||
|
||||
[package.extras]
|
||||
aiohttp = ["aiohttp (>=3.8.0,<4)", "aiohttp (>=3.9.0b0,<4)"]
|
||||
all = ["aiohttp (>=3.8.0,<4)", "aiohttp (>=3.9.0b0,<4)", "botocore (>=1.21,<2)", "httpx (>=0.23.1,<1)", "requests (>=2.26,<3)", "requests-toolbelt (>=1.0.0,<2)", "websockets (>=10,<12)"]
|
||||
aiohttp = ["aiohttp (>=3.8.0,<4) ; python_version <= \"3.11\"", "aiohttp (>=3.9.0b0,<4) ; python_version > \"3.11\""]
|
||||
all = ["aiohttp (>=3.8.0,<4) ; python_version <= \"3.11\"", "aiohttp (>=3.9.0b0,<4) ; python_version > \"3.11\"", "botocore (>=1.21,<2)", "httpx (>=0.23.1,<1)", "requests (>=2.26,<3)", "requests-toolbelt (>=1.0.0,<2)", "websockets (>=10,<12)"]
|
||||
botocore = ["botocore (>=1.21,<2)"]
|
||||
dev = ["aiofiles", "aiohttp (>=3.8.0,<4)", "aiohttp (>=3.9.0b0,<4)", "black (==22.3.0)", "botocore (>=1.21,<2)", "check-manifest (>=0.42,<1)", "flake8 (==3.8.1)", "httpx (>=0.23.1,<1)", "isort (==4.3.21)", "mock (==4.0.2)", "mypy (==0.910)", "parse (==1.15.0)", "pytest (==7.4.2)", "pytest-asyncio (==0.21.1)", "pytest-console-scripts (==1.3.1)", "pytest-cov (==3.0.0)", "requests (>=2.26,<3)", "requests-toolbelt (>=1.0.0,<2)", "sphinx (>=5.3.0,<6)", "sphinx-argparse (==0.2.5)", "sphinx-rtd-theme (>=0.4,<1)", "types-aiofiles", "types-mock", "types-requests", "vcrpy (==4.4.0)", "vcrpy (==7.0.0)", "websockets (>=10,<12)"]
|
||||
dev = ["aiofiles", "aiohttp (>=3.8.0,<4) ; python_version <= \"3.11\"", "aiohttp (>=3.9.0b0,<4) ; python_version > \"3.11\"", "black (==22.3.0)", "botocore (>=1.21,<2)", "check-manifest (>=0.42,<1)", "flake8 (==3.8.1)", "httpx (>=0.23.1,<1)", "isort (==4.3.21)", "mock (==4.0.2)", "mypy (==0.910)", "parse (==1.15.0)", "pytest (==7.4.2)", "pytest-asyncio (==0.21.1)", "pytest-console-scripts (==1.3.1)", "pytest-cov (==3.0.0)", "requests (>=2.26,<3)", "requests-toolbelt (>=1.0.0,<2)", "sphinx (>=5.3.0,<6)", "sphinx-argparse (==0.2.5)", "sphinx-rtd-theme (>=0.4,<1)", "types-aiofiles", "types-mock", "types-requests", "vcrpy (==4.4.0) ; python_version <= \"3.8\"", "vcrpy (==7.0.0) ; python_version > \"3.8\"", "websockets (>=10,<12)"]
|
||||
httpx = ["httpx (>=0.23.1,<1)"]
|
||||
requests = ["requests (>=2.26,<3)", "requests-toolbelt (>=1.0.0,<2)"]
|
||||
test = ["aiofiles", "aiohttp (>=3.8.0,<4)", "aiohttp (>=3.9.0b0,<4)", "botocore (>=1.21,<2)", "httpx (>=0.23.1,<1)", "mock (==4.0.2)", "parse (==1.15.0)", "pytest (==7.4.2)", "pytest-asyncio (==0.21.1)", "pytest-console-scripts (==1.3.1)", "pytest-cov (==3.0.0)", "requests (>=2.26,<3)", "requests-toolbelt (>=1.0.0,<2)", "vcrpy (==4.4.0)", "vcrpy (==7.0.0)", "websockets (>=10,<12)"]
|
||||
test-no-transport = ["aiofiles", "mock (==4.0.2)", "parse (==1.15.0)", "pytest (==7.4.2)", "pytest-asyncio (==0.21.1)", "pytest-console-scripts (==1.3.1)", "pytest-cov (==3.0.0)", "vcrpy (==4.4.0)", "vcrpy (==7.0.0)"]
|
||||
test = ["aiofiles", "aiohttp (>=3.8.0,<4) ; python_version <= \"3.11\"", "aiohttp (>=3.9.0b0,<4) ; python_version > \"3.11\"", "botocore (>=1.21,<2)", "httpx (>=0.23.1,<1)", "mock (==4.0.2)", "parse (==1.15.0)", "pytest (==7.4.2)", "pytest-asyncio (==0.21.1)", "pytest-console-scripts (==1.3.1)", "pytest-cov (==3.0.0)", "requests (>=2.26,<3)", "requests-toolbelt (>=1.0.0,<2)", "vcrpy (==4.4.0) ; python_version <= \"3.8\"", "vcrpy (==7.0.0) ; python_version > \"3.8\"", "websockets (>=10,<12)"]
|
||||
test-no-transport = ["aiofiles", "mock (==4.0.2)", "parse (==1.15.0)", "pytest (==7.4.2)", "pytest-asyncio (==0.21.1)", "pytest-console-scripts (==1.3.1)", "pytest-cov (==3.0.0)", "vcrpy (==4.4.0) ; python_version <= \"3.8\"", "vcrpy (==7.0.0) ; python_version > \"3.8\""]
|
||||
websockets = ["websockets (>=10,<12)"]
|
||||
|
||||
[[package]]
|
||||
@@ -1215,7 +1146,7 @@ httpcore = "==1.*"
|
||||
idna = "*"
|
||||
|
||||
[package.extras]
|
||||
brotli = ["brotli", "brotlicffi"]
|
||||
brotli = ["brotli ; platform_python_implementation == \"CPython\"", "brotlicffi ; platform_python_implementation != \"CPython\""]
|
||||
cli = ["click (==8.*)", "pygments (==2.*)", "rich (>=10,<14)"]
|
||||
http2 = ["h2 (>=3,<5)"]
|
||||
socks = ["socksio (==1.*)"]
|
||||
@@ -1252,12 +1183,12 @@ files = [
|
||||
zipp = ">=3.20"
|
||||
|
||||
[package.extras]
|
||||
check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1)"]
|
||||
check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1) ; sys_platform != \"cygwin\""]
|
||||
cover = ["pytest-cov"]
|
||||
doc = ["furo", "jaraco.packaging (>=9.3)", "jaraco.tidelift (>=1.4)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-lint"]
|
||||
enabler = ["pytest-enabler (>=2.2)"]
|
||||
perf = ["ipython"]
|
||||
test = ["flufl.flake8", "importlib_resources (>=1.3)", "jaraco.test (>=5.4)", "packaging", "pyfakefs", "pytest (>=6,!=8.1.*)", "pytest-perf (>=0.9.2)"]
|
||||
test = ["flufl.flake8", "importlib_resources (>=1.3) ; python_version < \"3.9\"", "jaraco.test (>=5.4)", "packaging", "pyfakefs", "pytest (>=6,!=8.1.*)", "pytest-perf (>=0.9.2)"]
|
||||
type = ["pytest-mypy"]
|
||||
|
||||
[[package]]
|
||||
@@ -1272,23 +1203,6 @@ files = [
|
||||
{file = "iniconfig-2.1.0.tar.gz", hash = "sha256:3abbd2e30b36733fee78f9c7f7308f2d0050e88f0087fd25c2645f63c773e1c7"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "inquirer"
|
||||
version = "3.4.0"
|
||||
description = "Collection of common interactive command line user interfaces, based on Inquirer.js"
|
||||
optional = false
|
||||
python-versions = ">=3.8.1"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "inquirer-3.4.0-py3-none-any.whl", hash = "sha256:bb0ec93c833e4ce7b51b98b1644b0a4d2bb39755c39787f6a504e4fee7a11b60"},
|
||||
{file = "inquirer-3.4.0.tar.gz", hash = "sha256:8edc99c076386ee2d2204e5e3653c2488244e82cb197b2d498b3c1b5ffb25d0b"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
blessed = ">=1.19.0"
|
||||
editor = ">=1.6.0"
|
||||
readchar = ">=4.2.0"
|
||||
|
||||
[[package]]
|
||||
name = "jinja2"
|
||||
version = "3.1.6"
|
||||
@@ -1307,22 +1221,6 @@ MarkupSafe = ">=2.0"
|
||||
[package.extras]
|
||||
i18n = ["Babel (>=2.7)"]
|
||||
|
||||
[[package]]
|
||||
name = "jinxed"
|
||||
version = "1.3.0"
|
||||
description = "Jinxed Terminal Library"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
groups = ["main"]
|
||||
markers = "platform_system == \"Windows\""
|
||||
files = [
|
||||
{file = "jinxed-1.3.0-py2.py3-none-any.whl", hash = "sha256:b993189f39dc2d7504d802152671535b06d380b26d78070559551cbf92df4fc5"},
|
||||
{file = "jinxed-1.3.0.tar.gz", hash = "sha256:1593124b18a41b7a3da3b078471442e51dbad3d77b4d4f2b0c26ab6f7d660dbf"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
ansicon = {version = "*", markers = "platform_system == \"Windows\""}
|
||||
|
||||
[[package]]
|
||||
name = "macholib"
|
||||
version = "1.16.3"
|
||||
@@ -1644,7 +1542,6 @@ description = "Fundamental package for array computing in Python"
|
||||
optional = false
|
||||
python-versions = ">=3.10"
|
||||
groups = ["main"]
|
||||
markers = "python_version == \"3.11\""
|
||||
files = [
|
||||
{file = "numpy-2.2.5-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:1f4a922da1729f4c40932b2af4fe84909c7a6e167e6e99f71838ce3a29f3fe26"},
|
||||
{file = "numpy-2.2.5-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:b6f91524d31b34f4a5fee24f5bc16dcd1491b668798b6d85585d836c1e633a6a"},
|
||||
@@ -2271,7 +2168,7 @@ files = [
|
||||
]
|
||||
|
||||
[package.extras]
|
||||
dev = ["Sphinx", "black", "build", "coverage", "docformatter", "flake8", "flake8-black", "flake8-bugbear", "flake8-isort", "importlib-metadata (<5)", "invoke", "isort", "pylint", "pytest", "pytest-cov", "sphinx-rtd-theme", "tox", "twine", "wheel"]
|
||||
dev = ["Sphinx", "black", "build", "coverage", "docformatter", "flake8", "flake8-black", "flake8-bugbear", "flake8-isort", "importlib-metadata (<5) ; python_version == \"3.7\"", "invoke", "isort", "pylint", "pytest", "pytest-cov", "sphinx-rtd-theme", "tox", "twine", "wheel"]
|
||||
|
||||
[[package]]
|
||||
name = "pygit2"
|
||||
@@ -2659,18 +2556,6 @@ files = [
|
||||
{file = "pyyaml-6.0.2.tar.gz", hash = "sha256:d584d9ec91ad65861cc08d42e834324ef890a082e591037abe114850ff7bbc3e"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "readchar"
|
||||
version = "4.2.1"
|
||||
description = "Library to easily read single chars and key strokes"
|
||||
optional = false
|
||||
python-versions = ">=3.8"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "readchar-4.2.1-py3-none-any.whl", hash = "sha256:a769305cd3994bb5fa2764aa4073452dc105a4ec39068ffe6efd3c20c60acc77"},
|
||||
{file = "readchar-4.2.1.tar.gz", hash = "sha256:91ce3faf07688de14d800592951e5575e9c7a3213738ed01d394dcc949b79adb"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "requests"
|
||||
version = "2.32.3"
|
||||
@@ -2831,21 +2716,6 @@ files = [
|
||||
{file = "ruff-0.5.7.tar.gz", hash = "sha256:8dfc0a458797f5d9fb622dd0efc52d796f23f0a1493a9527f4e49a550ae9a7e5"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "runs"
|
||||
version = "1.2.2"
|
||||
description = "🏃 Run a block of text as a subprocess 🏃"
|
||||
optional = false
|
||||
python-versions = ">=3.8"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "runs-1.2.2-py3-none-any.whl", hash = "sha256:0980dcbc25aba1505f307ac4f0e9e92cbd0be2a15a1e983ee86c24c87b839dfd"},
|
||||
{file = "runs-1.2.2.tar.gz", hash = "sha256:9dc1815e2895cfb3a48317b173b9f1eac9ba5549b36a847b5cc60c3bf82ecef1"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
xmod = "*"
|
||||
|
||||
[[package]]
|
||||
name = "segment-analytics-python"
|
||||
version = "2.3.3"
|
||||
@@ -2949,13 +2819,13 @@ files = [
|
||||
]
|
||||
|
||||
[package.extras]
|
||||
check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1)", "ruff (>=0.8.0)"]
|
||||
core = ["importlib_metadata (>=6)", "jaraco.functools (>=4)", "jaraco.text (>=3.7)", "more_itertools", "more_itertools (>=8.8)", "packaging (>=24.2)", "platformdirs (>=4.2.2)", "tomli (>=2.0.1)", "wheel (>=0.43.0)"]
|
||||
check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1) ; sys_platform != \"cygwin\"", "ruff (>=0.8.0) ; sys_platform != \"cygwin\""]
|
||||
core = ["importlib_metadata (>=6) ; python_version < \"3.10\"", "jaraco.functools (>=4)", "jaraco.text (>=3.7)", "more_itertools", "more_itertools (>=8.8)", "packaging (>=24.2)", "platformdirs (>=4.2.2)", "tomli (>=2.0.1) ; python_version < \"3.11\"", "wheel (>=0.43.0)"]
|
||||
cover = ["pytest-cov"]
|
||||
doc = ["furo", "jaraco.packaging (>=9.3)", "jaraco.tidelift (>=1.4)", "pygments-github-lexers (==0.0.5)", "pyproject-hooks (!=1.1)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-favicon", "sphinx-inline-tabs", "sphinx-lint", "sphinx-notfound-page (>=1,<2)", "sphinx-reredirects", "sphinxcontrib-towncrier", "towncrier (<24.7)"]
|
||||
enabler = ["pytest-enabler (>=2.2)"]
|
||||
test = ["build[virtualenv] (>=1.0.3)", "filelock (>=3.4.0)", "ini2toml[lite] (>=0.14)", "jaraco.develop (>=7.21)", "jaraco.envs (>=2.2)", "jaraco.path (>=3.7.2)", "jaraco.test (>=5.5)", "packaging (>=24.2)", "pip (>=19.1)", "pyproject-hooks (!=1.1)", "pytest (>=6,!=8.1.*)", "pytest-home (>=0.5)", "pytest-perf", "pytest-subprocess", "pytest-timeout", "pytest-xdist (>=3)", "tomli-w (>=1.0.0)", "virtualenv (>=13.0.0)", "wheel (>=0.44.0)"]
|
||||
type = ["importlib_metadata (>=7.0.2)", "jaraco.develop (>=7.21)", "mypy (==1.14.*)", "pytest-mypy"]
|
||||
test = ["build[virtualenv] (>=1.0.3)", "filelock (>=3.4.0)", "ini2toml[lite] (>=0.14)", "jaraco.develop (>=7.21) ; python_version >= \"3.9\" and sys_platform != \"cygwin\"", "jaraco.envs (>=2.2)", "jaraco.path (>=3.7.2)", "jaraco.test (>=5.5)", "packaging (>=24.2)", "pip (>=19.1)", "pyproject-hooks (!=1.1)", "pytest (>=6,!=8.1.*)", "pytest-home (>=0.5)", "pytest-perf ; sys_platform != \"cygwin\"", "pytest-subprocess", "pytest-timeout", "pytest-xdist (>=3)", "tomli-w (>=1.0.0)", "virtualenv (>=13.0.0)", "wheel (>=0.44.0)"]
|
||||
type = ["importlib_metadata (>=7.0.2) ; python_version < \"3.10\"", "jaraco.develop (>=7.21) ; sys_platform != \"cygwin\"", "mypy (==1.14.*)", "pytest-mypy"]
|
||||
|
||||
[[package]]
|
||||
name = "simpleeval"
|
||||
@@ -3203,23 +3073,11 @@ files = [
|
||||
]
|
||||
|
||||
[package.extras]
|
||||
brotli = ["brotli (>=1.0.9)", "brotlicffi (>=0.8.0)"]
|
||||
brotli = ["brotli (>=1.0.9) ; platform_python_implementation == \"CPython\"", "brotlicffi (>=0.8.0) ; platform_python_implementation != \"CPython\""]
|
||||
h2 = ["h2 (>=4,<5)"]
|
||||
socks = ["pysocks (>=1.5.6,!=1.5.7,<2.0)"]
|
||||
zstd = ["zstandard (>=0.18.0)"]
|
||||
|
||||
[[package]]
|
||||
name = "wcwidth"
|
||||
version = "0.2.13"
|
||||
description = "Measures the displayed width of unicode strings in a terminal"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "wcwidth-0.2.13-py2.py3-none-any.whl", hash = "sha256:3da69048e4540d84af32131829ff948f1e022c1c6bdb8d6102117aac784f6859"},
|
||||
{file = "wcwidth-0.2.13.tar.gz", hash = "sha256:72ea0c06399eb286d978fdedb6923a9eb47e1c486ce63e9b4e64fc18303972b5"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wrapt"
|
||||
version = "1.17.2"
|
||||
@@ -3321,18 +3179,6 @@ files = [
|
||||
{file = "xmltodict-0.13.0.tar.gz", hash = "sha256:341595a488e3e01a85a9d8911d8912fd922ede5fecc4dce437eb4b6c8d037e56"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "xmod"
|
||||
version = "1.8.1"
|
||||
description = "🌱 Turn any object into a module 🌱"
|
||||
optional = false
|
||||
python-versions = ">=3.8"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "xmod-1.8.1-py3-none-any.whl", hash = "sha256:a24e9458a4853489042522bdca9e50ee2eac5ab75c809a91150a8a7f40670d48"},
|
||||
{file = "xmod-1.8.1.tar.gz", hash = "sha256:38c76486b9d672c546d57d8035df0beb7f4a9b088bc3fb2de5431ae821444377"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "yarl"
|
||||
version = "1.20.0"
|
||||
@@ -3465,14 +3311,14 @@ files = [
|
||||
]
|
||||
|
||||
[package.extras]
|
||||
check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1)"]
|
||||
check = ["pytest-checkdocs (>=2.4)", "pytest-ruff (>=0.2.1) ; sys_platform != \"cygwin\""]
|
||||
cover = ["pytest-cov"]
|
||||
doc = ["furo", "jaraco.packaging (>=9.3)", "jaraco.tidelift (>=1.4)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-lint"]
|
||||
enabler = ["pytest-enabler (>=2.2)"]
|
||||
test = ["big-O", "importlib-resources", "jaraco.functools", "jaraco.itertools", "jaraco.test", "more-itertools", "pytest (>=6,!=8.1.*)", "pytest-ignore-flaky"]
|
||||
test = ["big-O", "importlib-resources ; python_version < \"3.9\"", "jaraco.functools", "jaraco.itertools", "jaraco.test", "more-itertools", "pytest (>=6,!=8.1.*)", "pytest-ignore-flaky"]
|
||||
type = ["pytest-mypy"]
|
||||
|
||||
[metadata]
|
||||
lock-version = "2.1"
|
||||
python-versions = "~3.11"
|
||||
content-hash = "33fe31639d0b0fcf89f8a426f56c7f2bc885406f97ad57fa93d304e46509e160"
|
||||
content-hash = "40348b618d48960db388c3f77018908eb7ad483306154591ae7ce797dde399a6"
|
||||
|
||||
@@ -23,7 +23,6 @@ airbyte-protocol-models = "^0.16.0"
|
||||
jinja2 = "^3.1.6"
|
||||
MarkupSafe = "==2.1.5" # pin transitive dependency to avoid issues with version resolution
|
||||
requests = "^2.31"
|
||||
airbyte-connectors-base-images = { path = "../base_images", develop = true }
|
||||
connector-ops = { path = "../connector_ops", develop = true }
|
||||
toml = "^0.10.2"
|
||||
types-requests = "^2.31"
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
# Migrated from airbyte-ci/connectors/base_images/README.md
|
||||
|
||||
# https://hub.docker.com/_/amazoncorretto/tags?name=21-al2023
|
||||
ARG BASE_IMAGE=docker.io/amazoncorretto:21-al2023
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
# Migrated from airbyte-ci/connectors/base_images/README.md
|
||||
|
||||
# https://hub.docker.com/_/python/tags?name=3.11.13-slim-bookworm
|
||||
ARG BASE_IMAGE=docker.io/python:3.11.13-slim-bookworm@sha256:139020233cc412efe4c8135b0efe1c7569dc8b28ddd88bddb109b764f8977e30
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
Reference in New Issue
Block a user