Update internal tool install instructions to include pyenv, editable, and force (#30858)

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
Co-authored-by: Prateek Mukhedkar <prateek@airbyte.io>

.python-version

@@ -0,0 +1 @@
+3.10

airbyte-ci/.python-version

@@ -0,0 +1 @@
+3.10

airbyte-ci/connectors/ci_credentials/README.md

@@ -11,20 +11,36 @@ This project requires Python 3.10 and pipx.
 
 The recommended way to install `ci_credentials` is using pipx. This ensures the tool and its dependencies are isolated from your other Python projects.
 
+If you havent installed pyenv, you can do it with brew:
+
+```bash
+brew update
+brew install pyenv
+```
+
 If you haven't installed pipx, you can do it with pip:
 
 ```bash
+cd airbyte-ci/connectors/ci_credentials/
+pyenv install # ensure you have the correct python version
 python -m pip install --user pipx
 python -m pipx ensurepath
 ```
 
-Once pipx is installed, navigate to the root directory of the project, then run:
+Once pyenv and pipx is installed then run the following:
 
 ```bash
-pipx install airbyte-ci/connectors/ci_credentials/
+pipx install --editable --force --version=python3.10 airbyte-ci/connectors/ci_credentials/
 ```
 
-This command installs ci_credentials and makes it globally available in your terminal.
+This command installs `ci_credentials` and makes it globally available in your terminal.
+
+_Note: `--force` is required to ensure updates are applied on subsequent installs._
+_Note: `--version=python3.10` is required to ensure the correct python version is used._
+_Note: `--editable` is required to ensure the correct python version is used._
+
+If you face any installation problem feel free to reach out the Airbyte Connectors Operations team.
+
 
 ## Get GSM access
 Download a Service account json key that has access to Google Secrets Manager.

airbyte-ci/connectors/connector_ops/.python-version

@@ -1 +0,0 @@
-3.10.8

airbyte-ci/connectors/pipelines/README.md

@@ -22,25 +22,43 @@ This project requires Python 3.10 and pipx.
 
 The recommended way to install `airbyte-ci` is using pipx. This ensures the tool and its dependencies are isolated from your other Python projects.
 
+If you havent installed pyenv, you can do it with brew:
+
+```bash
+brew update
+brew install pyenv
+```
+
 If you haven't installed pipx, you can do it with pip:
 
 ```bash
+pyenv install # ensure you have the correct python version
 python -m pip install --user pipx
 python -m pipx ensurepath
 ```
 
-Once pipx is installed, navigate to the root directory of the project, then run:
+Once pyenv and pipx is installed then run the following:
 
 ```bash
-pipx install airbyte-ci/connectors/pipelines/ --force
+# install airbyte-ci
+pipx install --editable --force --version=python3.10 airbyte-ci/connectors/pipelines/
 ```
 
 This command installs `airbyte-ci` and makes it globally available in your terminal.
 
 _Note: `--force` is required to ensure updates are applied on subsequent installs._
+_Note: `--version=python3.10` is required to ensure the correct python version is used._
+_Note: `--editable` is required to ensure the correct python version is used._
 
 If you face any installation problem feel free to reach out the Airbyte Connectors Operations team.
 
+### Updating the airbyte-ci tool
+To reinstall airbyte-ci:
+
+```sh
+pipx reinstall pipelines
+```
+
 ## Installation for development
 
 #### Pre-requisites
@@ -380,6 +398,7 @@ This command runs the Python tests for a airbyte-ci poetry package.
 ## Changelog
 | Version | PR                                                        | Description                                                                                               |
 |---------| --------------------------------------------------------- |-----------------------------------------------------------------------------------------------------------|
+| 1.4.3   | [#30595](https://github.com/airbytehq/airbyte/pull/30595) | Add --version and version check                                                                           |
 | 1.4.2   | [#30595](https://github.com/airbytehq/airbyte/pull/30595) | Remove directory name requirement                                                                         |
 | 1.4.1   | [#30595](https://github.com/airbytehq/airbyte/pull/30595) | Load base migration guide into QA Test container for strict encrypt variants                              |
 | 1.4.0   | [#30330](https://github.com/airbytehq/airbyte/pull/30330) | Add support for pyproject.toml as the prefered entry point for a connector package                        |
@@ -414,4 +433,56 @@ This command runs the Python tests for a airbyte-ci poetry package.
 
 ## More info
 This project is owned by the Connectors Operations team.
-We share project updates and remaining stories before its release to production in this [EPIC](https://github.com/airbytehq/airbyte/issues/24403).
+We share project updates and remaining stories before its release to production in this [EPIC](https://github.com/airbytehq/airbyte/issues/24403).
+
+# Troubleshooting
+## `airbyte-ci` is not found
+If you get the following error when running `airbyte-ci`:
+```bash
+$ airbyte-ci
+zsh: command not found: airbyte-ci
+```
+It means that the `airbyte-ci` command is not in your PATH.
+
+To fix this, you can either:
+* Ensure that airbyte-ci is installed with pipx. Run `pipx list` to check if airbyte-ci is installed.
+* Run `pipx ensurepath` to add the pipx binary directory to your PATH.
+* Add the pipx binary directory to your PATH manually. The pipx binary directory is usually `~/.local/bin`.
+
+
+## python3.10 not found
+If you get the following error when running `pipx install --editable --force --version=python3.10 airbyte-ci/connectors/pipelines/`:
+```bash
+$ pipx install --editable --force --version=python3.10 airbyte-ci/connectors/pipelines/
+Error: Python 3.10 not found on your system.
+```
+
+It means that you don't have Python 3.10 installed on your system.
+
+To fix this, you can either:
+* Install Python 3.10 with pyenv. Run `pyenv install 3.10` to install the latest Python version.
+* Install Python 3.10 with your system package manager. For instance, on Ubuntu you can run `sudo apt install python3.10`.
+* Ensure that Python 3.10 is in your PATH. Run `which python3.10` to check if Python 3.10 is installed and in your PATH.
+
+## Any type of pipeline failure
+First you should check that the version of the CLI you are using is the latest one.
+You can check the version of the CLI with the `--version` option:
+```bash
+$ airbyte-ci --version
+airbyte-ci, version 0.1.0
+```
+
+and compare it with the version in the pyproject.toml file:
+```bash
+$ cat airbyte-ci/connectors/pipelines/pyproject.toml | grep version
+```
+
+If you get any type of pipeline failure, you can run the pipeline with the `--show-dagger-logs` option to get more information about the failure.
+```bash
+$ airbyte-ci --show-dagger-logs connectors --name=source-pokeapi test
+```
+
+and when in doubt, you can reinstall the CLI with the `--force` option:
+```bash
+$ pipx reinstall pipelines --force
+```

airbyte-ci/connectors/pipelines/pipelines/commands/airbyte_ci.py

@@ -4,12 +4,16 @@
 
 """This module is the CLI entrypoint to the airbyte-ci commands."""
 
+import importlib
+import os
+import subprocess
 from typing import List
 
 import click
 from github import PullRequest
 from pipelines import github, main_logger
 from pipelines.bases import CIContext
+from pipelines.consts import LOCAL_PIPELINE_PACKAGE_PATH
 from pipelines.utils import (
     get_current_epoch_time,
     get_current_git_branch,
@@ -26,6 +30,31 @@ from .groups.tests import test
 
 # HELPERS
 
+__installed_version__ = importlib.metadata.version("pipelines")
+
+
+def check_up_to_date() -> bool:
+    """Check if the installed version of pipelines is up to date."""
+    # get the version of the latest release, which is just in the pyproject.toml file of the pipelines package
+    # as this is an internal tool, we don't need to check for the latest version on PyPI
+    latest_version = get_latest_version()
+    if latest_version != __installed_version__:
+        main_logger.warning(f"pipelines is not up to date. Installed version: {__installed_version__}. Latest version: {latest_version}")
+        main_logger.warning("Please run `pipx reinstall pipelines` to upgrade to the latest version.")
+        return False
+
+    main_logger.info(f"pipelines is up to date. Installed version: {__installed_version__}. Latest version: {latest_version}")
+    return True
+
+
+def get_latest_version() -> str:
+    path_to_pyproject_toml = LOCAL_PIPELINE_PACKAGE_PATH + "pyproject.toml"
+    with open(path_to_pyproject_toml, "r") as f:
+        for line in f.readlines():
+            if "version" in line:
+                return line.split("=")[1].strip().replace('"', "")
+    raise Exception("Could not find version in pyproject.toml. Please ensure you are running from the root of the airbyte repo.")
+
 
 def get_modified_files(
     git_branch: str, git_revision: str, diffed_branch: str, is_local: bool, ci_context: CIContext, pull_request: PullRequest
@@ -55,6 +84,7 @@ def get_modified_files(
 
 
 @click.group(help="Airbyte CI top-level command group.")
+@click.version_option(__installed_version__)
 @click.option("--is-local/--is-ci", default=True)
 @click.option("--git-branch", default=get_current_git_branch, envvar="CI_GIT_BRANCH")
 @click.option("--git-revision", default=get_current_git_revision, envvar="CI_GIT_REVISION")
@@ -99,6 +129,7 @@ def airbyte_ci(
     show_dagger_logs: bool,
 ):  # noqa D103
     ctx.ensure_object(dict)
+    check_up_to_date()
     ctx.obj["is_local"] = is_local
     ctx.obj["is_ci"] = not is_local
     ctx.obj["git_branch"] = git_branch

airbyte-ci/connectors/pipelines/pipelines/consts.py

@@ -32,6 +32,7 @@ GRADLE_CACHE_PATH = "/root/.gradle/caches"
 GRADLE_BUILD_CACHE_PATH = f"{GRADLE_CACHE_PATH}/build-cache-1"
 GRADLE_READ_ONLY_DEPENDENCY_CACHE_PATH = "/root/gradle_dependency_cache"
 LOCAL_REPORTS_PATH_ROOT = "airbyte-ci/connectors/pipelines/pipeline_reports/"
+LOCAL_PIPELINE_PACKAGE_PATH = "airbyte-ci/connectors/pipelines/"
 DOCS_DIRECTORY_ROOT_PATH = "docs/"
 GCS_PUBLIC_DOMAIN = "https://storage.cloud.google.com"
 DOCKER_HOST_NAME = "global-docker-host"

airbyte-ci/connectors/pipelines/pipelines/dagger_run.py

@@ -22,7 +22,7 @@ DAGGER_CLOUD_TOKEN_ENV_VAR_NAME_VALUE = (
     "_EXPERIMENTAL_DAGGER_CLOUD_TOKEN",
     "p.eyJ1IjogIjFiZjEwMmRjLWYyZmQtNDVhNi1iNzM1LTgxNzI1NGFkZDU2ZiIsICJpZCI6ICJlNjk3YzZiYy0yMDhiLTRlMTktODBjZC0yNjIyNGI3ZDBjMDEifQ.hT6eMOYt3KZgNoVGNYI3_v4CC-s19z8uQsBkGrBhU3k",
 )
-ARGS_DISABLING_TUI = ["--no-tui", "publish"]
+ARGS_DISABLING_TUI = ["--no-tui", "publish", "--version"]
 
 
 def get_dagger_path() -> Optional[str]:
@@ -97,6 +97,7 @@ def main():
         command = [dagger_path, "run", "airbyte-ci-internal"] + sys.argv[1:]
     try:
         try:
+            LOGGER.info(f"Running command: {command}")
             subprocess.run(command, check=True)
         except KeyboardInterrupt:
             LOGGER.info("Keyboard interrupt detected. Exiting...")

airbyte-ci/connectors/pipelines/pyproject.toml

@@ -4,12 +4,12 @@ build-backend = "poetry.core.masonry.api"
 
 [tool.poetry]
 name = "pipelines"
-version = "1.4.2"
+version = "1.4.3"
 description = "Packaged maintained by the connector operations team to perform CI for connectors' pipelines"
 authors = ["Airbyte <contact@airbyte.io>"]
 
 [tool.poetry.dependencies]
-python = "^3.10"
+python = "~3.10"
 dagger-io = "^0.6.4"
 asyncer = "^0.0.2"
 anyio = "^3.4.1"
@@ -19,7 +19,7 @@ semver = "^3.0.1"
 airbyte-protocol-models = "*"
 tabulate = "^0.8.9"
 jinja2 = "^3.0.2"
-requests = "^2.28.2"
+requests = "2.28.2" # Pinned as the requests 2.29.0 version is not compatible with the docker package
 connector-ops = {path = "../connector_ops", develop = true}
 toml = "^0.10.2"
 sentry-sdk = "^1.28.1"

airbyte-integrations/bases/connector-acceptance-test/README.md

@@ -37,7 +37,7 @@ To learn how to set up `ci_credentials` and your GSM Service account see [here](
 export GCP_GSM_CREDENTIALS=`cat <path-to-gsm-service-account-key-file>`
 
 # Install the credentials tool
-pipx install airbyte-ci/connectors/ci_credentials/
+pipx install airbyte-ci/connectors/ci_credentials/ --force --editable
 ```
 
 **Retrieve a connectors sandbox secrets**

octavia-cli/.python-version

@@ -1 +1 @@
-3.9.11
+3.9