1
0
mirror of synced 2025-12-26 14:02:10 -05:00
Commit Graph

22176 Commits

Author SHA1 Message Date
Airbyte
25802b6c4e 🐙 source-workflowmax: run up-to-date pipeline [2024-10-28] (#47527) 2024-10-28 16:10:39 +02:00
Airbyte
4a8e309850 🐙 source-pypi: run up-to-date pipeline [2024-10-28] (#47528) 2024-10-28 16:10:36 +02:00
Airbyte
fa1076128a 🐙 source-metabase: run up-to-date pipeline [2024-10-28] (#47531) 2024-10-28 16:10:29 +02:00
Airbyte
6b67aa40e5 🐙 source-goldcast: run up-to-date pipeline [2024-10-28] (#47533) 2024-10-28 16:10:24 +02:00
Airbyte
94ceec9dab 🐙 source-invoiced: run up-to-date pipeline [2024-10-28] (#47534) 2024-10-28 16:10:19 +02:00
Airbyte
e4576328ed 🐙 source-ezofficeinventory: run up-to-date pipeline [2024-10-28] (#47535) 2024-10-28 16:10:14 +02:00
Airbyte
cbc0b1b990 🐙 source-pennylane: run up-to-date pipeline [2024-10-28] (#47536) 2024-10-28 16:10:10 +02:00
Airbyte
8dd42bda24 🐙 source-intercom: run up-to-date pipeline [2024-10-28] (#47537) 2024-10-28 16:10:06 +02:00
Airbyte
d690a35645 🐙 source-mention: run up-to-date pipeline [2024-10-28] (#47538) 2024-10-28 16:10:01 +02:00
Airbyte
9396fe2fff 🐙 source-postmarkapp: run up-to-date pipeline [2024-10-28] (#47539) 2024-10-28 16:09:57 +02:00
Airbyte
5a8147a45b 🐙 source-teamtailor: run up-to-date pipeline [2024-10-28] (#47540) 2024-10-28 16:09:54 +02:00
Airbyte
a6d96e61fc 🐙 source-reddit: run up-to-date pipeline [2024-10-28] (#47542) 2024-10-28 16:09:50 +02:00
Airbyte
416f7ace92 🐙 source-visma-economic: run up-to-date pipeline [2024-10-28] (#47543) 2024-10-28 16:09:47 +02:00
Airbyte
87aa2311cc 🐙 source-microsoft-lists: run up-to-date pipeline [2024-10-28] (#47544) 2024-10-28 16:08:43 +02:00
tolik0
97514e6b36 🤖 Cut version 5.17.0 of source-declarative-manifest 2024-10-28 12:48:34 +00:00
Airbyte
141b27ffaf 🐙 source-breezometer: run up-to-date pipeline [2024-10-28] (#43777) 2024-10-28 14:15:00 +02:00
Airbyte
fa8086eb63 🐙 destination-firebolt: run up-to-date pipeline [2024-10-28] (#47100) 2024-10-28 14:13:48 +02:00
Airbyte
04e5eff20e 🐙 source-us-census: run up-to-date pipeline [2024-10-28] (#47119) 2024-10-28 14:13:37 +02:00
Airbyte
959e0fa40e 🐙 source-sonar-cloud: run up-to-date pipeline [2024-10-28] (#47469) 2024-10-28 14:13:30 +02:00
Airbyte
c2d32e774e 🐙 source-ip2whois: run up-to-date pipeline [2024-10-28] (#47471) 2024-10-28 14:13:24 +02:00
Airbyte
fc9389af58 🐙 source-high-level: run up-to-date pipeline [2024-10-28] (#47472) 2024-10-28 14:13:21 +02:00
Airbyte
8369523109 🐙 source-statsig: run up-to-date pipeline [2024-10-28] (#47473) 2024-10-28 14:13:17 +02:00
Airbyte
c21ce154ad 🐙 source-microsoft-entra-id: run up-to-date pipeline [2024-10-28] (#47479) 2024-10-28 14:13:13 +02:00
Airbyte
fe2799ffb7 🐙 source-iterable: run up-to-date pipeline [2024-10-28] (#47487) 2024-10-28 14:13:09 +02:00
Airbyte
38768790dc 🐙 source-veeqo: run up-to-date pipeline [2024-10-28] (#47488) 2024-10-28 14:13:03 +02:00
Airbyte
42a935ab76 🐙 source-mailjet-sms: run up-to-date pipeline [2024-10-28] (#47489) 2024-10-28 14:12:59 +02:00
Airbyte
3968ac5082 🐙 source-qonto: run up-to-date pipeline [2024-10-28] (#47490) 2024-10-28 14:12:56 +02:00
Airbyte
8690f5e8d2 🐙 source-nasa: run up-to-date pipeline [2024-10-28] (#47491) 2024-10-28 14:12:52 +02:00
Airbyte
702fe5982a 🐙 source-mux: run up-to-date pipeline [2024-10-28] (#47492) 2024-10-28 14:12:49 +02:00
Airbyte
620492d3e1 🐙 source-survicate: run up-to-date pipeline [2024-10-28] (#47494) 2024-10-28 14:12:45 +02:00
Airbyte
46c16cc6e0 🐙 source-zendesk-sell: run up-to-date pipeline [2024-10-28] (#47495) 2024-10-28 14:12:42 +02:00
Airbyte
112cf2d123 🐙 source-polygon-stock-api: run up-to-date pipeline [2024-10-28] (#47496) 2024-10-28 14:12:39 +02:00
Airbyte
bd191b244c 🐙 source-yahoo-finance-price: run up-to-date pipeline [2024-10-28] (#47497) 2024-10-28 14:12:35 +02:00
Airbyte
1e2c8354a9 🐙 source-persona: run up-to-date pipeline [2024-10-28] (#47498) 2024-10-28 14:12:32 +02:00
Airbyte
2e130ac16b 🐙 source-gnews: run up-to-date pipeline [2024-10-28] (#47499) 2024-10-28 14:12:29 +02:00
Airbyte
7de53bfa16 🐙 source-uservoice: run up-to-date pipeline [2024-10-28] (#47500) 2024-10-28 14:12:25 +02:00
tolik0
81ce6f234b 🤖 minor bump Python CDK to version 5.17.0 2024-10-28 12:06:44 +00:00
Anatolii Yatsuk
fb769bd0a0 feat(airbyte-cdk): Add Per Partition with Global fallback Cursor (#45125) 2024-10-28 14:03:41 +02:00
Augustin
917ddae846 airbyte-ci: rework VersionIncrementCheck for RC (#47386) 2024-10-28 12:24:44 +01:00
Augustin
ecf09cc30e airbyte-ci: build manifest only connectors correctly in up-to-date (#47483)
## What

* Problems: We saw build failure at `up-to-date` time, which let to unhandled errors.
* Solution: Manifest only connector build was not wired in the up-to-date pipeline, we used the python connector build logic on these connectors which led to these execeptions.
 
This pull request includes several changes aimed at improving the build logic and updating dependencies for the Airbyte CI connectors. The most important changes involve fixing the build logic to support any connector language, refactoring the import statements, and updating the pipeline version.

### Improvements to build logic:

* [`airbyte-ci/connectors/pipelines/README.md`](diffhunk://#diff-62eccd92928fbcd3d285983bfdaa2b0d4ca49016cb9c2f63d6d9fc968c59c541R853): Added a new entry to document the fix for build logic used in `up-to-date` to support any connector language.

### Refactoring:

* [`airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/up_to_date/pipeline.py`](diffhunk://#diff-cef433180658cd9d23043da2e4577f65c9bfb7bca5e9bacca87e3b24ad3e56bbL11-R11): Refactored import statements to use `run_connector_build` instead of `BuildConnectorImages`.
* [`airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/up_to_date/pipeline.py`](diffhunk://#diff-cef433180658cd9d23043da2e4577f65c9bfb7bca5e9bacca87e3b24ad3e56bbL129-R129): Updated the `run_connector_up_to_date_pipeline` function to use `run_connector_build` for building connector images.

### Dependency updates:

* [`airbyte-ci/connectors/pipelines/pyproject.toml`](diffhunk://#diff-087e2c37602bbd6824f875004abddcb4e1a374da12bf84201671ed0900882ce0L7-R7): Updated the version from `4.41.8` to `4.41.9`.
2024-10-28 12:02:40 +01:00
Augustin
b4496f07c7 up-to-date/base-image: cache dockerhub image listing (#47447)
## What

* Problems:`up-to-date`: to find the latest base image we have to list all the base images available and fetch their digest. If this operation is not cached it can easily lead to DockerHub rate limiting issue. 
* Solution: Add a `cache_ttl_seconds` parameters to cache the interaction with the `crane` client. In `up-to-date` we set this cache TTL to 1 day.

This pull request introduces a caching mechanism to avoid DockerHub rate limiting and updates various files to support this new feature. The most important changes include adding a cache TTL for base image listings, updating the `CraneClient` and related methods to accept a cache TTL, and adding logging for digest fetching.

### Caching Mechanism:

* [`airbyte-ci/connectors/base_images/README.md`](diffhunk://#diff-2e343fda68ecd85f5023c98f7f9a41af2125e7e1cc12d545bc13ace788a4d075R97-R99): Added a changelog entry for version 1.1.0, mentioning the cache TTL addition.
* [`airbyte-ci/connectors/base_images/base_images/utils/docker.py`](diffhunk://#diff-d92863e5bd18b276dba43c00c727c4f54d65193740f2044c7929948652e2d1b3L38-R48): Modified the `CraneClient` class to accept a `cache_ttl_seconds` parameter and updated the `bare_container` initialization to use this TTL.
* [`airbyte-ci/connectors/base_images/base_images/version_registry.py`](diffhunk://#diff-2b84383522cacca9880ff1dae76f40d08ec437c59208cd856e41f63971f4340dL108-R140): Updated methods to pass the `cache_ttl_seconds` parameter to the `CraneClient`. [[1]](diffhunk://#diff-2b84383522cacca9880ff1dae76f40d08ec437c59208cd856e41f63971f4340dL108-R140) [[2]](diffhunk://#diff-2b84383522cacca9880ff1dae76f40d08ec437c59208cd856e41f63971f4340dL144-R152) [[3]](diffhunk://#diff-2b84383522cacca9880ff1dae76f40d08ec437c59208cd856e41f63971f4340dL249-R274) [[4]](diffhunk://#diff-2b84383522cacca9880ff1dae76f40d08ec437c59208cd856e41f63971f4340dL267-R293)
* [`airbyte-ci/connectors/pipelines/README.md`](diffhunk://#diff-62eccd92928fbcd3d285983bfdaa2b0d4ca49016cb9c2f63d6d9fc968c59c541R853): Documented the use of `cache_ttl` for base image registry listing in version 4.41.8.
* [`airbyte-ci/connectors/pipelines/pyproject.toml`](diffhunk://#diff-087e2c37602bbd6824f875004abddcb4e1a374da12bf84201671ed0900882ce0L7-R7): Bumped the version to 4.41.8.

### Logging Enhancements:

* [`airbyte-ci/connectors/base_images/base_images/utils/docker.py`](diffhunk://#diff-d92863e5bd18b276dba43c00c727c4f54d65193740f2044c7929948652e2d1b3R90-R95): Added logging statements for digest fetching to help identify bottlenecks and improve debugging.
2024-10-28 11:58:39 +01:00
Augustin
6988679424 up-to-date: fix quote escape in workflow (#47444)
## Motivation
The single quote escaping syntax  used in GHA workflow to run `airbyte-ci` was wrong.
Separating `schedule` triggers from `workflow_dispatch` triggers makes it easier to get the syntax right.

### Workflow and Command Updates:

* [`.github/workflows/connectors_up_to_date.yml`](diffhunk://#diff-2d17c6d9c0d58a040bfab9725e36051d4330b78f1a76e2f9fb024f673b7a74d4L14-R14): Added a new metadata query to exclude `source-declarative-manifest` from the `connectors-options` input. Updated the workflow to handle both `workflow_dispatch` and `schedule` events separately. [[1]](diffhunk://#diff-2d17c6d9c0d58a040bfab9725e36051d4330b78f1a76e2f9fb024f673b7a74d4L14-R14) [[2]](diffhunk://#diff-2d17c6d9c0d58a040bfab9725e36051d4330b78f1a76e2f9fb024f673b7a74d4L28-R30) [[3]](diffhunk://#diff-2d17c6d9c0d58a040bfab9725e36051d4330b78f1a76e2f9fb024f673b7a74d4L42-R59)

### Documentation Changes:

* [`airbyte-ci/connectors/pipelines/README.md`](diffhunk://#diff-62eccd92928fbcd3d285983bfdaa2b0d4ca49016cb9c2f63d6d9fc968c59c541L528): Removed the `--ignore-connector` option from the documentation and added an entry to the changelog explaining the removal. [[1]](diffhunk://#diff-62eccd92928fbcd3d285983bfdaa2b0d4ca49016cb9c2f63d6d9fc968c59c541L528) [[2]](diffhunk://#diff-62eccd92928fbcd3d285983bfdaa2b0d4ca49016cb9c2f63d6d9fc968c59c541R853)

### Codebase Simplification:

* [`airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/up_to_date/commands.py`](diffhunk://#diff-e6ca381b7841536a83b88d6b45f24bbcdbe304f88248787054817373d8f73544L52-L57): Removed the `--ignore-connector` option and its related logic from the `up_to_date` command. [[1]](diffhunk://#diff-e6ca381b7841536a83b88d6b45f24bbcdbe304f88248787054817373d8f73544L52-L57) [[2]](diffhunk://#diff-e6ca381b7841536a83b88d6b45f24bbcdbe304f88248787054817373d8f73544L66-L75)

### Version Bump:

* [`airbyte-ci/connectors/pipelines/pyproject.toml`](diffhunk://#diff-087e2c37602bbd6824f875004abddcb4e1a374da12bf84201671ed0900882ce0L7-R7): Updated the version from `4.41.6` to `4.41.7`.
2024-10-28 11:55:02 +01:00
Airbyte
25018a86cb 🐙 source-pocket: run up-to-date pipeline [2024-10-28] (#47034) 2024-10-28 12:09:34 +02:00
Airbyte
44a78b18c5 🐙 destination-qdrant: run up-to-date pipeline [2024-10-28] (#47054) 2024-10-28 12:09:24 +02:00
Airbyte
f86967eaac 🐙 source-microsoft-onedrive: run up-to-date pipeline [2024-10-28] (#47060) 2024-10-28 12:09:19 +02:00
Airbyte
f3bb65396d 🐙 source-bigcommerce: run up-to-date pipeline [2024-10-28] (#47117) 2024-10-28 12:08:56 +02:00
Airbyte
9a002be9b5 🐙 source-drip: run up-to-date pipeline [2024-10-28] (#47446) 2024-10-28 12:08:51 +02:00
Airbyte
6091e87bbd 🐙 source-freshservice: run up-to-date pipeline [2024-10-28] (#47449) 2024-10-28 12:08:47 +02:00
Airbyte
0ff44429ef 🐙 source-emailoctopus: run up-to-date pipeline [2024-10-28] (#47450) 2024-10-28 12:08:44 +02:00
Airbyte
d5d941330d 🐙 source-datascope: run up-to-date pipeline [2024-10-28] (#47451) 2024-10-28 12:08:41 +02:00