1
0
mirror of synced 2025-12-30 12:04:43 -05:00
Commit Graph

83 Commits

Author SHA1 Message Date
Ben Church
5c9f0fa34e Dagster: Add a schedule and auto materialization to handle resource limits (#28190)
* Initial schedule

* Add dynamic partition

* Add auto materialization policy

* Add partition add job

* Add logging and format

* Add comment

* Add MAX_METADATA_PARTITION_RUN_REQUEST to config
2023-07-26 15:36:53 -07:00
Pedro S. Lopez
f1ffa2e148 rename method (#28741) 2023-07-26 14:31:08 -04:00
Pedro S. Lopez
f753dd04b6 fix: publish strict-encrypt connectors before regular variant (#28737)
* sort strict encrypt specially

* add secure variant handling
2023-07-26 13:10:43 -04:00
Ben Church
fb7258e2bd Move tools/ci_* projects to airbyte-ci, update to use Poetry, bump to python 3.10 (#27957)
* Move ci_connector_ops

* Move ci_credentials

* Move tools/ci_common_utils

* Rename tools to airbyte-ci

* Move to ci

* Convert ci_credentials

* Convert ci_common_utls

* Convert ci_connector_ops

* Get pipelines running

* Move pipelines to own poetry project

* Update readme

* Delete

* Add ci_code_validator

* Use pipx to install gha deps

* Fix'

* Ensure every thing is running

* Automated Commit - Formatting Changes

* Gitignore miss

* Add pipx installer

* Get local pipx dependencies

* Fix paths

* Install pipx

* ceremonial source-faker change

* Add installation step for ci_code_validator

* Add comment

* remove ci_code_validator

* Address code review comments

* add pipx install to acceptance-test-docker.sh

* Run formater

* Revert "ceremonial source-faker change"

This reverts commit 26884cd0db.

* gitignore lecacy pipeline report path

* update poetry.lock

* skip upload if logs do not exist

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
2023-07-26 15:49:59 +00:00
Ben Church
3404878a41 Update all metadata sensor from 10 hours to 10 min (#28717) 2023-07-25 16:02:08 -07:00
Ben Church
e53034921e Metadata: Ensure we run upload validate on the overrode metadata file (#28583)
* Ensure we do an upload validate on the overrode metadata file

* Change return value of upload

* Format
2023-07-24 15:35:46 -06:00
Ella Rohm-Ensing
f5056232e7 handle case where migrationDocumentationUrl is None (by default) (#28575)
* handle case where migrationDocumentationUrl is None (by default)

* change test data to reflect state of objects before the method
2023-07-21 12:54:45 -05:00
Ben Church
731097f86d Publish metadata on prerelease (#28067)
* First pass

* Get prerelease flag working

* Exclude none

* Move to json santized dict

* Add prepublish upload functionality

* Remove prerelease validation exception

* Format

* Fix optional arg

* Add tests

* Apply suggestions from code review

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* disable cache for all

* test main upload is called

* Enforce requirements

* Remove redundant docker_image_name

* Format and fix test

---------

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
2023-07-20 22:47:01 +00:00
Ben Church
1f0c817af4 Ensure you cannot add extra metadata fields outside of definition (#28544) 2023-07-20 14:27:14 -07:00
Edward Gao
b628878121 remove connectors ci checklist item (#28521) 2023-07-20 13:04:53 -07:00
Ben Church
a258873bed Parse registry entry as json (#28501) 2023-07-19 21:12:56 -05:00
Ella Rohm-Ensing
16be3b4075 improve DX of releasing breaking changes (docs, logs, bugfix) (#28486)
* cherry-pick me: add more logs to migration guide check

* cherry-pick me: migration guide should be named according to name in metadata

* cherry-pick me: change logging for expected headers

* cherry-pick me: include migration guide file path in qa checks step

* add breaking changes section to docs, update link to breaking changes section

* update link to semantic versioning, add point about migration guide

* drop other checklist link

* add to docs checklist

* update tests to make all 'test_qa_checks' tests pass locally. refactor to make them easier to understand (for me at least)

* add test files

* remove accidental file
2023-07-19 22:30:08 +00:00
Alexandre Girard
5ca1b41eb7 Move pyarrow to CDK extra (#28413)
* move pyarrow to extra

* Automated Commit - Formatting Changes

* remove parquet tests

* delete the import

* missing space

* Automated Commit - Formatting Changes

* comment parquet_parser too

* optimize imports

* comment out temporary file source

* add pyarrow to dev extra

* reset files

* share pyarrow dependency

* use alpine for declarative_source

* Automated Commit - Formatting Changes

* Revert "use alpine for declarative_source"

This reverts commit a3ad47ccca.

* pin cdk version

* reset the cdk version

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
2023-07-18 16:20:53 -05:00
Ella Rohm-Ensing
94574055e0 metadata validate: check that metadata with N.0.0 dockerImageTag contains relevant breakingChange entry for that version (#27785)
* re-add validation and test cases

* clean up test data

* add valid test case

* run valiation and make sure to return true if all good

* add exceptions

* fix name

* use semver library

* update exceptions

* changes to contribution docs

* update ignores
2023-07-18 15:51:49 -04:00
Ben Church
1c871756b3 Allow empty reg in registry entry generation (#28307)
* Handle empty or missing registries

* Update scaffolds to include cloud reg
2023-07-14 15:26:04 -07:00
Ben Church
679d475cae Break resources into dependency trees (#28166) 2023-07-11 14:25:13 -07:00
sh4sh
99cace9cab connector checklist: update pr naming conventions link (#28088) 2023-07-10 14:23:48 -04:00
Ben Church
6fd7a34a50 Trigger generate registry only on latest blobs (#28048) 2023-07-07 10:24:41 -05:00
Ben Church
6a0aec7f39 Fix Dagster Deploy Failure (#27955)
* Add pydantic

* Add pydantic to orchestration deploy pipeline
2023-07-04 18:14:56 -05:00
Ben Church
57945440cd Metadata: Persist Registry entries (#27766)
* DNC

* Update poetry

* Update dagster

* Apply partition

* Get metadata entry

* Use helpers

* Write registry entry to appropriate location

* Delete when registry removed

* Update to use new file (broken)

* Render registry from registry entries

* Run format

* Fix plural issue

* Update to all metadata file blobs

* Fix test

* Update to all blobs

* Add ignore validation error for version logic

* Rename to max_run_request

* Pedros review

* Ella suggestions

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* Update airbyte-ci/connectors/metadata_service/orchestrator/orchestrator/assets/registry_entry.py

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* Update naming

* Add tests for connector type and deletion

* Test safe parse

* Format

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
2023-07-04 10:41:24 -07:00
Augustin
f1f088d3c6 orchestrator: add html report url to connector health messages (#27665) 2023-06-24 11:09:26 +02:00
Ella Rohm-Ensing
44a01bd95d run formatter which wasn't run on pr to fix build (#27676)
* fix format issues

* run formatter
2023-06-23 16:36:07 -05:00
Ella Rohm-Ensing
beb0ed726a Validate that images referenced in breakingChanges exist on dockerhub (#27451)
* Add check that docker images in breakingChanges exist

* Add invalid test

* pull changes back together

* put em together

* Assert that the ValueError is a validation error

* make way too many changes at once lol

* fix typo

* update method description

* Run tests on all valid upload files, fix jsonschema issues

* Change mock logic

* don't check base:base image

* fix existing images and add new test cases

* reset mocks per test case

* attempt to improve renaming diff

* explicity check the base combos

* move override files - they're invalid if the base image doesn't exist

* make it clearer when a tag won't exist

Co-authored-by: Ben Church <ben@airbyte.io>

* named constant for versions that don't exist

---------

Co-authored-by: Ben Church <ben@airbyte.io>
2023-06-23 17:00:15 -04:00
Ella Rohm-Ensing
6c62f382e1 airbyte-ci: fix upload invalid metadata validation and testing (#27642)
* Assert that the ValueError is a validation error

* make way too many changes at once lol

* fix typo

* update method description

* generate it correctly

* generate other poetry lockfiles on 1.5.1

* Make validators non optional, also make a type for validator functions
2023-06-23 16:42:47 -04:00
Ben Church
14c57d79d3 Update to allow for extra fields (#27597)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-23 19:13:06 +02:00
Ella Rohm-Ensing
12c7da7f11 fix docker auth for metadata tests and orchestrator deploy (#27655)
* test change in metadata library to trigger tests

* add credentials when calling dagger from metadata_service_tests_dagger.yml

* Fix env for metadata_service_deploy_orchestrator_dagger.yml

* revert adding metadata api token input

* reorder inputs by how they are listed in workflow

* add context for orchestrator deploy
2023-06-23 14:57:55 +00:00
Octavia Squidington III
bad8157c44 🤖 Cloud Availability updater: new connectors to deploy [20230621] (#27542) 2023-06-21 10:26:00 +02:00
Ben Church
38a189e829 Breaking Changes: QA migration check (#27294)
* Add migration guide check

* Add test todo

* migration_guide to -migrations

* Update tests

* Valid tests

* Add tests

* Update tools/ci_connector_ops/ci_connector_ops/utils.py

Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>

* Ensure that the migration path is checked

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
2023-06-20 09:30:23 -07:00
Augustin
b0a63707f9 update connector checklist following CI changes (#27372) 2023-06-14 22:10:36 +02:00
Ben Church
59dbc609e0 Handle the case of no metadata on master for new connectors (#27377)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-14 15:01:22 -05:00
Ben Church
9dc8bb7a19 Update documentation for support url to documentation url (#27375)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-14 14:24:43 -05:00
Ben Church
68c0d6dbbc Connector CI: update badges test summary (#27215)
* Rename to connector test report

* Get all test outputs

* Get last 10 output for all connectors

* Get runs by connector

* Crudely write badge and summary

* Add render functions

* Update connector report badge and link

* Fix url link

* Fix qa report tests

* update qa-engine tests

* Move regex to resource

* Add cron

* Run format

* New line

* Make badge green if last is successful

* Fix test

* Remove report.sh and build_report.py

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-14 11:12:47 -07:00
Denys Davydov
6581b6992a 🐛 Source Snapchat Marketing: remove deprecated auth specification (#26358)
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* #26246 #26247 source Square, source Snapchat-Marketing: remove deprecated authSpecification in favour of advancedAuth

* #26246, #26247: upd changelogs

* Automated Change

* Automated Change

* revert square changes

* Automated Change

* Automated Commit - Formatting Changes

* Automated Commit - Format and Process Resources Changes

---------

Co-authored-by: davydov-d <davydov-d@users.noreply.github.com>
2023-06-13 13:48:36 +03:00
Ben Church
a816d39223 Add icon docs (#27295)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-12 20:51:28 -05:00
Ben Church
f8b9ee2289 Metadata: Add Breaking Changes entry (#27178)
* Add breaking changes metadata

* Add valid breaking change

* Add invalid cases

* Update comment

* Add invalid deadline check

* Convert to releases

* Update to releases

* Test for migration doc url

* Add registry override application

* Add even more tests

* Add default tests

* Apply format

* Add releases to registry model

* Add new test cases

* Set additional properties to false

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-12 18:54:10 -04:00
Ben Church
61a6f8ed68 Remove debug test (#27167)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-08 13:02:30 -05:00
Ben Church
1315f5a768 Nightly Reports: Generate markdown slack report from new nightlies (#26990)
* Add github asset

* reduce columns in asset

* Add sensor

* remove sensor calls

* Remove bucket manager

* Get nightly resources

* Add nightly report data

* TODO

* Generate usable data frame

* Add template

* Add all data to report

* Generalize gcs blob sensor

* Remove old metadata sensor

* Send report to slack

* Complete report

* Remove overly verbose comments

* Add generics

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-08 10:45:13 -07:00
Ben Church
cae63c0ac1 Metadata Upload Validation: Update to use registry url (#27047)
* Update to use registry url

* Update airbyte-ci/connectors/metadata_service/lib/metadata_service/docker_hub.py

Co-authored-by: Augustin <augustin@airbyte.io>

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
2023-06-06 14:16:57 -04:00
Ben Church
0a995bf9d6 disbale dockerhub (#26977)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-02 13:18:48 -05:00
Ben Church
623391bfa3 Update docs (#26935)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-06-02 09:54:48 -07:00
Augustin
f1476c06a8 qa-engine: remove sensitive columns from created PR (#26859) 2023-05-31 11:16:02 -07:00
Octavia Squidington III
1766460279 🤖 Cloud Availability updater: new connectors to deploy [20230531] (#26818) 2023-05-31 19:46:13 +02:00
Ben Church
5e0e317eac Move qa engine to aibyte-ci (#26377)
* Add change notes

* Add qa-engine poetry

* Port qa_engine code

* Add missing fixture

* Update import

* Update git code

* Delete old qa-engine files

* convert tests

* Add path

* Fix mocks

* Fix test_enable_in_cloud

* Fix metadata path test

* Run formater

* Automated Commit - Formatting Changes

* Update readme

* Remove duplicates

* Remove redundant if

* Add ruamel

* Use ruamel

* Remove safe load

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-05-31 05:26:28 +00:00
Ben Church
b083cde1ba add documentationUrl and mark required (#26813)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-05-30 20:17:14 -05:00
Ben Church
c4282942cb Publish destination-vertica: change metadata extension (#26770)
* Rename .yml to .yaml

* Add language tags

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-05-30 11:04:30 -05:00
Ben Church
1dabc6208e Metadata: add tags field (#26320)
* Add optional tags field

* Remove duplicate icons

* Add programming tags to all

* Update docs

* supportUrl -> documentationUrl

* Ensure one language tag is applied

* Add keyvalue check

* rebase and fix tests

* Format

* Add cache buster

* Improve test

* Automated Commit - Formatting Changes

* Update error

* Fix missing tags

* Fix scaffold

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-05-26 16:13:09 -07:00
Ben Church
dac35afdac Update icon urls (#26640)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
2023-05-26 09:38:15 -07:00
Ben Church
248bbf94c1 HACKDAY: Icon CDN (#26158)
* Move icons to connector folder

* Delete old icons

* Update upload logic

* Add icon url to definitions

* Update registry model

* Populate cdn url

* DNC butcher the pipeline

* Low hanging fruit fixes

* Fix bucket name

* Merge old and new approaches

* Fix metadata upload step

* Format

* Fix test
2023-05-24 17:25:41 -07:00
Augustin
80032f73f9 connectors-ci: deprecate slash publish (#25865) 2023-05-22 10:10:56 +02:00
Ben Church
cb2ac5adbf Dagster: Remove all code used for development of new metadata files (#26322)
* Remove unused

* Refactor to use registry in sensor

* Format
2023-05-19 16:39:42 -07:00