jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-19 18:14:56 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

806 Commits

Author SHA1 Message Date
Ben Church
bdf8eed759 Add dev job to regenerate catalog from metadata and view diff (#24545) 
* Create a diff between the catalogs

* Diff catalogs

* Move diff to dev

* Run black

* Add typing

* move all unnessesary postfix

* Apply all PR comments

* Add partial metadata model

* Add OutputDataFrame type

* Format code

* Handle small PR comments

* Run black

* Fix File write

* Fix action
 2023-04-05 11:16:41 -07:00
Ben Church
1245947189 Generate metadata files from catalog (#24336) 
* Get list of source controlled github assets

* Get list of which connectors are not source controlled

* Check spec caching

* Add skeleton for conversion

* Update to use apply

* validate metadata properly

* Remove type defs for now

* Add tmp file manager

* Get file write working

* Filter out null metadata values

* Clean up code

* rename v0 to v1

* Add overrides

* Add missing release stages

* Clean up todos

* Rename sourceType to connectionType

* Ensure we are doing overrides in a deep fashion

* Move spec to service

* Add comments

* Fix tests

* Move is_spec_cached to a list

* Update tests

* Add metadata tests

* Use pydash

* run flake and black

* Hoist fixtures to shared file

* Add comment

* rename connectiontype to connectorSubType
 2023-03-29 15:35:14 -07:00
Artem Inzhyyants
6f98a68e28 Source Yandex Metrica: certify to beta (#24188) 
* Source Yandex: http -> https

* Source Yandex: fix schema

* Source Yandex: bump to latest version

* Source Yandex: Ref Clean Job

* Source Yandex: Ref

* Source Yandex: update schemas; ref

* Source Yandex: refactor

* Source Yandex: refactor; add tests

* Source Yandex: bump version; update setup; update docs

* Source Yandex Metrica: fix tests

* Source Yandex Metrica: add allowed hosts; change releaseStage

* Source Yandex Metrica: fix docs

* Source Yandex Metrica: fix docs

* Automated Change

* Source Yandex Metrica: update docs

* Source Yandex Metrica: update docs

* Source Yandex Metrica: add expected_records for views stream

* auto-bump connector version

* Automated Commit - Formatting Changes

* Source Yandex Metrica: empty commit

---------

Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2023-03-20 21:26:21 +01:00
Ben Church
2d6f5ee88a Dagster PoC for Metadata service (#23989) 
* Add airbyte-ci folders

* Add poetry

* Add first dagster job

* Get sensors properly registering

* Trigger job on new files

* Add etag cursor

* Wire up resources and ops

* Parse destinations dataframe

* Add multiple dataframes

* Compute markdown

* Write html and markdown

* Move columns to variable

* move to a folder centered file structure

* Move to sensor factory

* Add resource def to sensor

* Use appropriate credentials

* Use GCSFileManager

* use catalog_directory_manager

* Generalize the gcp catalog resources

* Move bucket to env var

* Clean up and add comments

* Update readme

* Move dependencies into orchestrator

* Add gcs section to readme

* Clean up debug

* Add merge catalog tests

* Run code formatter

* Apply flake8 fixes

* Remove suffix

* Move tests up one level

* Folder rename

* Update readme and rename env

* Add jinja templates

* Rename connectors_ci to connectors for lib
 2023-03-20 12:28:10 -07:00
Benoit Moriceau
5cc7055cb5 Update sentry heartbeat timeout treshold (#24208) 
* Update sentry heartbeat timeout treshold

* Automated Commit - Formatting Changes

---------

Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
 2023-03-17 17:38:35 -07:00
Ben Church
1ed9adda05 Add metadata validator (#24198) 
* Add begining of lib folder

* Generate models from poetry command

* Run the validation script

* Add the catalog overrides type

* Add test for valid metadata files

* Add error state tests

* Expand valid and invalid test cases

* Update readme

* Run formatter

* Delete remaining catalogs
 2023-03-17 11:33:54 -07:00