Augustin
2af100f4e3
connector-acceptance-test: test it with airbyte-ci ( #29509 )
2023-08-17 17:59:00 -07:00
Ben Church
fb7258e2bd
Move tools/ci_* projects to airbyte-ci, update to use Poetry, bump to python 3.10 ( #27957 )
...
* Move ci_connector_ops
* Move ci_credentials
* Move tools/ci_common_utils
* Rename tools to airbyte-ci
* Move to ci
* Convert ci_credentials
* Convert ci_common_utls
* Convert ci_connector_ops
* Get pipelines running
* Move pipelines to own poetry project
* Update readme
* Delete
* Add ci_code_validator
* Use pipx to install gha deps
* Fix'
* Ensure every thing is running
* Automated Commit - Formatting Changes
* Gitignore miss
* Add pipx installer
* Get local pipx dependencies
* Fix paths
* Install pipx
* ceremonial source-faker change
* Add installation step for ci_code_validator
* Add comment
* remove ci_code_validator
* Address code review comments
* add pipx install to acceptance-test-docker.sh
* Run formater
* Revert "ceremonial source-faker change"
This reverts commit 26884cd0dbbnchrch@users.noreply.github.com >
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com >
2023-07-26 15:49:59 +00:00
Augustin
232b6272ff
connectors-ci: unique secret name at session level ( #28656 )
2023-07-25 16:31:16 +02:00
Augustin
387c651b12
connectors-ci: fix error handling ( #28562 )
2023-07-21 10:35:49 -05:00
Ben Church
731097f86d
Publish metadata on prerelease ( #28067 )
...
* First pass
* Get prerelease flag working
* Exclude none
* Move to json santized dict
* Add prepublish upload functionality
* Remove prerelease validation exception
* Format
* Fix optional arg
* Add tests
* Apply suggestions from code review
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com >
* disable cache for all
* test main upload is called
* Enforce requirements
* Remove redundant docker_image_name
* Format and fix test
---------
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com >
2023-07-20 22:47:01 +00:00
Augustin
77f147cef9
connectors-ci: centralize patches in hacks.py ( #28390 )
2023-07-20 14:31:32 +00:00
Helder Correia
f56ecaf6c4
Replace deprecated dagger APIs ( #28450 )
...
* Replace deprecated dagger APIs
Signed-off-by: Helder Correia <174525+helderco@users.noreply.github.com >
* Tidy up
Signed-off-by: Helder Correia <174525+helderco@users.noreply.github.com >
* Fix import and docstring.
Signed-off-by: Helder Correia <174525+helderco@users.noreply.github.com >
* Fix asyncer
Signed-off-by: Helder Correia <174525+helderco@users.noreply.github.com >
---------
Signed-off-by: Helder Correia <174525+helderco@users.noreply.github.com >
2023-07-19 18:04:36 -05:00
Ella Rohm-Ensing
16be3b4075
improve DX of releasing breaking changes (docs, logs, bugfix) ( #28486 )
...
* cherry-pick me: add more logs to migration guide check
* cherry-pick me: migration guide should be named according to name in metadata
* cherry-pick me: change logging for expected headers
* cherry-pick me: include migration guide file path in qa checks step
* add breaking changes section to docs, update link to breaking changes section
* update link to semantic versioning, add point about migration guide
* drop other checklist link
* add to docs checklist
* update tests to make all 'test_qa_checks' tests pass locally. refactor to make them easier to understand (for me at least)
* add test files
* remove accidental file
2023-07-19 22:30:08 +00:00
Augustin
65227f614c
connectors-ci: upgrade to Dagger 0.6.4 ( #28008 )
...
* connectors-ci: upgrade to Dagger 0.6.3
* redeclare _EXPERIMENTAL_DAGGER_RUNNER_HOST
* skip-ci: bump dagger version to 0.6.4
* remove unused dagger-cli-commit input
2023-07-19 18:24:36 -04:00
Augustin
241871fafb
connectors-ci: fix destination with normalization build ( #28464 )
2023-07-19 17:04:17 +00:00
Pedro S. Lopez
7befa93003
Stop publishing java connectors on merge to master ( #28344 )
...
* process language after modified flag
* modify some connectors
* empty commit
* another bump
* modified connectors are already filtered
* change some connectors
* change some connectors again
* revert to master version
* nit
* filter for python/lowcode connectors
2023-07-14 21:53:02 -04:00
Edward Gao
934acaa137
Destination bigquery: rerelease 1s1t behind gate ( #27936 )
...
* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268 )""
This reverts commit 348c577dbb#27981 )
* 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (#27856 )
* only run t+d code if it's enabled
* dockerfile+changelog
* remove changelog entry
* Destinations V2: handle optional fields for `object` and `array` types (#27898 )
* catch null schema
* fix null properties
* clean up
* consolidate + add more tests
* try catch
* empty json test
* Automated Commit - Formatting Changes
* remove todo
* destination bigquery: misc updates to 1s1t code (#28057 )
* switch to checkedconsumer
* add unit test for buildColumnId
* use flag
* restructure prefix check
* fix build
* more type-parsing fixes (#28100 )
* more type-parsing fixes
* handle duplicates
* Automated Commit - Format and Process Resources Changes
* add tests for asColumns
* Automated Commit - Format and Process Resources Changes
* log warnings instead of throwing exception
* better log message
* error level
---------
Co-authored-by: edgao <edgao@users.noreply.github.com >
* Automated Commit - Formatting Changes
* Improve protocol type parsing (#28126 )
* Automated Commit - Formatting Changes
* Change from T&D every 10k records to an increasing time based interval (#28130 )
* fifteen minute t&d
* add typing and deduping operation valve for increased intervals of typing and deduping
* Automated Commit - Format and Process Resources Changes
* resolve bizarre merge conflict
* Automated Commit - Format and Process Resources Changes
---------
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com >
* Simplify and speed up CDC delete support [DestinationsV2] (#28029 )
* Simplify and speed up CDC delete support [DestinationsV2]
* better QUOTE
* spotbugs?
* recompile dbt image for local arch and use that when building images
* things compile, but tests fail
* tests working-ish
* comment
* fix logic to re-insert deleted records for cursor comparison.
tests pass!
* remove comment
* Skip CDC re-include logic if there are no CDC columns
* stop hardcoding pk (#28092 )
* wip
* remove TODOs
---------
Co-authored-by: Edward Gao <edward.gao@airbyte.io >
* update method name
* Automated Commit - Formatting Changes
* depend on pinned normalization version
* implement 1s1t DATs for destination-bigquery (#27852 )
* intiial implementation
* Automated Commit - Formatting Changes
* add second sync to test
* do concurrent things
* Automated Commit - Formatting Changes
* clarify comment
* minor tweaks
* more stuff
* Automated Commit - Formatting Changes
* minor cleanup
* lots of fixes
* handle sql vs json null better
* verify extra columns
* only check deleted_at if in DEDUP mode and the column exists
* add full refresh append test case
* Automated Commit - Formatting Changes
* add tests for the remaining sync modes
* Automated Commit - Formatting Changes
* readability stuff
* Automated Commit - Formatting Changes
* add test for gcs mode
* remove static fields
* Automated Commit - Formatting Changes
* add more test cases, tweak test scaffold
* cleanup
* Automated Commit - Formatting Changes
* extract recorddiffer
* and use it in the sql generator test
* fix
* comment
* naming+comment
* one more comment
* better assert
* remove unnecessary thing
* one last thing
* Automated Commit - Formatting Changes
* enable concurrent execution on all java integration tests
* add test for default namespace
* Automated Commit - Formatting Changes
* implement a 2-stream test
* Automated Commit - Formatting Changes
* extract methods
* invert jsonNodesNotEquivalent
* Automated Commit - Formatting Changes
* fix conditional
* pull out diffSingleRecord
* Automated Commit - Formatting Changes
* handle nulls correctly
* remove raw-specific handling; break up methods
* Automated Commit - Formatting Changes
---------
Co-authored-by: edgao <edgao@users.noreply.github.com >
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com >
* Destinations V2: move create raw tables earlier (#28255 )
* move create raw tables
* better log message
* stop building normalization (#28256 )
* fix ability to run tests
* disable incremental t+d for now
* Automated Commit - Formatting Changes
---------
Co-authored-by: Evan Tahler <evan@airbyte.io >
Co-authored-by: Cynthia Yin <cynthia@airbyte.io >
Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com >
Co-authored-by: edgao <edgao@users.noreply.github.com >
Co-authored-by: Joe Bell <joseph.bell@airbyte.io >
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com >
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com >
2023-07-14 09:34:56 -05:00
Ben Church
4c6f14a016
Disable gradle build cache YOLO ( #28248 )
2023-07-12 17:39:35 -05:00
Joe Reuter
b52e88a977
🎉 New Destination Vector Database (powered by LangChain) ( #26184 )
...
* basic version
* polish
* iterate
* keep working
* fix spec
* wip
* improve destination
* basic unit tests
* move embedding dimensionality into embedder
* improve several things
* adjust documentation
* remove unnecessary call
* add some debug information
* fix local destination
* various small fixes
* bring tests into order
* document and add batching to pinecone
* checklist
* improve performance a bit and add test
* fix formatting
* fix metadata
* install C++ 11 on python base
* no more alpine for ci-connector-ops
* remove hard-to-run test
* more documentation
* better documentation
* add icon
* some small adjustments
* review comments
* format
* review comments
---------
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com >
Co-authored-by: Augustin <augustin@airbyte.io >
2023-07-12 20:06:59 +02:00
Augustin
086ddb750b
connectors-ci: revert to Dagger 0.5.4 ( #27976 )
2023-07-05 20:40:22 +02:00
Ben Church
6a0aec7f39
Fix Dagster Deploy Failure ( #27955 )
...
* Add pydantic
* Add pydantic to orchestration deploy pipeline
2023-07-04 18:14:56 -05:00
Augustin
efe21d1659
connectors-ci: better capture stderr ( #27833 )
2023-07-04 11:13:12 +02:00
Augustin
6183389254
connectors-ci: mount secrets in a late layer for CAT ( #27940 )
2023-07-04 02:32:59 -05:00
Augustin
816e83700e
connectors-ci/cat: validate image under test is the right one ( #27919 )
2023-07-03 18:19:52 +00:00
Ben Church
31c25443dd
Log connector dependencies ( #27760 )
...
* Log connector dependencies
* Update comment
* update to use path not parts
* ceremonial change to jdbc
* Add modified to test
* Revert ceremonial change
Did this using commit suggestions to keep from stopping some tests locally
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
2023-07-03 15:25:22 +00:00
Cynthia Yin
84ddfa8a5a
📚 CI credentials: fix GCP_GSM_CREDENTIALS export syntax ( #27851 )
2023-06-29 23:05:10 +00:00
Augustin
7c3111f4be
connectors-ci: use new domain for test reports ( #27826 )
2023-06-29 18:51:13 +02:00
Ben Church
95fb229ca9
Add api to dependencies ( #27834 )
...
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
2023-06-29 09:20:42 -07:00
Augustin
d9acea971e
connectors-ci: fix dagger logs url on publish ( #27830 )
2023-06-29 13:20:28 +02:00
Augustin
9b8a5a887b
connectors-ci: fix typo in duration ( #27829 )
...
* fix typo
* fix typo
2023-06-29 05:55:59 -05:00
Augustin
79987aa3ba
connectors-ci: add dagger logs url to report + better run duration formatting ( #27820 )
...
* format duration where missing
* format duration where missing
* add dagger logs url to report
* DEMO - to revert
* fix a type bug
* fix a type bug
* fix a type bug
* fix urls
* add a br
* add utc
* Revert "DEMO - to revert"
This reverts commit 6fa21acd0d
2023-06-29 03:51:54 -05:00
Augustin
df08d198d5
connectors-ci: fix how with ignore cat with gradle ( #27819 )
...
* ignore cat by disabling plugin
* DEMO - to revert
* patch the plugin directly
* Revert "DEMO - to revert"
This reverts commit c215d2f30f
2023-06-29 03:45:47 -05:00
Augustin
2081a0ceae
connectors-ci: improved caching ( #27721 )
2023-06-28 19:56:51 +00:00
Augustin
792b71d467
connectors-ci: only run integrationTest gradle task for Java connectors ( #27794 )
2023-06-28 18:22:42 +00:00
Ben Church
543897c2ad
Remove Affected connector report ( #27622 )
...
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
2023-06-26 09:54:58 -07:00
Augustin
c2f83bcd9f
mention connector ops instead of channel ( #27706 )
2023-06-26 09:34:41 -04:00
Augustin
d3c33a6321
connector-ci: better and lighter logging ( #27606 )
2023-06-24 09:17:46 +00:00
Evan Tahler
1f6aef98df
yum clean all after every yum install to save space (#27555 )
...
* `yum clean all` after every yum install to save space
* docs and versions
* update env clean
* fix python install confusion
---------
Co-authored-by: Augustin <augustin@airbyte.io >
2023-06-23 13:53:22 -07:00
Augustin
346c85b792
slugify branch name for better sanitization ( #27645 )
2023-06-23 02:58:44 -05:00
Augustin
e742f8d5b5
connectors-ci: better connector test debugging experience (1/2) ( #27550 )
2023-06-21 17:28:04 +00:00
Augustin
22480ac642
ref: broaden the use of the run-dagger-pipeline action ( #27506 )
2023-06-21 06:47:33 -05:00
Augustin
2e3ee2deed
qa-check: add pydash deps ( #27544 )
2023-06-21 02:28:50 -05:00
Augustin
473112548e
connectors-ci: fix CAT not run for java connectors ( #27474 )
2023-06-20 23:54:40 +02:00
Ben Church
38a189e829
Breaking Changes: QA migration check ( #27294 )
...
* Add migration guide check
* Add test todo
* migration_guide to -migrations
* Update tests
* Valid tests
* Add tests
* Update tools/ci_connector_ops/ci_connector_ops/utils.py
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com >
* Ensure that the migration path is checked
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com >
2023-06-20 09:30:23 -07:00
Augustin
e9d7c54adc
connectors ci: auto format connectors code ( #27353 )
2023-06-19 17:44:43 +00:00
Ben Church
48d09730f6
Remove old qa-engine logic ( #27439 )
...
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
2023-06-16 12:05:18 -05:00
Ben Church
59dbc609e0
Handle the case of no metadata on master for new connectors ( #27377 )
...
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
2023-06-14 15:01:22 -05:00
Ben Church
68c0d6dbbc
Connector CI: update badges test summary ( #27215 )
...
* Rename to connector test report
* Get all test outputs
* Get last 10 output for all connectors
* Get runs by connector
* Crudely write badge and summary
* Add render functions
* Update connector report badge and link
* Fix url link
* Fix qa report tests
* update qa-engine tests
* Move regex to resource
* Add cron
* Run format
* New line
* Make badge green if last is successful
* Fix test
* Remove report.sh and build_report.py
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
2023-06-14 11:12:47 -07:00
Augustin
5b8200181c
connectors-ci: deprecate slash test ( #27200 )
2023-06-14 18:19:13 +02:00
Augustin
ef860d3d2a
connectors-ci: minor cleaning ( #27348 )
...
* remove debug instructions
* do not consider markdown file in dependency resolution
2023-06-14 03:26:47 -05:00
Augustin
6e9be02172
connectors-ci: improve run duration computation ( #27304 )
2023-06-13 16:21:50 -07:00
Augustin
25ef3a2686
connectors-ci: disable format check on python connectors ( #27301 )
...
* disable format check on python connectors
* Automated Commit - Formatting Changes
* disable format check on python connectors
---------
Co-authored-by: alafanechere <alafanechere@users.noreply.github.com >
2023-06-13 06:17:05 -05:00
Augustin
9a040af212
connectors-ci: run dockerd in a background task ( #27267 )
...
* connectors-ci: long running but ephemeral dockerd service
* Automated Commit - Formatting Changes
* to revert
* Revert "to revert"
This reverts commit 168ed30695alafanechere@users.noreply.github.com >
2023-06-13 06:14:38 -05:00
Ben Church
fb1b58d004
Add CI_JOB_KEY to override log paths ( #27210 )
...
* Add CI_JOB_KEY
* Update to none
* ternary operator override
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com >
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com >
Co-authored-by: Augustin <augustin@airbyte.io >
2023-06-12 07:39:01 -05:00
Augustin
50c18d8825
connector_ci: more granular mgmt of concurrency ( #27204 )
2023-06-09 08:33:44 -05:00
