Ben Church
36b335872a
Metadata: set cache control headers ( #26091 )
...
* set cache control header to no-cache for metadata upload
* set cache control header to no-cache for spec cache upload
* set cache control header to no-cache for registry upload
* Run formatter
2023-05-16 09:10:39 -07:00
Augustin
29bb15ebc4
connectors-ci: slim down Java images ( #26095 )
2023-05-16 03:01:15 -05:00
Rodi Reich Zilberman
6374f4780f
destination harness ( #25030 )
...
* destination harness
* Automated Commit - Formatting Changes
* refactor to support multiple connectors
* better byte counting
* Automated Change
* fix main cp
* harness type
* harness type
* harness type
* harness type
* harness type
* harness type
* harness type
* speed up harness build
* harness type
* harness type
* harness type
* harness type
* 1m catalog
* 1m catalog
* Automated Commit - Formatting Changes
* 1m catalog
* Automated Commit - Formatting Changes
* 1m catalog
* Automated Commit - Formatting Changes
* 1m catalog
* 1m catalog
* Automated Commit - Formatting Changes
* 1m catalog
* Automated Commit - Formatting Changes
* Automated Change
* add some additional logging for debugging
* test
* test
* debugging code
* Automated Commit - Formatting Changes
* debugging code
* debugging code
* debugging code
* Automated Commit - Formatting Changes
* Checkpoint: This is working!.
* Automated Commit - Formatting Changes
* Updates plumbing for Snowflake with static records - Working
* Automated Commit - Formatting Changes
* working 10m
* Automated Commit - Formatting Changes
* add destination datasource
* Automated Commit - Formatting Changes
* sanity
* sanity
* Delete 1m_catalog.json
* sanity
* Update airbyte-integrations/connectors-performance/destination-harness/README.md
Co-authored-by: Davin Chia <davinchia@gmail.com >
* sanity
* Update airbyte-integrations/connectors/destination-e2e-test/src/main/java/io/airbyte/integrations/destination/e2e_test/TestingDestinations.java
Co-authored-by: Davin Chia <davinchia@gmail.com >
* Splits performance harness into separated methods
* Automated Commit - Formatting Changes
* Updates test description to force Github builds
---------
Co-authored-by: rodireich <rodireich@users.noreply.github.com >
Co-authored-by: Jeff Cowan (Airbyte) <4992320+jcowanpdx@users.noreply.github.com >
Co-authored-by: Davin Chia <davinchia@gmail.com >
Co-authored-by: davinchia <davinchia@users.noreply.github.com >
Co-authored-by: ryankfu <ryan.fu@airbyte.io >
Co-authored-by: ryankfu <ryankfu@users.noreply.github.com >
2023-05-16 06:37:00 +00:00
Ben Church
d2018514c4
hotfix: Update QA engine to use metadata to unblock PR ( #26098 )
...
* Change metadata to definition
* run black
2023-05-15 17:38:49 -05:00
Augustin
0e3ec70545
connectors-ci: check if the pushed image can be pulled ( #26088 )
2023-05-15 22:14:48 +00:00
Augustin
be6dbae6df
connectors-ci: sanitize the publish logic and write tests for it ( #26068 )
2023-05-15 19:52:07 +02:00
Augustin
d5f2b2bc67
connectors-ci: fix condition on image check ( #26044 )
2023-05-12 15:53:06 -05:00
Augustin
dc5cb9f080
connectors-ci: less scary but harmless error log on publish ( #26037 )
2023-05-12 19:11:34 +02:00
Augustin
00c758a0f0
connnectors-ci: improve publish reporting stability ( #26027 )
2023-05-12 17:33:16 +02:00
Augustin
5c5eab0308
connectors-ci: fix postgres integration testing ( #25942 )
2023-05-11 21:19:29 +02:00
Ben Church
83c807fb14
Revert metadata service upload to use env var over json file for creds ( #25979 )
...
* Remove service account
* Fix tests
* Format
* Remove test folder
2023-05-10 20:44:53 -05:00
Augustin
3e49e50990
connectors-ci: always run metadata validation and report success if no validation happened ( #25909 )
...
* Successful pipeline if no step ran
* run GHA workflow on all PRs
2023-05-09 16:59:30 -07:00
Augustin
ba1589808f
connectors-ci: clean gradle lock file ( #25912 )
2023-05-09 13:37:56 -07:00
Augustin
68f4a5a6fa
connectors-ci: remove slack channel mention on publish failures ( #25921 )
2023-05-09 10:17:49 -05:00
Augustin
eda12c2f1d
connectors-ci: augustin/connectors-ci/fix-build-include-for-java-connectors ( #25908 )
2023-05-09 03:57:33 -05:00
Augustin
7cc221812d
connectors-ci: handle strict encrypt connectors ( #25864 )
2023-05-09 10:26:55 +02:00
Augustin
7a61bf2c1f
connectors-ci: fix mounted secrets path ( #25871 )
2023-05-06 18:54:49 -05:00
Augustin
99fa4b1c80
connectors-ci: send slack message and set execute timeout ( #25858 )
...
* send slack message and set execute timeout
* set timeout to 1H
2023-05-06 09:34:09 -07:00
Ben Church
6ee6430734
Add metadata for file secure and bump ( #25860 )
...
* Add metadata for file secure
* Remove -secure during qa checks
2023-05-05 18:43:13 -07:00
Ben Church
a682955fb3
Exclude scaffold during validate all ( #25856 )
2023-05-05 16:43:00 -07:00
Augustin
b7538a3844
connectors-ci: reduce local/remote moving parts ( #25827 )
...
* cache buster on CAT
* remove image if exists before loading
* remove image if exists before loading
* fix errors in pytest
* append unit test results to global results
* always use local git to spot modified files
* Revert "always use local git to spot modified files"
This reverts commit 441521433f
2023-05-05 01:34:55 -05:00
Ben Church
2c2935eef8
Upload reports to separate files ( #25824 )
...
* Upload reports to separate files
* test > tests
---------
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com >
2023-05-04 17:05:49 -07:00
Edward Gao
ac03566f06
Replicate recent dockerfile changes into dagger ( #25653 )
...
* replicate dockerfile changes
* only add normalization stuff if needed
* also set entrypoint >.>
* fix order of constants
* clean up normalization mappings + fail loudly
* manually install urllib3 1.26.15
* fix declaration
2023-05-04 16:23:12 -07:00
Augustin
7310494846
qa-checks: check metadata version matches dockerfile version ( #25661 )
2023-05-04 16:08:19 -07:00
Ben Church
9d5a5f5059
Update connector catalog urls to new connector registry urls ( #25378 )
...
* Add staging urls to test with
* Remove stage prefix from urls
2023-05-04 07:40:18 -07:00
Augustin
2da66e9cf1
cache buster on CAT ( #25785 )
2023-05-03 13:49:45 -05:00
Augustin
9e785e559a
connectors-ci: new checks to validate connector version ( #25609 )
...
* connectors-ci: new checks to validate connector version
* tmp ref attempt wip
* Revert "tmp ref attempt wip"
This reverts commit 0950d39546
2023-04-28 17:57:39 +02:00
Augustin
69495ce5d7
qa-engine: update the inappropriate connectors list ( #25629 )
2023-04-28 08:52:33 +02:00
Augustin
46f62f6325
connectors-ci: load secrets on unit tests ( #25557 )
2023-04-27 13:13:22 +02:00
Augustin
2717940fdb
connectors-ci: disable the poetry cache ( #25553 )
2023-04-26 09:09:04 -05:00
Evan Tahler
b11657ac62
remove ENABLE_SENTRY from connector Dockerfiles ( #25442 )
2023-04-25 22:13:39 +00:00
Augustin
be51997fee
connectors-ci: use dockerfile for python connectors ( #25483 )
2023-04-25 22:31:56 +02:00
Augustin
b799c32114
connectors-ci: use pip install instead of python -m pip install in with_pip_packages ( #25489 )
2023-04-25 21:54:13 +02:00
Augustin
2cca05c856
connectors-ci: use new service account key to write to spec cache ( #25479 )
2023-04-25 06:18:42 -05:00
Augustin
e018c4172a
connectors-ci: do not use container id to name tar files ( #25477 )
...
* do not use container id to name tar files
* do not use container id to name tar files
2023-04-25 05:48:49 -05:00
Augustin
a552dd81a4
connectors-ci: fix modified files detection on master ( #25471 )
...
* connectors-ci: fix modified files detection on master
* destination-s3-glue: bump version in metadata
2023-04-25 03:34:30 -05:00
Ben Church
c01bef505f
Add docker hub check to metadata upload ( #25306 )
...
* Add docker hub check to metadata upload
* Add env vars to action
* Fake change
* Add docker hub env to upload
* Add new commit
* Use secret env var
2023-04-24 17:48:36 -07:00
Augustin
5b7b1924ee
connectors-ci: publish connectors on merge ( #25316 )
2023-04-24 22:25:55 +00:00
Augustin
666043290f
connectors-ci: package normalization inside destinations images supporting it ( #25359 )
2023-04-22 08:45:45 +00:00
Augustin
0ab82b06ab
connectors-ci: document the CLI ( #25391 )
2023-04-22 10:14:19 +02:00
Augustin
b30a23de7e
connectors-ci: use shared pip cache instead of locked ( #25370 )
2023-04-20 16:26:25 -05:00
Augustin
a342e14324
connectors-ci: build multi platform connector images ( #25286 )
2023-04-20 08:13:16 +00:00
Ben Church
20283fe156
Dagger: Return true on valid exit in Dagger Commands ( #25333 )
...
* Return true on valid exit
* Update return type to bool
2023-04-19 19:24:22 +00:00
Augustin
2cdde75ed9
connectors-ci: fix java connectors testing ( #25268 )
...
* connectors-ci: include destination-azure-blob-storage gradle project as a destination build dep
* fix integration test return value
2023-04-19 01:05:40 -05:00
Ben Church
b6f6baebc2
Add Dagster Deployment Action skeleton ( #24929 )
...
* Add code skeleton
* Remove pr deployments
* remove connectors change
* Add action git ignore
* Update YAML to deploy properly
* Add newline at eof
* Setup python-executable deploy
* Add temporary script
* Update readme
* Remove github workflow
* Add dagger pipeline for deploy
* Add github action
* Update readme
* Remove comment
* Move script into dagger pipeline
* Add run test helper and skip test logic
* Have the dagster orchestrator properly generate the setup.py file
* Remove deploy script
* Address PR comments
* Update lock
* Automated Commit - Formatting Changes
---------
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com >
2023-04-18 09:07:37 -07:00
Ryan Fu
0dcc653bd1
Shuffled destination-jdbc into bases folder ( #25120 )
...
* Testing changes for reshuffle
* Resolves dependency/race-condition with base classes and docker creation
* Removes dockerfile
* Updates java connectors gradle dependencies
2023-04-17 12:29:03 -07:00
Ben Church
3c01902a57
Add is_optional flag to connector tests ( #25211 )
2023-04-14 12:08:42 -07:00
Augustin
6bb47794df
airbyte-ci: orchestrate the upload of modified metadata.yaml files to GCS ( #25044 )
2023-04-14 10:40:49 +00:00
Augustin
ae5832f501
dagger: upgrade to 0.5.0 ( #25171 )
...
* dagger: upgrade to 0.5.0
* clean class ConnectorPackageInstall(Step):
2023-04-13 14:51:58 -05:00
Evan Tahler
0b6ee790d7
Add additional destination-bigquery datacenters ( #25122 )
...
* Add additional bigquery datacenters
* bump dockerfile
* revert source BQ bump
* docs
* builds depend on normalization now...
* no -base
* auto-bump connector version
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com >
2023-04-13 08:10:44 -07:00
