* start implementation of new persistence method
* add includingJobId and totalJobCount to job list request
* format
* update local openapi as well
* refactor queries into JOOQ and return empty list if target job cannot be found
* fix descriptions and undo changes from other branch
* switch including job to starting job
* fix job history handler tests
* rewrite jobs subqueries in jooq
* fix multiple config type querying
* remove unnecessary casts
* switch back to 'including' and return multiple of page size necessary to include job
* undo webapp changes
* fix test description
* format
* fix: revert extraEnv. Move updated extraEnv to env_vars
* fix: Add node pool selector feature for gke deployments
* fix: hotfix. Fix webapp values and imagepullsecrets
* fix: remove nodeSelector duplicate
* fix: revert imagePullSecrets changes
* fix nit
* fix: fix bootloader imagepullsecrets
* fix: fix image pull secrets again...
* feat: add acceptance tests for helm
* add step to install unzip
* fix incorrect command spelling(helm)
* update slack notification sending for Airbyte CI workflow
* Rename Helm E2E test report name
* fix: Update helm charts
* fix: fix temporal DB Port bug
* fix nits
* do a small testing for sake of science
* Update test chart.yaml with note block
* Revert changes for AC tests
* fix: Include #16166
* fix: add metrics to the ci workflow
* fix: revert changes for temporal service. move them to server service;
* fix: revert fullstory deletion
* fix: change way of defining image tag and make it to be set in global values. Add values.yaml.test for integration testing and local deployment
* fix: udpate docs, update values.yaml
* fix: fix helpers.tpl in all subcharts
* Update chart docs
Co-authored-by: Davin Chia <davinchia@gmail.com>
* save
* clean up more usages and remove withRefreshedCatalog
* make webapp use correct endpoint
* add back intercept
* fix acceptance test
* fix log
* remove 'new' from test name
It's difficult today to parse all the logs from tests. Engineers waste time scrolling through the log outputs and looking for the relevant stack trace.
This PR adds an action to generate a JUnit report so devs can understand test results at a glance. This generates 3 reports for each of the main build jobs when the build completes. We leave the frontend build out since this is aggregated by cypress.
See https://github.com/airbytehq/airbyte/pull/15271/checks?check_run_id=7683783016 for an example of how this works.
Use the https://github.com/dorny/test-reporter action and configure this to look at the Jacoco test report output for top level and second level builds. Note that most of the parameters into the action don't work.
Comment out the trap commands to output logs. Though this can be useful for debugging, there is little practical use in the day-to-day, and results in extremely noisy logs.
* sleep 1 second every connector lookup
to prevent running into docker hub API rate limits
* comment on why we sleep
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Google Sheets Destination flushes on every state message
* add PR number to readme
* suggestions
* reverted clean change
* added memory allocation check for records_buffer(stream)
* Update airbyte-integrations/connectors/destination-google-sheets/destination_google_sheets/writer.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* auto-bump connector version
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* set per stream feature flag to true for testing
* add a second table to cdc acceptance tests
* add partial reset test
* format
* add partial reset cdc tests
* test incremental after partial reset
* remove dev image from acceptance test
* fix flag and add comment
* Revert "set per stream feature flag to true for testing"
This reverts commit 164d7da05990268b09e315eb88ff297d3a9f52f4.
* set USE_STREAM_CAPABLE_STATE flag to true in acceptance test script
* call new update endpoint
* use methods in test harness instead
* remove comment
* format
* fix state check in basic acceptance test
* use test info for test name logging
* Add CONTAINER_ORCHESTRATOR=true flag to kube acceptance tests
* Add CONTAINER_ORCHESTRATOR=true flag to GKE acceptance tests too
* fix downtime test
* fix the other orchestrator tests
* move container orchestrator acceptance tests to their own class
* Add deployment mode to env shared with jobs
* Add adaptive runners
* Migrate postgres source to use adaptive runner
* Add an array of specs in docker image spec definition
* Add copyright
* Parse docker image spec with specs list
* Update spec yaml files
* Pass in DEPLOYMENT_MODE to docker compose file
* Revert "Parse docker image spec with specs list"
This reverts commit 8fe41dd3b7.
* Revert changes in docker image spec
* Read cloud specific spec files based on deployment mode
* Revert "Update spec yaml files"
This reverts commit 059f326432.
* Publish cloud spec file if necessary
* Fix upload script
* Move test files
* Update docker compose file
* Format code
* Add comment about spec filename
* Add unit tests
* Remove redundant jdbc acceptance test
When running `PostgresStrictEncryptJdbcSourceAcceptanceTest`, the `discover` method tests always fail because there are unexpected columns in the catalog:
- `wakeup_at`
- `last_visited_at`
- `last_comment_at`
These columns only exist in `PostgresJdbcSourceAcceptanceTest`. And this failure cannot be reproduced locally.
The hypothesis is that when the JDBC unit tests are run on CI, they are run in parallel, and the same testcontainer is used for both tests. That's why the strict encrypt test can discover columns from the oridinary unit test.
Given that the JDBC strict encrypt test is basically redundant, it is removed.
* explicitly use buildx for connector publish
* ... and use it
* use docker-container driver
* use `driver docker-container` only for normalization
* Update tools/integrations/manage.sh
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
